Introduction

Sometimes we need to know the current status of the storage usage of a service in order to find any possible bottleneck in storage side. This tutorial describes how to measure current IOPS and Throughput on your server.

Max IOPS and Throughput of storage:

In this section, we want to estimate the maximum IOPS and Throughput of storage.

dd command

dd if=/dev/zero of=/tmp/disk_test_dd.file bs=100M count=1 oflag=dsync

• By running this command, dd will read from /dev/zero (a stream of null bytes) and write 100 megabytes of data to the file /tmp/disk_test_dd.file. The write operation will be synchronized and physically written to the disk before dd exits, thanks to the oflag=dsync option. This command is often used to test the disk write performance or to create files filled with null bytes for various purposes.

Result:

1+0 records in
1+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.22367 s, 469 MB/s

• As you can see Throughput of the disk was 469 MB/s

fio command (Recommended)

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test_disk_fio --filename=test_disk_fio --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75

• By running this command, you will initiate a fio test that performs random read-write I/O operations with a block size of 4 kilobytes, using a 4-gigabyte test file/device. The test will use the libaio I/O engine with direct I/O enabled. The I/O depth is set to 64, and the reads-to-writes ratio is 75:25. The test results will provide performance metrics and insights into the disk's I/O capabilities under these conditions.

Result:

test_disk_fio: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=128MiB/s,w=41.0MiB/s][r=32.8k,w=10.7k IOPS][eta 00m:00s]
test_disk_fio: (groupid=0, jobs=1): err= 0: pid=68138: Mon May 15 10:52:23 2023
   read: IOPS=37.1k, BW=145MiB/s (152MB/s)(3070MiB/21170msec)
   bw (  KiB/s): min=105184, max=246976, per=100.00%, avg=148570.26, stdev=22072.40, samples=42
   iops        : min=26296, max=61744, avg=37142.55, stdev=5518.10, samples=42
  write: IOPS=12.4k, BW=48.5MiB/s (50.8MB/s)(1026MiB/21170msec)
   bw (  KiB/s): min=36384, max=82448, per=100.00%, avg=49650.48, stdev=7312.82, samples=42
   iops        : min= 9096, max=20612, avg=12412.60, stdev=1828.21, samples=42
  cpu          : usr=4.81%, sys=32.05%, ctx=72759, majf=0, minf=153
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=785920,262656,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=145MiB/s (152MB/s), 145MiB/s-145MiB/s (152MB/s-152MB/s), io=3070MiB (3219MB), run=21170-21170msec
  WRITE: bw=48.5MiB/s (50.8MB/s), 48.5MiB/s-48.5MiB/s (50.8MB/s-50.8MB/s), io=1026MiB (1076MB), run=21170-21170msec

Disk stats (read/write):
  sda: ios=785117/262419, merge=0/226, ticks=1027296/246228, in_queue=1273668, util=99.60%

• As you see fio provides more detailed result

• Mean IOPS for read was  37.1k and for write was 12.4k

• Mean Throughput for read was 152MB/s and for write was 50.8MB/s

Current IOPS and Throughput of storage:

In this section, we want to estimate current IOPS and Throughput of storage.

With iostat

iostat -xdmb 60 1440

• By running this command, iostat will start monitoring and displaying disk statistics for all disk partitions every 60 seconds. The statistics will include information about disk utilization, wait time, service time, and the timestamp when the statistics were collected. The monitoring will continue for approximately 1440 minutes (around 24 hours).

Note:
You can run iostat -xdmb 60 1440 >> iostat_res.txt to redirect results to a file.

Result:

05/13/2023 08:22:19 PM
Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
loop0            0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     1.60     0.00   0.00   0.00
sda            444.91   84.11  26980.23   2880.85     3.43    63.74   0.76  43.11    0.31    1.74   0.10    60.64    34.25   0.15   8.08
sda1             0.00    0.00      0.09      0.00     0.00     0.00  11.23   0.00    0.22   12.28   0.00    41.62     0.50   0.13   0.00
sda2           444.91   84.11  26980.14   2880.85     3.43    63.74   0.76  43.11    0.31    1.74   0.10    60.64    34.25   0.15   8.08
sdb            897.80  739.67  96125.22  35384.91    41.59   944.47   4.43  56.08    0.02    0.03   0.03   107.07    47.84   0.03   5.62

• You can see the detail of each column in iostat Doc
• At 05/13/2023 08:22:19 PM, sda has 444.91 IOPS for reading and 84.11 for write. (From r/s and w/s columns)
• As same, sda has 26980.23 kB/s Throughput for reading and 2880.85 kB/s for write. (From rkB/s and wkB/s columns)

Note:
This is a result of single 60 seconds. If you enter the command same as above, you will see 1440 tables like this after 1 day.

Analyze Output in jupyter notebook [Optional]

In case you have some servers, and you need to analyze disk activity on them, You can use provided Jupyter Notebook to get more insight into the disks on your servers.

You can see and download the Jupyter Notebook from this link

Conclusion

With provided commands and corresponding results, you can estimate needs of current service and see whether there is a bottleneck regarding to I/O or not.

References

• Calculate IOPS in linux
• Understanding I/O
• Also ChatGPT :)