iozone

How to install iozone

  1. Download iozone tarball from http://www.iozone.org/.

  2. Extract it:

1
# tar -xvf iozone3XXX.tar

  1. # cd iozone3XXX/src/current/

  2. Choose your platform(my platform is AMD64) and run # make AMD64

Features and Functions

Features

  • ANSII ‘C’ source.
  • POSIX async I/O.
  • Mmap() file I/O.
  • Normal file I/O.
  • Single stream measurement.
  • Multiple stream measurement.
  • POSIX pthreads.
  • Multi-process measurement.
  • Excel importable output for graph generation.
  • I/O Latency data for plots.
  • 64-bit compatible source.
  • Large file compatible.
  • Stonewalling in throughput tests to eliminate straggler effects.
  • Processor cache size configurable.
  • Selectable measurements with fsync, O_SYNC.
  • Options targeted for testing over NFS.

I/O operations

  • Read
    This test measures the performance of reading an existing file.

  • Write
    This test measures the performance of writing a new file. When a new file is written not only does the data need to be stored but also the overhead information for keeping track of where the data is located on the storage media. This overhead is called the “metadata” It consists of the directory information, the space allocation and any other data associated with a file that is not part of the data contained in the file. It is normal for the initial write performance to be lower than the performance of re-writing a file due to this overhead information.

  • Re-Read
    This test measures the performance of reading a file that was recently read. It is normal for the performance to be higher as the operating system generally maintains a cache of the data for files that were recently read. This cache can be used to satisfy reads and improves the performance.

  • Re-Write
    This test measures the performance of writing a file that already exists. When a file is written that already exists the work required is less as the metadata already exists. It is normal for the rewrite performance to be higher than the performance of writing a new file.

  • Read backwards
    This test measures the performance of reading a file backwards. This may seem like a strange way to read a file but in fact there are applications that do this. MSC Nastran is an example of an application that reads its files backwards. With MSC Nastran, these files are very large (Gbytes to Tbytes in size). Although many operating systems have special features that enable them to read a file forward more rapidly, there are very few operating systems that detect and enhance the performance of reading a file backwards.

  • Random Read
    This test measures the performance of reading a file with accesses being made to random locations within the file. The performance of a system under this type of activity can be impacted by several factors such as: Size of operating system’s cache, number of disks, seek latencies, and others.

  • Random Write
    This test measures the performance of writing a file with accesses being made to random locations within the file. Again the performance of a system under this type of activity can be impacted by several factors such as: Size of operating system’s cache, number of disks, seek latencies, and others.

  • Random Mix
    This test measures the performance of reading and writing a file with accesses being made to random locations within the file. Again the performance of a system under this type of activity can be impacted by several factors such as: Size of operating system’s cache, number of disks, seek latencies, and others. This test is only available in throughput mode. Each thread/process runs either the read or the write test. The distribution of read/write is done on a round robin basis. More than one thread/process is required for proper operation.

  • Backwards Read
    This test measures the performance of reading a file backwards. This may seem like a strange way to read a file but in fact there are applications that do this. MSC Nastran is an example of an application that reads its files backwards. With MSC Nastran, these files are very large (Gbytes to Tbytes in size). Although many operating systems have special features that enable them to read a file forward more rapidly, there are very few operating systems that detect and enhance the performance of reading a file backwards.

  • Record Rewrite
    This test measures the performance of writing and re-writing a particular spot within a file. This hot spot can have very interesting behaviors. If the size of the spot is small enough to fit in the CPU data cache then the performance is very high. If the size of the spot is bigger than the CPU data cache but still fits in the TLB then one gets a different level of performance. If the size of the spot is larger than the CPU data cache and larger than the TLB but still fits in the operating system cache then one gets another level of performance, and if the size of the spot is bigger than the operating system cache then one gets yet another level of performance.

  • Read Strided
    This test measures the performance of reading a file with a strided access behavior. An example would be: Read at offset zero for a length of 4 Kbytes, then seek 200 Kbytes, and then read for a length of 4 Kbytes, then seek 200 Kbytes and so on. Here the pattern is to read 4 Kbytes and then Seek 200 Kbytes and repeat the pattern. This again is a typical application behavior for applications that have data structures contained within a file and is accessing a particular region of the data structure. Most operating systems do not detect this behavior or implement any techniques to enhance the performance under this type of access behavior.

    This access behavior can also sometimes produce interesting performance anomalies. An example would be if the application’s stride causes a particular disk, in a striped file system, to become the bottleneck.

  • Fread
    This test measures the performance of reading a file using the library function fread(). This is a library routine that performs buffered & blocked read operations. The buffer is within the user’s address space. If an application were to read in very small size transfers then the buffered & blocked I/O functionality of fread() can enhance the performance of the application by reducing the number of actual operating system calls and increasing the size of the transfers when operating system calls are made.

  • Freread
    This test is the same as fread above except that in this test the file that is being read was read in the recent past. This should result in higher performance as the operating system is likely to have the file data in cache.

  • Fwrite
    This test measures the performance of writing a file using the library function fwrite(). This is a library routine that performs buffered write operations. The buffer is within the user’s address space. If an application were to write in very small size transfers then the buffered & blocked I/O functionality of fwrite() can enhance the performance of the application by reducing the number of actual operating system calls and increasing the size of the transfers when operating system calls are made. This test is writing a new file so again the overhead of the metadata is included in the measurement.

  • Frewrite
    This test measures the performance of writing a file using the library function fwrite().

    This is a library routine that performs buffered & blocked write operations. The buffer is within the user’s address space. If an application were to write in very small size transfers then the buffered & blocked I/O functionality of fwrite() can enhance the performance of the application by reducing the number of actual operating system calls and increasing the size of the transfers when operating system calls are made.

    This test is writing to an existing file so the performance should be higher as there are no metadata operations required.

  • Random Read/Write
    This test measures the performance of reading/writing a file with accesses being made to random locations within the file. The performance of a system under this type of activity can be impacted by several factors such as: Size of operating system’s cache, number of disks, seek latencies, and others.

  • Async I/O
    Another mechanism that is supported by many operating systems for performing I/O is POSIX async I/O. The application uses the POSIX standard async I/O interfaces to accomplish this. Example: aio_write(), aio_read(), aio_error(). This test measures the performance of the POSIX async I/O mechanism.

  • Mmap
    Many operating systems support the use of mmap() to map a file into a user’s address space. Once this mapping is in place then stores to this location in memory will result in the data being stored going to a file. This is handy if an application wishes to treat files as chunks of memory. An example would be to have an array in memory that is also being maintained as a file in the files system.

    The semantics of mmap files is somewhat different than normal files. If a store to the memory location is done then no actual file I/O may occur immediately. The use of the msyc() with the flags MS_SYNC, and MS_ASYNC control the coherency of the memory and the file. A call to msync() with MS_SYNC will force the contents of memory to the file and wait for it to be on storage before returning to the application. A call to msync() with the flag MS_ASYNC tells the operating system to flush the memory out to storage using an asynchronous mechanism so that the application may return into execution without waiting for the data to be written to storage.

    This test measures the performance of using the mmap() mechanism for performing I/O.

How to use iozone

For Xen disk performance testing, i choose the following command:

1
# iozone -azR -i 0 -i 1 -i 2 -n 1G -g 4G -r 64K -M result_file -f test_file

For each option meaings, see The main parameters section.

Note:
If you wish to get accurate results for the entire range of performance for a platform you need to make sure that the maximum file size that will be tested is bigger than the buffer cache. If you don’t know how big the buffer cache is, or if it is a dynamic buffer cache then just set the maximum file size to be greater than the total physical memory that is in the platform.

In general you should be able to see three or four plateaus.

  • File size fits in processor cache.
  • File size fits in buffer cache
  • File size is bigger than buffer cache.

You may see another plateau if the platform has a primary and secondary processor caches. If you don’t see at least 3 plateaus then you probably have the maximum file size set too small. Iozone will default to a maximum file size of 512 Mbytes. This is generally sufficient but for some very large systems you may need to use the –g option to increase the maximum file size. See the file Run_rules document in the distribution for further information.

Additional notes on how to make the graphs

Iozone sends Excel compatible output to standard out. This may be redirected to a file and then processed with Excel. The normal output for Iozone as well as the Excel portion are in the same output stream. So to get the graphs one needs to scroll down to the Excel portion of the file and graph the data in that section.

There are several sets of graph data. “Writer report” is one example. When importing the file be sure to tell Excel to import with “delimited” and then click next, then click on the “space delimited” button. To graph the data just highlight the region containing the file size and record size and then click on the graph wizard.

The type of graph used is “Surface”. When the next dialog box pops up you need to select “Columns”.
After that the rest should be straight forward.

Results analysis

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
# iozone -azR -i 0 -i 1 -i 2 -n 1G -g 4G -r 64K -M -b result_file.xls -f test_file
	Iozone: Performance Test of File I/O
	        Version $Revision: 3.414 $
		Compiled for 64 bit mode.
		Build: linux

	Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
	             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
	             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
	             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
	             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
	             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
	             Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
	             Vangel Bojaxhi, Ben England.

	Run began: Mon Apr 20 07:36:07 2015

	Auto Mode
	Cross over of record size disabled.
	Excel chart generation enabled
	Using minimum file size of 1024 kilobytes.
	Using maximum file size of 10240 kilobytes.
	Record Size 64 KB

	Machine = Linux dhcp-66-73-122.englab.nay.redhat.com 3.10.0-229.el7.x86_64 #1 SMP Thu Jan 29 18:37:38 EST 201
	Command line used: iozone -azR -i 0 -i 1 -i 2 -n 1M -g 10M -r 64K -M -b result_file.xls -f test_file
	Output is in Kbytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 Kbytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
            1024      64  564176 1943594  5813392  5813392 4589593 2358827
            2048      64  594302 2081497  5986827  3136379 4295384 2090107
            4096      64  591658 2084478  5926533  6195843 5634950 2345757
            8192      64  592850 1848808  4917099  5776876 5326425 1677328

iozone test complete.
Excel output is below:

"Writer report"
        "64"
"1024"   564176
"2048"   594302
"4096"   591658
"8192"   592850

"Re-writer report"
        "64"
"1024"   1943594
"2048"   2081497
"4096"   2084478
"8192"   1848808

"Reader report"
        "64"
"1024"   5813392
"2048"   5986827
"4096"   5926533
"8192"   4917099

"Re-Reader report"
        "64"
"1024"   5813392
"2048"   3136379
"4096"   6195843
"8192"   5776876

"Random read report"
        "64"
"1024"   4589593
"2048"   4295384
"4096"   5634950
"8192"   5326425

"Random write report"
        "64"
"1024"   2358827
"2048"   2090107
"4096"   2345757
"8192"   1677328

Use office tool to open the result_file.xls, you will find the results contain three part: file read/write method, records sizes, file sizes and the Throughput(Kbytes/sec).

The Throughput is the disk performance indicator, the larger the better. On a final note, you can generate graphs using the Generate_Graphs and gengnuplot.sh located under iozone3XXX/src/current/, based on the iozone output.

1
2
3
# cd iozone3XXX/src/current/
# iozone -azR -i 0 -i 1 -i 2 -n 1G -g 4G -r 64K -M -b result_file.xls -f test_file > test_results.txt
# ./Generate_Graphs test_results.txt

The main parameters

  • -a - Used to select full automatic mode. Produces output that covers all tested file operations for record sizes of 4k to 16M for file sizes of 64k to 512M.
  • -R - Generate Excel report. Iozone will generate an Excel compatible report to standard out. This file may be imported with Microsoft Excel (space delimited) and used to create a graph of the filesystem performance. Note: The 3D graphs are column oriented. You will need to select this when graphing as the default in Excel is row oriented data.
  • -b Filename - Iozone will create a binary file format file in Excel compatible output of results.
  • -g # - Set maximum file size (in Kbytes) for auto mode (or #m or #g)
  • -n # - Set minimum file size (in Kbytes) for auto mode (or #m or #g)

  • -i # - Used to specify which tests to run. (0=write/rewrite, 1=read/re-read, 2=random-read/write 3=Read-backwards, 4=Re-write-record, 5=stride-read, 6=fwrite/re-fwrite, 7=fread/Re-fread, 8=random mix, 9=pwrite/Re-pwrite, 10=pread/Re-pread, 11=pwritev/Re-pwritev, 12=preadv/Re-preadv).

    One will always need to specify 0 so that any of the following tests will have a file to measure.

    -i # -i # -i # is also supported so that one may select more than one test.

  • -s # - Used to specify the size, in Kbytes, of the file to test. One may also specify -s #k (size in Kbytes) or -s #m (size in Mbytes) or -s #g (size in Gbytes).
  • -z - Used in conjunction with -a to test all possible record sizes. Normally Iozone omits testing of small record sizes for very large files when used in full automatic mode. This option forces Iozone to include the small record sizes in the automatic tests also.
  • -B - Use mmap() files. This causes all of the temporary files being measured to be created and accessed with the mmap() interface. Some applications prefer to treat files as arrays of memory. These applications mmap() the file and then just access the array with loads and stores to perform file I/O.
  • -c - Include close() in the timing calculations. This is useful only if you suspect that close() is broken in the operating system currently under test. It can be useful for NFS Version 3 testing as well to help identify if the nfs3_commit is working well.
  • -D - Use msync(MS_ASYNC) on mmap files. This tells the operating system that all the data in the mmap space needs to be written to disk asynchronously.
  • -G - Use msync(MS_SYNC) on mmap files. This tells the operating system that all the data in the mmap space needs to be written to disk synchronously.
  • -o - Writes are synchronously written to disk. (O_SYNC). Iozone will open the files with the O_SYNC flag. This forces all writes to the file to go completely to disk before returning to the benchmark.
  • -H - Use POSIX async I/O with # async operations. Iozone will use POSIX async I/O with a bcopy from the async buffers back into the applications buffer. Some versions of MSC NASTRAN perform I/O this way. This technique is used by applications so that the async I/O may be performed in a library and requires no changes to the applications internal model.
  • -k # - Use POSIX async I/O (no bcopy) with # async operations. Iozone will use POSIX async I/O and will not perform any extra bcopys. The buffers used by Iozone will be handed to the async I/O system call directly.
  • -I - Use DIRECT I/O for all file operations. Tells the filesystem that all operations are to bypass the buffer cache and go directly to disk. This also will use VX_DIRECT on VxFS, and O_DIRECT on Linux, and directio() on Solaris.
  • -K - Generate some random accesses during the normal testing.
  • -l # - Set the lower limit on number of processes to run. When running throughput tests this option allows the user to specify the least number of processes or threads to start. This option should be used in conjunction with the -u option.
  • -L # - Set processor cache line size to value (in bytes). Tells Iozone the processor cache line size. This is used internally to help speed up the test.
  • -S # - Set processor cache size to value (in Kbytes). This tells Iozone the size of the processor cache. It is used internally for buffer alignment and for the purge functionality.
  • -M - Iozone will call uname() and will put the string in the output file.
  • -t # - Run Iozone in a throughput mode. This option allows the user to specify how many threads or processes to have active during the measurement.
  • -u # - Set the upper limit on number of processes to run. When running throughput tests this option allows the user to specify the greatest number of processes or threads to start. This option should be used in conjunction with the -l option.
  • -W - Lock files when reading or writing.
  • -y # - Set minimum record size (in Kbytes) for auto mode. One may also specify -y #k (size in Kbytes) or -y #m (size in Mbytes) or -y #g (size in Gbytes). See –q for setting maximum record size.
  • -q # - Set maximum record size (in Kbytes) for auto mode. One may also specify -q #k (size in Kbytes) or -q #m (size in Mbytes) or -q #g (size in Gbytes). See –y for setting minimum record size.
  • -r # - Used to specify the record size, in Kbytes, to test. One may also specify -r #k (size in Kbytes) or -r #m (size in Mbytes) or -r #g (size in Gbytes).
  • -f filename - Used to specify the filename for the temporary file under test. This is useful when the unmount option is used. When testing with unmount between tests it is necessary for the temporary file under test to be in a directory that can be unmounted. It is not possible to unmount the current working directory as the process Iozone is running in this directory.

Reference

Technology