fio

How to install fio

  1. Clone fio from git://git.kernel.dk/fio.git
1
# git clone git://git.kernel.dk/fio.git
  1. Install it:
1
2
3
4
# cd fio
# ./configure
# make
# make install

Features and Functions

fio is a tool that will spawn a number of threads or processes doing a particular type of io action as specified by the user. fio takes a number of global parameters, each inherited by the thread unless otherwise parameters given to them overriding that setting is given. The typical use of fio is to write a job file matching the io load one wants to simulate.

How fio works

The first step in getting fio to simulate a desired io workload, is writing a job file describing that specific setup. A job file may contain any number of threads and/or files - the typical contents of the job file is a global section defining shared parameters, and one or more job sections describing the jobs involved. When run, fio parses this file and sets everything up as described. If we break down a job from top to bottom, it contains the following basic parameters:

  • IO type - Defines the io pattern issued to the file(s). We may only be reading sequentially from this file(s), or we may be writing randomly. Or even mixing reads and writes, sequentially or randomly.
  • Block size - In how large chunks are we issuing io? This may be a single value, or it may describe a range of block sizes.
  • IO size - How much data are we going to be reading/writing.
  • IO engine - How do we issue io? We could be memory mapping the file, we could be using regular read/write, we could be using splice, async io, syslet, or even SG (SCSI generic sg).
  • IO depth - If the io engine is async, how large a queuing depth do we want to maintain?
  • IO type - Should we be doing buffered io, or direct/raw io?
  • Num files - How many files are we spreading the workload over.
  • Num threads - How many threads or processes should we spread this workload over.

The above are the basic parameters defined for a workload, in addition there’s a multitude of parameters that modify other aspects of how this job behaves.

Running fio

See the README file for command line parameters, there are only a few of them.

Running fio is normally the easiest part - you just give it the job file (or job files) as parameters:

$ fio job_file

and it will start doing what the job_file tells it to do. You can give more than one job file on the command line, fio will serialize the running of those files. Internally that is the same as using the ‘stonewall’ parameter described in the parameter section.

If the job file contains only one job, you may as well just give the parameters on the command line. The command line parameters are identical to the job parameters, with a few extra that control global parameters (see README). For example, for the job file parameter iodepth=2, the mirror command line option would be —iodepth 2 or —iodepth=2. You can also use the command line for giving more than one job entry. For each —name option that fio sees, it will start a new job with that name. Command line entries following a —name entry will apply to that job, until there are no more entries or a new —name entry is seen. This is similar to the job file options, where each option applies to the current job until a new [] job entry is seen.

fio does not need to run as root, except if the files or devices specified in the job section requires that. Some other options may also be restricted, such as memory locking, io scheduler switching, and decreasing the nice value.

Job file

As previously described, fio accepts one or more job files describing what it is supposed to do. The job file format is the classic ini file, where the names enclosed in [] brackets define the job name. You are free to use any ascii name you want, except ‘global’ which has special meaning. A global section sets defaults for the jobs described in that file. A job may override a global section parameter, and a job file may even have several global sections if so desired. A job is only affected by a global section residing above it. If the first character in a line is a ‘;’ or a ‘#’, the entire line is discarded as a comment.

So let’s look at a really simple job file that defines two processes, each randomly reading from a 128MB file.

1
2
3
4
5
6
7
8
9
10
; -- start job file --
[global]
rw=randread
size=128m

[job1]

[job2]

; -- end job file --

As you can see, the job file sections themselves are empty as all the described parameters are shared. As no filename= option is given, fio makes up a filename for each of the jobs as it sees fit. On the command line, this job would look as follows:

$ fio --name=global --rw=randread --size=128m --name=job1 --name=job2

Let’s look at an example that has a number of processes writing randomly to files.

1
2
3
4
5
6
7
8
9
10
; -- start job file --
[random-writers]
ioengine=libaio
iodepth=4
rw=randwrite
bs=32k
direct=0
size=64m
numjobs=4
; -- end job file --

Here we have no global section, as we only have one job defined anyway. We want to use async io here, with a depth of 4 for each file. We also increased the buffer size used to 32KB and define numjobs to 4 to fork 4 identical jobs. The result is 4 processes each randomly writing to their own 64MB file. Instead of using the above job file, you could have given the parameters on the command line. For this case, you would specify:

$ fio --name=random-writers --ioengine=libaio --iodepth=4 --rw=randwrite --bs=32k --direct=0 --size=64m --numjobs=4

When fio is utilized as a basis of any reasonably large test suite, it might be desirable to share a set of standardized settings across multiple job files. Instead of copy/pasting such settings, any section may pull in an external .fio file with ‘include filename’ directive, as in the following example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
; -- start job file including.fio --
[global]
filename=/tmp/test
filesize=1m
include glob-include.fio

[test]
rw=randread
bs=4k
time_based=1
runtime=10
include test-include.fio
; -- end job file including.fio --

; -- start job file glob-include.fio --
thread=1
group_reporting=1
; -- end job file glob-include.fio --

; -- start job file test-include.fio --
ioengine=libaio
iodepth=4
; -- end job file test-include.fio -

Settings pulled into a section apply to that section only (except global section). Include directives may be nested in that any included file may contain further include directive(s). Include files may not contain [] sections.

There are many parameters for job file, see fio/HOWTO(5.0 Detailed list of parameters) for details

Note:
There are some template job files, see fio/examples

Client/server

Normally fio is invoked as a stand-alone application on the machine where the IO workload should be generated. However, the frontend and backend of fio can be run separately. Ie the fio server can generate an IO workload on the “Device Under Test” while being controlled from another machine.

Start the server on the machine which has access to the storage DUT:

fio --server=args

where args defines what fio listens to. The arguments are of the form ‘type,hostname or IP,port’. ‘type’ is either ‘ip’ (or ip4) for TCP/IP v4, ‘ip6’ for TCP/IP v6, or ‘sock’ for a local unix domain socket. ‘hostname’ is either a hostname or IP address, and ‘port’ is the port to listen to (only valid for TCP/IP, not a local socket). Some examples:

1) fio --server

Start a fio server, listening on all interfaces on the default port (8765).

2) fio --server=ip:hostname,4444

Start a fio server, listening on IP belonging to hostname and on port 4444.

3) fio --server=ip6:::1,4444

Start a fio server, listening on IPv6 localhost ::1 and on port 4444.

4) fio --server=,4444

Start a fio server, listening on all interfaces on port 4444.

5) fio --server=1.2.3.4

Start a fio server, listening on IP 1.2.3.4 on the default port.

6) fio --server=sock:/tmp/fio.sock

Start a fio server, listening on the local socket /tmp/fio.sock.

Once a server is running, a “client” can connect to the fio server with:

fio --local-args --client=<server> --remote-args <job file(s)>

where —local-args are arguments for the client where it is running, ‘server’ is the connect string, and —remote-args and are sent to the server. The ‘server’ string follows the same format as it does on the server side, to allow IP/hostname/socket and port strings.

Fio can connect to multiple servers this way:

fio —client= —client=

If the job file is located on the fio server, then you can tell the server to load a local file as well. This is done by using —remote-config:

fio --client=server --remote-config /path/to/file.fio

Then the fio server will open this local (to the server) job file instead of being passed one from the client.

Xen fio job file

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
[global]
ioengine=libaio
size=190M
direct=1
runtime=7200
time_based
directory=/mnt
group_reporting
numjobs=16
filename=xen_block.file
write_bw_log=xen
write_lat_log=xen
write_iops_log=xen

[randread-4k-iodepth_1]
rw=read
bs=4k
iodepth=1
stonewall

[randwrite-4k-iodepth_1]
rw=randread
bs=4k
iodepth=1
stonewall

[randrw-4k-iodepth_1]
rw=randrw
bs=4k
iodepth=1
stonewall

[randread-256k-iodepth_64]
rw=read
bs=256k
iodepth=64
stonewall

[randwrite-256k-iodepth_64]
rw=randread
bs=256k
iodepth=64
stonewall

[randrw-256k-iodepth_64]
rw=randrw
bs=256k
iodepth=64
stonewall

If you want to run this job, just issue:

# fio $xen_job_file

Results analysis

Issue # fio $xen_job_file --output=xen_fio_results.txt

# cat xen_fio_results.txtThe results will be something like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
randread-4k-iodepth_1: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1
...
randwrite-4k-iodepth_1: (g=1): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1
...
randrw-4k-iodepth_1: (g=2): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1
...
randread-256k-iodepth_64: (g=3): rw=read, bs=256K-256K/256K-256K/256K-256K, ioengine=libaio, iodepth=64
...
randwrite-256k-iodepth_64: (g=4): rw=randread, bs=256K-256K/256K-256K/256K-256K, ioengine=libaio, iodepth=64
...
randrw-256k-iodepth_64: (g=5): rw=randrw, bs=256K-256K/256K-256K/256K-256K, ioengine=libaio, iodepth=64
...
fio-2.1.7
Starting 12 processes

randread-4k-iodepth_1: (groupid=0, jobs=2): err= 0: pid=19833: Tue Apr 21 21:37:14 2015
  read : io=4768.1MB, bw=67824KB/s, iops=16955, runt= 72001msec
    slat (usec): min=6, max=2097, avg=13.71, stdev= 4.32
    clat (usec): min=1, max=5581, avg=100.62, stdev=18.40
     lat (usec): min=44, max=5593, avg=114.64, stdev=19.03
    clat percentiles (usec):
     |  1.00th=[   66],  5.00th=[   88], 10.00th=[   90], 20.00th=[   93],
     | 30.00th=[   95], 40.00th=[   97], 50.00th=[   98], 60.00th=[  100],
     | 70.00th=[  102], 80.00th=[  106], 90.00th=[  114], 95.00th=[  126],
     | 99.00th=[  155], 99.50th=[  167], 99.90th=[  217], 99.95th=[  253],
     | 99.99th=[  358]
    bw (KB  /s): min=30824, max=37912, per=49.97%, avg=33893.51, stdev=1242.74
    lat (usec) : 2=0.02%, 4=0.03%, 10=0.01%, 50=0.02%, 100=58.24%
    lat (usec) : 250=41.64%, 500=0.05%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=8.43%, sys=24.57%, ctx=1220128, majf=0, minf=65
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1220840/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1
     ......
     ......
     ......
randrw-256k-iodepth_64: (groupid=5, jobs=2): err= 0: pid=19850: Tue Apr 21 21:37:14 2015
  read : io=30666MB, bw=436078KB/s, iops=1703, runt= 72010msec
    slat (usec): min=29, max=1577.6K, avg=596.93, stdev=13016.56
    clat (msec): min=3, max=1612, avg=34.20, stdev=96.81
     lat (msec): min=3, max=1612, avg=34.80, stdev=97.77
    clat percentiles (msec):
     |  1.00th=[   11],  5.00th=[   15], 10.00th=[   16], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   23], 60.00th=[   25],
     | 70.00th=[   28], 80.00th=[   31], 90.00th=[   37], 95.00th=[   41],
     | 99.00th=[  474], 99.50th=[  824], 99.90th=[ 1467], 99.95th=[ 1598],
     | 99.99th=[ 1614]
    bw (KB  /s): min=  378, max=191247, per=12.54%, avg=54676.03, stdev=47675.86
  write: io=26461MB, bw=376286KB/s, iops=1469, runt= 72010msec
    slat (usec): min=39, max=1577.6K, avg=615.76, stdev=11338.38
    clat (msec): min=3, max=1625, avg=43.17, stdev=107.59
     lat (msec): min=3, max=1626, avg=43.78, stdev=108.31
    clat percentiles (msec):
     |  1.00th=[   12],  5.00th=[   18], 10.00th=[   20], 20.00th=[   23],
     | 30.00th=[   25], 40.00th=[   28], 50.00th=[   30], 60.00th=[   33],
     | 70.00th=[   36], 80.00th=[   40], 90.00th=[   46], 95.00th=[   51],
     | 99.00th=[  578], 99.50th=[  938], 99.90th=[ 1483], 99.95th=[ 1598],
     | 99.99th=[ 1614]
    bw (KB  /s): min= 2509, max=159744, per=13.09%, avg=49253.29, stdev=39566.46
    lat (msec) : 4=0.01%, 10=0.58%, 20=23.55%, 50=72.10%, 100=2.23%
    lat (msec) : 250=0.06%, 500=0.42%, 750=0.41%, 1000=0.27%, 2000=0.38%
  cpu          : usr=3.89%, sys=31.56%, ctx=99711, majf=0, minf=52
  IO depths    : 1=0.1%, 2=0.3%, 4=0.5%, 8=1.1%, 16=2.1%, 32=4.2%, >=64=91.7%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=122664/w=105845/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=4768.1MB, aggrb=67823KB/s, minb=67823KB/s, maxb=67823KB/s, mint=72001msec, maxt=72001msec

Run status group 1 (all jobs):
   READ: io=4720.6MB, aggrb=67135KB/s, minb=67135KB/s, maxb=67135KB/s, mint=72001msec, maxt=72001msec

Run status group 2 (all jobs):
   READ: io=355724KB, aggrb=4924KB/s, minb=4924KB/s, maxb=4924KB/s, mint=72237msec, maxt=72237msec
  WRITE: io=355916KB, aggrb=4927KB/s, minb=4927KB/s, maxb=4927KB/s, mint=72237msec, maxt=72237msec

Run status group 3 (all jobs):
   READ: io=104547MB, aggrb=1452.3MB/s, minb=1452.3MB/s, maxb=1452.3MB/s, mint=72001msec, maxt=72001msec

Run status group 4 (all jobs):
   READ: io=107474MB, aggrb=1492.7MB/s, minb=1492.7MB/s, maxb=1492.7MB/s, mint=72001msec, maxt=72001msec

Run status group 5 (all jobs):
   READ: io=30666MB, aggrb=436078KB/s, minb=436078KB/s, maxb=436078KB/s, mint=72010msec, maxt=72010msec
  WRITE: io=26461MB, aggrb=376285KB/s, minb=376285KB/s, maxb=376285KB/s, mint=72010msec, maxt=72010msec

Disk stats (read/write):
  xvdb: ios=8341258/722656, merge=0/29, ticks=11020547/7441881, in_queue=18466369, util=76.14%

The client number is printed, along with the group id and error of that thread. Below is the io statistics, here for writes. In the order listed, they denote:

  • io= - Number of megabytes io performed
  • bw= - Average bandwidth rate
  • iops= - Average IOs performed per second
  • runt= - The runtime of that thread
  • slat= - Submission latency (avg being the average, stdev being the standard deviation). This is the time it took to submit the io. For sync io, the slat is really the completion latency, since queue/complete is one operation there. This value can be in milliseconds or microseconds, fio will choose the most appropriate base and print that. In the example above, milliseconds is the best scale. Note: in —minimal mode latencies are always expressed in microseconds.
  • clat= - Completion latency. Same names as slat, this denotes the time from submission to completion of the io pieces. For sync io, clat will usually be equal (or very close) to 0, as the time from submit to complete is basically just CPU time (io has already been done, see slat explanation).
  • cpu= - CPU usage. User and system time, along with the number of context switches this thread went through, usage of system and user time, and finally the number of major and minor page faults.

After each client has been listed, the group statistics are printed. They will look like this:

1
2
3
Run status group 5 (all jobs):
   READ: io=30666MB, aggrb=436078KB/s, minb=436078KB/s, maxb=436078KB/s, mint=72010msec, maxt=72010msec
  WRITE: io=26461MB, aggrb=376285KB/s, minb=376285KB/s, maxb=376285KB/s, mint=72010msec, maxt=72010msec

For each data direction, it prints:

  • io= - Number of megabytes io performed.
  • aggrb= - Aggregate bandwidth of threads in this group.
  • minb= - The minimum average bandwidth a thread saw.
  • maxb= - The maximum average bandwidth a thread saw.
  • mint= - The smallest runtime of the threads in that group.
  • maxt= - The longest runtime of the threads in that group.

And finally, the disk statistics are printed. They will look like this:

1
2
Disk stats (read/write):
  sda: ios=16398/16511, merge=30/162, ticks=6853/819634, in_queue=826487, util=100.00%

Each value is printed for both reads and writes, with reads first. The numbers denote:

  • ios= - Number of ios performed by all groups.
  • merge= - Number of merges io the io scheduler.
  • ticks= - Number of ticks we kept the disk busy.
  • io_queue= - Total time spent in the disk queue.
  • util= - The disk utilization. A value of 100% means we kept the disk busy constantly, 50% would be a disk idling half of the time.

It is also possible to get fio to dump the current output while it is running, without terminating the job. To do that, send fio the USR1 signal. You can also get regularly timed dumps by using the —status-interval parameter, or by creating a file in /tmp named fio-dump-status. If fio sees this file, it will unlink it and dump the current output status.

The performance indicators i use for xen are bw, iops and slat/clat/lat. bw, iops the larger the better. slat/clat/lat the smaller the better.

For xen performance testing, i also add write_bw_log=xen, write_lat_log=xen, write_iops_log=xen in the fio job file to generate xen_bw.log, xen_iops.log, xen_clat.log, xen_slat.log, xen_lat.log. You can use fio/tools/fio_generate_plots to generate svg format picture:

1
./fio_generate_plots xen

Note: fio_generate_plots only take one argument, which is the prefix(xen) of xen_bw.log, xen_clat.log, xen_iops.log, xen_lat.log, xen_slat.log.

For more details, see fio/HOWTO(6.0 Interpreting the output) section.

Terse output

For scripted usage where you typically want to generate tables or graphs of the results, fio can output the results in a semicolon separated format. The format is one long line of values, such as:

1
2
2;card0;0;0;7139336;121836;60004;1;10109;27.932460;116.933948;220;126861;3495.446807;1085.368601;226;126864;3523.635629;1089.012448;24063;99944;50.275485%;59818.274627;5540.657370;7155060;122104;60004;1;8338;29.086342;117.839068;388;128077;5032.488518;1234.785715;391;128085;5061.839412;1236.909129;23436;100928;50.287926%;59964.832030;5644.844189;14.595833%;19.394167%;123706;0;7313;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.02%;0.05%;0.16%;6.04%;40.40%;52.68%;0.64%;0.01%;0.00%;0.01%;0.00%;0.00%;0.00%;0.00%;0.00%
A description of this job goes here.

To enable terse output, use the —minimal command line option. The first value is the version of the terse output format. If the output has to be changed for some reason, this number will be incremented by 1 to signify that change.

For more details, see fio/HOWTO(7.0 Terse output) section.

The main parameters for fio command line

  • —showcmd - Turn a job file into command line options
  • —output - Write output to file
  • —bandwidth-log - Generate per-job bandwidth logs
  • —minimal - Minimal (terse) output
  • —output-format=type - Output format (terse,json,normal)
  • —enghelp=engine - Print ioengine help, or list available ioengines
  • —enghelp=engine,cmd - Print help for an ioengine cmd
  • —server=args - Start backend server. See Client/Server section.
  • —client=host - Connect to specified backend.

Reference

  • fio HOWTO
  • For more details, please read fio/README and fio/HOWTO
Technology