Re: fio version change gives different throughput results

[Vincent Fu <vincentfu@gmail.com>]

On 3/5/24 08:53, Thomas Weber wrote:
> Hi all,
> 
> I noticed a change in the reported throughput when switching from fio version 3.17 to 3.36.
> 
> I narrowed it down to fio-3.26 which seem to introduce the throughput change.
> 
> The exemplary fio workload attached gives me different results since then.
> 
> Was that change intended?
> 
> Thanks Thomas
> 
> fio-3.25:
> Run status group 0 (all jobs):
>     READ: bw=3658MiB/s (3835MB/s), 3658MiB/s-3658MiB/s (3835MB/s-3835MB/s), io=429GiB (460GB), run=120001-120001msec
>    WRITE: bw=2438MiB/s (2557MB/s), 2438MiB/s-2438MiB/s (2557MB/s-2557MB/s), io=286GiB (307GB), run=120001-120001msec
> 
> fio-3.26:
> Run status group 0 (all jobs):
>     READ: bw=474MiB/s (497MB/s), 474MiB/s-474MiB/s (497MB/s-497MB/s), io=55.5GiB (59.6GB), run=120001-120001msec
>    WRITE: bw=317MiB/s (332MB/s), 317MiB/s-317MiB/s (332MB/s-332MB/s), io=37.1GiB (39.9GB), run=120001-120001msec
> 
> 
> FIO_FREEPARM1=description=na FIO_FREEPARM2=description=na FIO_FREEPARM3=description=na FIO_FREEPARM4=runtime=120 FIO_FREEPARM5=time_based FIO_TARGETSPEC=directory=./fsstress_data FIO_NUMJOB=1 FIO_BS=64k FIO_SIZE=1g FIO_DIRECT=0 FIO_IOENGINE=sync FIO_IODEPTH=1 fio ./fs-stress2.fiojob --output-format=normal,terse --terse-version=3 --warnings-fatal --eta=never
> 
> fs-stress2.fiojob file:
> [global]
> ioengine=${FIO_IOENGINE}
> iodepth=${FIO_IODEPTH}
> size=${FIO_SIZE}
> direct=${FIO_DIRECT}
> numjobs=${FIO_NUMJOB}
> ${FIO_TARGETSPEC}
> 
> # free parameters that can be overwritten including the var= specifier
> ${FIO_FREEPARM1}
> ${FIO_FREEPARM2}
> ${FIO_FREEPARM3}
> ${FIO_FREEPARM4}
> ${FIO_FREEPARM5}
> 
> # environment setup
> norandommap
> allrandrepeat=1
> startdelay=50ms-2s
> ramp_time=5s
> 
> [smallstuff-seq-slow]
> rw=rw
> nrfiles=1000
> openfiles=25
> filesize=1k-1m
> bssplit=4k/20:128k/80
> rwmixread=60
> 

There was a bug fix in fio-3.26 which seems relevant to your configuration:

https://github.com/axboe/fio/commit/4ef1562a013513fd0a0048cca4048f28d308a90f

commit 4ef1562a013513fd0a0048cca4048f28d308a90f
Author: Adam Kupczyk <akupczyk@redhat.com>
Date:   Mon Dec 28 14:55:17 2020 +0100

     io_u: Fix bad interaction with --openfiles and non-sequential file 
selection policy

     Problem happens when --openfiles is set and file_service_type != 
FIO_FSERVICE_SEQ.
     In function __get_next_file, we decrement file_service_left and if 
0, we select
     next file to operate on.
     However, get_next_file_rand can return -EBUSY if too many files are 
already opened,
     and __get_next_file exits with error.

     In next invocation of __get_next_file, we decrement 
file_service_left again (from 0),
     wrapping around to 2^32-1, effectively locking __get_next_file to 
always select the same.

     Algorithm to observe bad behavior:
     fio --randseed=1 --ioengine=libaio --rw=randwrite --nrfiles=256 
--bs=4k --size=256m \
     --loops=50 --allow_file_create=1 --write_iolog=log.txt 
--file_service_type=normal:20 \
     --filename_format=object.\$filenum --name=x --openfiles=100

     cat log.txt |grep write |cut -f 1 -d " " |sort |uniq -c | sort -n | 
sed "s/[.]/ /" \
     | while read a b c; do echo $c $b $a; done |sort -n
     ....
     70 object 17
     71 object 19
     72 object 22
     73 object 65296
     74 object 65255
     75 object 33
     76 object 27
     77 object 25
     78 object 65243
     79 object 36
     80 object 49
     81 object 47
     ....

     Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>