* fio version change gives different throughput results
@ 2024-03-05 13:53 Thomas Weber
2024-03-05 17:11 ` Vincent Fu
0 siblings, 1 reply; 4+ messages in thread
From: Thomas Weber @ 2024-03-05 13:53 UTC (permalink / raw)
To: fio@vger.kernel.org
Hi all,
I noticed a change in the reported throughput when switching from fio version 3.17 to 3.36.
I narrowed it down to fio-3.26 which seem to introduce the throughput change.
The exemplary fio workload attached gives me different results since then.
Was that change intended?
Thanks Thomas
fio-3.25:
Run status group 0 (all jobs):
READ: bw=3658MiB/s (3835MB/s), 3658MiB/s-3658MiB/s (3835MB/s-3835MB/s), io=429GiB (460GB), run=120001-120001msec
WRITE: bw=2438MiB/s (2557MB/s), 2438MiB/s-2438MiB/s (2557MB/s-2557MB/s), io=286GiB (307GB), run=120001-120001msec
fio-3.26:
Run status group 0 (all jobs):
READ: bw=474MiB/s (497MB/s), 474MiB/s-474MiB/s (497MB/s-497MB/s), io=55.5GiB (59.6GB), run=120001-120001msec
WRITE: bw=317MiB/s (332MB/s), 317MiB/s-317MiB/s (332MB/s-332MB/s), io=37.1GiB (39.9GB), run=120001-120001msec
FIO_FREEPARM1=description=na FIO_FREEPARM2=description=na FIO_FREEPARM3=description=na FIO_FREEPARM4=runtime=120 FIO_FREEPARM5=time_based FIO_TARGETSPEC=directory=./fsstress_data FIO_NUMJOB=1 FIO_BS=64k FIO_SIZE=1g FIO_DIRECT=0 FIO_IOENGINE=sync FIO_IODEPTH=1 fio ./fs-stress2.fiojob --output-format=normal,terse --terse-version=3 --warnings-fatal --eta=never
fs-stress2.fiojob file:
[global]
ioengine=${FIO_IOENGINE}
iodepth=${FIO_IODEPTH}
size=${FIO_SIZE}
direct=${FIO_DIRECT}
numjobs=${FIO_NUMJOB}
${FIO_TARGETSPEC}
# free parameters that can be overwritten including the var= specifier
${FIO_FREEPARM1}
${FIO_FREEPARM2}
${FIO_FREEPARM3}
${FIO_FREEPARM4}
${FIO_FREEPARM5}
# environment setup
norandommap
allrandrepeat=1
startdelay=50ms-2s
ramp_time=5s
[smallstuff-seq-slow]
rw=rw
nrfiles=1000
openfiles=25
filesize=1k-1m
bssplit=4k/20:128k/80
rwmixread=60
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: fio version change gives different throughput results
2024-03-05 13:53 fio version change gives different throughput results Thomas Weber
@ 2024-03-05 17:11 ` Vincent Fu
2024-03-06 9:44 ` Thomas Weber
0 siblings, 1 reply; 4+ messages in thread
From: Vincent Fu @ 2024-03-05 17:11 UTC (permalink / raw)
To: Thomas Weber, fio@vger.kernel.org
On 3/5/24 08:53, Thomas Weber wrote:
> Hi all,
>
> I noticed a change in the reported throughput when switching from fio version 3.17 to 3.36.
>
> I narrowed it down to fio-3.26 which seem to introduce the throughput change.
>
> The exemplary fio workload attached gives me different results since then.
>
> Was that change intended?
>
> Thanks Thomas
>
> fio-3.25:
> Run status group 0 (all jobs):
> READ: bw=3658MiB/s (3835MB/s), 3658MiB/s-3658MiB/s (3835MB/s-3835MB/s), io=429GiB (460GB), run=120001-120001msec
> WRITE: bw=2438MiB/s (2557MB/s), 2438MiB/s-2438MiB/s (2557MB/s-2557MB/s), io=286GiB (307GB), run=120001-120001msec
>
> fio-3.26:
> Run status group 0 (all jobs):
> READ: bw=474MiB/s (497MB/s), 474MiB/s-474MiB/s (497MB/s-497MB/s), io=55.5GiB (59.6GB), run=120001-120001msec
> WRITE: bw=317MiB/s (332MB/s), 317MiB/s-317MiB/s (332MB/s-332MB/s), io=37.1GiB (39.9GB), run=120001-120001msec
>
>
> FIO_FREEPARM1=description=na FIO_FREEPARM2=description=na FIO_FREEPARM3=description=na FIO_FREEPARM4=runtime=120 FIO_FREEPARM5=time_based FIO_TARGETSPEC=directory=./fsstress_data FIO_NUMJOB=1 FIO_BS=64k FIO_SIZE=1g FIO_DIRECT=0 FIO_IOENGINE=sync FIO_IODEPTH=1 fio ./fs-stress2.fiojob --output-format=normal,terse --terse-version=3 --warnings-fatal --eta=never
>
> fs-stress2.fiojob file:
> [global]
> ioengine=${FIO_IOENGINE}
> iodepth=${FIO_IODEPTH}
> size=${FIO_SIZE}
> direct=${FIO_DIRECT}
> numjobs=${FIO_NUMJOB}
> ${FIO_TARGETSPEC}
>
> # free parameters that can be overwritten including the var= specifier
> ${FIO_FREEPARM1}
> ${FIO_FREEPARM2}
> ${FIO_FREEPARM3}
> ${FIO_FREEPARM4}
> ${FIO_FREEPARM5}
>
> # environment setup
> norandommap
> allrandrepeat=1
> startdelay=50ms-2s
> ramp_time=5s
>
> [smallstuff-seq-slow]
> rw=rw
> nrfiles=1000
> openfiles=25
> filesize=1k-1m
> bssplit=4k/20:128k/80
> rwmixread=60
>
There was a bug fix in fio-3.26 which seems relevant to your configuration:
https://github.com/axboe/fio/commit/4ef1562a013513fd0a0048cca4048f28d308a90f
commit 4ef1562a013513fd0a0048cca4048f28d308a90f
Author: Adam Kupczyk <akupczyk@redhat.com>
Date: Mon Dec 28 14:55:17 2020 +0100
io_u: Fix bad interaction with --openfiles and non-sequential file
selection policy
Problem happens when --openfiles is set and file_service_type !=
FIO_FSERVICE_SEQ.
In function __get_next_file, we decrement file_service_left and if
0, we select
next file to operate on.
However, get_next_file_rand can return -EBUSY if too many files are
already opened,
and __get_next_file exits with error.
In next invocation of __get_next_file, we decrement
file_service_left again (from 0),
wrapping around to 2^32-1, effectively locking __get_next_file to
always select the same.
Algorithm to observe bad behavior:
fio --randseed=1 --ioengine=libaio --rw=randwrite --nrfiles=256
--bs=4k --size=256m \
--loops=50 --allow_file_create=1 --write_iolog=log.txt
--file_service_type=normal:20 \
--filename_format=object.\$filenum --name=x --openfiles=100
cat log.txt |grep write |cut -f 1 -d " " |sort |uniq -c | sort -n |
sed "s/[.]/ /" \
| while read a b c; do echo $c $b $a; done |sort -n
....
70 object 17
71 object 19
72 object 22
73 object 65296
74 object 65255
75 object 33
76 object 27
77 object 25
78 object 65243
79 object 36
80 object 49
81 object 47
....
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: fio version change gives different throughput results
2024-03-05 17:11 ` Vincent Fu
@ 2024-03-06 9:44 ` Thomas Weber
2024-03-06 16:51 ` Vincent Fu
0 siblings, 1 reply; 4+ messages in thread
From: Thomas Weber @ 2024-03-06 9:44 UTC (permalink / raw)
To: fio@vger.kernel.org, vincentfu@gmail.com
On Tue, 2024-03-05 at 12:11 -0500, Vincent Fu wrote:
> On 3/5/24 08:53, Thomas Weber wrote:
> > Hi all,
> >
> > I noticed a change in the reported throughput when switching from
> > fio version 3.17 to 3.36.
> >
> > I narrowed it down to fio-3.26 which seem to introduce the
> > throughput change.
> >
> > The exemplary fio workload attached gives me different results
> > since then.
> >
> > Was that change intended?
> >
> > Thanks Thomas
> >
> > fio-3.25:
> > Run status group 0 (all jobs):
> > READ: bw=3658MiB/s (3835MB/s), 3658MiB/s-3658MiB/s (3835MB/s-
> > 3835MB/s), io=429GiB (460GB), run=120001-120001msec
> > WRITE: bw=2438MiB/s (2557MB/s), 2438MiB/s-2438MiB/s (2557MB/s-
> > 2557MB/s), io=286GiB (307GB), run=120001-120001msec
> >
> > fio-3.26:
> > Run status group 0 (all jobs):
> > READ: bw=474MiB/s (497MB/s), 474MiB/s-474MiB/s (497MB/s-
> > 497MB/s), io=55.5GiB (59.6GB), run=120001-120001msec
> > WRITE: bw=317MiB/s (332MB/s), 317MiB/s-317MiB/s (332MB/s-
> > 332MB/s), io=37.1GiB (39.9GB), run=120001-120001msec
> >
> >
> > FIO_FREEPARM1=description=na FIO_FREEPARM2=description=na
> > FIO_FREEPARM3=description=na FIO_FREEPARM4=runtime=120
> > FIO_FREEPARM5=time_based FIO_TARGETSPEC=directory=./fsstress_data
> > FIO_NUMJOB=1 FIO_BS=64k FIO_SIZE=1g FIO_DIRECT=0 FIO_IOENGINE=sync
> > FIO_IODEPTH=1 fio ./fs-stress2.fiojob --output-format=normal,terse
> > --terse-version=3 --warnings-fatal --eta=never
> >
> > fs-stress2.fiojob file:
> > [global]
> > ioengine=${FIO_IOENGINE}
> > iodepth=${FIO_IODEPTH}
> > size=${FIO_SIZE}
> > direct=${FIO_DIRECT}
> > numjobs=${FIO_NUMJOB}
> > ${FIO_TARGETSPEC}
> >
> > # free parameters that can be overwritten including the var=
> > specifier
> > ${FIO_FREEPARM1}
> > ${FIO_FREEPARM2}
> > ${FIO_FREEPARM3}
> > ${FIO_FREEPARM4}
> > ${FIO_FREEPARM5}
> >
> > # environment setup
> > norandommap
> > allrandrepeat=1
> > startdelay=50ms-2s
> > ramp_time=5s
> >
> > [smallstuff-seq-slow]
> > rw=rw
> > nrfiles=1000
> > openfiles=25
> > filesize=1k-1m
> > bssplit=4k/20:128k/80
> > rwmixread=60
> >
>
> There was a bug fix in fio-3.26 which seems relevant to your
> configuration:
>
>
https://github.com/axboe/fio/commit/4ef1562a013513fd0a0048cca4048f28d308a90f
>
>
> commit 4ef1562a013513fd0a0048cca4048f28d308a90f
> Author: Adam Kupczyk <
> akupczyk@redhat.com
> >
> Date: Mon Dec 28 14:55:17 2020 +0100
>
> io_u: Fix bad interaction with --openfiles and non-sequential
> file
> selection policy
>
> Problem happens when --openfiles is set and file_service_type
> !=
> FIO_FSERVICE_SEQ.
> In function __get_next_file, we decrement file_service_left and
> if
> 0, we select
> next file to operate on.
> However, get_next_file_rand can return -EBUSY if too many files
> are
> already opened,
> and __get_next_file exits with error.
>
> In next invocation of __get_next_file, we decrement
> file_service_left again (from 0),
> wrapping around to 2^32-1, effectively locking __get_next_file
> to
> always select the same.
>
> Algorithm to observe bad behavior:
> fio --randseed=1 --ioengine=libaio --rw=randwrite --nrfiles=256
> --bs=4k --size=256m \
> --loops=50 --allow_file_create=1 --write_iolog=log.txt
> --file_service_type=normal:20 \
> --filename_format=object.\$filenum --name=x --openfiles=100
>
> cat log.txt |grep write |cut -f 1 -d " " |sort |uniq -c | sort
> -n |
> sed "s/[.]/ /" \
> | while read a b c; do echo $c $b $a; done |sort -n
> ....
> 70 object 17
> 71 object 19
> 72 object 22
> 73 object 65296
> 74 object 65255
> 75 object 33
> 76 object 27
> 77 object 25
> 78 object 65243
> 79 object 36
> 80 object 49
> 81 object 47
> ....
>
> Signed-off-by: Adam Kupczyk <
> akupczyk@redhat.com
> >
>
Thank you. I noticed that fix as well in the fio changelog.
From the description it seems to affect only non-sequential IO.
However I saw huge throughput degradations for both rw=rw and
rw=randrw.
Do you think this is related to that bug fix too?
[smallstuff-seq-slow]
rw=rw
nrfiles=1000
openfiles=25
filesize=1k-1m
bssplit=4k/20:128k/80
rwmixread=60
[smallstuff-rand-slow]
rw=randrw
nrfiles=1000
openfiles=25
filesize=1k-1m
bssplit=4k/50:8k/25:64k/25
rwmixread=75
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: fio version change gives different throughput results
2024-03-06 9:44 ` Thomas Weber
@ 2024-03-06 16:51 ` Vincent Fu
0 siblings, 0 replies; 4+ messages in thread
From: Vincent Fu @ 2024-03-06 16:51 UTC (permalink / raw)
To: Thomas Weber, fio@vger.kernel.org
On 3/6/24 04:44, Thomas Weber wrote:
<snip>
>
> Thank you. I noticed that fix as well in the fio changelog.
> From the description it seems to affect only non-sequential IO.
> However I saw huge throughput degradations for both rw=rw and
> rw=randrw.
>
> Do you think this is related to that bug fix too?
>
> [smallstuff-seq-slow]
> rw=rw
> nrfiles=1000
> openfiles=25
> filesize=1k-1m
> bssplit=4k/20:128k/80
> rwmixread=60
>
> [smallstuff-rand-slow]
> rw=randrw
> nrfiles=1000
> openfiles=25
> filesize=1k-1m
> bssplit=4k/50:8k/25:64k/25
> rwmixread=75
>
The patch description mentions file_service_type != sequential but makes
no comment about whether offsets are chosen randomly or sequentially. In
fact the example job in the patch description chooses offsets randomly.
I tried your smallstuff-seq-slow job and found the following:
fio-3.25+ [1]:----- 424 MiB/s read, 384 MiB/s write
fio-3.36:---------- 384 MiB/s read, 347 MiB/s write
fio-3.36-97-g9b699: 517 MiB/s read, 467 MiB/s write
[1] fio-3.25 needs 382975557e632efb506836bc1709789e615c9094 to build
with recent kernels
If you are able to reliably reproduce the problem I suggest that you run
git bisect to identify the commit responsible for the difference you
have observed.
Vincent
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2024-03-06 16:51 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-03-05 13:53 fio version change gives different throughput results Thomas Weber
2024-03-05 17:11 ` Vincent Fu
2024-03-06 9:44 ` Thomas Weber
2024-03-06 16:51 ` Vincent Fu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox