fio problems

fio problems

[Danny Kukawka]

[-- Attachment #1: Type: text/plain, Size: 1665 bytes --]

Hi,

we try to run fio against raw RBD devices on a ceph cluster. Here are
the parameter we use to run fio:

[test]
filename=/dev/rbd0
time_based
runtime=3h
rw=randrw
rwmixread=80
rwmixwrite=20
blocksize_range=4k-1024k
ioengine=libaio
iodepth=64
direct=1
verify=crc32c-intel
verify_fatal=1
verify_async=16
lockfile=exclusive

We see some problems:

1) fio reports higher READ speeds than the used 1Gbit network interface
could provide (the interface isn't even working to capacity in this
situation). That happens if we use --rwmixread/rwmixwrite:

 READ: io=91560MB, aggrb=156206KB/s, minb=159955KB/s,
        maxb=159955KB/s, mint=600216msec, maxt=600216msec
 WRITE: io=22944MB, aggrb=39143KB/s, minb=40083KB/s, maxb=40083KB/s,
        mint=600216msec, maxt=600216msec

2) fio 2.0.6 and also the current git tends to fail with this error:

   fio 2.0.6
   Starting 1 process
   fio: iolog.c:242: log_io_piece: Assertion `ipo->len == __ipo->len'
   failed.:08m:31s]

It seems that we can prevent it when we use the --size parameter, but we
would like to prevent the usage of the parameter, if possible

3) fio always reports minb == maxb and aggrb < minb as you can see
above. From my understanding this would be false info.

4) If we define rw=randrw, rwmixread=80, rwmixwrite=20 and the verify
options, what does the READ process if there isn't enough data written
out yet? Which data does READ read and verify?

Thanks and Regards,

Danny
-- 
Danny Kukawka
 SUSE LINUX Products GmbH
  Maxfeldstrasse 5, D-90409 Nuremberg, Germany
  GF: Jeff Hawn,Jennifer Guild,Felix Imendörffer,HRB16746 (AG Nürnberg)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: fio problems

[Jens Axboe]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 2012-04-04 00:39, Danny Kukawka wrote:
> Hi,
> 
> we try to run fio against raw RBD devices on a ceph cluster. Here are
> the parameter we use to run fio:
> 
> [test]
> filename=/dev/rbd0
> time_based
> runtime=3h
> rw=randrw
> rwmixread=80
> rwmixwrite=20
> blocksize_range=4k-1024k
> ioengine=libaio
> iodepth=64
> direct=1
> verify=crc32c-intel
> verify_fatal=1
> verify_async=16
> lockfile=exclusive
> 
> We see some problems:
> 
> 1) fio reports higher READ speeds than the used 1Gbit network interface
> could provide (the interface isn't even working to capacity in this
> situation). That happens if we use --rwmixread/rwmixwrite:
> 
>  READ: io=91560MB, aggrb=156206KB/s, minb=159955KB/s,
>         maxb=159955KB/s, mint=600216msec, maxt=600216msec
>  WRITE: io=22944MB, aggrb=39143KB/s, minb=40083KB/s, maxb=40083KB/s,
>         mint=600216msec, maxt=600216msec

OK that's very odd. How are these numbers matching up with watching
vmstat 1 while the job runs? I've never seen fio over-report before.

> 2) fio 2.0.6 and also the current git tends to fail with this error:
> 
>    fio 2.0.6
>    Starting 1 process
>    fio: iolog.c:242: log_io_piece: Assertion `ipo->len == __ipo->len'
>    failed.:08m:31s]
> 
> It seems that we can prevent it when we use the --size parameter, but we
> would like to prevent the usage of the parameter, if possible

Strangely, I had two reports of this today. Did a bit of digging, and
it's due to this commit:

commit c04e4661e4da3b6079f415897e4507cf8e610c54
Author: Daniel Ehrenberg <dehrenberg@google.com>
Date:   Fri Mar 16 18:54:15 2012 +0100

    time_based: Avoid restarting main I/O loop

I think what is going on here is that your job is time based, and when
we fail getting a new offset, we clear the offset bitmap. But we could
have verifies pending at this point, we'd need to prune this out as
well.

> 3) fio always reports minb == maxb and aggrb < minb as you can see
> above. From my understanding this would be false info.

Yeah, it's a cosmetic thing. I've never really taken the time to look at
that...

> 4) If we define rw=randrw, rwmixread=80, rwmixwrite=20 and the verify
> options, what does the READ process if there isn't enough data written
> out yet? Which data does READ read and verify?

If you have a split workload like that and are also doing verifies, the
read can be either a read verify or just an offset read. In the latter
case, it doesn't verify anything. It'll just read a block (or multiple
blocks) at some offset instead of writing it.

- -- 
Jens Axboe

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBCAAGBQJPfKXtAAoJEPfTWPspceCmqeMP/1p+btObGgcYDgCDa5ohSi/D
usBx/MYkxhgyeleC8ketcC0QVl9qU9dW4krUso1Fe7lyYvjtS+vyvu/ayxISY6hp
CJ0c6m/py+jNDZNANgQQNR0bC8pZ03GbcJnAg0Ari8hewRu8oF2s6razcpr9PstT
nhg0wSrrx+1LNalfYk0GmQe7VDX8QNgOvcBp+gN2v1R9sxYjvR86F6MGBib19LeR
g6fwnQ/zBQRHHVCY3e6AQrozLJk47QOfdwF8pIKTkxVcOCK51Qynbi5YK6zxArto
jFaCgdbQ+DxvUSH6korIaQv3PLmf2wmokQz7TmTjxYQyz8Q6RMW3M/ZdcZlxqHhe
VkBIxC/JYaU97wpLI+Y4XqMDJr+3vrp8JaS4DPrlb7+hNY4sDMFqKH9MCRy3Th1v
12/WZy244SVtqIF9iboS4Gx9QbxfQzZfL5tpQcJaormknGKaTpP3YHZ//egeI/pK
tZvZ34l8CRHS58wfWyh4qPaYyy/hjwoojNJ0/CZGFQnUa6tH6hnWKhRX++oVsEP7
24FbqI2xLG3tBTDMeQIYg0/4SZo65D10vvLF8zzvyqy9hhaZJTon2AmnxpOsm+EA
pUSDWmB7OdpbnIPwA8PpK0sATxBUA638S8UmGFYjqubjwCWpKr5x6HPR6aVj+XN+
O5FfQel0Tz8w4g+ARjju
=cM7O
-----END PGP SIGNATURE-----

Re: fio problems

[Jens Axboe]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 2012-04-04 13:50, Jens Axboe wrote:
>> 2) fio 2.0.6 and also the current git tends to fail with this error:
> 
>>    fio 2.0.6
>>    Starting 1 process
>>    fio: iolog.c:242: log_io_piece: Assertion `ipo->len == __ipo->len'
>>    failed.:08m:31s]
> 
>> It seems that we can prevent it when we use the --size parameter, but we
>> would like to prevent the usage of the parameter, if possible
> 
> Strangely, I had two reports of this today. Did a bit of digging, and
> it's due to this commit:
> 
> commit c04e4661e4da3b6079f415897e4507cf8e610c54
> Author: Daniel Ehrenberg <dehrenberg@google.com>
> Date:   Fri Mar 16 18:54:15 2012 +0100
> 
>     time_based: Avoid restarting main I/O loop
> 
> I think what is going on here is that your job is time based, and when
> we fail getting a new offset, we clear the offset bitmap. But we could
> have verifies pending at this point, we'd need to prune this out as
> well.

Actually, since it happens so rarely, I think the better approach is
just to ignore it. I've updated fio to reflect that.

- -- 
Jens Axboe

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBCAAGBQJPfKvBAAoJEPfTWPspceCmuLIP+wccx5iGP9f5qpD7J0GJbQ7m
ClUlBZEPbKJw15KUW27bydRjaPZjBw71aX/ho5UzAPTH7vR97jqgQVve6brg7lB0
xxoUVa1ww3JQHpT5rDsZ+wy+JeFxbiKVWgqQJquJuayX128j6gGxyIap5qGatCym
N9yS3BrZxBmujEuY4O1OOLPi30eIXq3ncLqWq5Xi1HBlo3Wog+/22ZD0Lp2WcAZT
A8oCUNcvtEuhSseMqc4B/xBDZHBPnEI0JmiHB+COkKN3oRi536LfO/7SZMx4rm3x
LHUGwUtE5zdyz3HtiTcy9TxjPFUK83JfwaOQ4dayIhSyJw1SPQN7ZlPXuwd1A8Q8
lFpZffFkU5IBoBRXXE/VUxpoKTB8YBMbtp1uELp5nYnVjVsJA2eYmaStQP7jU7RD
v8UqSNTCJCAJBNFdliuVxSYjdiRM9nGxBLcOt1knereWCJTjdzf+FnjCKFrS9Pop
m3IOpkrH1Jr44xtOs9wZkrPP1XYOfqCAiNiZqnfQUiWK7Po3jIvVuHRymzkomDBB
S4ellXIx1MTyl5OkIniMYwJuP5gN2LDLQ3cicTcLggve821o2Shr0CK67zn49WQt
0THsyw0fOcgbJYYZ9FKaN98rM42pL+ctujjUe6yaA4Vk3ylp9qSSZRDS7gPziMaw
Yf/J5XC3leAxr2hLDuyT
=ajen
-----END PGP SIGNATURE-----

Re: fio problems

[Danny Kukawka]

[-- Attachment #1: Type: text/plain, Size: 2480 bytes --]

Am 04.04.2012 21:50, schrieb Jens Axboe:
> On 2012-04-04 00:39, Danny Kukawka wrote:
>> Hi,
> 
>> we try to run fio against raw RBD devices on a ceph cluster. Here are
>> the parameter we use to run fio:
> 
>> [test]
>> filename=/dev/rbd0
>> time_based
>> runtime=3h
>> rw=randrw
>> rwmixread=80
>> rwmixwrite=20
>> blocksize_range=4k-1024k
>> ioengine=libaio
>> iodepth=64
>> direct=1
>> verify=crc32c-intel
>> verify_fatal=1
>> verify_async=16
>> lockfile=exclusive
> 
>> We see some problems:
> 
>> 1) fio reports higher READ speeds than the used 1Gbit network interface
>> could provide (the interface isn't even working to capacity in this
>> situation). That happens if we use --rwmixread/rwmixwrite:
> 
>>  READ: io=91560MB, aggrb=156206KB/s, minb=159955KB/s,
>>         maxb=159955KB/s, mint=600216msec, maxt=600216msec
>>  WRITE: io=22944MB, aggrb=39143KB/s, minb=40083KB/s, maxb=40083KB/s,
>>         mint=600216msec, maxt=600216msec
> 
> OK that's very odd. How are these numbers matching up with watching
> vmstat 1 while the job runs? I've never seen fio over-report before.

I will take a look at it and let you know.

>> 3) fio always reports minb == maxb and aggrb < minb as you can see
>> above. From my understanding this would be false info.
> 
> Yeah, it's a cosmetic thing. I've never really taken the time to look at
> that...

It's not cosmetic if you need correct values for your tests ...

>> 4) If we define rw=randrw, rwmixread=80, rwmixwrite=20 and the verify
>> options, what does the READ process if there isn't enough data written
>> out yet? Which data does READ read and verify?
> 
> If you have a split workload like that and are also doing verifies, the
> read can be either a read verify or just an offset read. In the latter
> case, it doesn't verify anything. It'll just read a block (or multiple
> blocks) at some offset instead of writing it.

That would may explain the fio over-report in 1).

We create a new RBD on the cluster, mount it on the client and let fio
start it's test on the RBD. The RBD is completely empty at this moment.
Not sure what happens if fio reads some random offset that may isn't
filled with data at this moment. Or is fio only reading from a prior
written block/offset?

Danny
-- 
Danny Kukawka
 SUSE LINUX Products GmbH
  Maxfeldstrasse 5, D-90409 Nuremberg, Germany
  GF: Jeff Hawn,Jennifer Guild,Felix Imendörffer,HRB16746 (AG Nürnberg)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: fio problems

[Danny Kukawka]

[-- Attachment #1: Type: text/plain, Size: 1406 bytes --]

Am 04.04.2012 22:15, schrieb Jens Axboe:
> On 2012-04-04 13:50, Jens Axboe wrote:
>>> 2) fio 2.0.6 and also the current git tends to fail with this error:
> 
>>>    fio 2.0.6
>>>    Starting 1 process
>>>    fio: iolog.c:242: log_io_piece: Assertion `ipo->len == __ipo->len'
>>>    failed.:08m:31s]
> 
>>> It seems that we can prevent it when we use the --size parameter, but we
>>> would like to prevent the usage of the parameter, if possible
> 
>> Strangely, I had two reports of this today. Did a bit of digging, and
>> it's due to this commit:
> 
>> commit c04e4661e4da3b6079f415897e4507cf8e610c54
>> Author: Daniel Ehrenberg <dehrenberg@google.com>
>> Date:   Fri Mar 16 18:54:15 2012 +0100
> 
>>     time_based: Avoid restarting main I/O loop
> 
>> I think what is going on here is that your job is time based, and when
>> we fail getting a new offset, we clear the offset bitmap. But we could
>> have verifies pending at this point, we'd need to prune this out as
>> well.
> 
> Actually, since it happens so rarely, I think the better approach is
> just to ignore it. I've updated fio to reflect that.

Btw. I can suppress the error if I add --size with a high number e.g.
--size=1p.

Danny
-- 
Danny Kukawka
 SUSE LINUX Products GmbH
  Maxfeldstrasse 5, D-90409 Nuremberg, Germany
  GF: Jeff Hawn,Jennifer Guild,Felix Imendörffer,HRB16746 (AG Nürnberg)




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: fio problems

[Jens Axboe]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 2012-04-05 02:51, Danny Kukawka wrote:
>>> 3) fio always reports minb == maxb and aggrb < minb as you can see
>>> above. From my understanding this would be false info.
>>
>> Yeah, it's a cosmetic thing. I've never really taken the time to look at
>> that...
> 
> It's not cosmetic if you need correct values for your tests ...

Most people don't use the group reporting, but yes of course it should
be correct. I have checked in a patch to correct it. The min/max group
report bandwidth were off by 1.024, the aggregate was correct (and
matched the "real" data output).

>>> 4) If we define rw=randrw, rwmixread=80, rwmixwrite=20 and the verify
>>> options, what does the READ process if there isn't enough data written
>>> out yet? Which data does READ read and verify?
>>
>> If you have a split workload like that and are also doing verifies, the
>> read can be either a read verify or just an offset read. In the latter
>> case, it doesn't verify anything. It'll just read a block (or multiple
>> blocks) at some offset instead of writing it.
> 
> That would may explain the fio over-report in 1).
> 
> We create a new RBD on the cluster, mount it on the client and let fio
> start it's test on the RBD. The RBD is completely empty at this moment.
> Not sure what happens if fio reads some random offset that may isn't
> filled with data at this moment. Or is fio only reading from a prior
> written block/offset?

OK, if we back up a bit. If you have a workload that is rwmixread=50,
hence doing equal amount of random reads and writes. Then the job will
read 50% of the disk, and write 50% of the disk. None of the read and
write offsets will be the same. So if the drive is initially empty, you
are reading non-initialized data.

Same thing is true for your workload. So some of the reads you see will
be the verifies, but some of them will also be for parts of the disk
that you will then never write.

- -- 
Jens Axboe

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBCAAGBQJPfa85AAoJEPfTWPspceCmPX0P/jyhie1ZJnLOgZ8iAuBK0AIY
fxY/X9z8h1rx7VqCSbYkEs9MQsMZzuw8piHK76TagE+VBzLNjP6eXNMGwAZU9YMS
4e9gSRWmBLIRTLhEthW7d6aUKV5N6bFDdywhMqFjlk8v6BMuTzsiakMV11HILdA6
WLlmazhyxIsaFc024wXZyQbwovD2R6iUEZb3uydN/t05gOj1R1EJ6nSReUsvJ6HN
F6iy3nLLxrqMsegaZPPBQmHyTdrQB1u2dAlFjGEiQ+ROG0SlnxI34kpqJmXd+fIl
7G5parEu9zkOQOcU2SUVWJMRYh2BM5+F2lT9D31bEdhtBowNIJDh7tr0YRdZst/y
s3BKl/6sskZTLlI4IJDEauaYZYJEdaYzZPTzTqwGLZejBai8YuDikCooK4Q/a0yH
qHJFlDL86k+XZ3nrW65op0juILbH8dWs8VIJjG49y8ChXUIu9tTu4j2Zl2mm/WOD
apweCVPUH6cGBfgkLin62T129PdSc8iKDbAJNn39UZGY5OxeAvEgSXQf79nMmTYu
w0Ntw8WvZe1g7gT6d/hk32QP76tguAwyNTnoPYuKU+bablHaHMMFNGtneXGfT5SI
gYtUobCx1oUW1WHByGerAM1E6ADaEByaXbYzYWM6VGiNRrLNPoZAYx4mP1negCG1
0oFxyRGT3ZwOKskaRFrC
=lhlr
-----END PGP SIGNATURE-----

Re: fio problems

[Jens Axboe]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 2012-04-05 03:24, Danny Kukawka wrote:
> Am 04.04.2012 22:15, schrieb Jens Axboe:
>> On 2012-04-04 13:50, Jens Axboe wrote:
>>>> 2) fio 2.0.6 and also the current git tends to fail with this error:
>>
>>>>    fio 2.0.6
>>>>    Starting 1 process
>>>>    fio: iolog.c:242: log_io_piece: Assertion `ipo->len == __ipo->len'
>>>>    failed.:08m:31s]
>>
>>>> It seems that we can prevent it when we use the --size parameter, but we
>>>> would like to prevent the usage of the parameter, if possible
>>
>>> Strangely, I had two reports of this today. Did a bit of digging, and
>>> it's due to this commit:
>>
>>> commit c04e4661e4da3b6079f415897e4507cf8e610c54
>>> Author: Daniel Ehrenberg <dehrenberg@google.com>
>>> Date:   Fri Mar 16 18:54:15 2012 +0100
>>
>>>     time_based: Avoid restarting main I/O loop
>>
>>> I think what is going on here is that your job is time based, and when
>>> we fail getting a new offset, we clear the offset bitmap. But we could
>>> have verifies pending at this point, we'd need to prune this out as
>>> well.
>>
>> Actually, since it happens so rarely, I think the better approach is
>> just to ignore it. I've updated fio to reflect that.
> 
> Btw. I can suppress the error if I add --size with a high number e.g.
> --size=1p.

It'll still happen eventually, it just takes longer. But the fix is
committed, so you should be fine going forward.

- -- 
Jens Axboe

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBCAAGBQJPfbAlAAoJEPfTWPspceCmNBYP/2L5T6H/2rqy7VKEJgSCR+VW
ky/P+kEIPslWPgGK3yyy4A+c/uIaYpXZux2LSeXG0eoEsFNMv4Bp+bHEmm33G9SP
uJMieUK4F0yif0OrSNgeSFLLhYc86t6WgkBZCS+d/KFKH0VfHnUTD+MZY/6O24sd
A6+fXX3xXriYeThkIM+ffzKpsLiH7uxWtRpUN+1n9XhrhOqmIpuRZSOWPdh37R9D
81mbPwnLkicLyknji9jaIiZn7NrXR988wsxumKrPe27+acSg9pE/xbynks3OEBUU
A+LlwNw9//5q0iyF0sxFMmGs5hOp8Fl5atjYwUs9S2R3ZODOgUeONpHO3RNAAwsZ
5446gs+2HwCzZuSpSLbEQa4FSJH8IPV9b2z/RgOpVmMb2dnIuiOKGvpdOhwqikpE
S7TQ1Mq+uF/H+30VyEbnErYoKE7DdfJWok0C7V5yDVlnMQaV8BM+l40jLWfEiH0C
9FcmDKjNeI3imeHIwan/TCcQxLB/QG6RdVt6801bJN8Ol+HdpOiMRiT5yAaECTl0
oBMs5k4MYlcy1dkKOdzBKu7MbXYYzpT42dOzwvtTX6Wq+2SIyPnci/qcBCpEpsjo
of6Arl3mqgimCvtScqZTgeSviGUrkD7JtonYGYDm5MciDtFT8zWCcxnIKq7M9Wot
mXMTCXuPdaajyco2DBiH
=O8h4
-----END PGP SIGNATURE-----

Re: fio problems

[Danny Kukawka]

[-- Attachment #1: Type: text/plain, Size: 2520 bytes --]

Am 05.04.2012 16:42, schrieb Jens Axboe:
> On 2012-04-05 02:51, Danny Kukawka wrote:
>>>> 3) fio always reports minb == maxb and aggrb < minb as you can see
>>>> above. From my understanding this would be false info.
>>>
>>> Yeah, it's a cosmetic thing. I've never really taken the time to look at
>>> that...
> 
>> It's not cosmetic if you need correct values for your tests ...
> 
> Most people don't use the group reporting, but yes of course it should
> be correct. I have checked in a patch to correct it. The min/max group
> report bandwidth were off by 1.024, the aggregate was correct (and
> matched the "real" data output).
> 
>>>> 4) If we define rw=randrw, rwmixread=80, rwmixwrite=20 and the verify
>>>> options, what does the READ process if there isn't enough data written
>>>> out yet? Which data does READ read and verify?
>>>
>>> If you have a split workload like that and are also doing verifies, the
>>> read can be either a read verify or just an offset read. In the latter
>>> case, it doesn't verify anything. It'll just read a block (or multiple
>>> blocks) at some offset instead of writing it.
> 
>> That would may explain the fio over-report in 1).
> 
>> We create a new RBD on the cluster, mount it on the client and let fio
>> start it's test on the RBD. The RBD is completely empty at this moment.
>> Not sure what happens if fio reads some random offset that may isn't
>> filled with data at this moment. Or is fio only reading from a prior
>> written block/offset?
> 
> OK, if we back up a bit. If you have a workload that is rwmixread=50,
> hence doing equal amount of random reads and writes. Then the job will
> read 50% of the disk, and write 50% of the disk. None of the read and
> write offsets will be the same. So if the drive is initially empty, you
> are reading non-initialized data.
> 
> Same thing is true for your workload. So some of the reads you see will
> be the verifies, but some of them will also be for parts of the disk
> that you will then never write.

We solved it now by running an initial fio write over the full RBD
before running any verify tests:

 fio --name=fill-rbd --filename=/dev/rbd1 --direct=1 --rw=write \
     --bs=4M --size=25g --ioengine=libaio --iodepth=128

Now we don't see problems with over-reporting fio anymore.

Danny
-- 
Danny Kukawka
 SUSE LINUX Products GmbH
  Maxfeldstrasse 5, D-90409 Nuremberg, Germany
  GF: Jeff Hawn,Jennifer Guild,Felix Imendörffer,HRB16746 (AG Nürnberg)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: fio problems

[Jens Axboe]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 2012-04-05 09:30, Danny Kukawka wrote:
>>> We create a new RBD on the cluster, mount it on the client and let fio
>>> start it's test on the RBD. The RBD is completely empty at this moment.
>>> Not sure what happens if fio reads some random offset that may isn't
>>> filled with data at this moment. Or is fio only reading from a prior
>>> written block/offset?
>>
>> OK, if we back up a bit. If you have a workload that is rwmixread=50,
>> hence doing equal amount of random reads and writes. Then the job will
>> read 50% of the disk, and write 50% of the disk. None of the read and
>> write offsets will be the same. So if the drive is initially empty, you
>> are reading non-initialized data.
>>
>> Same thing is true for your workload. So some of the reads you see will
>> be the verifies, but some of them will also be for parts of the disk
>> that you will then never write.
> 
> We solved it now by running an initial fio write over the full RBD
> before running any verify tests:
> 
>  fio --name=fill-rbd --filename=/dev/rbd1 --direct=1 --rw=write \
>      --bs=4M --size=25g --ioengine=libaio --iodepth=128
> 
> Now we don't see problems with over-reporting fio anymore.

Excellent, so it was down to reading un-written data. Thanks for
confirming.

- -- 
Jens Axboe

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBCAAGBQJPfb0qAAoJEPfTWPspceCmzawP/RLbtj62iOVuFJm5rK42Mmq0
Texy8U9RJz4D4h5oc9xAQkvlavBGPWnf0WgsnbkSr7tc/STtqVg5GW6Y8ybvfAkw
2OlZt5T/vb23uz8L9OKp+HCjV6p6Mq7lbksAcddeZdPp9GyzE+v/Cby7LMjIRgjd
OPtq759KCb/fr2gWYpxJyyyZlXost1RrRnE+OsnKhThqIYFg42G4IpdmaZGbLeTi
3EeXTNE5RDSplnuu1BsE9jffaTZuTbjm4HbCiMyixXEtp/dZBrS4tlmClC7bpjdt
SQMp68wBvji26f7K91JcjENvagGBEmi6SRJxCUgNmmZhKr3Kaf39JzbN9zEPIHOp
zOlWomS0D9Qpj0Sh24x81nlQXbWDP0aLmrggH6w5rlqLGl3pq1kz3Wgn+RkwcObw
ri8JYy8aVqT5YCVNIriIW5rQCg7kmJp170ZH6YaPd7+SeiifX8ICWFplbKHjF5Vs
vnnysT9gYEH0PLqpcoVHaYV2SFKEWVtHo7yvpqsyVevdP6uwFFXg83pAERDPsxcc
ro2VOhdJON+AhcgdgUmyfWxfcbDQnpyuw5v3/lMc2LB1oqgfKjXgW6CxqOh+tPmX
IUrpomRX+6B4WxPBxCm1Ivpr7Vja9DdFspKBZhRH4i7LErwWKTVMBKv6CaS+zxZn
bK8sy3IbjEeGP+Mv+aHs
=RYtK
-----END PGP SIGNATURE-----

Usage of group reporting

[Lucian Grijincu]

On 4/5/12 7:42 AM, "Jens Axboe" <axboe@kernel.dk> wrote:
> Most people don't use the group reporting


Why do you say that?

For example I'm trying to find those parameters (direct-io, fadvise,
number of threads, libaio/mmap/sync, etc.) that give an overall "best
performance" (whatever that means) on a simulated workload.

I'm using group reporting, because I want numbers for the entire system,
not for an individual thread.

Am I using "fio" wrong or is "fio" mostly used for other kinds of
benchmarking?
--
Lucian

Re: Usage of group reporting

[Jens Axboe]

On 2012-04-05 12:29, Lucian Grijincu wrote:
> 
> On 4/5/12 7:42 AM, "Jens Axboe" <axboe@kernel.dk> wrote:
>> Most people don't use the group reporting
> 
> 
> Why do you say that?
> 
> For example I'm trying to find those parameters (direct-io, fadvise,
> number of threads, libaio/mmap/sync, etc.) that give an overall "best
> performance" (whatever that means) on a simulated workload.
> 
> I'm using group reporting, because I want numbers for the entire system,
> not for an individual thread.
> 
> Am I using "fio" wrong or is "fio" mostly used for other kinds of
> benchmarking?

I think there's a confusion of termininology here, it's my fault. So
when I said group reporting, I meant the group status reporting. The
part that shows up at the end of the reporting:

Run status group 0 (all jobs):
   READ: io=2369KB, aggrb=1024KB/s, minb=1024KB/s, maxb=1024KB/s, mint=2313msec, maxt=2313msec

This is what you control with the new_group, forcing fio to group
multiple jobs together for reporting.

You are probably referring to group_reporting, which is used quite
extensively. That is not the one I meant :-)

-- 
Jens Axboe