Re: fio problems

[Danny Kukawka <dkukawka@suse.de>]

[-- Attachment #1: Type: text/plain, Size: 2480 bytes --]

Am 04.04.2012 21:50, schrieb Jens Axboe:
> On 2012-04-04 00:39, Danny Kukawka wrote:
>> Hi,
> 
>> we try to run fio against raw RBD devices on a ceph cluster. Here are
>> the parameter we use to run fio:
> 
>> [test]
>> filename=/dev/rbd0
>> time_based
>> runtime=3h
>> rw=randrw
>> rwmixread=80
>> rwmixwrite=20
>> blocksize_range=4k-1024k
>> ioengine=libaio
>> iodepth=64
>> direct=1
>> verify=crc32c-intel
>> verify_fatal=1
>> verify_async=16
>> lockfile=exclusive
> 
>> We see some problems:
> 
>> 1) fio reports higher READ speeds than the used 1Gbit network interface
>> could provide (the interface isn't even working to capacity in this
>> situation). That happens if we use --rwmixread/rwmixwrite:
> 
>>  READ: io=91560MB, aggrb=156206KB/s, minb=159955KB/s,
>>         maxb=159955KB/s, mint=600216msec, maxt=600216msec
>>  WRITE: io=22944MB, aggrb=39143KB/s, minb=40083KB/s, maxb=40083KB/s,
>>         mint=600216msec, maxt=600216msec
> 
> OK that's very odd. How are these numbers matching up with watching
> vmstat 1 while the job runs? I've never seen fio over-report before.

I will take a look at it and let you know.

>> 3) fio always reports minb == maxb and aggrb < minb as you can see
>> above. From my understanding this would be false info.
> 
> Yeah, it's a cosmetic thing. I've never really taken the time to look at
> that...

It's not cosmetic if you need correct values for your tests ...

>> 4) If we define rw=randrw, rwmixread=80, rwmixwrite=20 and the verify
>> options, what does the READ process if there isn't enough data written
>> out yet? Which data does READ read and verify?
> 
> If you have a split workload like that and are also doing verifies, the
> read can be either a read verify or just an offset read. In the latter
> case, it doesn't verify anything. It'll just read a block (or multiple
> blocks) at some offset instead of writing it.

That would may explain the fio over-report in 1).

We create a new RBD on the cluster, mount it on the client and let fio
start it's test on the RBD. The RBD is completely empty at this moment.
Not sure what happens if fio reads some random offset that may isn't
filled with data at this moment. Or is fio only reading from a prior
written block/offset?

Danny
-- 
Danny Kukawka
 SUSE LINUX Products GmbH
  Maxfeldstrasse 5, D-90409 Nuremberg, Germany
  GF: Jeff Hawn,Jennifer Guild,Felix Imendörffer,HRB16746 (AG Nürnberg)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]