Re: IOPS higher than expected on randwrite, direct=1 tests

[Jens Axboe <jaxboe@fusionio.com>]

On 2010-11-10 18:18, Sebastian Kayser wrote:
> * Jens Axboe <jaxboe@fusionio.com> wrote:
>> On 2010-11-10 09:22, Sebastian Kayser wrote:
>>> Just to make sure my understanding is correct:
>>> - direct=1 should mitigate (disable?) OS caching effects
>>> - sync=1, iodepth=1 should make sure that an I/O has really made it to
>>>   disk before the next on is issued, i.e. should de-facto disable
>>>   I/O coalescing or device caching
>>>
>>> Are these sane/valid assumptions?
>>
>> Yes, your assumptions are valid. I think the issue here is that you give
>> randwrite, but don't also specify that you want overwrites. So what
>> happens is that the file will be truncated and then randomly written,
>> allowing the file system to place randomly chosen file block together.
>> If you remove the test file and re-run with overwrite=1 added, then see
>> if those results are more in line with what you expect.
> 
> Thanks for the additional aspect! So the possible pitfall is that w/o
> overwrite=1 the test file could be sparse and the file system then
> re-arrange random file addresses to fall onto consecutive device blocks?
> 
> (If so, wouldn't that be evil / deliberate data fragmentation?)

Depends on whether you care about the write performance or the read-back
performance :-)

> Tested it again with overwrite=1, same IOPS figures. Test file size as
> measured with du -ks is the same in both cases btw. (which combined with
> the limited runtime=120 doesn't seem to indicate sparseness in case of
> overwrite=0, right?).

Was the file pre-created? I'm pretty sure that fio will not lay it out
first with a write workload, even for random writes (I could be wrong
though, and if the file is indeed non-sparse and the test stopped before
all IO was written, then it would have been sparse for this case).

> What I will do now is to export the whole 2TB of the disk (instead of
> just 10GB) and increase size= to see whether that makes any difference
> (hopefully). Other than that, further ideas?

Lets see what that yields.

-- 
Jens Axboe