Re: fio is being killed by the oom-killer after fio verify runs for some time ~13 hours

[Jens Axboe <axboe@kernel.dk>]

On 2013-01-11 10:10, Jens Axboe wrote:
> On Thu, Jan 10 2013, brian arb wrote:
>> Seems fio is being killed by the oom-killer after fio verify runs for
>> some time ~13 hours. What parameters can I tweak or how can I run my
>> test differently so the test will be completed with out interruption?
> 
> You are probably running into OOM issues since each completed write will
> log some meta data to help verify that later. The easiest fix for you
> would be to verify continously, setting a backlog of how old data can
> get before being verified. See verify_backlog and verify_async for that.
> 
> You should also upgrade your fio. Fio uses a random map for tracking
> what has been written. It's static memory, so it wont cause your OOM
> during runtime, but it will gobble up some memory when you start. If you
> upgrade to 2.0.13 and use random_distribution=lfsr, then that memory
> consumption will go away.
> 
> There's room for a bit of improvement on fio for verification. Since IO
> buffer contents and offsets etc are fully randomized with specific
> seeding, it is possible to verify what has been written without storing
> this meta data. Basically verify can just re-create the contents for
> verification, instead of storing a checksum of it. That will cost some
> CPU, but it will get you more predictable (and much lower) memory
> consumption numbers. I will look into that. But as a starter, the above
> suggestions should help you out.

I committed the first part of this. Now I just need to double check that
we re-seed properly, then we can dump the meta data storage for the
"normal" verify workload (that has both a write and a read phase).

-- 
Jens Axboe