Re: fio memory usage

[Jens Axboe <axboe@kernel.dk>]

On 04/19/2012 12:59 AM, Vikram Seth wrote:
> On Wed, Apr 18, 2012 at 11:54 AM, Jens Axboe <axboe@kernel.dk> wrote:
>> On 2012-04-18 20:49, Jens Axboe wrote:
>>> On 2012-04-18 17:41, Vikram Seth wrote:
>>>> Hi Jens,
>>>>
>>>> What's the max memory used per job (per device) for fio? Is there a
>>>> rule of thumb for min memory needed for fio?
>>>> Like, if I am running N #threads with numjobs on M #devices in the
>>>> system, then I'd like to know if I have enough memory before I start
>>>> the test rather than I wait for it to crashes days later.
>>>>
>>>> Also, in case fio finds that it is running out of memory while
>>>> running, then does it generate an OOM kind of message in the output
>>>> file that can be used to track the reason for a crashed test.
>>>
>>> Generally, any memory that is being used is allocated before fio starts
>>> running anything. That's not always strictly true. For verify workloads,
>>> fio will store meta data for written blocks. So memory foot print could
>>> grow for that. But that's the only case that isn't limited in that
>>> sense. Fio will alloc small items while running, but now we are in the
>>> sub-kb category. Things that should not fail. And they are continually
>>> freed as well, so not persistent items.
>>>
>>> Usually IO buffers will take the most memory. You can easily calculate
>>> that, that would be queue_depth * max_buffer_size * number_of_jobs.
>>> Outside of that, fio sets up a shared memory segment. Default on Linux
>>> is 32MB. If you use a random workload and don't set norandommap, fio
>>> will allocate a device/file sized bitmap for tracking which blocks have
>>> been written or not. That consumes 1 byte per block per not-shared file.
>>> So for a 500gb drive using 4kb blocks as the minimum IO size, that'd be
>>> 122070313 blocks or ~116MB of memory for that bitmap. That'd be your
>>> biggest consumer of persistent memory, but one you can usually
>>> eliminate.
>>>
>>> Hope that helps...
>>
>> Oh, and any sort of logging of IO will also continually allocate memory.
>> That'd be options like write_*_log.
>>
>> And I forgot to touch on what fio does if memory allocations fail. It
>> crashes. No attempts are being made at handling memory allocation
>> failures. That is something that should most likely be improved. In
>> reality most people run on Linux where allocations just don't fail with
>> the default settings, so what would happen is that the OOM killer would
>> terminate fio instead. I'm not sure what other platforms default to
>> here, but this fact has made making handling allocation failures a
>> smaller priority. In reality, what should be done, is have fio just exit
>> and report status in case of allocation failures. It would not be hard
>> to add.
>>
>> --
>> Jens Axboe
>>
> 
> Thanks Jens. That was really helpful.
> 
> OOM killer should output the details in syslog, about what process it
> killed etc.
> As long as that is true, we have a way to figure out after a crash
> whether if fio ran out of memory so that we know where to look
> further.

Correct.

> However, I think having fio report fail status in output file on
> memory allocation failure, will remove any dependencies on OS and its
> reporting of such a failure.

That would indeed be ideal and not too hard to do. Mainly fio would need
to ensure that it does not do any allocations in the exit report path.

-- 
Jens Axboe