From: Jens Axboe <axboe@kernel.dk>
To: brian arb <brianjamesarb@gmail.com>
Cc: fio@vger.kernel.org
Subject: Re: fio is being killed by the oom-killer after fio verify runs for some time ~13 hours
Date: Fri, 11 Jan 2013 10:10:10 +0100 [thread overview]
Message-ID: <20130111091010.GA32674@kernel.dk> (raw)
In-Reply-To: <CABYizFKetDGfE_fOARgbehmJo_uDTqy32bMTQTDVyny-Nk5riQ@mail.gmail.com>
On Thu, Jan 10 2013, brian arb wrote:
> Seems fio is being killed by the oom-killer after fio verify runs for
> some time ~13 hours. What parameters can I tweak or how can I run my
> test differently so the test will be completed with out interruption?
You are probably running into OOM issues since each completed write will
log some meta data to help verify that later. The easiest fix for you
would be to verify continously, setting a backlog of how old data can
get before being verified. See verify_backlog and verify_async for that.
You should also upgrade your fio. Fio uses a random map for tracking
what has been written. It's static memory, so it wont cause your OOM
during runtime, but it will gobble up some memory when you start. If you
upgrade to 2.0.13 and use random_distribution=lfsr, then that memory
consumption will go away.
There's room for a bit of improvement on fio for verification. Since IO
buffer contents and offsets etc are fully randomized with specific
seeding, it is possible to verify what has been written without storing
this meta data. Basically verify can just re-create the contents for
verification, instead of storing a checksum of it. That will cost some
CPU, but it will get you more predictable (and much lower) memory
consumption numbers. I will look into that. But as a starter, the above
suggestions should help you out.
--
Jens Axboe
next prev parent reply other threads:[~2013-01-11 9:10 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-01-10 20:57 fio is being killed by the oom-killer after fio verify runs for some time ~13 hours brian arb
2013-01-10 21:09 ` Chris Worley
2013-01-11 9:10 ` Jens Axboe [this message]
2013-01-12 8:51 ` Jens Axboe
2013-01-18 17:25 ` brian arb
2013-01-18 20:40 ` Jens Axboe
2013-01-21 21:29 ` Jens Axboe
2013-01-23 18:21 ` Jens Axboe
2013-01-23 18:52 ` brian arb
2013-01-25 20:03 ` brian arb
2013-01-25 20:06 ` brian arb
2013-01-25 21:22 ` Jens Axboe
2013-01-25 21:28 ` brian arb
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20130111091010.GA32674@kernel.dk \
--to=axboe@kernel.dk \
--cc=brianjamesarb@gmail.com \
--cc=fio@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.