From: Michal Hocko <mhocko@suse.cz>
To: Ralf-Peter Rohbeck <Ralf-Peter.Rohbeck@quantum.com>
Cc: Vlastimil Babka <vbabka@suse.cz>,
Linus Torvalds <torvalds@linux-foundation.org>,
Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>,
Oleg Nesterov <oleg@redhat.com>,
Vladimir Davydov <vdavydov@parallels.com>,
Andrew Morton <akpm@linux-foundation.org>,
Markus Trippelsdorf <markus@trippelsdorf.de>,
Arkadiusz Miskiewicz <a.miskiewicz@gmail.com>,
Jiri Slaby <jslaby@suse.com>, Olaf Hering <olaf@aepfle.de>,
Joonsoo Kim <js1304@gmail.com>, linux-mm <linux-mm@kvack.org>
Subject: Re: More OOM problems
Date: Tue, 11 Oct 2016 08:44:26 +0200 [thread overview]
Message-ID: <20161011064426.GA31996@dhcp22.suse.cz> (raw)
In-Reply-To: <982671bd-5733-0cd5-c15d-112648ff14c5@Quantum.com>
[Let's restore the CC list]
On Mon 10-10-16 10:20:27, Ralf-Peter Rohbeck wrote:
> I ran my torture test overnight (after finding the last linux-next branch
> that compiled, sigh...):
> Wrote two 4TB USB3 drives, compiled a kernel and ran my btrfs dedup script
> in parallel.
Thanks for testing and good to hear that premature OOMs are gone
> There were a few allocation failures but I didn't notice anything amiss but
> the log entries.
> Logs are at
> https://filebin.net/duj4c1bv64uohm5q/OOM_4.8.0-rc7-next-20160920.tar.bz2.
Oct 10 03:35:18 fs kernel: kworker/1:202: page allocation failure: order:0, mode:0x2204000(GFP_NOWAIT|__GFP_COMP|__GFP_NOTRACK)
Oct 10 03:35:18 fs kernel: kworker/1:214: page allocation failure: order:0, mode:0x2204000(GFP_NOWAIT|__GFP_COMP|__GFP_NOTRACK)
Oct 10 03:35:18 fs kernel: kworker/1:236: page allocation failure: order:0, mode:0x2204000(GFP_NOWAIT|__GFP_COMP|__GFP_NOTRACK)
Oct 10 03:35:18 fs kernel: kworker/1:236: page allocation failure: order:0, mode:0x2204000(GFP_NOWAIT|__GFP_COMP|__GFP_NOTRACK)
Oct 10 03:35:18 fs kernel: kworker/1:224: page allocation failure: order:0, mode:0x2204000(GFP_NOWAIT|__GFP_COMP|__GFP_NOTRACK)
Oct 10 03:35:18 fs kernel: kworker/1:224: page allocation failure: order:0, mode:0x2204000(GFP_NOWAIT|__GFP_COMP|__GFP_NOTRACK)
Oct 10 03:35:18 fs kernel: kworker/1:172: page allocation failure: order:0, mode:0x2204000(GFP_NOWAIT|__GFP_COMP|__GFP_NOTRACK)
Oct 10 03:35:18 fs kernel: kworker/1:227: page allocation failure: order:0, mode:0x2204000(GFP_NOWAIT|__GFP_COMP|__GFP_NOTRACK)
Oct 10 03:35:18 fs kernel: kworker/1:226: page allocation failure: order:0, mode:0x2204000(GFP_NOWAIT|__GFP_COMP|__GFP_NOTRACK)
Oct 10 03:35:18 fs kernel: kworker/1:229: page allocation failure: order:0, mode:0x2204000(GFP_NOWAIT|__GFP_COMP|__GFP_NOTRACK)
Oct 10 06:45:54 fs kernel: kworker/3:91: page allocation failure: order:0, mode:0x2204000(GFP_NOWAIT|__GFP_COMP|__GFP_NOTRACK)
Oct 10 06:45:54 fs kernel: kworker/3:91: page allocation failure: order:0, mode:0x2204000(GFP_NOWAIT|__GFP_COMP|__GFP_NOTRACK)
So those are all atomic (aka not sleeping) 4K allocations failing
because you are running low on memory and this kind of allocation
requests cannot reclaim any memory.
: Oct 10 03:35:18 fs kernel: Node 0 active_anon:28004kB inactive_anon:532404kB active_file:5665056kB inactive_file:1290052kB unevictable:64kB isolated(anon):0kB isolated(file):128kB mapped:46196kB dirty:686200kB writeback:124196kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 17920kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
: Oct 10 03:35:18 fs kernel: Node 0 DMA free:14236kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15980kB managed:15896kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:1660kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
: Oct 10 03:35:18 fs kernel: lowmem_reserve[]: 0 1939 7939 7939 7939
: Oct 10 03:35:18 fs kernel: Node 0 DMA32 free:40476kB min:16480kB low:20600kB high:24720kB active_anon:6472kB inactive_anon:14408kB active_file:1073784kB inactive_file:740536kB unevictable:0kB writepending:470432kB present:2072256kB managed:2006688kB mlocked:0kB slab_reclaimable:60376kB slab_unreclaimable:32844kB kernel_stack:8352kB pagetables:1984kB bounce:0kB free_pcp:164kB local_pcp:0kB free_cma:0kB
: Oct 10 03:35:18 fs kernel: lowmem_reserve[]: 0 0 5999 5999 5999
These two zones are above min watermark but still under if we consider
lowmemory reserves.
: Oct 10 03:35:18 fs kernel: Node 0 Normal free:50928kB min:50968kB low:63708kB high:76448kB active_anon:21532kB inactive_anon:517996kB active_file:4591272kB inactive_file:549636kB unevictable:64kB writepending:339940kB present:6291456kB managed:6147908kB mlocked:64kB slab_reclaimable:105320kB slab_unreclaimable:146140kB kernel_stack:17664kB pagetables:43872kB bounce:0kB free_pcp:340kB local_pcp:0kB free_cma:0kB
: Oct 10 03:35:18 fs kernel: lowmem_reserve[]: 0 0 0 0 0
and this zone is below the min watermark. I haven't checked other
allocation failures but I assume a similar situation. It looks that you
have a peak memory pressure load and kswapd just cannot catch up with it
for a moment. Note that most of those failures come within a second. You
can ignore these warnings.
I will just note that all those failures come from bcache.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next parent reply other threads:[~2016-10-11 6:44 UTC|newest]
Thread overview: 31+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <eafb59b5-0a2b-0e28-ca79-f044470a2851@Quantum.com>
[not found] ` <20160930214448.GB28379@dhcp22.suse.cz>
[not found] ` <982671bd-5733-0cd5-c15d-112648ff14c5@Quantum.com>
2016-10-11 6:44 ` Michal Hocko [this message]
2016-10-11 7:10 ` More OOM problems Vlastimil Babka
2016-10-30 4:17 ` Simon Kirby
2016-10-31 21:41 ` Vlastimil Babka
2016-10-31 21:51 ` Vlastimil Babka
2016-09-18 20:03 Linus Torvalds
2016-09-18 20:26 ` Lorenzo Stoakes
2016-09-18 20:58 ` Linus Torvalds
2016-09-18 21:13 ` Vlastimil Babka
2016-09-18 21:34 ` Lorenzo Stoakes
2016-09-19 8:32 ` Michal Hocko
2016-09-19 8:42 ` Lorenzo Stoakes
2016-09-19 8:53 ` Michal Hocko
2016-09-25 21:48 ` Lorenzo Stoakes
2016-09-26 7:48 ` Michal Hocko
2016-09-18 21:00 ` Vlastimil Babka
2016-09-18 21:18 ` Linus Torvalds
2016-09-19 6:27 ` Jiri Slaby
2016-09-19 7:01 ` Michal Hocko
2016-09-19 7:52 ` Michal Hocko
2016-09-19 1:07 ` Andi Kleen
[not found] ` <alpine.DEB.2.20.1609190836540.12121@east.gentwo.org>
2016-09-19 14:31 ` Andi Kleen
2016-09-19 14:39 ` Michal Hocko
2016-09-19 14:41 ` Vlastimil Babka
2016-09-19 18:18 ` Linus Torvalds
2016-09-19 19:57 ` Christoph Lameter
2016-09-18 22:00 ` Vlastimil Babka
2016-09-19 6:56 ` Michal Hocko
2016-09-19 6:48 ` Michal Hocko
2016-09-21 7:04 ` Raymond Jennings
2016-09-21 7:29 ` Michal Hocko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20161011064426.GA31996@dhcp22.suse.cz \
--to=mhocko@suse.cz \
--cc=Ralf-Peter.Rohbeck@quantum.com \
--cc=a.miskiewicz@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=js1304@gmail.com \
--cc=jslaby@suse.com \
--cc=linux-mm@kvack.org \
--cc=markus@trippelsdorf.de \
--cc=olaf@aepfle.de \
--cc=oleg@redhat.com \
--cc=penguin-kernel@i-love.sakura.ne.jp \
--cc=torvalds@linux-foundation.org \
--cc=vbabka@suse.cz \
--cc=vdavydov@parallels.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).