From: Christoph Hellwig <hch@lst.de>
To: Andrew Morton <akpm@linux-foundation.org>
Cc: Christoph Hellwig <hch@lst.de>, Al Viro <viro@zeniv.linux.org.uk>,
bugzilla-daemon@bugzilla.kernel.org, gaetan@nexylan.com,
linux-mm@kvack.org, Alexey Dobriyan <adobriyan@gmail.com>,
Josef Bacik <josef@toxicpanda.com>
Subject: Re: [Bug 209245] New: 5.8.x Cannot allocate memory requesting sysctl
Date: Sun, 13 Sep 2020 08:33:02 +0200 [thread overview]
Message-ID: <20200913063302.GA6620@lst.de> (raw)
In-Reply-To: <20200912173854.8ec0189616e541691644c4d0@linux-foundation.org>
On Sat, Sep 12, 2020 at 05:38:54PM -0700, Andrew Morton wrote:
> (switched to email. Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
This is probably fixed by the
"[PATCH 0/6] Some buffer management fixes for proc"
series from Josef.
>
> On Sat, 12 Sep 2020 09:12:18 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:
>
> > https://bugzilla.kernel.org/show_bug.cgi?id=209245
> >
> > Bug ID: 209245
> > Summary: 5.8.x Cannot allocate memory requesting sysctl
> > Product: Memory Management
> > Version: 2.5
> > Kernel Version: 5.8.x
> > Hardware: All
> > OS: Linux
> > Tree: Mainline
> > Status: NEW
> > Severity: high
> > Priority: P1
> > Component: Page Allocator
> > Assignee: akpm@linux-foundation.org
> > Reporter: gaetan@nexylan.com
> > Regression: No
>
> Actually, "Regression: yes".
>
> > Created attachment 292483
> > --> https://bugzilla.kernel.org/attachment.cgi?id=292483&action=edit
> > dmesg
> >
> > Since we upgraded a few nodes to 5.8.x (Gentoo Linux), we are having random
> > "Cannot allocate memory" when requesting infos from sysctl despite system has
> > free memory available.
> >
> > Steps to reproduce :
> >
> > 1/ Have system with heavy workload under 5.8.2 (reproduced with 5.8.8)
> > 2/ At some point, accessing sysctl is not possible anymore :
> >
> >
> > ~# cat /proc/sys/net/netfilter/nf_conntrack_count
> > cat: /proc/sys/net/netfilter/nf_conntrack_count: Cannot allocate memory
> >
> >
> > 3/ Dmesg drops memory allocation errors (see attached)
> >
> >
> > Memory usage :
> >
> > total used free shared buff/cache available
> > Mem: 32009 22029 1547 0 8431 9506
> > Swap: 7811 621 7190
> >
> > Node is running a single mongodb process.
> >
>
> The out-of-memory dump is below.
>
> proc_sys_call_handler() is attempting an order-5 kmalloc for a read of
> a simple int. Presumably because cat(1) is starting out with a 128kb
> read().
>
> AFAICT this is due to changes made in 32927393dc1cc ("sysctl: pass
> kernel pointers to ->proc_handler")?
>
>
>
>
> [Sat Sep 12 11:07:55 2020] cat: page allocation failure: order:5, mode:0x40dc0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null),cpuset=/,mems_allowed=0
> [Sat Sep 12 11:07:55 2020] CPU: 10 PID: 32410 Comm: cat Not tainted 5.8.8-gentoo #1
> [Sat Sep 12 11:07:55 2020] Hardware name: Supermicro MBI-6119M-T2N/B2SC1-CPU, BIOS 1.1 06/25/2019
> [Sat Sep 12 11:07:55 2020] Call Trace:
> [Sat Sep 12 11:07:55 2020] dump_stack+0x6d/0x8b
> [Sat Sep 12 11:07:55 2020] warn_alloc+0xf6/0x160
> [Sat Sep 12 11:07:55 2020] __alloc_pages_slowpath.constprop.123+0xb28/0xb70
> [Sat Sep 12 11:07:55 2020] ? __alloc_pages_nodemask+0x144/0x2e0
> [Sat Sep 12 11:07:55 2020] __alloc_pages_nodemask+0x2af/0x2e0
> [Sat Sep 12 11:07:55 2020] kmalloc_order+0x13/0x70
> [Sat Sep 12 11:07:55 2020] kmalloc_order_trace+0x19/0xa0
> [Sat Sep 12 11:07:55 2020] proc_sys_call_handler+0xf9/0x170
> [Sat Sep 12 11:07:55 2020] vfs_read+0x96/0x170
> [Sat Sep 12 11:07:55 2020] ksys_read+0x9c/0xd0
> [Sat Sep 12 11:07:55 2020] do_syscall_64+0x44/0x70
> [Sat Sep 12 11:07:55 2020] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [Sat Sep 12 11:07:55 2020] RIP: 0033:0x7f2c7a19ac7b
> [Sat Sep 12 11:07:55 2020] Code: 00 3d 00 00 41 00 74 07 31 c0 e9 e9 fe ff ff 50 48 8d 3d 60 63 09 00 e8 3b c4 01 00 64 8b 04 25 18 00 00 00 85 c0 75 0b 0f 05 <48> 3d 00 f0 ff ff 77 4c c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
> [Sat Sep 12 11:07:55 2020] RSP: 002b:00007ffec0af8ec8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
> [Sat Sep 12 11:07:55 2020] RAX: ffffffffffffffda RBX: 0000000000020000 RCX: 00007f2c7a19ac7b
> [Sat Sep 12 11:07:55 2020] RDX: 0000000000020000 RSI: 00007f2c7a097000 RDI: 0000000000000003
> [Sat Sep 12 11:07:55 2020] RBP: 00007f2c7a097000 R08: 00007f2c7a096010 R09: 0000000000000000
> [Sat Sep 12 11:07:55 2020] R10: 0000559b45d2aa00 R11: 0000000000000246 R12: 00007f2c7a097000
> [Sat Sep 12 11:07:55 2020] R13: 0000000000000003 R14: 0000000000000fff R15: 0000000000020000
> [Sat Sep 12 11:07:55 2020] Mem-Info:
> [Sat Sep 12 11:07:55 2020] active_anon:5210006 inactive_anon:373749 isolated_anon:0
> active_file:972354 inactive_file:990026 isolated_file:0
> unevictable:1958 dirty:70 writeback:0
> slab_reclaimable:198141 slab_unreclaimable:19557
> mapped:8172 shmem:29 pagetables:11756 bounce:0
> free:391832 free_pcp:0 free_cma:0
> [Sat Sep 12 11:07:55 2020] Node 0 active_anon:20840024kB inactive_anon:1494996kB active_file:3889416kB inactive_file:3960104kB unevictable:7832kB isolated(anon):0kB isolated(file):0kB mapped:32688kB dirty:280kB writeback:0kB shmem:116kB writeback_tmp:0kB all_unreclaimable? no
> [Sat Sep 12 11:07:55 2020] Node 0 DMA free:11800kB min:8kB low:20kB high:32kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15980kB managed:15896kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
> [Sat Sep 12 11:07:55 2020] lowmem_reserve[]: 0 2108 31993 31993
> [Sat Sep 12 11:07:55 2020] Node 0 DMA32 free:153020kB min:1504kB low:3656kB high:5808kB reserved_highatomic:0KB active_anon:1169844kB inactive_anon:206804kB active_file:180128kB inactive_file:209060kB unevictable:0kB writepending:0kB present:2257504kB managed:2159144kB mlocked:0kB kernel_stack:16kB pagetables:1848kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
> [Sat Sep 12 11:07:55 2020] lowmem_reserve[]: 0 0 29884 29884
> [Sat Sep 12 11:07:55 2020] Node 0 Normal free:1402508kB min:21380kB low:51980kB high:82580kB reserved_highatomic:0KB active_anon:19670180kB inactive_anon:1288192kB active_file:3709496kB inactive_file:3751000kB unevictable:7832kB writepending:280kB present:31178752kB managed:30602236kB mlocked:7832kB kernel_stack:4496kB pagetables:45176kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
> [Sat Sep 12 11:07:55 2020] lowmem_reserve[]: 0 0 0 0
> [Sat Sep 12 11:07:55 2020] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 2*4096kB (M) = 11800kB
> [Sat Sep 12 11:07:55 2020] Node 0 DMA32: 2120*4kB (UME) 9907*8kB (UME) 3974*16kB (UM) 56*32kB (UM) 3*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 153304kB
> [Sat Sep 12 11:07:55 2020] Node 0 Normal: 8036*4kB (UME) 62654*8kB (UME) 54156*16kB (UM) 155*32kB (UME) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1404832kB
> [Sat Sep 12 11:07:55 2020] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> [Sat Sep 12 11:07:55 2020] 1994278 total pagecache pages
> [Sat Sep 12 11:07:55 2020] 33986 pages in swap cache
> [Sat Sep 12 11:07:55 2020] Swap cache stats: add 205324, delete 171339, find 1160015/1183131
> [Sat Sep 12 11:07:55 2020] Free swap = 7363576kB
> [Sat Sep 12 11:07:55 2020] Total swap = 7999480kB
> [Sat Sep 12 11:07:55 2020] 8363059 pages RAM
> [Sat Sep 12 11:07:55 2020] 0 pages HighMem/MovableOnly
> [Sat Sep 12 11:07:55 2020] 168740 pages reserved
> [Sat Sep 12 11:07:55 2020] 0 pages hwpoisoned
---end quoted text---
next prev parent reply other threads:[~2020-09-13 6:33 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <bug-209245-27@https.bugzilla.kernel.org/>
2020-09-13 0:38 ` [Bug 209245] New: 5.8.x Cannot allocate memory requesting sysctl Andrew Morton
2020-09-13 6:33 ` Christoph Hellwig [this message]
2020-09-13 23:57 ` Andrew Morton
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200913063302.GA6620@lst.de \
--to=hch@lst.de \
--cc=adobriyan@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=bugzilla-daemon@bugzilla.kernel.org \
--cc=gaetan@nexylan.com \
--cc=josef@toxicpanda.com \
--cc=linux-mm@kvack.org \
--cc=viro@zeniv.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.