From: Luis Henriques <lhenriques@suse.com>
To: Dennis Zhou <dennisszhou@gmail.com>
Cc: Tejun Heo <tj@kernel.org>, Christoph Lameter <cl@linux.com>,
linux-mm@kvack.org, linux-kernel@vger.kernel.org
Subject: Re: percpu allocation failures
Date: Wed, 27 Sep 2017 10:05:44 +0100 [thread overview]
Message-ID: <87lgl0o6jb.fsf@hermes> (raw)
In-Reply-To: <20170927015323.GA19100@Big-Sky.local> (Dennis Zhou's message of "Tue, 26 Sep 2017 20:54:00 -0500")
[-- Attachment #1: Type: text/plain, Size: 1038 bytes --]
Dennis Zhou <dennisszhou@gmail.com> writes:
> Hi Luis,
>
> This seems to be an issue with the reserved chunk being unable to
> allocate memory when loading kernel modules. Unfortunately, I have not
> been successful in reproducing this with the reserved chunk allocation
> path exposed or by inserting the nft_meta module.
>
> Could you please send me the output when ran with the following patch
> and the output of the percpu memory statistics file before and after
> inserting the module (PERCPU_STATS)? The stats are in
> /sys/kernel/debug/percpu_stats.
Please find attached all the info you requested. Hope it helps.
Also, here's the nft script I'm using to trigger the issue locally:
#!/usr/sbin/nft -f
flush ruleset
table inet filter {
chain input {
type filter hook input priority 0;
iif lo accept
ct state established,related accept
tcp dport { 22 } ct state new accept
ip6 nexthdr icmpv6 icmpv6 type { nd-neighbor-solicit, nd-router-advert, nd-neighbor-advert } accept
counter drop
}
}
Cheers,
--
Luis
[-- Attachment #2: percpu_stats.before --]
[-- Type: application/octet-stream, Size: 2031 bytes --]
Percpu Memory Statistics
Allocation Info:
----------------------------------------
unit_size : 262144
static_size : 110488
reserved_size : 8192
dyn_size : 28776
atom_size : 2097152
alloc_size : 2097152
Global Stats:
----------------------------------------
nr_alloc : 2204
nr_dealloc : 764
nr_cur_alloc : 1440
nr_max_alloc : 1638
nr_chunks : 2
nr_max_chunks : 2
min_alloc_size : 4
max_alloc_size : 4104
empty_pop_pages : 2
Per Chunk Stats:
----------------------------------------
Chunk: <- Reserved Chunk
nr_alloc : 4
max_alloc_size : 4104
empty_pop_pages : 0
first_bit : 999
free_bytes : 4004
contig_bytes : 8
sum_frag : 0
max_frag : 0
cur_min_alloc : 0
cur_med_alloc : 0
cur_max_alloc : 0
Chunk: <- First Chunk
nr_alloc : 113
max_alloc_size : 976
empty_pop_pages : 0
first_bit : 8192
free_bytes : 0
contig_bytes : 0
sum_frag : 0
max_frag : 0
cur_min_alloc : 4
cur_med_alloc : 976
cur_max_alloc : 976
Chunk:
nr_alloc : 1323
max_alloc_size : 4096
empty_pop_pages : 2
first_bit : 11005
free_bytes : 162604
contig_bytes : 155648
sum_frag : 6956
max_frag : 2808
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
[-- Attachment #3: percpu_stats.after --]
[-- Type: application/octet-stream, Size: 2031 bytes --]
Percpu Memory Statistics
Allocation Info:
----------------------------------------
unit_size : 262144
static_size : 110488
reserved_size : 8192
dyn_size : 28776
atom_size : 2097152
alloc_size : 2097152
Global Stats:
----------------------------------------
nr_alloc : 2215
nr_dealloc : 765
nr_cur_alloc : 1450
nr_max_alloc : 1638
nr_chunks : 2
nr_max_chunks : 2
min_alloc_size : 4
max_alloc_size : 4104
empty_pop_pages : 2
Per Chunk Stats:
----------------------------------------
Chunk: <- Reserved Chunk
nr_alloc : 7
max_alloc_size : 4104
empty_pop_pages : 0
first_bit : 2052
free_bytes : 3976
contig_bytes : 3976
sum_frag : 0
max_frag : 0
cur_min_alloc : 0
cur_med_alloc : 0
cur_max_alloc : 0
Chunk: <- First Chunk
nr_alloc : 113
max_alloc_size : 976
empty_pop_pages : 0
first_bit : 8192
free_bytes : 0
contig_bytes : 0
sum_frag : 0
max_frag : 0
cur_min_alloc : 4
cur_med_alloc : 976
cur_max_alloc : 976
Chunk:
nr_alloc : 1330
max_alloc_size : 4096
empty_pop_pages : 2
first_bit : 11005
free_bytes : 160492
contig_bytes : 155648
sum_frag : 4844
max_frag : 2808
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
[-- Attachment #4: dmesg.debug --]
[-- Type: application/octet-stream, Size: 5911 bytes --]
[ 87.709692] percpu: reserved chunk: 4004, 2, 2030, 999, 3992, 104, 3
[ 87.709694] percpu: rchunk md blocks
[ 87.709697] percpu: 1, 999, 0, 0, 999
[ 87.709699] percpu: 2, 1006, 0, 0, 1006
[ 87.709701] percpu: 998, 0, 998, 0, 0
[ 87.709704] percpu: pcpu_find_block_fit: -1, 4, 4
[ 87.709707] percpu: allocation failed, size=16 align=16 atomic=0, alloc from reserved chunk failed to find fit
[ 87.709717] CPU: 0 PID: 1328 Comm: modprobe Tainted: G E 4.14.0-rc2+ #123
[ 87.709721] Hardware name: Dell Inc. Precision 5510/0N8J4R, BIOS 1.2.25 05/07/2017
[ 87.709723] Call Trace:
[ 87.709738] dump_stack+0x63/0x89
[ 87.709745] pcpu_alloc+0x6c8/0x720
[ 87.709753] __alloc_reserved_percpu+0x18/0x20
[ 87.709760] load_module+0x733/0x2c00
[ 87.709769] ? kernel_read_file+0x1a3/0x1d0
[ 87.709776] SYSC_finit_module+0xfc/0x120
[ 87.709781] ? SYSC_finit_module+0xfc/0x120
[ 87.709789] SyS_finit_module+0xe/0x10
[ 87.709795] entry_SYSCALL_64_fastpath+0x1e/0xa9
[ 87.709800] RIP: 0033:0x7f211163f219
[ 87.709804] RSP: 002b:00007ffe68de7318 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[ 87.709809] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f211163f219
[ 87.709812] RDX: 0000000000000000 RSI: 000055dc7a86a8cc RDI: 0000000000000000
[ 87.709815] RBP: 00007ffe68de6310 R08: 0000000000000000 R09: 0000000000000001
[ 87.709818] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000005
[ 87.709821] R13: 000055dc7c54c600 R14: 00007ffe68de62f0 R15: 0000000000000005
[ 87.709827] nft_meta: Could not allocate 16 bytes percpu data
[ 87.749131] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[ 87.749791] percpu: reserved chunk: 4004, 2, 2030, 999, 3992, 104, 3
[ 87.749793] percpu: rchunk md blocks
[ 87.749796] percpu: 1, 999, 0, 0, 999
[ 87.749798] percpu: 2, 1006, 0, 0, 1006
[ 87.749800] percpu: 998, 0, 998, 0, 0
[ 87.749802] percpu: pcpu_find_block_fit: 2030, 2, 2
[ 87.749806] percpu: pcpu_alloc_area: 8120, 2, 2
[ 87.753575] percpu: reserved chunk: 3996, 1, 999, 999, 3992, 104, 3
[ 87.753578] percpu: rchunk md blocks
[ 87.753580] percpu: 1, 999, 0, 0, 999
[ 87.753582] percpu: 0, 1006, 0, 0, 1024
[ 87.753584] percpu: 998, 0, 998, 0, 0
[ 87.753587] percpu: pcpu_find_block_fit: -1, 4, 4
[ 87.753590] percpu: allocation failed, size=16 align=16 atomic=0, alloc from reserved chunk failed to find fit
[ 87.753599] CPU: 1 PID: 1334 Comm: modprobe Tainted: G E 4.14.0-rc2+ #123
[ 87.753603] Hardware name: Dell Inc. Precision 5510/0N8J4R, BIOS 1.2.25 05/07/2017
[ 87.753605] Call Trace:
[ 87.753619] dump_stack+0x63/0x89
[ 87.753626] pcpu_alloc+0x6c8/0x720
[ 87.753635] __alloc_reserved_percpu+0x18/0x20
[ 87.753642] load_module+0x733/0x2c00
[ 87.753650] ? kernel_read_file+0x1a3/0x1d0
[ 87.753657] SYSC_finit_module+0xfc/0x120
[ 87.753663] ? SYSC_finit_module+0xfc/0x120
[ 87.753671] SyS_finit_module+0xe/0x10
[ 87.753677] entry_SYSCALL_64_fastpath+0x1e/0xa9
[ 87.753681] RIP: 0033:0x7fb1b2421219
[ 87.753685] RSP: 002b:00007ffe287a6d88 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[ 87.753690] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fb1b2421219
[ 87.753693] RDX: 0000000000000000 RSI: 00005622387988cc RDI: 0000000000000000
[ 87.753696] RBP: 00007ffe287a5d80 R08: 0000000000000000 R09: 0000000000000001
[ 87.753699] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000005
[ 87.753702] R13: 0000562239046600 R14: 00007ffe287a5d60 R15: 0000000000000005
[ 87.753708] nft_meta: Could not allocate 16 bytes percpu data
[ 87.782057] percpu: reserved chunk: 3996, 1, 999, 999, 3992, 104, 3
[ 87.782060] percpu: rchunk md blocks
[ 87.782062] percpu: 1, 999, 0, 0, 999
[ 87.782065] percpu: 0, 1006, 0, 0, 1024
[ 87.782067] percpu: 998, 0, 998, 0, 0
[ 87.782069] percpu: pcpu_find_block_fit: -1, 4, 4
[ 87.782072] percpu: allocation failed, size=16 align=16 atomic=0, alloc from reserved chunk failed to find fit
[ 87.782081] CPU: 0 PID: 1336 Comm: modprobe Tainted: G E 4.14.0-rc2+ #123
[ 87.782085] Hardware name: Dell Inc. Precision 5510/0N8J4R, BIOS 1.2.25 05/07/2017
[ 87.782087] Call Trace:
[ 87.782099] dump_stack+0x63/0x89
[ 87.782107] pcpu_alloc+0x6c8/0x720
[ 87.782115] __alloc_reserved_percpu+0x18/0x20
[ 87.782121] load_module+0x733/0x2c00
[ 87.782129] ? kernel_read_file+0x1a3/0x1d0
[ 87.782136] SYSC_finit_module+0xfc/0x120
[ 87.782141] ? SYSC_finit_module+0xfc/0x120
[ 87.782149] SyS_finit_module+0xe/0x10
[ 87.782155] entry_SYSCALL_64_fastpath+0x1e/0xa9
[ 87.782159] RIP: 0033:0x7f4dd4073219
[ 87.782163] RSP: 002b:00007ffefd82e658 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[ 87.782169] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f4dd4073219
[ 87.782172] RDX: 0000000000000000 RSI: 000055d5cd3fd8cc RDI: 0000000000000000
[ 87.782175] RBP: 00007ffefd82d650 R08: 0000000000000000 R09: 0000000000000001
[ 87.782178] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000005
[ 87.782181] R13: 000055d5ce138600 R14: 00007ffefd82d630 R15: 0000000000000005
[ 87.782186] nft_meta: Could not allocate 16 bytes percpu data
[ 87.822555] percpu: reserved chunk: 3996, 1, 999, 999, 3992, 104, 3
[ 87.822557] percpu: rchunk md blocks
[ 87.822560] percpu: 1, 999, 0, 0, 999
[ 87.822562] percpu: 0, 1006, 0, 0, 1024
[ 87.822564] percpu: 998, 0, 998, 0, 0
[ 87.822567] percpu: pcpu_find_block_fit: 999, 1, 1
[ 87.822571] percpu: pcpu_alloc_area: 3996, 1, 1
[ 87.897805] percpu: reserved chunk: 3992, 998, 2048, 2048, 3992, 104, 3
[ 87.897807] percpu: rchunk md blocks
[ 87.897809] percpu: 0, 999, 0, 0, 1024
[ 87.897812] percpu: 0, 1006, 0, 0, 1024
[ 87.897814] percpu: 998, 0, 998, 0, 0
[ 87.897816] percpu: pcpu_find_block_fit: 2048, 4, 4
prev parent reply other threads:[~2017-09-27 9:05 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-09-26 15:37 percpu allocation failures Luis Henriques
2017-09-26 15:37 ` Luis Henriques
2017-09-27 1:54 ` Dennis Zhou
2017-09-27 1:54 ` Dennis Zhou
2017-09-27 9:05 ` Luis Henriques [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87lgl0o6jb.fsf@hermes \
--to=lhenriques@suse.com \
--cc=cl@linux.com \
--cc=dennisszhou@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.