* percpu allocation failures
@ 2017-09-26 15:37 Luis Henriques
2017-09-27 1:54 ` Dennis Zhou
0 siblings, 1 reply; 3+ messages in thread
From: Luis Henriques @ 2017-09-26 15:37 UTC (permalink / raw)
To: Tejun Heo, Christoph Lameter, Dennis Zhou; +Cc: linux-mm, linux-kernel
Hi,
Probably already reported, but I couldn't find anything so here it
goes:
starting with 4.14-rc1 I see the following during boot:
[ 25.199053] percpu: allocation failed, size=16 align=16 atomic=0, alloc from reserved chunk failed
[ 25.200195] CPU: 5 PID: 723 Comm: modprobe Tainted: G E 4.14.0-rc2 #103
[ 25.201290] Hardware name: Dell Inc. Precision 5510/0N8J4R, BIOS 1.2.25 05/07/2017
[ 25.202430] Call Trace:
[ 25.203509] dump_stack+0x63/0x89
[ 25.204364] pcpu_alloc+0x5cd/0x5f0
[ 25.205302] __alloc_reserved_percpu+0x18/0x20
[ 25.206355] load_module+0x733/0x2c00
[ 25.207444] ? kernel_read_file+0x1a3/0x1d0
[ 25.208596] SYSC_finit_module+0xfc/0x120
[ 25.209634] ? SYSC_finit_module+0xfc/0x120
[ 25.210733] SyS_finit_module+0xe/0x10
[ 25.211747] entry_SYSCALL_64_fastpath+0x1e/0xa9
[ 25.212763] RIP: 0033:0x7f70d9e86219
[ 25.213508] RSP: 002b:00007ffcd3ff8f38 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[ 25.214391] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f70d9e86219
[ 25.215107] RDX: 0000000000000000 RSI: 00005642ddf158cc RDI: 0000000000000000
[ 25.215949] RBP: 00007ffcd3ff7f30 R08: 0000000000000000 R09: 0000000000000001
[ 25.216592] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000005
[ 25.217194] R13: 00005642de3d65f0 R14: 00007ffcd3ff7f10 R15: 0000000000000005
[ 25.217812] nft_meta: Could not allocate 16 bytes percpu data
A few more failures follow.
A bisect ended up with the merge commit a7cbfd05f427 ("Merge branch
'for-4.14' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu").
Cheers,
--
Luis
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: percpu allocation failures
2017-09-26 15:37 percpu allocation failures Luis Henriques
@ 2017-09-27 1:54 ` Dennis Zhou
2017-09-27 9:05 ` Luis Henriques
0 siblings, 1 reply; 3+ messages in thread
From: Dennis Zhou @ 2017-09-27 1:54 UTC (permalink / raw)
To: Luis Henriques; +Cc: Tejun Heo, Christoph Lameter, linux-mm, linux-kernel
Hi Luis,
This seems to be an issue with the reserved chunk being unable to
allocate memory when loading kernel modules. Unfortunately, I have not
been successful in reproducing this with the reserved chunk allocation
path exposed or by inserting the nft_meta module.
Could you please send me the output when ran with the following patch
and the output of the percpu memory statistics file before and after
inserting the module (PERCPU_STATS)? The stats are in
/sys/kernel/debug/percpu_stats.
Thanks,
Dennis
---
mm/percpu.c | 32 ++++++++++++++++++++++++++++++--
1 file changed, 30 insertions(+), 2 deletions(-)
diff --git a/mm/percpu.c b/mm/percpu.c
index 59d44d6..031fd91 100644
--- a/mm/percpu.c
+++ b/mm/percpu.c
@@ -1335,6 +1335,7 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved,
{
static int warn_limit = 10;
struct pcpu_chunk *chunk;
+ struct pcpu_block_md *block;
const char *err;
bool is_atomic = (gfp & GFP_KERNEL) != GFP_KERNEL;
int slot, off, cpu, ret;
@@ -1371,17 +1372,43 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved,
if (reserved && pcpu_reserved_chunk) {
chunk = pcpu_reserved_chunk;
+ printk(KERN_DEBUG "percpu: reserved chunk: %d, %d, %d, %d, %d, %d, %d",
+ chunk->free_bytes, chunk->contig_bits,
+ chunk->contig_bits_start, chunk->first_bit,
+ chunk->start_offset, chunk->end_offset,
+ chunk->nr_pages);
+
+ printk(KERN_DEBUG "percpu: rchunk md blocks");
+ for (block = chunk->md_blocks;
+ block < chunk->md_blocks + pcpu_chunk_nr_blocks(chunk);
+ block++) {
+ printk(KERN_DEBUG " percpu: %d, %d, %d, %d, %d",
+ block->contig_hint,
+ block->contig_hint_start,
+ block->left_free,
+ block->right_free,
+ block->first_free);
+ }
+
off = pcpu_find_block_fit(chunk, bits, bit_align, is_atomic);
+
+ printk(KERN_DEBUG "percpu: pcpu_find_block_fit: %d, %zu, %zu",
+ off, bits, bit_align);
+
if (off < 0) {
- err = "alloc from reserved chunk failed";
+ err = "alloc from reserved chunk failed to find fit";
goto fail_unlock;
}
off = pcpu_alloc_area(chunk, bits, bit_align, off);
+
+ printk(KERN_DEBUG "percpu: pcpu_alloc_area: %d, %zu, %zu",
+ off, bits, bit_align);
+
if (off >= 0)
goto area_found;
- err = "alloc from reserved chunk failed";
+ err = "alloc from reserved chunk failed to alloc area";
goto fail_unlock;
}
@@ -1547,6 +1574,7 @@ void __percpu *__alloc_reserved_percpu(size_t size, size_t align)
{
return pcpu_alloc(size, align, true, GFP_KERNEL);
}
+EXPORT_SYMBOL_GPL(__alloc_reserved_percpu);
/**
* pcpu_balance_workfn - manage the amount of free chunks and populated pages
--
1.8.3.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: percpu allocation failures
2017-09-27 1:54 ` Dennis Zhou
@ 2017-09-27 9:05 ` Luis Henriques
0 siblings, 0 replies; 3+ messages in thread
From: Luis Henriques @ 2017-09-27 9:05 UTC (permalink / raw)
To: Dennis Zhou; +Cc: Tejun Heo, Christoph Lameter, linux-mm, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 1038 bytes --]
Dennis Zhou <dennisszhou@gmail.com> writes:
> Hi Luis,
>
> This seems to be an issue with the reserved chunk being unable to
> allocate memory when loading kernel modules. Unfortunately, I have not
> been successful in reproducing this with the reserved chunk allocation
> path exposed or by inserting the nft_meta module.
>
> Could you please send me the output when ran with the following patch
> and the output of the percpu memory statistics file before and after
> inserting the module (PERCPU_STATS)? The stats are in
> /sys/kernel/debug/percpu_stats.
Please find attached all the info you requested. Hope it helps.
Also, here's the nft script I'm using to trigger the issue locally:
#!/usr/sbin/nft -f
flush ruleset
table inet filter {
chain input {
type filter hook input priority 0;
iif lo accept
ct state established,related accept
tcp dport { 22 } ct state new accept
ip6 nexthdr icmpv6 icmpv6 type { nd-neighbor-solicit, nd-router-advert, nd-neighbor-advert } accept
counter drop
}
}
Cheers,
--
Luis
[-- Attachment #2: percpu_stats.before --]
[-- Type: application/octet-stream, Size: 2031 bytes --]
Percpu Memory Statistics
Allocation Info:
----------------------------------------
unit_size : 262144
static_size : 110488
reserved_size : 8192
dyn_size : 28776
atom_size : 2097152
alloc_size : 2097152
Global Stats:
----------------------------------------
nr_alloc : 2204
nr_dealloc : 764
nr_cur_alloc : 1440
nr_max_alloc : 1638
nr_chunks : 2
nr_max_chunks : 2
min_alloc_size : 4
max_alloc_size : 4104
empty_pop_pages : 2
Per Chunk Stats:
----------------------------------------
Chunk: <- Reserved Chunk
nr_alloc : 4
max_alloc_size : 4104
empty_pop_pages : 0
first_bit : 999
free_bytes : 4004
contig_bytes : 8
sum_frag : 0
max_frag : 0
cur_min_alloc : 0
cur_med_alloc : 0
cur_max_alloc : 0
Chunk: <- First Chunk
nr_alloc : 113
max_alloc_size : 976
empty_pop_pages : 0
first_bit : 8192
free_bytes : 0
contig_bytes : 0
sum_frag : 0
max_frag : 0
cur_min_alloc : 4
cur_med_alloc : 976
cur_max_alloc : 976
Chunk:
nr_alloc : 1323
max_alloc_size : 4096
empty_pop_pages : 2
first_bit : 11005
free_bytes : 162604
contig_bytes : 155648
sum_frag : 6956
max_frag : 2808
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
[-- Attachment #3: percpu_stats.after --]
[-- Type: application/octet-stream, Size: 2031 bytes --]
Percpu Memory Statistics
Allocation Info:
----------------------------------------
unit_size : 262144
static_size : 110488
reserved_size : 8192
dyn_size : 28776
atom_size : 2097152
alloc_size : 2097152
Global Stats:
----------------------------------------
nr_alloc : 2215
nr_dealloc : 765
nr_cur_alloc : 1450
nr_max_alloc : 1638
nr_chunks : 2
nr_max_chunks : 2
min_alloc_size : 4
max_alloc_size : 4104
empty_pop_pages : 2
Per Chunk Stats:
----------------------------------------
Chunk: <- Reserved Chunk
nr_alloc : 7
max_alloc_size : 4104
empty_pop_pages : 0
first_bit : 2052
free_bytes : 3976
contig_bytes : 3976
sum_frag : 0
max_frag : 0
cur_min_alloc : 0
cur_med_alloc : 0
cur_max_alloc : 0
Chunk: <- First Chunk
nr_alloc : 113
max_alloc_size : 976
empty_pop_pages : 0
first_bit : 8192
free_bytes : 0
contig_bytes : 0
sum_frag : 0
max_frag : 0
cur_min_alloc : 4
cur_med_alloc : 976
cur_max_alloc : 976
Chunk:
nr_alloc : 1330
max_alloc_size : 4096
empty_pop_pages : 2
first_bit : 11005
free_bytes : 160492
contig_bytes : 155648
sum_frag : 4844
max_frag : 2808
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
[-- Attachment #4: dmesg.debug --]
[-- Type: application/octet-stream, Size: 5911 bytes --]
[ 87.709692] percpu: reserved chunk: 4004, 2, 2030, 999, 3992, 104, 3
[ 87.709694] percpu: rchunk md blocks
[ 87.709697] percpu: 1, 999, 0, 0, 999
[ 87.709699] percpu: 2, 1006, 0, 0, 1006
[ 87.709701] percpu: 998, 0, 998, 0, 0
[ 87.709704] percpu: pcpu_find_block_fit: -1, 4, 4
[ 87.709707] percpu: allocation failed, size=16 align=16 atomic=0, alloc from reserved chunk failed to find fit
[ 87.709717] CPU: 0 PID: 1328 Comm: modprobe Tainted: G E 4.14.0-rc2+ #123
[ 87.709721] Hardware name: Dell Inc. Precision 5510/0N8J4R, BIOS 1.2.25 05/07/2017
[ 87.709723] Call Trace:
[ 87.709738] dump_stack+0x63/0x89
[ 87.709745] pcpu_alloc+0x6c8/0x720
[ 87.709753] __alloc_reserved_percpu+0x18/0x20
[ 87.709760] load_module+0x733/0x2c00
[ 87.709769] ? kernel_read_file+0x1a3/0x1d0
[ 87.709776] SYSC_finit_module+0xfc/0x120
[ 87.709781] ? SYSC_finit_module+0xfc/0x120
[ 87.709789] SyS_finit_module+0xe/0x10
[ 87.709795] entry_SYSCALL_64_fastpath+0x1e/0xa9
[ 87.709800] RIP: 0033:0x7f211163f219
[ 87.709804] RSP: 002b:00007ffe68de7318 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[ 87.709809] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f211163f219
[ 87.709812] RDX: 0000000000000000 RSI: 000055dc7a86a8cc RDI: 0000000000000000
[ 87.709815] RBP: 00007ffe68de6310 R08: 0000000000000000 R09: 0000000000000001
[ 87.709818] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000005
[ 87.709821] R13: 000055dc7c54c600 R14: 00007ffe68de62f0 R15: 0000000000000005
[ 87.709827] nft_meta: Could not allocate 16 bytes percpu data
[ 87.749131] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[ 87.749791] percpu: reserved chunk: 4004, 2, 2030, 999, 3992, 104, 3
[ 87.749793] percpu: rchunk md blocks
[ 87.749796] percpu: 1, 999, 0, 0, 999
[ 87.749798] percpu: 2, 1006, 0, 0, 1006
[ 87.749800] percpu: 998, 0, 998, 0, 0
[ 87.749802] percpu: pcpu_find_block_fit: 2030, 2, 2
[ 87.749806] percpu: pcpu_alloc_area: 8120, 2, 2
[ 87.753575] percpu: reserved chunk: 3996, 1, 999, 999, 3992, 104, 3
[ 87.753578] percpu: rchunk md blocks
[ 87.753580] percpu: 1, 999, 0, 0, 999
[ 87.753582] percpu: 0, 1006, 0, 0, 1024
[ 87.753584] percpu: 998, 0, 998, 0, 0
[ 87.753587] percpu: pcpu_find_block_fit: -1, 4, 4
[ 87.753590] percpu: allocation failed, size=16 align=16 atomic=0, alloc from reserved chunk failed to find fit
[ 87.753599] CPU: 1 PID: 1334 Comm: modprobe Tainted: G E 4.14.0-rc2+ #123
[ 87.753603] Hardware name: Dell Inc. Precision 5510/0N8J4R, BIOS 1.2.25 05/07/2017
[ 87.753605] Call Trace:
[ 87.753619] dump_stack+0x63/0x89
[ 87.753626] pcpu_alloc+0x6c8/0x720
[ 87.753635] __alloc_reserved_percpu+0x18/0x20
[ 87.753642] load_module+0x733/0x2c00
[ 87.753650] ? kernel_read_file+0x1a3/0x1d0
[ 87.753657] SYSC_finit_module+0xfc/0x120
[ 87.753663] ? SYSC_finit_module+0xfc/0x120
[ 87.753671] SyS_finit_module+0xe/0x10
[ 87.753677] entry_SYSCALL_64_fastpath+0x1e/0xa9
[ 87.753681] RIP: 0033:0x7fb1b2421219
[ 87.753685] RSP: 002b:00007ffe287a6d88 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[ 87.753690] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fb1b2421219
[ 87.753693] RDX: 0000000000000000 RSI: 00005622387988cc RDI: 0000000000000000
[ 87.753696] RBP: 00007ffe287a5d80 R08: 0000000000000000 R09: 0000000000000001
[ 87.753699] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000005
[ 87.753702] R13: 0000562239046600 R14: 00007ffe287a5d60 R15: 0000000000000005
[ 87.753708] nft_meta: Could not allocate 16 bytes percpu data
[ 87.782057] percpu: reserved chunk: 3996, 1, 999, 999, 3992, 104, 3
[ 87.782060] percpu: rchunk md blocks
[ 87.782062] percpu: 1, 999, 0, 0, 999
[ 87.782065] percpu: 0, 1006, 0, 0, 1024
[ 87.782067] percpu: 998, 0, 998, 0, 0
[ 87.782069] percpu: pcpu_find_block_fit: -1, 4, 4
[ 87.782072] percpu: allocation failed, size=16 align=16 atomic=0, alloc from reserved chunk failed to find fit
[ 87.782081] CPU: 0 PID: 1336 Comm: modprobe Tainted: G E 4.14.0-rc2+ #123
[ 87.782085] Hardware name: Dell Inc. Precision 5510/0N8J4R, BIOS 1.2.25 05/07/2017
[ 87.782087] Call Trace:
[ 87.782099] dump_stack+0x63/0x89
[ 87.782107] pcpu_alloc+0x6c8/0x720
[ 87.782115] __alloc_reserved_percpu+0x18/0x20
[ 87.782121] load_module+0x733/0x2c00
[ 87.782129] ? kernel_read_file+0x1a3/0x1d0
[ 87.782136] SYSC_finit_module+0xfc/0x120
[ 87.782141] ? SYSC_finit_module+0xfc/0x120
[ 87.782149] SyS_finit_module+0xe/0x10
[ 87.782155] entry_SYSCALL_64_fastpath+0x1e/0xa9
[ 87.782159] RIP: 0033:0x7f4dd4073219
[ 87.782163] RSP: 002b:00007ffefd82e658 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[ 87.782169] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f4dd4073219
[ 87.782172] RDX: 0000000000000000 RSI: 000055d5cd3fd8cc RDI: 0000000000000000
[ 87.782175] RBP: 00007ffefd82d650 R08: 0000000000000000 R09: 0000000000000001
[ 87.782178] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000005
[ 87.782181] R13: 000055d5ce138600 R14: 00007ffefd82d630 R15: 0000000000000005
[ 87.782186] nft_meta: Could not allocate 16 bytes percpu data
[ 87.822555] percpu: reserved chunk: 3996, 1, 999, 999, 3992, 104, 3
[ 87.822557] percpu: rchunk md blocks
[ 87.822560] percpu: 1, 999, 0, 0, 999
[ 87.822562] percpu: 0, 1006, 0, 0, 1024
[ 87.822564] percpu: 998, 0, 998, 0, 0
[ 87.822567] percpu: pcpu_find_block_fit: 999, 1, 1
[ 87.822571] percpu: pcpu_alloc_area: 3996, 1, 1
[ 87.897805] percpu: reserved chunk: 3992, 998, 2048, 2048, 3992, 104, 3
[ 87.897807] percpu: rchunk md blocks
[ 87.897809] percpu: 0, 999, 0, 0, 1024
[ 87.897812] percpu: 0, 1006, 0, 0, 1024
[ 87.897814] percpu: 998, 0, 998, 0, 0
[ 87.897816] percpu: pcpu_find_block_fit: 2048, 4, 4
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2017-09-27 9:05 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-09-26 15:37 percpu allocation failures Luis Henriques
2017-09-27 1:54 ` Dennis Zhou
2017-09-27 9:05 ` Luis Henriques
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).