percpu allocation failures

linux-mm.kvack.org archive mirror
 help / color / mirror / Atom feed

* percpu allocation failures
@ 2017-09-26 15:37 Luis Henriques
  2017-09-27  1:54 ` Dennis Zhou
  0 siblings, 1 reply; 3+ messages in thread
From: Luis Henriques @ 2017-09-26 15:37 UTC (permalink / raw)
  To: Tejun Heo, Christoph Lameter, Dennis Zhou; +Cc: linux-mm, linux-kernel

Hi,

Probably already reported, but I couldn't find anything so here it
goes:

starting with 4.14-rc1 I see the following during boot:

[   25.199053] percpu: allocation failed, size=16 align=16 atomic=0, alloc from reserved chunk failed
[   25.200195] CPU: 5 PID: 723 Comm: modprobe Tainted: G            E   4.14.0-rc2 #103
[   25.201290] Hardware name: Dell Inc. Precision 5510/0N8J4R, BIOS 1.2.25 05/07/2017
[   25.202430] Call Trace:
[   25.203509]  dump_stack+0x63/0x89
[   25.204364]  pcpu_alloc+0x5cd/0x5f0
[   25.205302]  __alloc_reserved_percpu+0x18/0x20
[   25.206355]  load_module+0x733/0x2c00
[   25.207444]  ? kernel_read_file+0x1a3/0x1d0
[   25.208596]  SYSC_finit_module+0xfc/0x120
[   25.209634]  ? SYSC_finit_module+0xfc/0x120
[   25.210733]  SyS_finit_module+0xe/0x10
[   25.211747]  entry_SYSCALL_64_fastpath+0x1e/0xa9
[   25.212763] RIP: 0033:0x7f70d9e86219
[   25.213508] RSP: 002b:00007ffcd3ff8f38 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[   25.214391] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f70d9e86219
[   25.215107] RDX: 0000000000000000 RSI: 00005642ddf158cc RDI: 0000000000000000
[   25.215949] RBP: 00007ffcd3ff7f30 R08: 0000000000000000 R09: 0000000000000001
[   25.216592] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000005
[   25.217194] R13: 00005642de3d65f0 R14: 00007ffcd3ff7f10 R15: 0000000000000005
[   25.217812] nft_meta: Could not allocate 16 bytes percpu data

A few more failures follow.

A bisect ended up with the merge commit a7cbfd05f427 ("Merge branch
'for-4.14' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu").

Cheers,
-- 
Luis

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: percpu allocation failures
  2017-09-26 15:37 percpu allocation failures Luis Henriques
@ 2017-09-27  1:54 ` Dennis Zhou
  2017-09-27  9:05   ` Luis Henriques
  0 siblings, 1 reply; 3+ messages in thread
From: Dennis Zhou @ 2017-09-27  1:54 UTC (permalink / raw)
  To: Luis Henriques; +Cc: Tejun Heo, Christoph Lameter, linux-mm, linux-kernel

Hi Luis,

This seems to be an issue with the reserved chunk being unable to
allocate memory when loading kernel modules. Unfortunately, I have not 
been successful in reproducing this with the reserved chunk allocation
path exposed or by inserting the nft_meta module.

Could you please send me the output when ran with the following patch
and the output of the percpu memory statistics file before and after
inserting the module (PERCPU_STATS)? The stats are in
/sys/kernel/debug/percpu_stats.

Thanks,
Dennis

---
 mm/percpu.c | 32 ++++++++++++++++++++++++++++++--
 1 file changed, 30 insertions(+), 2 deletions(-)

diff --git a/mm/percpu.c b/mm/percpu.c
index 59d44d6..031fd91 100644
--- a/mm/percpu.c
+++ b/mm/percpu.c
@@ -1335,6 +1335,7 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved,
 {
 	static int warn_limit = 10;
 	struct pcpu_chunk *chunk;
+	struct pcpu_block_md *block;
 	const char *err;
 	bool is_atomic = (gfp & GFP_KERNEL) != GFP_KERNEL;
 	int slot, off, cpu, ret;
@@ -1371,17 +1372,43 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved,
 	if (reserved && pcpu_reserved_chunk) {
 		chunk = pcpu_reserved_chunk;
 
+		printk(KERN_DEBUG "percpu: reserved chunk: %d, %d, %d, %d, %d, %d, %d",
+		       chunk->free_bytes, chunk->contig_bits,
+		       chunk->contig_bits_start, chunk->first_bit,
+		       chunk->start_offset, chunk->end_offset,
+		       chunk->nr_pages);
+
+		printk(KERN_DEBUG "percpu: rchunk md blocks");
+		for (block = chunk->md_blocks;
+		     block < chunk->md_blocks + pcpu_chunk_nr_blocks(chunk);
+		     block++) {
+			printk(KERN_DEBUG "   percpu: %d, %d, %d, %d, %d",
+			       block->contig_hint,
+			       block->contig_hint_start,
+			       block->left_free,
+			       block->right_free,
+			       block->first_free);
+		}
+
 		off = pcpu_find_block_fit(chunk, bits, bit_align, is_atomic);
+
+		printk(KERN_DEBUG "percpu: pcpu_find_block_fit: %d, %zu, %zu",
+		       off, bits, bit_align);
+
 		if (off < 0) {
-			err = "alloc from reserved chunk failed";
+			err = "alloc from reserved chunk failed to find fit";
 			goto fail_unlock;
 		}
 
 		off = pcpu_alloc_area(chunk, bits, bit_align, off);
+
+		printk(KERN_DEBUG "percpu: pcpu_alloc_area: %d, %zu, %zu",
+		       off, bits, bit_align);
+
 		if (off >= 0)
 			goto area_found;
 
-		err = "alloc from reserved chunk failed";
+		err = "alloc from reserved chunk failed to alloc area";
 		goto fail_unlock;
 	}
 
@@ -1547,6 +1574,7 @@ void __percpu *__alloc_reserved_percpu(size_t size, size_t align)
 {
 	return pcpu_alloc(size, align, true, GFP_KERNEL);
 }
+EXPORT_SYMBOL_GPL(__alloc_reserved_percpu);
 
 /**
  * pcpu_balance_workfn - manage the amount of free chunks and populated pages
-- 
1.8.3.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply related	[flat|nested] 3+ messages in thread

* Re: percpu allocation failures
  2017-09-27  1:54 ` Dennis Zhou
@ 2017-09-27  9:05   ` Luis Henriques
  0 siblings, 0 replies; 3+ messages in thread
From: Luis Henriques @ 2017-09-27  9:05 UTC (permalink / raw)
  To: Dennis Zhou; +Cc: Tejun Heo, Christoph Lameter, linux-mm, linux-kernel

[-- Attachment #1: Type: text/plain, Size: 1038 bytes --]

Dennis Zhou <dennisszhou@gmail.com> writes:

> Hi Luis,
>
> This seems to be an issue with the reserved chunk being unable to
> allocate memory when loading kernel modules. Unfortunately, I have not 
> been successful in reproducing this with the reserved chunk allocation
> path exposed or by inserting the nft_meta module.
>
> Could you please send me the output when ran with the following patch
> and the output of the percpu memory statistics file before and after
> inserting the module (PERCPU_STATS)? The stats are in
> /sys/kernel/debug/percpu_stats.

Please find attached all the info you requested.  Hope it helps.

Also, here's the nft script I'm using to trigger the issue locally:

#!/usr/sbin/nft -f

flush ruleset

table inet filter {
	chain input {
		type filter hook input priority 0;
		iif lo accept
		ct state established,related accept
		tcp dport { 22 } ct state new accept
		ip6 nexthdr icmpv6 icmpv6 type { nd-neighbor-solicit,  nd-router-advert, nd-neighbor-advert } accept
		counter drop
	}
}

Cheers,
-- 
Luis


[-- Attachment #2: percpu_stats.before --]
[-- Type: application/octet-stream, Size: 2031 bytes --]

Percpu Memory Statistics
Allocation Info:
----------------------------------------
  unit_size           :       262144
  static_size         :       110488
  reserved_size       :         8192
  dyn_size            :        28776
  atom_size           :      2097152
  alloc_size          :      2097152

Global Stats:
----------------------------------------
  nr_alloc            :         2204
  nr_dealloc          :          764
  nr_cur_alloc        :         1440
  nr_max_alloc        :         1638
  nr_chunks           :            2
  nr_max_chunks       :            2
  min_alloc_size      :            4
  max_alloc_size      :         4104
  empty_pop_pages     :            2

Per Chunk Stats:
----------------------------------------
Chunk: <- Reserved Chunk
  nr_alloc            :            4
  max_alloc_size      :         4104
  empty_pop_pages     :            0
  first_bit           :          999
  free_bytes          :         4004
  contig_bytes        :            8
  sum_frag            :            0
  max_frag            :            0
  cur_min_alloc       :            0
  cur_med_alloc       :            0
  cur_max_alloc       :            0

Chunk: <- First Chunk
  nr_alloc            :          113
  max_alloc_size      :          976
  empty_pop_pages     :            0
  first_bit           :         8192
  free_bytes          :            0
  contig_bytes        :            0
  sum_frag            :            0
  max_frag            :            0
  cur_min_alloc       :            4
  cur_med_alloc       :          976
  cur_max_alloc       :          976

Chunk:
  nr_alloc            :         1323
  max_alloc_size      :         4096
  empty_pop_pages     :            2
  first_bit           :        11005
  free_bytes          :       162604
  contig_bytes        :       155648
  sum_frag            :         6956
  max_frag            :         2808
  cur_min_alloc       :            4
  cur_med_alloc       :            8
  cur_max_alloc       :         4096


[-- Attachment #3: percpu_stats.after --]
[-- Type: application/octet-stream, Size: 2031 bytes --]

Percpu Memory Statistics
Allocation Info:
----------------------------------------
  unit_size           :       262144
  static_size         :       110488
  reserved_size       :         8192
  dyn_size            :        28776
  atom_size           :      2097152
  alloc_size          :      2097152

Global Stats:
----------------------------------------
  nr_alloc            :         2215
  nr_dealloc          :          765
  nr_cur_alloc        :         1450
  nr_max_alloc        :         1638
  nr_chunks           :            2
  nr_max_chunks       :            2
  min_alloc_size      :            4
  max_alloc_size      :         4104
  empty_pop_pages     :            2

Per Chunk Stats:
----------------------------------------
Chunk: <- Reserved Chunk
  nr_alloc            :            7
  max_alloc_size      :         4104
  empty_pop_pages     :            0
  first_bit           :         2052
  free_bytes          :         3976
  contig_bytes        :         3976
  sum_frag            :            0
  max_frag            :            0
  cur_min_alloc       :            0
  cur_med_alloc       :            0
  cur_max_alloc       :            0

Chunk: <- First Chunk
  nr_alloc            :          113
  max_alloc_size      :          976
  empty_pop_pages     :            0
  first_bit           :         8192
  free_bytes          :            0
  contig_bytes        :            0
  sum_frag            :            0
  max_frag            :            0
  cur_min_alloc       :            4
  cur_med_alloc       :          976
  cur_max_alloc       :          976

Chunk:
  nr_alloc            :         1330
  max_alloc_size      :         4096
  empty_pop_pages     :            2
  first_bit           :        11005
  free_bytes          :       160492
  contig_bytes        :       155648
  sum_frag            :         4844
  max_frag            :         2808
  cur_min_alloc       :            4
  cur_med_alloc       :            8
  cur_max_alloc       :         4096


[-- Attachment #4: dmesg.debug --]
[-- Type: application/octet-stream, Size: 5911 bytes --]

[   87.709692] percpu: reserved chunk: 4004, 2, 2030, 999, 3992, 104, 3
[   87.709694] percpu: rchunk md blocks
[   87.709697]    percpu: 1, 999, 0, 0, 999
[   87.709699]    percpu: 2, 1006, 0, 0, 1006
[   87.709701]    percpu: 998, 0, 998, 0, 0
[   87.709704] percpu: pcpu_find_block_fit: -1, 4, 4
[   87.709707] percpu: allocation failed, size=16 align=16 atomic=0, alloc from reserved chunk failed to find fit
[   87.709717] CPU: 0 PID: 1328 Comm: modprobe Tainted: G            E   4.14.0-rc2+ #123
[   87.709721] Hardware name: Dell Inc. Precision 5510/0N8J4R, BIOS 1.2.25 05/07/2017
[   87.709723] Call Trace:
[   87.709738]  dump_stack+0x63/0x89
[   87.709745]  pcpu_alloc+0x6c8/0x720
[   87.709753]  __alloc_reserved_percpu+0x18/0x20
[   87.709760]  load_module+0x733/0x2c00
[   87.709769]  ? kernel_read_file+0x1a3/0x1d0
[   87.709776]  SYSC_finit_module+0xfc/0x120
[   87.709781]  ? SYSC_finit_module+0xfc/0x120
[   87.709789]  SyS_finit_module+0xe/0x10
[   87.709795]  entry_SYSCALL_64_fastpath+0x1e/0xa9
[   87.709800] RIP: 0033:0x7f211163f219
[   87.709804] RSP: 002b:00007ffe68de7318 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[   87.709809] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f211163f219
[   87.709812] RDX: 0000000000000000 RSI: 000055dc7a86a8cc RDI: 0000000000000000
[   87.709815] RBP: 00007ffe68de6310 R08: 0000000000000000 R09: 0000000000000001
[   87.709818] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000005
[   87.709821] R13: 000055dc7c54c600 R14: 00007ffe68de62f0 R15: 0000000000000005
[   87.709827] nft_meta: Could not allocate 16 bytes percpu data
[   87.749131] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[   87.749791] percpu: reserved chunk: 4004, 2, 2030, 999, 3992, 104, 3
[   87.749793] percpu: rchunk md blocks
[   87.749796]    percpu: 1, 999, 0, 0, 999
[   87.749798]    percpu: 2, 1006, 0, 0, 1006
[   87.749800]    percpu: 998, 0, 998, 0, 0
[   87.749802] percpu: pcpu_find_block_fit: 2030, 2, 2
[   87.749806] percpu: pcpu_alloc_area: 8120, 2, 2
[   87.753575] percpu: reserved chunk: 3996, 1, 999, 999, 3992, 104, 3
[   87.753578] percpu: rchunk md blocks
[   87.753580]    percpu: 1, 999, 0, 0, 999
[   87.753582]    percpu: 0, 1006, 0, 0, 1024
[   87.753584]    percpu: 998, 0, 998, 0, 0
[   87.753587] percpu: pcpu_find_block_fit: -1, 4, 4
[   87.753590] percpu: allocation failed, size=16 align=16 atomic=0, alloc from reserved chunk failed to find fit
[   87.753599] CPU: 1 PID: 1334 Comm: modprobe Tainted: G            E   4.14.0-rc2+ #123
[   87.753603] Hardware name: Dell Inc. Precision 5510/0N8J4R, BIOS 1.2.25 05/07/2017
[   87.753605] Call Trace:
[   87.753619]  dump_stack+0x63/0x89
[   87.753626]  pcpu_alloc+0x6c8/0x720
[   87.753635]  __alloc_reserved_percpu+0x18/0x20
[   87.753642]  load_module+0x733/0x2c00
[   87.753650]  ? kernel_read_file+0x1a3/0x1d0
[   87.753657]  SYSC_finit_module+0xfc/0x120
[   87.753663]  ? SYSC_finit_module+0xfc/0x120
[   87.753671]  SyS_finit_module+0xe/0x10
[   87.753677]  entry_SYSCALL_64_fastpath+0x1e/0xa9
[   87.753681] RIP: 0033:0x7fb1b2421219
[   87.753685] RSP: 002b:00007ffe287a6d88 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[   87.753690] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fb1b2421219
[   87.753693] RDX: 0000000000000000 RSI: 00005622387988cc RDI: 0000000000000000
[   87.753696] RBP: 00007ffe287a5d80 R08: 0000000000000000 R09: 0000000000000001
[   87.753699] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000005
[   87.753702] R13: 0000562239046600 R14: 00007ffe287a5d60 R15: 0000000000000005
[   87.753708] nft_meta: Could not allocate 16 bytes percpu data
[   87.782057] percpu: reserved chunk: 3996, 1, 999, 999, 3992, 104, 3
[   87.782060] percpu: rchunk md blocks
[   87.782062]    percpu: 1, 999, 0, 0, 999
[   87.782065]    percpu: 0, 1006, 0, 0, 1024
[   87.782067]    percpu: 998, 0, 998, 0, 0
[   87.782069] percpu: pcpu_find_block_fit: -1, 4, 4
[   87.782072] percpu: allocation failed, size=16 align=16 atomic=0, alloc from reserved chunk failed to find fit
[   87.782081] CPU: 0 PID: 1336 Comm: modprobe Tainted: G            E   4.14.0-rc2+ #123
[   87.782085] Hardware name: Dell Inc. Precision 5510/0N8J4R, BIOS 1.2.25 05/07/2017
[   87.782087] Call Trace:
[   87.782099]  dump_stack+0x63/0x89
[   87.782107]  pcpu_alloc+0x6c8/0x720
[   87.782115]  __alloc_reserved_percpu+0x18/0x20
[   87.782121]  load_module+0x733/0x2c00
[   87.782129]  ? kernel_read_file+0x1a3/0x1d0
[   87.782136]  SYSC_finit_module+0xfc/0x120
[   87.782141]  ? SYSC_finit_module+0xfc/0x120
[   87.782149]  SyS_finit_module+0xe/0x10
[   87.782155]  entry_SYSCALL_64_fastpath+0x1e/0xa9
[   87.782159] RIP: 0033:0x7f4dd4073219
[   87.782163] RSP: 002b:00007ffefd82e658 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[   87.782169] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f4dd4073219
[   87.782172] RDX: 0000000000000000 RSI: 000055d5cd3fd8cc RDI: 0000000000000000
[   87.782175] RBP: 00007ffefd82d650 R08: 0000000000000000 R09: 0000000000000001
[   87.782178] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000005
[   87.782181] R13: 000055d5ce138600 R14: 00007ffefd82d630 R15: 0000000000000005
[   87.782186] nft_meta: Could not allocate 16 bytes percpu data
[   87.822555] percpu: reserved chunk: 3996, 1, 999, 999, 3992, 104, 3
[   87.822557] percpu: rchunk md blocks
[   87.822560]    percpu: 1, 999, 0, 0, 999
[   87.822562]    percpu: 0, 1006, 0, 0, 1024
[   87.822564]    percpu: 998, 0, 998, 0, 0
[   87.822567] percpu: pcpu_find_block_fit: 999, 1, 1
[   87.822571] percpu: pcpu_alloc_area: 3996, 1, 1
[   87.897805] percpu: reserved chunk: 3992, 998, 2048, 2048, 3992, 104, 3
[   87.897807] percpu: rchunk md blocks
[   87.897809]    percpu: 0, 999, 0, 0, 1024
[   87.897812]    percpu: 0, 1006, 0, 0, 1024
[   87.897814]    percpu: 998, 0, 998, 0, 0
[   87.897816] percpu: pcpu_find_block_fit: 2048, 4, 4

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2017-09-27  9:05 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-09-26 15:37 percpu allocation failures Luis Henriques
2017-09-27  1:54 ` Dennis Zhou
2017-09-27  9:05   ` Luis Henriques

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).