Re: [PATCH v3 7/7] mm: switch deferred split shrinker to list_lru - [s390] panic in __memcg_list_lru_alloc

Re: [PATCH v3 7/7] mm: switch deferred split shrinker to list_lru - [s390] panic in __memcg_list_lru_alloc

[Mikhail Zaslonko]

On 18-Mar-26 20:53, Johannes Weiner wrote:
> The deferred split queue handles cgroups in a suboptimal fashion. 
> The queue is per-NUMA node or per-cgroup, not the intersection. That 
> means on a cgrouped system, a node-restricted allocation entering 
> reclaim can end up splitting large pages on other nodes:
> 
> alloc/unmap deferred_split_folio() list_add_tail(memcg-
> >split_queue) set_shrinker_bit(memcg, node, deferred_shrinker_id)
> 
> for_each_zone_zonelist_nodemask(restricted_nodes) mem_cgroup_iter() 
> shrink_slab(node, memcg) shrink_slab_memcg(node, memcg) if 
> test_shrinker_bit(memcg, node, deferred_shrinker_id) 
> deferred_split_scan() walks memcg->split_queue
> 
> The shrinker bit adds an imperfect guard rail. As soon as the
> cgroup has a single large page on the node of interest, all large
> pages owned by that memcg, including those on other nodes, will be
> split.
> 
> list_lru properly sets up per-node, per-cgroup lists. As a bonus,
> it streamlines a lot of the list operations and reclaim walks. It's 
> used widely by other major shrinkers already. Convert the deferred 
> split queue as well.
> 
> The list_lru per-memcg heads are instantiated on demand when the 
> first object of interest is allocated for a cgroup, by calling 
> folio_memcg_list_lru_alloc(). Add calls to where splittable pages 
> are created: anon faults, swapin faults, khugepaged collapse.
> 
> These calls create all possible node heads for the cgroup at once, 
> so the migration code (between nodes) doesn't need any special care.
> 
> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> --- include/ 
> linux/huge_mm.h    |   6 +- include/linux/memcontrol.h |   4 - 
> include/linux/mmzone.h     |  12 -- mm/huge_memory.c           | 342 
> ++++++++++++------------------------- mm/internal.h              | 2
> +- mm/khugepaged.c            |   7 + mm/memcontrol.c |  12 +- mm/
> memory.c                |  52 +++--- mm/ mm_init.c               |
> 15 -- 9 files changed, 151 insertions(+), 301 deletions(-)
> 

Hi,

with this series in linux-next (since next-20260324) I see a reproducible panic on s390 in the
dump kernel when running NVMe standalone dump (ngdump).
This only happens in the 'capture kernel', normal boot of the same kernel works fine.

[   14.350676] Unable to handle kernel pointer dereference in virtual kernel address space
[   14.350682] Failing address: 4000000000000000 TEID: 4000000000000803 ESOP-2 FSI
[   14.350686] Fault in home space mode while using kernel ASCE.
[   14.350689] AS:0000000002798007 R3:000000002d2c4007 S:000000002d2c3001 P:000000000000013d 
[   14.350730] Oops: 0038 ilc:3 [#1]SMP 
[   14.350735] Modules linked in: dm_service_time zfcp scsi_transport_fc uvdevice diag288_wdt nvme prng aes_s390 nvme_core des_s390 libdes zcrypt_cex4 dm_mirror dm_region_hash dm_log scsi_dh_rdac scsi_dh_emc scsi_dh_alua paes_s390 crypto_engine pkey_cca pkey_ep11 zcrypt rng_core pkey_pckmo pkey dm_multipath autofs4
[   14.350760] CPU: 0 UID: 0 PID: 32 Comm: khugepaged Not tainted 7.0.0-rc5-next-20260324 
[   14.350762] Hardware name: IBM 3931 A01 704 (LPAR)
[   14.350764] Krnl PSW : 0704d00180000000 000003ffe0443a82 (__memcg_list_lru_alloc+0x52/0x1d0)
[   14.350774]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3
[   14.350776] Krnl GPRS: 0000000000000402 00000000000bece0 0000000000000000 000003ffe1c17928
[   14.350778]            00000000001c24ca 0000000000000000 0000000000000000 000003ffe1c17948
[   14.350780]            0000000000000000 00000000000824c0 0000037200098000 4000000000000000
[   14.350782]            0000000000782400 0000000000000001 0000037fe00f39b8 0000037fe00f3918
[   14.350788] Krnl Code: 000003ffe0443a72: a7690000            lghi    %r6,0
[   14.350788]            000003ffe0443a76: e380f0a00004        lg      %r8,160(%r15)
[   14.350788]           *000003ffe0443a7c: e3b080b80004        lg      %r11,184(%r8)
[   14.350788]           >000003ffe0443a82: e330b9400012        lt      %r3,2368(%r11)
[   14.350788]            000003ffe0443a88: a7a40065            brc     10,000003ffe0443b52
[   14.350788]            000003ffe0443a8c: e3b0f0a00004        lg      %r11,160(%r15)
[   14.350788]            000003ffe0443a92: ec68006f007c        cgij    %r6,0,8,000003ffe0443b70
[   14.350788]            000003ffe0443a98: e300b9400014        lgf     %r0,2368(%r11)
[   14.350825] Call Trace:
[   14.350826]  [<000003ffe0443a82>] __memcg_list_lru_alloc+0x52/0x1d0 
[   14.350831]  [<000003ffe044529a>] folio_memcg_list_lru_alloc+0xba/0x150 
[   14.350834]  [<000003ffe04f279a>] alloc_charge_folio+0x18a/0x250 
[   14.350839]  [<000003ffe04f34dc>] collapse_huge_page+0x8c/0x890 
[   14.350841]  [<000003ffe04f4222>] collapse_scan_pmd+0x542/0x690 
[   14.350844]  [<000003ffe04f65b4>] collapse_single_pmd+0x144/0x240 
[   14.350847]  [<000003ffe04f69ce>] collapse_scan_mm_slot.constprop.0+0x31e/0x480 
[   14.350849]  [<000003ffe04f6d3c>] khugepaged+0x20c/0x210 
[   14.350852]  [<000003ffe019b0a8>] kthread+0x148/0x170 
[   14.350856]  [<000003ffe0119fec>] __ret_from_fork+0x3c/0x240 
[   14.350860]  [<000003ffe0ffa4b2>] ret_from_fork+0xa/0x30 
[   14.350865] Last Breaking-Event-Address:
[   14.350865]  [<000003ffe0445294>] folio_memcg_list_lru_alloc+0xb4/0x150
[   14.350870] Kernel panic - not syncing: Fatal exception: panic_on_oops


Environment:
Arch: s390x (IBM LPAR)
Kernel: next-20260324
Config: (can provide if needed)
Reproducible: always
Steps to Reproduce: 
  Install ngdump to an NVMe device partition via 'zipl -d' and initiate a dump (same issue with DASD ldipl-dump).

I have bisected to this specific commit. 
  good: 230bbdc110b3 ("mm: list_lru: introduce folio_memcg_list_lru_alloc()")
  bad:  b0f512f6e36c ("mm: switch deferred split shrinker to list_lru")
Reverting it on top of linux-next <next-20260327> restores normal Standalone Dump operation.
 
Let me know if I can provide any other data.

Thanks,
Mikhail

Re: [PATCH v3 7/7] mm: switch deferred split shrinker to list_lru - [s390] panic in __memcg_list_lru_alloc

[Andrew Morton]

On Mon, 30 Mar 2026 18:37:01 +0200 Mikhail Zaslonko <zaslonko@linux.ibm.com> wrote:

> with this series in linux-next (since next-20260324) I see a reproducible panic on s390 in the
> dump kernel when running NVMe standalone dump (ngdump).
> This only happens in the 'capture kernel', normal boot of the same kernel works fine.

Thanks.  It seems that hannes is working on a significant respin, so
I'll remove this version of the series from mm.git/linux-next.

Re: [PATCH v3 7/7] mm: switch deferred split shrinker to list_lru - [s390] panic in __memcg_list_lru_alloc

[Johannes Weiner]

Hello Mikhail,

On Mon, Mar 30, 2026 at 06:37:01PM +0200, Mikhail Zaslonko wrote:
> with this series in linux-next (since next-20260324) I see a reproducible panic on s390 in the
> dump kernel when running NVMe standalone dump (ngdump).
> This only happens in the 'capture kernel', normal boot of the same kernel works fine.
> 
> [   14.350676] Unable to handle kernel pointer dereference in virtual kernel address space
> [   14.350682] Failing address: 4000000000000000 TEID: 4000000000000803 ESOP-2 FSI
> [   14.350686] Fault in home space mode while using kernel ASCE.
> [   14.350689] AS:0000000002798007 R3:000000002d2c4007 S:000000002d2c3001 P:000000000000013d 
> [   14.350730] Oops: 0038 ilc:3 [#1]SMP 
> [   14.350735] Modules linked in: dm_service_time zfcp scsi_transport_fc uvdevice diag288_wdt nvme prng aes_s390 nvme_core des_s390 libdes zcrypt_cex4 dm_mirror dm_region_hash dm_log scsi_dh_rdac scsi_dh_emc scsi_dh_alua paes_s390 crypto_engine pkey_cca pkey_ep11 zcrypt rng_core pkey_pckmo pkey dm_multipath autofs4
> [   14.350760] CPU: 0 UID: 0 PID: 32 Comm: khugepaged Not tainted 7.0.0-rc5-next-20260324 
> [   14.350762] Hardware name: IBM 3931 A01 704 (LPAR)
> [   14.350764] Krnl PSW : 0704d00180000000 000003ffe0443a82 (__memcg_list_lru_alloc+0x52/0x1d0)
> [   14.350774]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3
> [   14.350776] Krnl GPRS: 0000000000000402 00000000000bece0 0000000000000000 000003ffe1c17928
> [   14.350778]            00000000001c24ca 0000000000000000 0000000000000000 000003ffe1c17948
> [   14.350780]            0000000000000000 00000000000824c0 0000037200098000 4000000000000000
> [   14.350782]            0000000000782400 0000000000000001 0000037fe00f39b8 0000037fe00f3918
> [   14.350788] Krnl Code: 000003ffe0443a72: a7690000            lghi    %r6,0
> [   14.350788]            000003ffe0443a76: e380f0a00004        lg      %r8,160(%r15)
> [   14.350788]           *000003ffe0443a7c: e3b080b80004        lg      %r11,184(%r8)
> [   14.350788]           >000003ffe0443a82: e330b9400012        lt      %r3,2368(%r11)
> [   14.350788]            000003ffe0443a88: a7a40065            brc     10,000003ffe0443b52
> [   14.350788]            000003ffe0443a8c: e3b0f0a00004        lg      %r11,160(%r15)
> [   14.350788]            000003ffe0443a92: ec68006f007c        cgij    %r6,0,8,000003ffe0443b70
> [   14.350788]            000003ffe0443a98: e300b9400014        lgf     %r0,2368(%r11)
> [   14.350825] Call Trace:
> [   14.350826]  [<000003ffe0443a82>] __memcg_list_lru_alloc+0x52/0x1d0 
> [   14.350831]  [<000003ffe044529a>] folio_memcg_list_lru_alloc+0xba/0x150 
> [   14.350834]  [<000003ffe04f279a>] alloc_charge_folio+0x18a/0x250 
> [   14.350839]  [<000003ffe04f34dc>] collapse_huge_page+0x8c/0x890 
> [   14.350841]  [<000003ffe04f4222>] collapse_scan_pmd+0x542/0x690 
> [   14.350844]  [<000003ffe04f65b4>] collapse_single_pmd+0x144/0x240 
> [   14.350847]  [<000003ffe04f69ce>] collapse_scan_mm_slot.constprop.0+0x31e/0x480 
> [   14.350849]  [<000003ffe04f6d3c>] khugepaged+0x20c/0x210 
> [   14.350852]  [<000003ffe019b0a8>] kthread+0x148/0x170 
> [   14.350856]  [<000003ffe0119fec>] __ret_from_fork+0x3c/0x240 
> [   14.350860]  [<000003ffe0ffa4b2>] ret_from_fork+0xa/0x30 
> [   14.350865] Last Breaking-Event-Address:
> [   14.350865]  [<000003ffe0445294>] folio_memcg_list_lru_alloc+0xb4/0x150
> [   14.350870] Kernel panic - not syncing: Fatal exception: panic_on_oops

Thanks for the report. Would you be able to correlate the function
offset in __memcg_list_lru_alloc() to C code? It would be very useful
to understand *which* pointer or data structure it's tripping over.

thp_shrinker_init() is called before start_stop_khugepaged(), so I am
not sure how we could have invalid/uninitialized list_lru structures.

Could you extract the instruction stream sequence surrounding the
crash? I don't see a Code: line, but maybe you can correlate it with
objdump? And then correlate it with C code (make mm/list_lru.lst e.g.)

That would be super helpful. Sorry about the breakage.

Re: [PATCH v3 7/7] mm: switch deferred split shrinker to list_lru - [s390] panic in __memcg_list_lru_alloc

[Johannes Weiner]

On Mon, Mar 30, 2026 at 04:41:16PM -0400, Johannes Weiner wrote:
> Hello Mikhail,
> 
> On Mon, Mar 30, 2026 at 06:37:01PM +0200, Mikhail Zaslonko wrote:
> > with this series in linux-next (since next-20260324) I see a reproducible panic on s390 in the
> > dump kernel when running NVMe standalone dump (ngdump).
> > This only happens in the 'capture kernel', normal boot of the same kernel works fine.
> > 
> > [   14.350676] Unable to handle kernel pointer dereference in virtual kernel address space
> > [   14.350682] Failing address: 4000000000000000 TEID: 4000000000000803 ESOP-2 FSI
> > [   14.350686] Fault in home space mode while using kernel ASCE.
> > [   14.350689] AS:0000000002798007 R3:000000002d2c4007 S:000000002d2c3001 P:000000000000013d 
> > [   14.350730] Oops: 0038 ilc:3 [#1]SMP 
> > [   14.350735] Modules linked in: dm_service_time zfcp scsi_transport_fc uvdevice diag288_wdt nvme prng aes_s390 nvme_core des_s390 libdes zcrypt_cex4 dm_mirror dm_region_hash dm_log scsi_dh_rdac scsi_dh_emc scsi_dh_alua paes_s390 crypto_engine pkey_cca pkey_ep11 zcrypt rng_core pkey_pckmo pkey dm_multipath autofs4
> > [   14.350760] CPU: 0 UID: 0 PID: 32 Comm: khugepaged Not tainted 7.0.0-rc5-next-20260324 
> > [   14.350762] Hardware name: IBM 3931 A01 704 (LPAR)
> > [   14.350764] Krnl PSW : 0704d00180000000 000003ffe0443a82 (__memcg_list_lru_alloc+0x52/0x1d0)
> > [   14.350774]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3
> > [   14.350776] Krnl GPRS: 0000000000000402 00000000000bece0 0000000000000000 000003ffe1c17928
> > [   14.350778]            00000000001c24ca 0000000000000000 0000000000000000 000003ffe1c17948
> > [   14.350780]            0000000000000000 00000000000824c0 0000037200098000 4000000000000000
> > [   14.350782]            0000000000782400 0000000000000001 0000037fe00f39b8 0000037fe00f3918
> > [   14.350788] Krnl Code: 000003ffe0443a72: a7690000            lghi    %r6,0
> > [   14.350788]            000003ffe0443a76: e380f0a00004        lg      %r8,160(%r15)
> > [   14.350788]           *000003ffe0443a7c: e3b080b80004        lg      %r11,184(%r8)
> > [   14.350788]           >000003ffe0443a82: e330b9400012        lt      %r3,2368(%r11)
> > [   14.350788]            000003ffe0443a88: a7a40065            brc     10,000003ffe0443b52
> > [   14.350788]            000003ffe0443a8c: e3b0f0a00004        lg      %r11,160(%r15)
> > [   14.350788]            000003ffe0443a92: ec68006f007c        cgij    %r6,0,8,000003ffe0443b70
> > [   14.350788]            000003ffe0443a98: e300b9400014        lgf     %r0,2368(%r11)
> > [   14.350825] Call Trace:
> > [   14.350826]  [<000003ffe0443a82>] __memcg_list_lru_alloc+0x52/0x1d0 
> > [   14.350831]  [<000003ffe044529a>] folio_memcg_list_lru_alloc+0xba/0x150 
> > [   14.350834]  [<000003ffe04f279a>] alloc_charge_folio+0x18a/0x250 
> > [   14.350839]  [<000003ffe04f34dc>] collapse_huge_page+0x8c/0x890 
> > [   14.350841]  [<000003ffe04f4222>] collapse_scan_pmd+0x542/0x690 
> > [   14.350844]  [<000003ffe04f65b4>] collapse_single_pmd+0x144/0x240 
> > [   14.350847]  [<000003ffe04f69ce>] collapse_scan_mm_slot.constprop.0+0x31e/0x480 
> > [   14.350849]  [<000003ffe04f6d3c>] khugepaged+0x20c/0x210 
> > [   14.350852]  [<000003ffe019b0a8>] kthread+0x148/0x170 
> > [   14.350856]  [<000003ffe0119fec>] __ret_from_fork+0x3c/0x240 
> > [   14.350860]  [<000003ffe0ffa4b2>] ret_from_fork+0xa/0x30 
> > [   14.350865] Last Breaking-Event-Address:
> > [   14.350865]  [<000003ffe0445294>] folio_memcg_list_lru_alloc+0xb4/0x150
> > [   14.350870] Kernel panic - not syncing: Fatal exception: panic_on_oops

Can you verify whether the kdump kernel boots with
cgroup_disable=memory?

I think there is an issue with how we call __list_lru_init(). The
existing callsites had their own memcg_kmem_online() guards. But the
THP one does not, so we're creating a memcg-aware list_lru, but the
do-while hierarchy walk in __memcg_list_lru_alloc() runs into a NULL
memcg.

Can you try the below on top of that -next checkout?

diff --git a/mm/list_lru.c b/mm/list_lru.c
index 1ccdd45b1d14..7c7024e33653 100644
--- a/mm/list_lru.c
+++ b/mm/list_lru.c
@@ -637,7 +637,7 @@ int __list_lru_init(struct list_lru *lru, bool memcg_aware, struct shrinker *shr
 	else
 		lru->shrinker_id = -1;
 
-	if (mem_cgroup_kmem_disabled())
+	if (mem_cgroup_disabled() || mem_cgroup_kmem_disabled())
 		memcg_aware = false;
 #endif

Re: [PATCH v3 7/7] mm: switch deferred split shrinker to list_lru - [s390] panic in __memcg_list_lru_alloc

[Vasily Gorbik]

On Mon, Mar 30, 2026 at 04:56:56PM -0400, Johannes Weiner wrote:
> On Mon, Mar 30, 2026 at 04:41:16PM -0400, Johannes Weiner wrote:
> > Hello Mikhail,
> > 
> > On Mon, Mar 30, 2026 at 06:37:01PM +0200, Mikhail Zaslonko wrote:
> > > with this series in linux-next (since next-20260324) I see a reproducible panic on s390 in the
> > > dump kernel when running NVMe standalone dump (ngdump).
> > > This only happens in the 'capture kernel', normal boot of the same kernel works fine.
> 
> Can you verify whether the kdump kernel boots with
> cgroup_disable=memory?

Yes, that is the only special thing about that dump kernel.
Sorry for not including the command line. x86 crashes more or less the
same way with cgroup_disable=memory. But while I was doing the repro on
x86, you already found the root cause.

> Can you try the below on top of that -next checkout?
> 
> diff --git a/mm/list_lru.c b/mm/list_lru.c
> index 1ccdd45b1d14..7c7024e33653 100644
> --- a/mm/list_lru.c
> +++ b/mm/list_lru.c
> @@ -637,7 +637,7 @@ int __list_lru_init(struct list_lru *lru, bool memcg_aware, struct shrinker *shr
>  	else
>  		lru->shrinker_id = -1;
>  
> -	if (mem_cgroup_kmem_disabled())
> +	if (mem_cgroup_disabled() || mem_cgroup_kmem_disabled())
>  		memcg_aware = false;
>  #endif

Yes, that fixes it for me on s390 and x86. Thank you!

Re: [PATCH v3 7/7] mm: switch deferred split shrinker to list_lru - [s390] panic in __memcg_list_lru_alloc

[Mikhail Zaslonko]

On 30-Mar-26 22:56, Johannes Weiner wrote:
> On Mon, Mar 30, 2026 at 04:41:16PM -0400, Johannes Weiner wrote:
>> Hello Mikhail,
>>

> 
> Can you verify whether the kdump kernel boots with
> cgroup_disable=memory?

It was not a kdump, but s390 specific dump tool. Yes, here is the cmdline (forgot to include):
kernel parmline...: 'reset_devices cgroup_disable=memory nokaslr numa=off irqpoll nr_cpus=1'
> 
> I think there is an issue with how we call __list_lru_init(). The
> existing callsites had their own memcg_kmem_online() guards. But the
> THP one does not, so we're creating a memcg-aware list_lru, but the
> do-while hierarchy walk in __memcg_list_lru_alloc() runs into a NULL
> memcg.
> 
> Can you try the below on top of that -next checkout?
> 
> diff --git a/mm/list_lru.c b/mm/list_lru.c
> index 1ccdd45b1d14..7c7024e33653 100644
> --- a/mm/list_lru.c
> +++ b/mm/list_lru.c
> @@ -637,7 +637,7 @@ int __list_lru_init(struct list_lru *lru, bool memcg_aware, struct shrinker *shr
>  	else
>  		lru->shrinker_id = -1;
>  
> -	if (mem_cgroup_kmem_disabled())
> +	if (mem_cgroup_disabled() || mem_cgroup_kmem_disabled())
>  		memcg_aware = false;
>  #endif
> 

I confirm, this resolves the issue.

Thanks.