[REGRESSION] mm/memcg: list_del corruption on memcg LRU on 7.2.11 despite bf4ade7dbd76 (single NUMA node machines)

[Noah Feldt <noah@feldt.systems>]

[-- Attachment #1.1: Type: text/plain, Size: 6765 bytes --]

Hi,

we are hitting list_del corruption on memcg LRU lists on 7.2.11. The
signature is the same as in Karl's report [1], but 7.2.11 already
contains both fixes from that thread:

  bf4ade7dbd76 ("memcg: keep folio's objcg same as its node")
               -> 7.2.y as 6165478eaa, in v7.2.5
  0e0ac326c511 ("memcg: move LRU size accounting on reparenting
                instead of copying it")
               -> 7.2.y as 007bc18423, in v7.2.6

All three affected machines are single-socket with a single NUMA node
(AMD EPYC 9655 and EPYC 7702P, "NUMA node(s): 1"). So cross-node
migration, the trigger fixed by bf4ade7dbd76 and a requirement for
Karl's reproducer, cannot be involved. This looks like a different race
in the reparenting/LRU code.

We saw three panics in eight days on one cluster, on hardware from two
vendors. Nothing in v7.2.12..v7.2.15, v7.3.3, v7.4-rc2 or the
mm-unstable/mm-hotfixes-unstable branches looks like it addresses this.

Kernel:   Debian 7.2.11-
1 (7.2.11+deb14+1-amd64), PREEMPT(lazy), MGLRU on,
          cgroup v2
Workload: Kubernetes nodes (CRI-O) with high pod and cgroup churn:
          constant pod start/stop, frequent XFS mount/umount of zvols,
          memcg OOM kills with memory.oom.group, and memory.high
          reclaim. Every crash was preceded by memcg OOM kills and/or
          heavy mount/umount churn minutes to hours before.

In all three cases the corrupted pointer is an LRU list head inside a
mem_cgroup_per_node (slab start + offset == the bad pointer):

  #1: next == head, but head->prev == head (head list is empty)
  #2: prev == head, but head->next == head (head list is empty)
  #3: prev == head (different list, offset 368), but head->next points
      to another folio

So in #1 and #2 a folio still points at a lruvec list head that is
already empty, which looks like a folio stranded on the LRU of a memcg
whose lists have already been spliced to the parent. #3 may or may not
be the same
 event.

The entry paths differ (memory.high reclaim via lru_gen_look_around,
and compaction from an XFS mount), which suggests that the list is
already corrupted and the next LRU operation trips over it. Both
signatures match Karl's in-vivo cases #1 (lru_activate) and #2
(isolate_migratepages_block) in [1].

#1: 2026-11-02 08:19:59 UTC, Giga Computing R163-ZG2 (1 socket, 1 NUMA node)
-------------------------------------------------------------------------------
[1165000.986266]  slab mem_cgroup_per_node start ff31bfa36a3c9e00 pointer offset 528 size 1920
[1165000.995872] list_del corruption. next->prev should be ffacf881ccc70e08, but was ff31bfa36a3ca010. (next=ff31bfa36a3ca010)
[1165001.014102] kernel BUG at lib/list_debug.c:65!
[1165001.019464] Oops: invalid opcode: 0000 [#1] SMP NOPTI
[1165001.048442] Hardware name: Giga Computing R163-ZG2-AAJ2-000/MZG3-GU0-000, BIOS R20_F18 10/23/2025
[1165001.058744] RIP: 0010:__list_del_entry_valid_or_report+0xd2/0xf0
Call Trace
:
 lru_gen_del_folio.constprop.0+0x194/0x390
 lru_activate+0x80/0x380
 folio_batch_move_lru+0xb5/0x280
 __folio_batch_add_and_move+0x7c/0x100
 lru_gen_look_around+0x3ea/0x6b0
 folio_referenced_one+0x262/0x4e0
 __rmap_walk_file+0xb4/0x1a0
 folio_referenced+0x109/0x130
 shrink_folio_list+0x910/0xfd0
 evict_folios+0x126/0x5b0
 try_to_shrink_lruvec+0xb4/0x200
 shrink_lruvec+0x119/0xfe0
 shrink_node+0x22f/0xd00
 do_try_to_free_pages+0xb3/0x4d0
 try_to_free_mem_cgroup_pages+0xf0/0x210
 reclaim_high+0x83/0xd0
 __mem_cgroup_handle_over_high+0xf1/0x2c0
 exit_to_user_mode_loop+0x1c4/0x560
 do_syscall_64+0x263/0x640
[1165001.906773] Kernel panic - not syncing: Fatal exception

#2: 2026-11-08 ~04:58 UTC, Giga Computing R163-ZG2 (1 socket, 1 NUMA node)
-------------------------------------------------------------------------------
[1670289.398643]  slab mem_cgroup_per_node start ff16459986b32580 pointer offset 528 size 1920
[1670289.408376] list_del corruption. prev->next s
hould be ffc1e319d5020a08, but was ff16459986b32790. (prev=ff16459986b32790)
[1670289.426642] kernel BUG at lib/list_debug.c:62!
[1670289.431977] Oops: invalid opcode: 0000 [#1] SMP NOPTI
[1670289.461060] Hardware name: Giga Computing R163-ZG2-AAJ2-000/MZG3-GU0-000, BIOS R20_F18 10/23/2025
[1670289.471224] RIP: 0010:__list_del_entry_valid_or_report+0xb2/0xf0
Call Trace:
 isolate_migratepages_block+0x9d1/0x1720
 compact_zone+0x6da/0x10f0
 compact_zone_order+0xa1/0xd0
 try_to_compact_pages+0x124/0x290
 __alloc_pages_slowpath.constprop.0+0x70b/0x1400
 __alloc_frozen_pages_noprof+0x2e9/0x330
 alloc_pages_mpol+0xa6/0x180
 ___kmalloc_large_node+0xbc/0xf0
 __kvmalloc_node_noprof+0x4b5/0x730
 xlog_find_verify_log_record+0x43/0x210 [xfs]
 xlog_find_zeroed+0x19c/0x1f0 [xfs]
 xlog_find_head+0x52/0x370 [xfs]
 xlog_find_tail+0x53/0x380 [xfs]
 xlog_recover+0x3d/0x190 [xfs]
 xfs_log_mount+0xb5/0x1f0 [xfs]
 xfs_mountfs+0x439/0xad0 [xfs]
 xfs_fs_fill_super+0x756/0xa00 [xfs]
 get_
tree_bdev_flags+0x132/0x1d0
 vfs_get_tree+0x29/0xd0
 fc_mount+0x13/0x90
 path_mount+0x34a/0xc40
 __x64_sys_mount+0x121/0x150
 do_syscall_64+0xea/0x640
[1670290.504046] Kernel panic - not syncing: Fatal exception

#3: 2026-11-10 09:53:55 UTC, Supermicro H11SSL-i (1 socket, 1 NUMA node)
-------------------------------------------------------------------------------
[585952.686020]  slab mem_cgroup_per_node start ffff8af9df25e180 pointer offset 368 size 1920
[585952.695593] list_del corruption. prev->next should be fffff38a68aff788, but was fffff38a202f8808. (prev=ffff8af9df25e2f0)
[585952.713751] kernel BUG at lib/list_debug.c:62!
[585952.719475] Oops: invalid opcode: 0000 [#1] SMP NOPTI
[585952.748716] Hardware name: Supermicro Super Server/H11SSL-i, BIOS 3.2 01/24/2025
[585952.757460] RIP: 0010:__list_del_entry_valid_or_report+0xb2/0xf0
Call Trace:
 lru_gen_del_folio.constprop.0+0x194/0x390
 lru_activate+0x80/0x380
 folio_batch_move_lru+0xb5/0x280
 __folio_batch_add
_and_move+0x7c/0x100
 lru_gen_look_around+0x3ea/0x6b0
 folio_referenced_one+0x262/0x4e0
 __rmap_walk_file+0xb4/0x1a0
 folio_referenced+0x109/0x130
 shrink_folio_list+0x910/0xfd0
 evict_folios+0x126/0x5b0
 try_to_shrink_lruvec+0xb4/0x200
 shrink_lruvec+0x119/0xfe0
 shrink_node+0x22f/0xd00
 do_try_to_free_pages+0xb3/0x4d0
 try_to_free_mem_cgroup_pages+0xf0/0x210
 reclaim_high+0x83/0xd0
 __mem_cgroup_handle_over_high+0xf1/0x2c0
 try_charge_memcg+0x4e3/0x650
 charge_memcg+0x55/0xc0
 __mem_cgroup_charge+0x2c/0x80
 filemap_add_folio+0x61/0x210
 page_cache_ra_unbounded+0x13e/0x250
 do_page_cache_ra+0x5e/0x80
 filemap_fault+0x7e0/0x12b0
 ...
 asm_exc_page_fault+0x26/0x30
[585953.604861] Kernel panic - not syncing: Fatal exception

We are happy to test patches or debug options on affected nodes.

[1] https://lore.kernel.org/all/anMmd1ADrDVwMO6v@work/


Thanks,
Noah Feldt
mittwald

[-- Attachment #1.2: publickey - noah@feldt.systems - 0x9B02BBB4.asc --]
[-- Type: application/pgp-keys, Size: 698 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 322 bytes --]