Re: hunting memory corruption bug in 6.18.x

[Nikola Ciprich <nikola.ciprich@linuxbox.cz>]

> 
> So I took a look. The one you added the sources for is this: 20261008-lbxv1f
> and below is what I could decipher it to.
> 
> Long story short, I *think* that dentry pointer becomes funky:
> 
>         hlist_bl_for_each_entry_rcu(dentry, node, b, d_hash) {
>                 unsigned seq;
> 
>                 /*
>                  * The dentry sequence count protects us from concurrent
>                  * renames, and thus protects parent and name fields.
>                  *
>                  * The caller must perform a seqcount check in order
>                  * to do anything useful with the returned dentry.
>                  *
>                  * NOTE! We do a "raw" seqcount_begin here. That means that
>                  * we don't wait for the sequence count to stabilize if it
>                  * is in the middle of a sequence change. If we do the slow
>                  * dentry compare, we will do seqretries until it is stable,
>                  * and if we end up with a successful lookup, we actually
>                  * want to exit RCU lookup anyway.
>                  *
>                  * Note that raw_seqcount_begin still *does* smp_rmb(), so
>                  * we are still guaranteed NUL-termination of ->d_name.name.
>                  */
>                 seq = raw_seqcount_begin(&dentry->d_seq);
> 					 ^^^^^^^^^^^^^^
> 
> Now, the $10^6 question is, do you have the crash dump still and if so, can
> you figure out what the real address of that dentry is?
> 
> The wrong one is: 0xffffffff0c930020 so I'm guessing that 0 after the ffff's
> should actually be 81-something as that's kernel memory.

I don’t have the vmcore for 20261008-lbxv1f (that box got fenced), but I have
a full vmcore for 20260930-flev1c, which is the same bug: same crash in the
dcache hash walk, and the same bad value — flev1c has 0x0fffffff0c930020
(top byte sheared), lbxv1f has 0xffffffff0c930020 (canonical). Identical low
48 bits (...0c930020). flev1c is __d_lookup (6.18.53), lbxv1f is
__d_lookup_rcu (6.18.55), so they may surface at slightly different points in
the walk — mapping what I see on flev1c onto lbxv1f is for you to judge.

What I can show from the flev1c dump:

The bad value is NOT a sheared dentry pointer — it’s the hash bucket head
that’s corrupt. __d_lookup computed the bucket from the hash
(dentry_hashtable=0xffffb18900a00000, d_hash_shift=7) as
0xffffb18905f60450 and loaded the first node from it: crash> rd -64 0xffffb18905f60450 1
ffffb18905f60450: 0fffffff0c930020 i.e. the garbage was already sitting in the bucket slot; the walk
dereferenced it as node->d_name.hash (+0x18) and took the GP fault.
The dentry the lookup was actually for (“app.slice”,
0xffff989ddf09b5c0) is fully intact, and its seqcount is even/stable: crash> struct dentry.d_seq 0xffff989ddf09b5c0
d_seq = { seqcount = { sequence = 2 } } So no rename/update was in progress — this doesn’t look like a d_seq /
raw_seqcount_begin race on flev1c; the bucket head itself holds foreign
data.
On the real address question: on this box page_offset_base =
0xffff985a80000000 (not 0xffff81…), so a direct-map dentry would be
0xffff985a……, and 0x…0c930020 doesn’t reconstruct to a valid dentry
under that base. That’s because the value isn’t a corrupted pointer at
all — it appears to be foreign structured data written into the bucket: crash> search -p 0x0fffffff0c930020
finds this exact value in ~10 physically-scattered frames, always at page
offsets 0x400/0x420/0x430/0x440/0x450 (relative to a 0x400 base:
+0x00/+0x20/+0x30/+0x40/+0x50; none at +0x10). Those frames all hold the
same fixed-format record: constant header 0x00ff00ff00100010 /
0xbdcc802700060042, a couple of varying address/length-looking fields,
and 0xcc padding. The frames’ struct page also carries stale page_pool
metadata (pp_magic set, pp_ref_count=0) while in use elsewhere.

So on flev1c it looks less like “a dentry pointer went funky” and more like
“some other structure’s content got written into dcache (and other) frames,
one copy landing in a hash bucket.” Does that header signature
(0x00ff00ff00100010 / 0xbdcc802700060042) ring a bell for any structure?

Happy to run anything else against this vmcore — struct page dumps for the
affected frames, the full record contents, the physical-search hit list, etc.

BR

nik




> 
> Hmmm?
> 
> Thx.
> 
> [11402.940943] BUG: unable to handle page fault for address: ffffffff0c93001c
> [11402.942273] #PF: supervisor read access in kernel mode
> [11402.943629] #PF: error_code(0x0000) - not-present page
> [11402.945025] PGD 6d6f83a067 P4D 6d6f83b067 PUD 0 
> [11402.946469] Oops: Oops: 0000 [#1] SMP NOPTI
> [11402.947950] CPU: 23 UID: 0 PID: 704950 Comm: servercare-moni Kdump: loaded Tainted: G            E       6.18.55lb9.01 #1 PREEMPT(voluntary) 
> [11402.951254] Tainted: [E]=UNSIGNED_MODULE
> [11402.952913] Hardware name: ASUSTeK COMPUTER INC. RS720A-E12-RS12/K14PP-D24 Series, BIOS 1201 08/25/2023
> [11402.956569] RIP: 0010:__d_lookup_rcu+0x4d/0xe0
> 
> ffffffff8165ae30 <__d_lookup_rcu>
> 		+0x4d = 0xffffffff8165ae7d
> 
> [11402.958452] Code: 48 8d 04 c2 f6 07 02 0f 85 a0 00 00 00 48 8b 10 48 89 d0 48 83 e0 fe 48 83 fa 01 77 0d e9 80 00 00 00 48 8b 00 48 85 c0 74 78 <44> 8b 58 fc 48 39 78 10 75 ee 48 83 78 08 00 74 e7 48 39 58 18 75
> 
> [11402.958452] Code: 48 8d 04 c2 f6 07 02 0f 85 a0 00 00 00 48 8b 10 48 89 d0 48 83 e0 fe 48 83 fa 01 77 0d e9 80 00 00 00 48 8b 00 48 85 c0 74 78 <44> 8b 58 fc 48 39 78 10 75 ee 48 83 78 08 00 74 e7 48 39 58 18 75
> 
> ffffffff8165ae6e:       77 0d                   ja     ffffffff8165ae7d <__d_lookup_rcu+0x4d>
> ffffffff8165ae70:       e9 80 00 00 00          jmp    ffffffff8165aef5 <__d_lookup_rcu+0xc5>
> ffffffff8165ae75:       48 8b 00                mov    (%rax),%rax
> ffffffff8165ae78:       48 85 c0                test   %rax,%rax
> ffffffff8165ae7b:       74 78                   je     ffffffff8165aef5 <__d_lookup_rcu+0xc5>
> ffffffff8165ae7d:       44 8b 58 fc             mov    -0x4(%rax),%r11d
> 
> <--- here
> 
> that looks like:
> 
> struct dentry *__d_lookup_rcu(const struct dentry *parent,
> ...
> 
> 	                seq = raw_seqcount_begin(&dentry->d_seq);
> 			...
> 
> 			#define raw_read_seqcount(s)                                            \
> ({                                                                      \
>         unsigned __seq = seqprop_sequence(s);                           \
> 
> which is that unreadable gunk:
> 
> # ./include/linux/seqlock.h:227: SEQCOUNT_LOCKNAME(spinlock,     spinlock_t,      __SEQ_RT, spin)
> #NO_APP
> 	movl	-4(%rax), %r11d	# MEM[(const volatile unsigned int *)node_49 + -4B], _25
> # ./include/linux/seqlock.h:227: SEQCOUNT_LOCKNAME(spinlock,     spinlock_t,      __SEQ_RT, spin)
> 
> i.e., that dentry->d_seq _Generic selector pointer thing:
> 
>   seq = ({ ({ unsigned __seq =
> 
>         _Generic(*(&dentry->d_seq), seqcount_t:
>                 __seqprop_sequence, seqcount_raw_spinlock_t:
>                 __seqprop_raw_spinlock_sequence, seqcount_spinlock_t:
>                 __seqprop_spinlock_sequence, seqcount_rwlock_t:
>                 __seqprop_rwlock_sequence,  seqcount_mutex_t:
>                 __seqprop_mutex_sequence)(&dentry->d_seq);
>   kcsan_atomic_next(1000); __seq; }) & ~1; });
> 
> 
> ffffffff8165ae81:       48 39 78 10             cmp    %rdi,0x10(%rax)
> ffffffff8165ae85:       75 ee                   jne    ffffffff8165ae75 <__d_lookup_rcu+0x45>
> ffffffff8165ae87:       48 83 78 08 00          cmpq   $0x0,0x8(%rax)
> ffffffff8165ae8c:       74 e7                   je     ffffffff8165ae75 <__d_lookup_rcu+0x45>
> ffffffff8165ae8e:       48 39 58 18             cmp    %rbx,0x18(%rax)
> ffffffff8165ae92:       75 e1                   jne    ffffffff8165ae75 <__d_lookup_rcu+0x45
> 
> All code
> ========
>    0:   48 8d 04 c2             lea    (%rdx,%rax,8),%rax
>    4:   f6 07 02                testb  $0x2,(%rdi)
>    7:   0f 85 a0 00 00 00       jne    0xad
>    d:   48 8b 10                mov    (%rax),%rdx
>   10:   48 89 d0                mov    %rdx,%rax
>   13:   48 83 e0 fe             and    $0xfffffffffffffffe,%rax
>   17:   48 83 fa 01             cmp    $0x1,%rdx
>   1b:   77 0d                   ja     0x2a
>   1d:   e9 80 00 00 00          jmp    0xa2
>   22:   48 8b 00                mov    (%rax),%rax
>   25:   48 85 c0                test   %rax,%rax
>   28:   74 78                   je     0xa2
>   2a:*  44 8b 58 fc             mov    -0x4(%rax),%r11d         <-- trapping instruction
> 
> rax = ffffffff0c930020
> 
>   2e:   48 39 78 10             cmp    %rdi,0x10(%rax)
>   32:   75 ee                   jne    0x22
>   34:   48 83 78 08 00          cmpq   $0x0,0x8(%rax)
>   39:   74 e7                   je     0x22
>   3b:   48 39 58 18             cmp    %rbx,0x18(%rax)
>   3f:   75                      .byte 0x75
> 
> Code starting with the faulting instruction
> ===========================================
>    0:   44 8b 58 fc             mov    -0x4(%rax),%r11d
>    4:   48 39 78 10             cmp    %rdi,0x10(%rax)
>    8:   75 ee                   jne    0xfffffffffffffff8
>    a:   48 83 78 08 00          cmpq   $0x0,0x8(%rax)
>    f:   74 e7                   je     0xfffffffffffffff8
>   11:   48 39 58 18             cmp    %rbx,0x18(%rax)
>   15:   75                      .byte 0x75
> 
> 
> [11402.964498] RSP: 0018:ff3eefdafcacfc40 EFLAGS: 00010292
> [11402.966642] RAX: ffffffff0c930020 RBX: 000000061356452c RCX: 0000000000000006
> [11402.968884] RDX: ffffffff0c930020 RSI: ff3eefdafcacfd70 RDI: ff2d1acd00409140
> [11402.971167] RBP: ff3eefdafcacfda4 R08: 8080808080808080 R09: fefefefefefefeff
> [11402.973512] R10: 0000303539343037 R11: 0000000000000004 R12: ff2d1acd8cb0b7e0
> [11402.975915] R13: ff3eefdafcacfd70 R14: 0000000000000000 R15: 0000000000000000
> [11402.978356] FS:  00007f757eaed640(0000) GS:ff2d1b89b5a42000(0000) knlGS:0000000000000000
> [11402.980842] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [11402.983374] CR2: ffffffff0c93001c CR3: 00000004ba18e002 CR4: 0000000000771ef0
> [11402.986004] PKRU: 55555554
> [11402.988640] Call Trace:
> [11402.991231]  <TASK>
> [11402.993831]  lookup_fast+0x2b/0x100
> [11402.996468]  walk_component+0x1f/0x150
> [11402.999080]  link_path_walk+0x105/0x2a0
> [11403.001714]  path_openat+0x99/0x2b0
> [11403.004319]  ? srso_alias_return_thunk+0x5/0xfbef5
> [11403.006983]  ? up_read+0x5a/0x90
> [11403.009610]  ? srso_alias_return_thunk+0x5/0xfbef5
> [11403.012250]  ? do_user_addr_fault+0x190/0x6a0
> [11403.014927]  do_filp_open+0xd3/0x180
> [11403.017635]  ? __pfx_kfree_link+0x10/0x10
> [11403.020383]  ? srso_alias_return_thunk+0x5/0xfbef5
> [11403.023097]  do_sys_openat2+0x8a/0xe0
> [11403.025765]  __x64_sys_openat+0x69/0xa0
> [11403.028424]  do_syscall_64+0x64/0xba0
> [11403.031066]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [11403.033733] RIP: 0033:0x7f75816ff1e4
> [11403.036336] Code: 24 20 eb 8f 66 90 44 89 54 24 0c e8 26 8c f8 ff 44 8b 54 24 0c 44 89 e2 48 89 ee 41 89 c0 bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 89 44 24 0c e8 78 8c f8 ff 8b 44
> [11403.044367] RSP: 002b:00007f757eaeb870 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
> [11403.047072] RAX: ffffffffffffffda RBX: 00007f757eaed5e0 RCX: 00007f75816ff1e4
> [11403.049795] RDX: 0000000000080000 RSI: 00007f757edd90e2 RDI: 00000000ffffff9c
> [11403.052470] RBP: 00007f757edd90e2 R08: 0000000000000000 R09: 0000000000000000
> [11403.055070] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000080000
> [11403.057594] R13: 00007f758187fe50 R14: 00007f75818e5040 R15: 0000000000000001
> [11403.060096]  </TASK>
> [11403.062521] Modules linked in: vhost_net(E) vhost(E) vhost_iotlb(E) tap(E) tun(E) ceph(E) libceph(E) cts(E) krb5enc(E) authenc(E) camellia_aesni_avx2(E) camellia_aesni_avx_x86_64(E) camellia_x86_64(E) camellia_generic(E) cmac(E) krb5(E) dns_resolver(E) netfs(E) nf_tables(E) tcp_diag(E) udp_diag(E) inet_diag(E) netconsole(E) sctp(E) ip6_udp_tunnel(E) udp_tunnel(E) rfkill(E) nfnetlink_cttimeout(E) nfnetlink(E) openvswitch(E) nf_conncount(E) nf_nat(E) nsh(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) sunrpc(E) kvm_amd(E) sr_mod(E) kvm(E) irqbypass(E) amd_atl(E) cdrom(E) sd_mod(E) amd64_edac(E) wmi_bmof(E) cdc_ether(E) acpi_ipmi(E) sg(E) edac_mce_amd(E) acpi_cpufreq(E) usbnet(E) pcspkr(E) ipmi_si(E) mii(E) ipmi_ssif(E) i2c_piix4(E) ipmi_devintf(E) vfat(E) i2c_smbus(E) k10temp(E) ipmi_msghandler(E) wmi(E) fat(E) i2c_core(E) sch_fq_codel(E) fuse(E) ext4(E) crc16(E) mbcache(E) jbd2(E) raid1(E) usb_storage(E) nvme(E) ahci(E) libahci(E) mpt3sas(E) raid_class(E) xhci_pci(E) nvme_core(E) ghash_clmulni_intel(E) libata(E)
> [11403.062611]  scsi_transport_sas(E) i40e(E) nvme_keyring(E) libie(E) nvme_auth(E) xhci_hcd(E) sp5100_tco(E) libie_adminq(E) hkdf(E) dm_mod(E) dax(E) aesni_intel(E)
> [11403.100478] CR2: ffffffff0c93001c
> 
> 
> -- 
> Regards/Gruss,
>     Boris.
> 
> https://people.kernel.org/tglx/notes-about-netiquette
> 

-- 
Ing. Nikola CIPRICH
technický ředitel

+420 591 166 214
+420 777 093 799
nikola.ciprich@linuxbox.cz

www.linuxbox.cz