Re: 6.18.37 has problems with nfs4 (server), 6.18.36 works

Re: 6.18.37 has problems with nfs4 (server), 6.18.36 works

[Chuck Lever]

Hi Wolfgang,

Thanks for the report, and for narrowing it to 6.18.36 vs 6.18.37.

You've picked the right commit to suspect: 95f9eb19d5e6 ("Revert
'NFSD: Defer sub-object cleanup in export put callbacks'") is the
*only* change to the NFS server between 6.18.36 and 6.18.37, so an
A/B test around it is exactly the right experiment. Two things would
let us pin this down.

1. The full kernel log. The trace you sent begins in the middle of a
register dump, and the task that actually triggered the stall isn't
in it -- the RCU stall names CPU 13 / PID 8887 (nfsd), and that
backtrace is above where your paste starts. Could you send the
complete log from the first "soft lockup" / "rcu ... stall" /
"hung task" line onward, with all CPU backtraces? The part before
what you already sent is the piece I need. If the machine wedges
hard, a serial console or netconsole capture (or pstore/ramoops read
back after reboot) will get the whole thing.

While you're at it: roughly what was the server doing (client count,
NFS version, and was an "exportfs -r", mount, or umount in play), and
does it reproduce or was it a one-off after ~1 day?

2. The revert test, if you're willing to spend the rebuild. On a
v6.18.37 tree:

    git revert --no-edit 95f9eb19d5e6

That reverts the revert -- i.e. restores the 6.18.36 behavior for this
code. If that build stays healthy, it strongly implicates the change;
if it still locks up, we've ruled it out and should look at the NFSv4
laundromat/grace-period path instead. One caveat: this also brings
back a separate problem the revert fixed (a lingering mount reference
that can make "exportfs -r" followed by umount fail with EBUSY), so
treat it as a diagnostic build, not something to run long term.

With the full backtrace I can usually tell quickly whether the export
cache change is even on the code path that hung, or whether 6.18.37
just happened to expose something else.

--
Chuck Lever

Re: 6.18.37 has problems with nfs4 (server), 6.18.36 works

[Wolfgang Walter]

Hi!

Am 2026-07-02 01:43, schrieb Chuck Lever:
> Hi Wolfgang,
> 
> Thanks for the report, and for narrowing it to 6.18.36 vs 6.18.37.
> 
> You've picked the right commit to suspect: 95f9eb19d5e6 ("Revert
> 'NFSD: Defer sub-object cleanup in export put callbacks'") is the
> *only* change to the NFS server between 6.18.36 and 6.18.37, so an
> A/B test around it is exactly the right experiment. Two things would
> let us pin this down.
> 
> 1. The full kernel log. The trace you sent begins in the middle of a
> register dump, and the task that actually triggered the stall isn't
> in it -- the RCU stall names CPU 13 / PID 8887 (nfsd), and that
> backtrace is above where your paste starts. Could you send the
> complete log from the first "soft lockup" / "rcu ... stall" /
> "hung task" line onward, with all CPU backtraces? The part before
> what you already sent is the piece I need. If the machine wedges
> hard, a serial console or netconsole capture (or pstore/ramoops read
> back after reboot) will get the whole thing.

systemd-journald didn't catch it. But /var/log/messages seems to have 
logged it, hope this is it:

Jul  1 16:27:24 fileserver kernel: [76950.437185] PGD 0 P4D 0
Jul  1 16:27:24 fileserver kernel: [76950.437193] Oops: Oops: 0000 [#1] 
SMP NOPTI
Jul  1 16:27:24 fileserver kernel: [76950.437202] CPU: 2 UID: 0 PID: 
8844 Comm: nfsd Not tainted 6.18.37-debian64.all+1.3 #1 PREEMPT(full)
Jul  1 16:27:24 fileserver kernel: [76950.437215] Hardware name: 
Supermicro X10DRi/X10DRI-T, BIOS 1.1a 10/16/2015
Jul  1 16:27:24 fileserver kernel: [76950.437225] RIP: 
0010:__list_del_entry_valid_or_report+0x8/0x110
Jul  1 16:27:24 fileserver kernel: [76950.437240] Code: 48 89 c1 e8 ea 
18 94 ff 0f 0b 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 
90 90 90 90 90 f3 0f 1e fa 48 83 ec 10 <48> 8b 17 48 8b 4f 08 48 89 fe 
48 85 d2 0f 84 b8 a3 95 ff 48 85 c9
Jul  1 16:27:24 fileserver kernel: [76950.437262] RSP: 
0018:ffffd3aba4433c88 EFLAGS: 00010286
Jul  1 16:27:24 fileserver kernel: [76950.437271] RAX: 0000000000000000 
RBX: 0000000000000000 RCX: ffff8acfa99d4e10
Jul  1 16:27:24 fileserver kernel: [76950.437281] RDX: 0000000000000001 
RSI: 0000000000000001 RDI: 0000000000000000
Jul  1 16:27:24 fileserver kernel: [76950.437290] RBP: ffffffff99659a00 
R08: 0000000000480000 R09: ffff8ae70680f000
Jul  1 16:27:24 fileserver kernel: [76950.437299] R10: 0000000000480000 
R11: 0000000000000002 R12: ffffd3aba4433ca8
Jul  1 16:27:24 fileserver kernel: [76950.437308] R13: ffff8acfa99d4e10 
R14: ffff8acfa99d4f68 R15: ffff8ae7072d8000
Jul  1 16:27:24 fileserver kernel: [76950.437318] FS:  
0000000000000000(0000) GS:ffff8ae7065cf000(0000) knlGS:0000000000000000
Jul  1 16:27:24 fileserver kernel: [76950.437328] CS:  0010 DS: 0000 ES: 
0000 CR0: 0000000080050033
Jul  1 16:27:24 fileserver kernel: [76950.437336] CR2: 0000000000000000 
CR3: 00000011a3624002 CR4: 00000000001726f0
Jul  1 16:27:24 fileserver kernel: [76950.437345] Call Trace:
Jul  1 16:27:24 fileserver kernel: [76950.437351]  <TASK>
Jul  1 16:27:24 fileserver kernel: [76950.437358]  
remove_blocked_locks+0x91/0x1d0 [nfsd]
Jul  1 16:27:24 fileserver kernel: [76950.437453]  
__destroy_client+0x1b4/0x2a0 [nfsd]
Jul  1 16:27:24 fileserver kernel: [76950.437500]  
nfsd4_destroy_clientid+0xe6/0x1c0 [nfsd]
Jul  1 16:27:24 fileserver kernel: [76950.437545]  
nfsd4_proc_compound+0x325/0x680 [nfsd]
Jul  1 16:27:24 fileserver kernel: [76950.437594]  
nfsd_dispatch+0xc6/0x210 [nfsd]
Jul  1 16:27:24 fileserver kernel: [76950.437652]  
svc_process_common+0x4c3/0x6a0 [sunrpc]
Jul  1 16:27:24 fileserver kernel: [76950.437772]  ? 
__pfx_nfsd_dispatch+0x10/0x10 [nfsd]
Jul  1 16:27:24 fileserver kernel: [76950.437837]  
svc_process+0x142/0x210 [sunrpc]
Jul  1 16:27:24 fileserver kernel: [76950.437900]  svc_recv+0x7e5/0x9b0 
[sunrpc]
Jul  1 16:27:24 fileserver kernel: [76950.437955]  ? 
__pfx_nfsd+0x10/0x10 [nfsd]
Jul  1 16:27:24 fileserver kernel: [76950.438008]  nfsd+0x8f/0xf0 [nfsd]
Jul  1 16:27:24 fileserver kernel: [76950.438058]  kthread+0xfc/0x230
Jul  1 16:27:24 fileserver kernel: [76950.438068]  ? 
__pfx_kthread+0x10/0x10
Jul  1 16:27:24 fileserver kernel: [76950.438077]  
ret_from_fork+0x231/0x260
Jul  1 16:27:24 fileserver kernel: [76950.438086]  ? 
__pfx_kthread+0x10/0x10
Jul  1 16:27:24 fileserver kernel: [76950.438094]  
ret_from_fork_asm+0x1a/0x30
Jul  1 16:27:24 fileserver kernel: [76950.438104]  </TASK>
Jul  1 16:27:24 fileserver kernel: [76950.438108] Modules linked in: 
rpcsec_gss_krb5 msr 8021q garp stp llc mrp binfmt_misc intel_rapl_msr 
intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp 
ipmi_ssif kvm_intel kvm snd_pcm irqbypass polyval_clmulni snd_timer 
ghash_clmulni_intel rapl ast snd intel_cstate drm_client_lib vga16fb 
soundcore drm_shmem_helper intel_uncore drm_kms_helper vgastate pcspkr 
iTCO_wdt mei_me intel_pmc_bxt iTCO_vendor_support i2c_algo_bit mei 
watchdog ioatdma evdev joydev acpi_power_meter ipmi_si acpi_ipmi 
ipmi_devintf ipmi_msghandler button sg nfsd nfs_acl lockd 
chacha20poly1305 fileserverth_rpcgss aesni_intel grace cryptd 
nfs_localio drbd drm sunrpc fuse lru_cache loop efi_pstore configfs 
ip_tables x_tables fileservertofs4 ext4 crc16 mbcache jbd2 efivarfs 
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor 
async_tx xor raid6_pq raid0 linear dm_mod raid1 hid_generic md_mod ses 
enclosure usbhid hid sd_mod ixgbe libie_fwlog xfrm_algo dca mdio_devres 
of_mdio ahci fixed_phy xhci_pci libahci fwnode_mdio mpt3sas ehci_pci
Jul  1 16:27:24 fileserver kernel: [76950.438181]  libphy raid_class 
xhci_hcd libata ehci_hcd mdio_bus scsi_transport_sas i2c_i801 usbcore 
ptp i2c_smbus lpc_ich scsi_mod pps_core usb_common mdio scsi_common wmi
Jul  1 16:27:24 fileserver kernel: [76950.439549] CR2: 0000000000000000
Jul  1 16:27:24 fileserver kernel: [76950.439820] ---[ end trace 
0000000000000000 ]---


> 
> While you're at it: roughly what was the server doing (client count,
> NFS version, and was an "exportfs -r", mount, or umount in play), and
> does it reproduce or was it a one-off after ~1 day?

NFS 4.2 with sec=krb5p

The kernel of the clients vary, a lot have a 6.18 kernel, but a lot also 
use their distribution kernels.

I would think that at that moment when it crashed about 50 people were 
still using it. Probably quit some people where shuting down der clients 
at that time as it was almost 16:30.

It happened suddenly. Nothing special on the server.

It wasn't a reproduce, it was the first time 6.18.37 was running, I 
upgraded to 6.18.37 the evening before.

We had to hard reset the server after about 1/2 our as neither rebooting 
nor syncing seemed to make progress.

As this did some damage to the filesystem we hat to fsck it which needed 
some time so I decided to downgrade to 6.18.36 again.

> 
> 2. The revert test, if you're willing to spend the rebuild. On a
> v6.18.37 tree:
> 
>     git revert --no-edit 95f9eb19d5e6
> 
> That reverts the revert -- i.e. restores the 6.18.36 behavior for this
> code. If that build stays healthy, it strongly implicates the change;
> if it still locks up, we've ruled it out and should look at the NFSv4
> laundromat/grace-period path instead. One caveat: this also brings
> back a separate problem the revert fixed (a lingering mount reference
> that can make "exportfs -r" followed by umount fail with EBUSY), so
> treat it as a diagnostic build, not something to run long term.
> 

Ok, I think this would be acceptable. Maybe I encountered that in 
6.18.19, I wrote a report that nfsd got stuck when rebooting, but maybe 
it is something different as it seems to have disappeared in the last 
6.18 kernels.

So I will give 3.18.37 with this change reverted a try.

Regards and thanks for your help,
-- 
Wolfgang Walter
Studierendenwerk München Oberbayern
Anstalt des öffentlichen Rechts

Re: 6.18.37 has problems with nfs4 (server), 6.18.36 works

[Chuck Lever]

Hi Wolfgang, and stable@ --

Short version for stable@: 6.18.37 does not need a revert of
95f9eb19d5e6 ("Revert 'NFSD: Defer sub-object cleanup in export
put callbacks'").  That commit is correct for 6.18, and it is
not the cause of Wolfgang's crash.  Please leave it in place.

The reasoning: 95f9eb19d5e6 touches only fs/nfsd/export.c,
export.h, and nfsctl.c.  Wolfgang's oops is in
remove_blocked_locks() -> __destroy_client() ->
nfsd4_destroy_clientid(), entirely within fs/nfsd/nfs4state.c,
which the revert does not modify.  That path is byte-for-byte
identical across 6.18.36, 6.18.37, and current mainline, so the
revert cannot have introduced the bug and no missing backport
repairs it.  The 6.18.36-good / 6.18.37-bad split is a timing
coincidence; I believe the same latent bug is present in both.

Because the defect is present upstream as well, the fix belongs
in mainline first and is then backported to 6.18.y and the other
affected trees.

Wolfgang - to confirm this and capture the allocation and free
stacks, a KASAN-enabled kernel would settle it.  On a v6.18.37
tree:

  1. Add to your .config (keep your usual CONFIG_DEBUG_INFO so
     symbols resolve):

       CONFIG_KASAN=y
       CONFIG_KASAN_GENERIC=y
       CONFIG_KASAN_INLINE=y
       CONFIG_STACKTRACE=y

  2. Build and boot that kernel.  Stay on 6.18.37 -- you do not
     need the revert-the-revert build I suggested earlier; that
     experiment no longer tells us anything.

  3. When it trips, KASAN prints a "BUG: KASAN: use-after-free"
     report with "Allocated by" and "Freed by" call stacks.
     That report, in full, is what I need -- it should land in
     /var/log/messages just as the last oops did.

One caveat: KASAN roughly doubles memory use and adds CPU cost,
so weigh that before running it on the production server.  If
that is not practical, a full log from the first stall line
onward, with all CPU backtraces, captured over netconsole or
serial, is a useful second best.

I will draft a candidate upstream fix from the analysis so far
and send it separately.  If KASAN on the production box is not
an option, testing that patch may be the least disruptive way
to confirm.

Thanks for the careful report and the bisect.

Chuck

Re: 6.18.37 has problems with nfs4 (server), 6.18.36 works

[Wolfgang Walter]

Hello Chuck,

Am 2026-07-03 18:03, schrieb Chuck Lever:
> Hi Wolfgang, and stable@ --
> 
> Short version for stable@: 6.18.37 does not need a revert of
> 95f9eb19d5e6 ("Revert 'NFSD: Defer sub-object cleanup in export
> put callbacks'").  That commit is correct for 6.18, and it is
> not the cause of Wolfgang's crash.  Please leave it in place.

Ok. I run v6.18.37 with the patch reverted since about a day (just for 
the record). But according to your analysis, that's just a coincidence.

> 
> The reasoning: 95f9eb19d5e6 touches only fs/nfsd/export.c,
> export.h, and nfsctl.c.  Wolfgang's oops is in
> remove_blocked_locks() -> __destroy_client() ->
> nfsd4_destroy_clientid(), entirely within fs/nfsd/nfs4state.c,
> which the revert does not modify.  That path is byte-for-byte
> identical across 6.18.36, 6.18.37, and current mainline, so the
> revert cannot have introduced the bug and no missing backport
> repairs it.  The 6.18.36-good / 6.18.37-bad split is a timing
> coincidence; I believe the same latent bug is present in both.
> 
> Because the defect is present upstream as well, the fix belongs
> in mainline first and is then backported to 6.18.y and the other
> affected trees.
> 
> Wolfgang - to confirm this and capture the allocation and free
> stacks, a KASAN-enabled kernel would settle it.  On a v6.18.37
> tree:
> 
>   1. Add to your .config (keep your usual CONFIG_DEBUG_INFO so
>      symbols resolve):
> 
>        CONFIG_KASAN=y
>        CONFIG_KASAN_GENERIC=y
>        CONFIG_KASAN_INLINE=y
>        CONFIG_STACKTRACE=y
> 
>   2. Build and boot that kernel.  Stay on 6.18.37 -- you do not
>      need the revert-the-revert build I suggested earlier; that
>      experiment no longer tells us anything.
> 
>   3. When it trips, KASAN prints a "BUG: KASAN: use-after-free"
>      report with "Allocated by" and "Freed by" call stacks.
>      That report, in full, is what I need -- it should land in
>      /var/log/messages just as the last oops did.
> 
> One caveat: KASAN roughly doubles memory use and adds CPU cost,
> so weigh that before running it on the production server.  If
> that is not practical, a full log from the first stall line
> onward, with all CPU backtraces, captured over netconsole or
> serial, is a useful second best.
> 
> I will draft a candidate upstream fix from the analysis so far
> and send it separately.  If KASAN on the production box is not
> an option, testing that patch may be the least disruptive way
> to confirm.
> 

I think the memory usage should not be a problem, higher cpu usage 
neither.

But as it is a coincidence the probability to catch that error is 
probably very low. We use v6.18 kernels since v6.18.1 on that fileserver 
and this error never occured before.

Or do you think it happens more often, but without symptoms, and KASAN 
would detect it?

So I will try running a v3.18.37 + your patch applied. This of course 
can not prove that it fixes the problem because it almost never happens, 
but probably this would detect if if the patch had side effects.

Regards,
-- 
Wolfgang Walter
Studierendenwerk München Oberbayern
Anstalt des öffentlichen Rechts

Re: 6.18.37 has problems with nfs4 (server), 6.18.36 works

[Chuck Lever]

On Fri, Jul 3, 2026, at 2:30 PM, Wolfgang Walter wrote:
> Hello Chuck,
>
> Am 2026-07-03 18:03, schrieb Chuck Lever:
>> Hi Wolfgang, and stable@ --
>> 
>> Short version for stable@: 6.18.37 does not need a revert of
>> 95f9eb19d5e6 ("Revert 'NFSD: Defer sub-object cleanup in export
>> put callbacks'").  That commit is correct for 6.18, and it is
>> not the cause of Wolfgang's crash.  Please leave it in place.
>
> Ok. I run v6.18.37 with the patch reverted since about a day (just for 
> the record). But according to your analysis, that's just a coincidence.
>
>> 
>> The reasoning: 95f9eb19d5e6 touches only fs/nfsd/export.c,
>> export.h, and nfsctl.c.  Wolfgang's oops is in
>> remove_blocked_locks() -> __destroy_client() ->
>> nfsd4_destroy_clientid(), entirely within fs/nfsd/nfs4state.c,
>> which the revert does not modify.  That path is byte-for-byte
>> identical across 6.18.36, 6.18.37, and current mainline, so the
>> revert cannot have introduced the bug and no missing backport
>> repairs it.  The 6.18.36-good / 6.18.37-bad split is a timing
>> coincidence; I believe the same latent bug is present in both.
>> 
>> Because the defect is present upstream as well, the fix belongs
>> in mainline first and is then backported to 6.18.y and the other
>> affected trees.
>> 
>> Wolfgang - to confirm this and capture the allocation and free
>> stacks, a KASAN-enabled kernel would settle it.  On a v6.18.37
>> tree:
>> 
>>   1. Add to your .config (keep your usual CONFIG_DEBUG_INFO so
>>      symbols resolve):
>> 
>>        CONFIG_KASAN=y
>>        CONFIG_KASAN_GENERIC=y
>>        CONFIG_KASAN_INLINE=y
>>        CONFIG_STACKTRACE=y
>> 
>>   2. Build and boot that kernel.  Stay on 6.18.37 -- you do not
>>      need the revert-the-revert build I suggested earlier; that
>>      experiment no longer tells us anything.
>> 
>>   3. When it trips, KASAN prints a "BUG: KASAN: use-after-free"
>>      report with "Allocated by" and "Freed by" call stacks.
>>      That report, in full, is what I need -- it should land in
>>      /var/log/messages just as the last oops did.
>> 
>> One caveat: KASAN roughly doubles memory use and adds CPU cost,
>> so weigh that before running it on the production server.  If
>> that is not practical, a full log from the first stall line
>> onward, with all CPU backtraces, captured over netconsole or
>> serial, is a useful second best.
>> 
>> I will draft a candidate upstream fix from the analysis so far
>> and send it separately.  If KASAN on the production box is not
>> an option, testing that patch may be the least disruptive way
>> to confirm.
>> 
>
> I think the memory usage should not be a problem, higher cpu usage 
> neither.
>
> But as it is a coincidence the probability to catch that error is 
> probably very low. We use v6.18 kernels since v6.18.1 on that fileserver 
> and this error never occured before.
>
> Or do you think it happens more often, but without symptoms, and KASAN 
> would detect it?
>
> So I will try running a v3.18.37 + your patch applied. This of course 
> can not prove that it fixes the problem because it almost never happens, 
> but probably this would detect if if the patch had side effects.

Correct: your reproduction of the crash does not appear to
be strongly correlated with any particular kernel release. I
based my analysis strictly on the additional stack trace data
you sent earlier today.

I think it's more likely that your 50 client workload hit a
particular race that exposed a pre-existing UAF. KASAN will
change execution timing, certainly, but I can't predict
whether it will make the race window bigger.

So you can only test whether my patch causes new regressions,
not whether it prevents your crasher. :-(


-- 
Chuck Lever