Re: 6.18.37 has problems with nfs4 (server), 6.18.36 works

[Chuck Lever <cel@kernel.org>]

Hi Wolfgang,

Thanks for the report, and for narrowing it to 6.18.36 vs 6.18.37.

You've picked the right commit to suspect: 95f9eb19d5e6 ("Revert
'NFSD: Defer sub-object cleanup in export put callbacks'") is the
*only* change to the NFS server between 6.18.36 and 6.18.37, so an
A/B test around it is exactly the right experiment. Two things would
let us pin this down.

1. The full kernel log. The trace you sent begins in the middle of a
register dump, and the task that actually triggered the stall isn't
in it -- the RCU stall names CPU 13 / PID 8887 (nfsd), and that
backtrace is above where your paste starts. Could you send the
complete log from the first "soft lockup" / "rcu ... stall" /
"hung task" line onward, with all CPU backtraces? The part before
what you already sent is the piece I need. If the machine wedges
hard, a serial console or netconsole capture (or pstore/ramoops read
back after reboot) will get the whole thing.

While you're at it: roughly what was the server doing (client count,
NFS version, and was an "exportfs -r", mount, or umount in play), and
does it reproduce or was it a one-off after ~1 day?

2. The revert test, if you're willing to spend the rebuild. On a
v6.18.37 tree:

    git revert --no-edit 95f9eb19d5e6

That reverts the revert -- i.e. restores the 6.18.36 behavior for this
code. If that build stays healthy, it strongly implicates the change;
if it still locks up, we've ruled it out and should look at the NFSv4
laundromat/grace-period path instead. One caveat: this also brings
back a separate problem the revert fixed (a lingering mount reference
that can make "exportfs -r" followed by umount fail with EBUSY), so
treat it as a diagnostic build, not something to run long term.

With the full backtrace I can usually tell quickly whether the export
cache change is even on the code path that hung, or whether 6.18.37
just happened to expose something else.

--
Chuck Lever