Re: 6.18.37 has problems with nfs4 (server), 6.18.36 works

[Chuck Lever <cel@kernel.org>]

Hi Wolfgang, and stable@ --

Short version for stable@: 6.18.37 does not need a revert of
95f9eb19d5e6 ("Revert 'NFSD: Defer sub-object cleanup in export
put callbacks'").  That commit is correct for 6.18, and it is
not the cause of Wolfgang's crash.  Please leave it in place.

The reasoning: 95f9eb19d5e6 touches only fs/nfsd/export.c,
export.h, and nfsctl.c.  Wolfgang's oops is in
remove_blocked_locks() -> __destroy_client() ->
nfsd4_destroy_clientid(), entirely within fs/nfsd/nfs4state.c,
which the revert does not modify.  That path is byte-for-byte
identical across 6.18.36, 6.18.37, and current mainline, so the
revert cannot have introduced the bug and no missing backport
repairs it.  The 6.18.36-good / 6.18.37-bad split is a timing
coincidence; I believe the same latent bug is present in both.

Because the defect is present upstream as well, the fix belongs
in mainline first and is then backported to 6.18.y and the other
affected trees.

Wolfgang - to confirm this and capture the allocation and free
stacks, a KASAN-enabled kernel would settle it.  On a v6.18.37
tree:

  1. Add to your .config (keep your usual CONFIG_DEBUG_INFO so
     symbols resolve):

       CONFIG_KASAN=y
       CONFIG_KASAN_GENERIC=y
       CONFIG_KASAN_INLINE=y
       CONFIG_STACKTRACE=y

  2. Build and boot that kernel.  Stay on 6.18.37 -- you do not
     need the revert-the-revert build I suggested earlier; that
     experiment no longer tells us anything.

  3. When it trips, KASAN prints a "BUG: KASAN: use-after-free"
     report with "Allocated by" and "Freed by" call stacks.
     That report, in full, is what I need -- it should land in
     /var/log/messages just as the last oops did.

One caveat: KASAN roughly doubles memory use and adds CPU cost,
so weigh that before running it on the production server.  If
that is not practical, a full log from the first stall line
onward, with all CPU backtraces, captured over netconsole or
serial, is a useful second best.

I will draft a candidate upstream fix from the analysis so far
and send it separately.  If KASAN on the production box is not
an option, testing that patch may be the least disruptive way
to confirm.

Thanks for the careful report and the bisect.

Chuck