[REGRESSION] v6.16 system hangs (bisected to nf_conntrack fix)

[Dan Moulding <dan@danm.net>]

Hello netfilter folks,

Since v6.16-rc7 I've been hitting a vexing system hang (no kernel
panic is being produced that I can see). I did not have this problem
when running rc6. I first noticed it the morning after upgrading to
rc7. I found the machine unresponsive. Checking logs after restarting
it, I could see it had been in the middle of being backed up by an
rsync-based backup system. This same sequence repeated the following
day.

Then I also started experiencing the hang when running a build on a
proprietary codebase that I work on. The machine that is hanging is a
virtual machine host, with a fleet of VMs that I use for doing various
development tasks. One of those VMs is where I build the proprietary
system. I do that by SSHing to the VM from the host and invoking the
build from there. The strange thing is that at the point the hang
occurs, there's nothing overtly "networky" that the build system is
doing. It's just compressing and creating a self-extracting archive on
the build VM. But it happens with 100% consistency which allowed me to
bisect it down to commit 2d72afb34065 (netfilter: nf_conntrack: fix
crash due to removal of uninitialised entry).

The hang is still present on the final v6.16 just released
yesterday. I have confirmed that if I revert the above commit on top
of v6.16, I can no longer reproduce the problem.

I know this doesn't provide a lot detail about the cause of the
problem, but the nature of the hang prevents me from being able to
check logs since the whole system becomes unresponsive. Any ideas on
next steps I might be able to take to gather more information, if
needed, are welcome.

Cheers,

-- Dan

#regzbot introduced: 2d72afb34065