Re: file server freezes with all nfsds stuck in D state after upgrade to Debian bookworm

[Chuck Lever III <chuck.lever@oracle.com>]

> On Apr 6, 2023, at 11:33 AM, Christian Herzog <herzog@phys.ethz.ch> wrote:
> 
> Dear Chuck,
> 
>>> for our researchers we are running file servers in the hundreds-of-TiB to
>>> low-PiB range that export via NFS and SMB. Storage is iSCSI-over-Infiniband
>>> LUNs LVM'ed into individual XFS file systems. With Ubuntu 18.04 nearing EOL,
>>> we prepared an upgrade to Debian bookworm and tests went well. About a week
>>> after one of the upgrades, we ran into the first occurence of our problem: all
>>> of a sudden, all nfsds enter the D state and are not recoverable. However, the
>>> underlying file systems seem fine and can be read and written to. The only way
>>> out appears to be to reboot the server. The only clues are the frozen nfsds
>>> and strack traces like
>>> 
>>> [<0>] rq_qos_wait+0xbc/0x130
>>> [<0>] wbt_wait+0xa2/0x110
>> 
>> Hi Christian, you have a pretty deep storage stack!
>> rq_qos_wait is a few layers below NFSD. Jens Axboe
>> and linux-block are the folks who maintain that.
> are you saying the root cause isn't nfs*, but the file system?

I can't possibly know what the root cause is at this point.


> That was our first idea too, but we haven't found any indication that this is the case. The xfs file systems seem perfectly fine when all nfsds are in D state, and we can
> read from them and write to them. If xfs were to block nfs IO, this should
> affect other processes too, right?

It's possible that the NFSD threads are waiting on I/O to a particular filesystem block. XFS is not likely to block other activity in this case.

I'm merely suggesting that you should start troubleshooting at the bottom of the stack instead of the top. The wait is far outside the realm of NFSD.


--
Chuck Lever