ext4-nfsd interaction causes sporadic hang on rwsem_down_write_failed

ext4-nfsd interaction causes sporadic hang on rwsem_down_write_failed

[Kevin Liu]

Hi,

I recently submitted an NFS bug
(https://bugzilla.kernel.org/show_bug.cgi?id=201655) where nfsd randomly
locks up on rwsem_down_write_failed:

Nov 10 15:29:55 rem kernel: INFO: task nfsd:7464 blocked for more than
120 seconds.
Nov 10 15:29:55 rem kernel:       Tainted: P           O      4.19.1
#1-NixOS
Nov 10 15:29:55 rem kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 10 15:29:55 rem kernel: nfsd            D    0  7464      2 0x80000000
Nov 10 15:29:55 rem kernel: Call Trace:
Nov 10 15:29:55 rem kernel:  ? __schedule+0x1d3/0x6f0
Nov 10 15:29:55 rem kernel:  schedule+0x28/0x80
Nov 10 15:29:55 rem kernel:  rwsem_down_write_failed+0x15e/0x350
Nov 10 15:29:55 rem kernel:  ? call_rwsem_down_write_failed+0x13/0x20
Nov 10 15:29:55 rem kernel:  call_rwsem_down_write_failed+0x13/0x20
Nov 10 15:29:55 rem kernel:  down_write+0x29/0x40
Nov 10 15:29:55 rem kernel:  ext4_file_write_iter+0x91/0x3d0 [ext4]
Nov 10 15:29:55 rem kernel:  ? nfsd_proc_write+0x160/0x160 [nfsd]
Nov 10 15:29:55 rem kernel:  ? exportfs_decode_fh+0xf2/0x2b0
...

(more details in the bugzilla)

And according to Bruce Fields:

> I'm guessing it's the inode_lock at the start of ext4_file_write_iter that's blocking.  On a quick look I don't see any of the callers taking any locks.  So I'd expect that elsewhere there'd be a process holding that inode lock and blocking on something else.
> 
> Based just on this might first guess would be a vfs or maybe ext4 bug rather than an nfsd bug, but I'm not seeing how to reassign.  May be worth reporting to the relevant mailing lists.

So, starting with ext4, I was wondering if you had an idea of what the
cause might be or where the fault truly lies.

Kevin Liu

Re: ext4-nfsd interaction causes sporadic hang on rwsem_down_write_failed

[Theodore Y. Ts'o]

On Mon, Nov 12, 2018 at 04:38:34AM +0000, Kevin Liu wrote:
> Hi,
> 
> I recently submitted an NFS bug
> (https://bugzilla.kernel.org/show_bug.cgi?id=201655) where nfsd randomly
> locks up on rwsem_down_write_failed:

> So, starting with ext4, I was wondering if you had an idea of what the
> cause might be or where the fault truly lies.

Sorry, this isn't something I've seen before.  And it's not at all
obvious from the information in Bugzilla what's causing the deadlock.

The down_read() up appears to be in mm/memory.c, in
__access_remote_vm() getting called from proc_pid_cmdline_read().
access_Remote_vm is apparently trying to get a shared lock on
&mm->mmap_sem.  How this would get involved with the inode_lock() is
not immediately obvious.

Things I would suggest.

1) Try running your kernel console log throughn
./scripts/decode_stacktrace.sh so we can be sure we've correctly
assessed where the kernel is grabbing which lock.  Enabling
CONFIG_DEBUG_INFO and CONFIG_DEBUG_INFO_REDUCED will be helpful.

2) Try turning on CONFIG_LOCKDEP and see if this reports some
potential deadlock.

3) Try using sysrq-d to find all held locks (running the resulting
kernel console output through decode_stacktrace.sh will also eb
helpful).

Cheers,

					- Ted