* Locking problems with Linux 4.9 with NFSD and `fs/iomap.c`
@ 2017-05-07 19:09 Paul Menzel
2017-05-08 13:18 ` Brian Foster
0 siblings, 1 reply; 13+ messages in thread
From: Paul Menzel @ 2017-05-07 19:09 UTC (permalink / raw)
To: linux-nfs, linux-xfs; +Cc: it+linux-nfs
Dear Linux folks,
There seems to be a regression in Linux 4.9 compared to 4.4. Maybe you
have an idea.
The system used 4.4.38 without issues, and was updated to 4.9.23 on
April 24th. Since Friday, the NFS exports where not accessible anymore.
Rebooting the system into 4.9.24, 4.9.24, and 4.9.25 didn’t change
anything, and the system went into the some lock right away. Booting
4.4.38 fixed the issue though.
Here is more information.
NFS doesn’t respond to a null call.
```
# tshark -r x.x
1 0.000000 141.14.16.23 -> 141.14.26.2 TCP 74 1005 → 2049 [SYN]
Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=327954228 TSecr=0
WS=512
2 0.000245 141.14.26.2 -> 141.14.16.23 TCP 74 2049 → 1005 [SYN,
ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=1126765132
TSecr=327954228 WS=512
3 0.000269 141.14.16.23 -> 141.14.26.2 TCP 66 1005 → 2049 [ACK]
Seq=1 Ack=1 Win=29696 Len=0 TSval=327954229 TSecr=1126765132
4 0.000300 141.14.16.23 -> 141.14.26.2 NFS 110 V4 NULL Call
5 0.202652 141.14.16.23 -> 141.14.26.2 TCP 110 [TCP Retransmission]
1005 → 2049 [PSH, ACK] Seq=1 Ack=1 Win=29696 Len=44 TSval=327954432
TSecr=1126765132
6 0.410651 141.14.16.23 -> 141.14.26.2 TCP 110 [TCP Retransmission]
1005 → 2049 [PSH, ACK] Seq=1 Ack=1 Win=29696 Len=44 TSval=327954640
TSecr=1126765132
7 0.818649 141.14.16.23 -> 141.14.26.2 TCP 110 [TCP Retransmission]
1005 → 2049 [PSH, ACK] Seq=1 Ack=1 Win=29696 Len=44 TSval=327955048
TSecr=1126765132
8 1.012429 141.14.26.2 -> 141.14.16.23 TCP 74 [TCP Spurious
Retransmission] 2049 → 1005 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0
MSS=1460 SACK_PERM=1 TSval=1126766144 TSecr=327955048 WS=512
9 1.012448 141.14.16.23 -> 141.14.26.2 TCP 66 [TCP Dup ACK 3#1]
1005 → 2049 [ACK] Seq=45 Ack=1 Win=29696 Len=0 TSval=327955241
TSecr=1126765132
10 1.674650 141.14.16.23 -> 141.14.26.2 TCP 110 [TCP Retransmission]
1005 → 2049 [PSH, ACK] Seq=1 Ack=1 Win=29696 Len=44 TSval=327955904
TSecr=1126765132
11 3.060502 141.14.26.2 -> 141.14.16.23 TCP 74 [TCP Spurious
Retransmission] 2049 → 1005 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0
MSS=1460 SACK_PERM=1 TSval=1126768193 TSecr=327955904 WS=512
12 3.060521 141.14.16.23 -> 141.14.26.2 TCP 66 [TCP Dup ACK 3#2]
1005 → 2049 [ACK] Seq=45 Ack=1 Win=29696 Len=0 TSval=327957289
TSecr=1126765132
13 3.338653 141.14.16.23 -> 141.14.26.2 TCP 110 [TCP Retransmission]
1005 → 2049 [PSH, ACK] Seq=1 Ack=1 Win=29696 Len=44 TSval=327957568
TSecr=1126765132
```
``
# tshark -r x.x
1 0.000000 141.14.16.23 -> 141.14.26.2 TCP 74 690 → 2049 [SYN]
Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=328229577 TSecr=0
WS=512
2 0.000039 141.14.26.2 -> 141.14.16.23 TCP 74 2049 → 690 [SYN, ACK]
Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=1127040480
TSecr=328229577 WS=512
3 0.000155 141.14.16.23 -> 141.14.26.2 TCP 66 690 → 2049 [ACK]
Seq=1 Ack=1 Win=29696 Len=0 TSval=328229577 TSecr=1127040480
4 0.000180 141.14.16.23 -> 141.14.26.2 NFS 110 V4 NULL Call
5 0.206937 141.14.16.23 -> 141.14.26.2 TCP 110 [TCP Retransmission]
690 → 2049 [PSH, ACK] Seq=1 Ack=1 Win=29696 Len=44 TSval=328229784
TSecr=1127040480
6 0.414925 141.14.16.23 -> 141.14.26.2 TCP 110 [TCP Retransmission]
690 → 2049 [PSH, ACK] Seq=1 Ack=1 Win=29696 Len=44 TSval=328229992
TSecr=1127040480
7 0.822928 141.14.16.23 -> 141.14.26.2 TCP 110 [TCP Retransmission]
690 → 2049 [PSH, ACK] Seq=1 Ack=1 Win=29696 Len=44 TSval=328230400
TSecr=1127040480
8 1.056498 141.14.26.2 -> 141.14.16.23 TCP 74 [TCP Spurious
Retransmission] 2049 → 690 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0
MSS=1460 SACK_PERM=1 TSval=1127041536 TSecr=328230400 WS=512
```
Looking at the stack traces of the nfsd process with
```
for p in $(pgrep 'nfsd$'); do echo $p; cat /proc/$p/stack; done
```
all but one have the trace below.
```
[<ffffffff813eded7>] call_rwsem_down_write_failed+0x17/0x30
[<ffffffff8133aa1c>] xfs_file_buffered_aio_write+0x5c/0x2d0
[<ffffffff8133ad2e>] xfs_file_write_iter+0x9e/0x150
[<ffffffff81196b50>] do_iter_readv_writev+0xa0/0xf0
[<ffffffff81197d4a>] do_readv_writev+0x18a/0x230
[<ffffffff8119803c>] vfs_writev+0x3c/0x50
[<ffffffffa02826b5>] nfsd_vfs_write+0xc5/0x280 [nfsd]
[<ffffffffa028e48a>] nfsd4_write+0x17a/0x1d0 [nfsd]
[<ffffffffa02902f9>] nfsd4_proc_compound+0x369/0x5e0 [nfsd]
[<ffffffffa027e218>] nfsd_dispatch+0xa8/0x180 [nfsd]
[<ffffffffa01a66af>] svc_process_common+0x3ff/0x580 [sunrpc]
[<ffffffffa01a696b>] svc_process+0x13b/0x1b0 [sunrpc]
[<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd]
[<ffffffff8107b0ba>] kthread+0xca/0xe0
[<ffffffff81a97092>] ret_from_fork+0x22/0x30
[<ffffffffffffffff>] 0xffffffffffffffff
```
The one problematic one has the stack trace below.
```
[<ffffffff811f6f6c>] iomap_write_begin+0x8c/0x120
[<ffffffff811f758b>] iomap_zero_range_actor+0xeb/0x210 # oder
iomap_zero_range_actor+0x90/0x210
[<ffffffff811f77e2>] iomap_apply+0xa2/0x110
[<ffffffff811f79b8>] iomap_zero_range+0x58/0x80
[<ffffffff8133a38e>] xfs_zero_eof+0x4e/0xb0
[<ffffffff8133a58d>] xfs_file_aio_write_checks+0x19d/0x1c0
[<ffffffff8133aa39>] xfs_file_buffered_aio_write+0x79/0x2d0
[<ffffffff8133ad2e>] xfs_file_write_iter+0x9e/0x150
[<ffffffff81196b50>] do_iter_readv_writev+0xa0/0xf0
[<ffffffff81197d4a>] do_readv_writev+0x18a/0x230
[<ffffffff8119803c>] vfs_writev+0x3c/0x50
[<ffffffffa02826b5>] nfsd_vfs_write+0xc5/0x280 [nfsd]
[<ffffffffa028e48a>] nfsd4_write+0x17a/0x1d0 [nfsd]
[<ffffffffa02902f9>] nfsd4_proc_compound+0x369/0x5e0 [nfsd]
[<ffffffffa027e218>] nfsd_dispatch+0xa8/0x180 [nfsd]
[<ffffffffa01a66af>] svc_process_common+0x3ff/0x580 [sunrpc]
[<ffffffffa01a696b>] svc_process+0x13b/0x1b0 [sunrpc]
[<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd]
[<ffffffff8107b0ba>] kthread+0xca/0xe0
[<ffffffff81a97092>] ret_from_fork+0x22/0x30
[<ffffffffffffffff>] 0xffffffffffffffff
```
Here is the corresponding lines in the source code.
```
info line *0xffffffff811f6f6c Line 124 of "fs/iomap.c" starts at address
0xffffffff811f6f6c <iomap_write_begin+140> and ends at
0xffffffff811f6f70 <iomap_write_begin+144>.
info line *0xffffffff811f758b Line 345 of "fs/iomap.c" starts at address
0xffffffff811f758b <iomap_zero_range_actor+235> and ends at
0xffffffff811f758f <iomap_zero_range_actor+239>.
info line *0xffffffff811f7530 Line 385 of "fs/iomap.c" starts at address
0xffffffff811f7530 <iomap_zero_range_actor+144> and ends at
0xffffffff811f7538 <iomap_zero_range_actor+152>.
```
Here is the code.
```
[…]
109 static int
110 iomap_write_begin(struct inode *inode, loff_t pos, unsigned len,
unsigned flags,
111 struct page **pagep, struct iomap *iomap)
112 {
113 pgoff_t index = pos >> PAGE_SHIFT;
114 struct page *page;
115 int status = 0;
116
117 BUG_ON(pos + len > iomap->offset + iomap->length);
118
119 if (fatal_signal_pending(current))
120 return -EINTR;
121
122 page = grab_cache_page_write_begin(inode->i_mapping, index,
flags);
123 if (!page)
124 return -ENOMEM;
125
126 status = __block_write_begin_int(page, pos, len, NULL,
iomap);
127 if (unlikely(status)) {
128 unlock_page(page);
129 put_page(page);
130 page = NULL;
131
132 iomap_write_failed(inode, pos, len);
133 }
134
135 *pagep = page;
136 return status;
137 }
[…]
```
So it looks like insufficent memory? But the machine has 1 TB of RAM,
and I didn’t see any suspicious running `htop`. Also it should have
returned from with that error, shouldn’t it have?
Kind regards,
Paul
^ permalink raw reply [flat|nested] 13+ messages in thread* Re: Locking problems with Linux 4.9 with NFSD and `fs/iomap.c` 2017-05-07 19:09 Locking problems with Linux 4.9 with NFSD and `fs/iomap.c` Paul Menzel @ 2017-05-08 13:18 ` Brian Foster 2017-05-09 9:05 ` Christoph Hellwig 2017-05-10 9:08 ` Locking problems with Linux 4.9 " Paul Menzel 0 siblings, 2 replies; 13+ messages in thread From: Brian Foster @ 2017-05-08 13:18 UTC (permalink / raw) To: Paul Menzel; +Cc: linux-nfs, linux-xfs, it+linux-nfs, Christoph Hellwig cc Christoph, who's more familiar with nfs and wrote the iomap bits. On Sun, May 07, 2017 at 09:09:49PM +0200, Paul Menzel wrote: > Dear Linux folks, > > > There seems to be a regression in Linux 4.9 compared to 4.4. Maybe you have > an idea. > > The system used 4.4.38 without issues, and was updated to 4.9.23 on April > 24th. Since Friday, the NFS exports where not accessible anymore. Rebooting > the system into 4.9.24, 4.9.24, and 4.9.25 didn’t change anything, and the > system went into the some lock right away. Booting 4.4.38 fixed the issue > though. > The buffered write path was rewritten with the iomap mechanism around 4.7 or so, so there's a pretty big functionality gap between 4.4 and 4.9. > Here is more information. > > NFS doesn’t respond to a null call. > What exactly is a NULL call? Can this be reproduced easily? Otherwise, it's not clear to me whether you've hit a deadlock or some kind of livelock. Have you checked syslog for any crash or hung task messages? Please also provide the hung task output (echo w > /proc/sysrq-trigger) once you've hit this state. It would be particularly interesting to see whether the iomap_zero_range() path is included in that output. It may also be interesting to enable the xfs_zero_eof() tracepoint (trace-cmd start -e 'xfs:xfs_zero_eof') and see what the last few entries are from /sys/kernel/debug/tracing/trace_pipe. Brian > ``` > # tshark -r x.x > 1 0.000000 141.14.16.23 -> 141.14.26.2 TCP 74 1005 → 2049 [SYN] Seq=0 > Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=327954228 TSecr=0 WS=512 > 2 0.000245 141.14.26.2 -> 141.14.16.23 TCP 74 2049 → 1005 [SYN, ACK] > Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=1126765132 > TSecr=327954228 WS=512 > 3 0.000269 141.14.16.23 -> 141.14.26.2 TCP 66 1005 → 2049 [ACK] Seq=1 > Ack=1 Win=29696 Len=0 TSval=327954229 TSecr=1126765132 > 4 0.000300 141.14.16.23 -> 141.14.26.2 NFS 110 V4 NULL Call > 5 0.202652 141.14.16.23 -> 141.14.26.2 TCP 110 [TCP Retransmission] > 1005 → 2049 [PSH, ACK] Seq=1 Ack=1 Win=29696 Len=44 TSval=327954432 > TSecr=1126765132 > 6 0.410651 141.14.16.23 -> 141.14.26.2 TCP 110 [TCP Retransmission] > 1005 → 2049 [PSH, ACK] Seq=1 Ack=1 Win=29696 Len=44 TSval=327954640 > TSecr=1126765132 > 7 0.818649 141.14.16.23 -> 141.14.26.2 TCP 110 [TCP Retransmission] > 1005 → 2049 [PSH, ACK] Seq=1 Ack=1 Win=29696 Len=44 TSval=327955048 > TSecr=1126765132 > 8 1.012429 141.14.26.2 -> 141.14.16.23 TCP 74 [TCP Spurious > Retransmission] 2049 → 1005 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 > SACK_PERM=1 TSval=1126766144 TSecr=327955048 WS=512 > 9 1.012448 141.14.16.23 -> 141.14.26.2 TCP 66 [TCP Dup ACK 3#1] 1005 → > 2049 [ACK] Seq=45 Ack=1 Win=29696 Len=0 TSval=327955241 TSecr=1126765132 > 10 1.674650 141.14.16.23 -> 141.14.26.2 TCP 110 [TCP Retransmission] > 1005 → 2049 [PSH, ACK] Seq=1 Ack=1 Win=29696 Len=44 TSval=327955904 > TSecr=1126765132 > 11 3.060502 141.14.26.2 -> 141.14.16.23 TCP 74 [TCP Spurious > Retransmission] 2049 → 1005 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 > SACK_PERM=1 TSval=1126768193 TSecr=327955904 WS=512 > 12 3.060521 141.14.16.23 -> 141.14.26.2 TCP 66 [TCP Dup ACK 3#2] 1005 → > 2049 [ACK] Seq=45 Ack=1 Win=29696 Len=0 TSval=327957289 TSecr=1126765132 > 13 3.338653 141.14.16.23 -> 141.14.26.2 TCP 110 [TCP Retransmission] > 1005 → 2049 [PSH, ACK] Seq=1 Ack=1 Win=29696 Len=44 TSval=327957568 > TSecr=1126765132 > ``` > > `` > # tshark -r x.x > 1 0.000000 141.14.16.23 -> 141.14.26.2 TCP 74 690 → 2049 [SYN] Seq=0 > Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=328229577 TSecr=0 WS=512 > 2 0.000039 141.14.26.2 -> 141.14.16.23 TCP 74 2049 → 690 [SYN, ACK] > Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=1127040480 > TSecr=328229577 WS=512 > 3 0.000155 141.14.16.23 -> 141.14.26.2 TCP 66 690 → 2049 [ACK] Seq=1 > Ack=1 Win=29696 Len=0 TSval=328229577 TSecr=1127040480 > 4 0.000180 141.14.16.23 -> 141.14.26.2 NFS 110 V4 NULL Call > 5 0.206937 141.14.16.23 -> 141.14.26.2 TCP 110 [TCP Retransmission] 690 > → 2049 [PSH, ACK] Seq=1 Ack=1 Win=29696 Len=44 TSval=328229784 > TSecr=1127040480 > 6 0.414925 141.14.16.23 -> 141.14.26.2 TCP 110 [TCP Retransmission] 690 > → 2049 [PSH, ACK] Seq=1 Ack=1 Win=29696 Len=44 TSval=328229992 > TSecr=1127040480 > 7 0.822928 141.14.16.23 -> 141.14.26.2 TCP 110 [TCP Retransmission] 690 > → 2049 [PSH, ACK] Seq=1 Ack=1 Win=29696 Len=44 TSval=328230400 > TSecr=1127040480 > 8 1.056498 141.14.26.2 -> 141.14.16.23 TCP 74 [TCP Spurious > Retransmission] 2049 → 690 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 > SACK_PERM=1 TSval=1127041536 TSecr=328230400 WS=512 > ``` > > Looking at the stack traces of the nfsd process with > > ``` > for p in $(pgrep 'nfsd$'); do echo $p; cat /proc/$p/stack; done > ``` > > all but one have the trace below. > > > ``` > [<ffffffff813eded7>] call_rwsem_down_write_failed+0x17/0x30 > [<ffffffff8133aa1c>] xfs_file_buffered_aio_write+0x5c/0x2d0 > [<ffffffff8133ad2e>] xfs_file_write_iter+0x9e/0x150 > [<ffffffff81196b50>] do_iter_readv_writev+0xa0/0xf0 > [<ffffffff81197d4a>] do_readv_writev+0x18a/0x230 > [<ffffffff8119803c>] vfs_writev+0x3c/0x50 > [<ffffffffa02826b5>] nfsd_vfs_write+0xc5/0x280 [nfsd] > [<ffffffffa028e48a>] nfsd4_write+0x17a/0x1d0 [nfsd] > [<ffffffffa02902f9>] nfsd4_proc_compound+0x369/0x5e0 [nfsd] > [<ffffffffa027e218>] nfsd_dispatch+0xa8/0x180 [nfsd] > [<ffffffffa01a66af>] svc_process_common+0x3ff/0x580 [sunrpc] > [<ffffffffa01a696b>] svc_process+0x13b/0x1b0 [sunrpc] > [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] > [<ffffffff8107b0ba>] kthread+0xca/0xe0 > [<ffffffff81a97092>] ret_from_fork+0x22/0x30 > [<ffffffffffffffff>] 0xffffffffffffffff > ``` > > The one problematic one has the stack trace below. > > ``` > [<ffffffff811f6f6c>] iomap_write_begin+0x8c/0x120 > [<ffffffff811f758b>] iomap_zero_range_actor+0xeb/0x210 # oder > iomap_zero_range_actor+0x90/0x210 > [<ffffffff811f77e2>] iomap_apply+0xa2/0x110 > [<ffffffff811f79b8>] iomap_zero_range+0x58/0x80 > [<ffffffff8133a38e>] xfs_zero_eof+0x4e/0xb0 > [<ffffffff8133a58d>] xfs_file_aio_write_checks+0x19d/0x1c0 > [<ffffffff8133aa39>] xfs_file_buffered_aio_write+0x79/0x2d0 > [<ffffffff8133ad2e>] xfs_file_write_iter+0x9e/0x150 > [<ffffffff81196b50>] do_iter_readv_writev+0xa0/0xf0 > [<ffffffff81197d4a>] do_readv_writev+0x18a/0x230 > [<ffffffff8119803c>] vfs_writev+0x3c/0x50 > [<ffffffffa02826b5>] nfsd_vfs_write+0xc5/0x280 [nfsd] > [<ffffffffa028e48a>] nfsd4_write+0x17a/0x1d0 [nfsd] > [<ffffffffa02902f9>] nfsd4_proc_compound+0x369/0x5e0 [nfsd] > [<ffffffffa027e218>] nfsd_dispatch+0xa8/0x180 [nfsd] > [<ffffffffa01a66af>] svc_process_common+0x3ff/0x580 [sunrpc] > [<ffffffffa01a696b>] svc_process+0x13b/0x1b0 [sunrpc] > [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] > [<ffffffff8107b0ba>] kthread+0xca/0xe0 > [<ffffffff81a97092>] ret_from_fork+0x22/0x30 > [<ffffffffffffffff>] 0xffffffffffffffff > ``` > > Here is the corresponding lines in the source code. > > ``` > info line *0xffffffff811f6f6c Line 124 of "fs/iomap.c" starts at address > 0xffffffff811f6f6c <iomap_write_begin+140> and ends at 0xffffffff811f6f70 > <iomap_write_begin+144>. > info line *0xffffffff811f758b Line 345 of "fs/iomap.c" starts at address > 0xffffffff811f758b <iomap_zero_range_actor+235> and ends at > 0xffffffff811f758f <iomap_zero_range_actor+239>. > info line *0xffffffff811f7530 Line 385 of "fs/iomap.c" starts at address > 0xffffffff811f7530 <iomap_zero_range_actor+144> and ends at > 0xffffffff811f7538 <iomap_zero_range_actor+152>. > ``` > > Here is the code. > > ``` > […] > 109 static int > 110 iomap_write_begin(struct inode *inode, loff_t pos, unsigned len, > unsigned flags, > 111 struct page **pagep, struct iomap *iomap) > 112 { > 113 pgoff_t index = pos >> PAGE_SHIFT; > 114 struct page *page; > 115 int status = 0; > 116 > 117 BUG_ON(pos + len > iomap->offset + iomap->length); > 118 > 119 if (fatal_signal_pending(current)) > 120 return -EINTR; > 121 > 122 page = grab_cache_page_write_begin(inode->i_mapping, index, > flags); > 123 if (!page) > 124 return -ENOMEM; > 125 > 126 status = __block_write_begin_int(page, pos, len, NULL, iomap); > 127 if (unlikely(status)) { > 128 unlock_page(page); > 129 put_page(page); > 130 page = NULL; > 131 > 132 iomap_write_failed(inode, pos, len); > 133 } > 134 > 135 *pagep = page; > 136 return status; > 137 } > […] > ``` > > So it looks like insufficent memory? But the machine has 1 TB of RAM, and I > didn’t see any suspicious running `htop`. Also it should have returned from > with that error, shouldn’t it have? > > > Kind regards, > > Paul > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Locking problems with Linux 4.9 with NFSD and `fs/iomap.c` 2017-05-08 13:18 ` Brian Foster @ 2017-05-09 9:05 ` Christoph Hellwig [not found] ` <7ae18b0d-38e3-9b12-0989-ede68956ad43@molgen.mpg.de> 2017-05-10 9:08 ` Locking problems with Linux 4.9 " Paul Menzel 1 sibling, 1 reply; 13+ messages in thread From: Christoph Hellwig @ 2017-05-09 9:05 UTC (permalink / raw) To: Brian Foster Cc: Paul Menzel, linux-nfs, linux-xfs, it+linux-nfs, Christoph Hellwig On Mon, May 08, 2017 at 09:18:46AM -0400, Brian Foster wrote: > What exactly is a NULL call? Can this be reproduced easily? It's a sunrpc concept of a dummy procedure that does nothing but ensures the communication works. > It may also be interesting to enable the xfs_zero_eof() tracepoint > (trace-cmd start -e 'xfs:xfs_zero_eof') and see what the last few > entries are from /sys/kernel/debug/tracing/trace_pipe. Yeah. I have a customer that uses Linux 4.9-stable with XFS and nfsd exports in a product, so it's certainly not something that happens too easily, but memory pressure might be a factor. I'll see if I can figure out what happens if I inject ENOMEM in the said spot. ^ permalink raw reply [flat|nested] 13+ messages in thread
[parent not found: <7ae18b0d-38e3-9b12-0989-ede68956ad43@molgen.mpg.de>]
[parent not found: <358037e8-6784-ebca-9fbb-ec7eef3977d6@molgen.mpg.de>]
[parent not found: <20170510171757.GA10534@localhost.localdomain>]
* Re: Locking problems with Linux 4.9 and 4.11 with NFSD and `fs/iomap.c` [not found] ` <20170510171757.GA10534@localhost.localdomain> @ 2017-06-27 11:59 ` Paul Menzel 2017-06-28 16:41 ` Christoph Hellwig 2017-08-01 17:49 ` Paul Menzel 0 siblings, 2 replies; 13+ messages in thread From: Paul Menzel @ 2017-06-27 11:59 UTC (permalink / raw) To: Brian Foster Cc: Christoph Hellwig, it+linux-nfs, linux-nfs, linux-xfs, J. Bruce Fields, Jeff Layton [-- Attachment #1: Type: text/plain, Size: 901 bytes --] Dear Brian, dear Christoph, Just a small update that we were hit by the problem on a different machine (identical model) with Linux 4.9.32 and the exact same symptoms. ``` $ sudo cat /proc/2085/stack [<ffffffff811f920c>] iomap_write_begin+0x8c/0x120 [<ffffffff811f982b>] iomap_zero_range_actor+0xeb/0x210 [<ffffffff811f9a82>] iomap_apply+0xa2/0x110 [<ffffffff811f9c58>] iomap_zero_range+0x58/0x80 [<ffffffff8133c7de>] xfs_zero_eof+0x4e/0xb0 [<ffffffff8133c9dd>] xfs_file_aio_write_checks+0x19d/0x1c0 [<ffffffff8133ce89>] xfs_file_buffered_aio_write+0x79/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffffffffff>] 0xffffffffffffffff ``` We haven’t had time to set up a test system yet to analyze that further. Kind regards, Paul [-- Attachment #2: 20170627–linux_4.9.32–nfsd_locking.txt --] [-- Type: text/plain, Size: 52848 bytes --] 2033 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2034 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2035 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2036 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2037 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2038 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2039 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2040 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2041 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2042 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2043 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2044 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2045 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2046 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2047 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2048 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2049 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2050 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2051 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2052 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2053 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2054 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2055 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2056 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2057 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2058 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2060 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2061 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2062 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2063 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2064 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2066 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2067 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2068 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2069 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2070 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2071 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2073 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2074 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2075 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2076 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2077 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2078 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2079 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2080 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2081 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2082 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2083 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2084 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2085 [<ffffffff811f920c>] iomap_write_begin+0x8c/0x120 [<ffffffff811f97d0>] iomap_zero_range_actor+0x90/0x210 [<ffffffff811f9a82>] iomap_apply+0xa2/0x110 [<ffffffff811f9c58>] iomap_zero_range+0x58/0x80 [<ffffffff8133c7de>] xfs_zero_eof+0x4e/0xb0 [<ffffffff8133c9dd>] xfs_file_aio_write_checks+0x19d/0x1c0 [<ffffffff8133ce89>] xfs_file_buffered_aio_write+0x79/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2086 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2087 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2088 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2089 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2090 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2091 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2092 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2093 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2095 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2096 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2097 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2098 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2099 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff 2100 [<ffffffff813f0a97>] call_rwsem_down_write_failed+0x17/0x30 [<ffffffff8133ce6c>] xfs_file_buffered_aio_write+0x5c/0x2d0 [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffa0282705>] nfsd_vfs_write+0xc5/0x280 [nfsd] [<ffffffffa028e4ea>] nfsd4_write+0x17a/0x1d0 [nfsd] [<ffffffffa029034e>] nfsd4_proc_compound+0x35e/0x5d0 [nfsd] [<ffffffffa027e21d>] nfsd_dispatch+0xad/0x1d0 [nfsd] [<ffffffffa00a26af>] svc_process_common+0x3ff/0x580 [sunrpc] [<ffffffffa00a296b>] svc_process+0x13b/0x1b0 [sunrpc] [<ffffffffa027dc7d>] nfsd+0xed/0x160 [nfsd] [<ffffffff8107d117>] kthread+0xd7/0xf0 [<ffffffff81a99d92>] ret_from_fork+0x22/0x30 [<ffffffffffffffff>] 0xffffffffffffffff ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Locking problems with Linux 4.9 and 4.11 with NFSD and `fs/iomap.c` 2017-06-27 11:59 ` Locking problems with Linux 4.9 and 4.11 " Paul Menzel @ 2017-06-28 16:41 ` Christoph Hellwig 2017-08-01 17:49 ` Paul Menzel 1 sibling, 0 replies; 13+ messages in thread From: Christoph Hellwig @ 2017-06-28 16:41 UTC (permalink / raw) To: Paul Menzel Cc: Brian Foster, Christoph Hellwig, it+linux-nfs, linux-nfs, linux-xfs, J. Bruce Fields, Jeff Layton On Tue, Jun 27, 2017 at 01:59:46PM +0200, Paul Menzel wrote: > Dear Brian, dear Christoph, > > > Just a small update that we were hit by the problem on a different machine > (identical model) with Linux 4.9.32 and the exact same symptoms. Thanks for the update. I have to admit this issue completely dropped off my table, but I'll try to get back to it in the next days. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Locking problems with Linux 4.9 and 4.11 with NFSD and `fs/iomap.c` 2017-06-27 11:59 ` Locking problems with Linux 4.9 and 4.11 " Paul Menzel 2017-06-28 16:41 ` Christoph Hellwig @ 2017-08-01 17:49 ` Paul Menzel 2017-08-01 22:51 ` Dave Chinner 1 sibling, 1 reply; 13+ messages in thread From: Paul Menzel @ 2017-08-01 17:49 UTC (permalink / raw) To: Brian Foster, Christoph Hellwig Cc: it+linux-nfs, linux-nfs, linux-xfs, J. Bruce Fields, Jeff Layton Dear Brian, dear Christoph, On 06/27/17 13:59, Paul Menzel wrote: > Just a small update that we were hit by the problem on a different > machine (identical model) with Linux 4.9.32 and the exact same symptoms. > > ``` > $ sudo cat /proc/2085/stack > [<ffffffff811f920c>] iomap_write_begin+0x8c/0x120 > [<ffffffff811f982b>] iomap_zero_range_actor+0xeb/0x210 > [<ffffffff811f9a82>] iomap_apply+0xa2/0x110 > [<ffffffff811f9c58>] iomap_zero_range+0x58/0x80 > [<ffffffff8133c7de>] xfs_zero_eof+0x4e/0xb0 > [<ffffffff8133c9dd>] xfs_file_aio_write_checks+0x19d/0x1c0 > [<ffffffff8133ce89>] xfs_file_buffered_aio_write+0x79/0x2d0 > [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 > [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 > [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 > [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 > [<ffffffffffffffff>] 0xffffffffffffffff > ``` > > We haven’t had time to set up a test system yet to analyze that further. Today, two systems with Linux 4.9.23 exhibited the problem of `top` showing that `nfsd` is at 100 %. Restarting one machine into Linux *4.9.38* showed the same problem. One of them with a 1 GBit/s network device got traffic from a 10 GBit/s system, so the connection was saturated. But looking at the current processes, process with ID 829 does not show that CPU usage value. (I don’t remember what the behavior during the other incidents on other machines was.) This is from 4.9.38. The stack trace of the process with ID 857 is below. ``` $ sudo cat /proc/857/stack [<ffffffff811f94fc>] iomap_write_begin+0x6c/0x120 [<ffffffffffffffff>] 0xffffffffffffffff ``` Then I executed Brian’s commands. ``` $ trace-cmd start -p function -l iomap* -P <pid> $ cat /sys/kernel/debug/tracing/trace_pipe ``` This produced output. ``` nfsd-857 [005] .... 426.650413: iomap_write_end <-iomap_zero_range_actor nfsd-857 [005] .... 426.650414: iomap_write_begin <-iomap_zero_range_actor CPU:5 [LOST 115621695 EVENTS] nfsd-857 [005] .... 453.070439: iomap_write_begin <-iomap_zero_range_actor nfsd-857 [005] .... 453.070440: iomap_write_end <-iomap_zero_range_actor ``` This continues endlessly. Kind regards, Paul ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Locking problems with Linux 4.9 and 4.11 with NFSD and `fs/iomap.c` 2017-08-01 17:49 ` Paul Menzel @ 2017-08-01 22:51 ` Dave Chinner 2017-08-10 14:11 ` Paul Menzel 0 siblings, 1 reply; 13+ messages in thread From: Dave Chinner @ 2017-08-01 22:51 UTC (permalink / raw) To: Paul Menzel Cc: Brian Foster, Christoph Hellwig, it+linux-nfs, linux-nfs, linux-xfs, J. Bruce Fields, Jeff Layton On Tue, Aug 01, 2017 at 07:49:50PM +0200, Paul Menzel wrote: > Dear Brian, dear Christoph, > > > On 06/27/17 13:59, Paul Menzel wrote: > > >Just a small update that we were hit by the problem on a different > >machine (identical model) with Linux 4.9.32 and the exact same > >symptoms. > > > >``` > >$ sudo cat /proc/2085/stack > >[<ffffffff811f920c>] iomap_write_begin+0x8c/0x120 > >[<ffffffff811f982b>] iomap_zero_range_actor+0xeb/0x210 > >[<ffffffff811f9a82>] iomap_apply+0xa2/0x110 > >[<ffffffff811f9c58>] iomap_zero_range+0x58/0x80 > >[<ffffffff8133c7de>] xfs_zero_eof+0x4e/0xb0 > >[<ffffffff8133c9dd>] xfs_file_aio_write_checks+0x19d/0x1c0 > >[<ffffffff8133ce89>] xfs_file_buffered_aio_write+0x79/0x2d0 > >[<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 > >[<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 > >[<ffffffff81199fba>] do_readv_writev+0x18a/0x230 > >[<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 > >[<ffffffffffffffff>] 0xffffffffffffffff > >``` > > > >We haven’t had time to set up a test system yet to analyze that further. > > Today, two systems with Linux 4.9.23 exhibited the problem of `top` > showing that `nfsd` is at 100 %. Restarting one machine into Linux > *4.9.38* showed the same problem. One of them with a 1 GBit/s > network device got traffic from a 10 GBit/s system, so the > connection was saturated. So the question is this: is there IO being issued here, is the page cache growing, or is it in a tight loop doing nothing? Details of your hardware, XFS config and NFS server config is kinda important here, too. For example, if the NFS server IO patterns trigger a large speculative delayed allocation, then the client does a write at the end of the speculative delalloc range, we will zero the entire speculative delalloc range. That could be several GB of zeros that need to be written here. It's sub-optimal, yes, and but large zeroing is rare enough that we haven't needed to optimise it by allocating unwritten extents instead. It would be really handy to know what application the NFS client is running as that might give insight into the trigger behaviour and whether you are hitting this case. Also, if the NFS client is only writing to one file, then all the other writes that are on the wire will end up being serviced by nfsd threads that then block waiting for the inode lock. If the client issues more writes on the wire thant he NFS server has worker threads, the client side write will starve the NFS server of worker threads until the zeroing completes. This is the behaviour you are seeing - it's a common server side config error that's been known for at least 15 years... FWIW, it used to be that a linux NFS client could have 16 concurrent outstanding NFS RPCs to a server at a time - I don't know if that limit still exists or whether it's been increased. However, the typical knfsd default is (still) only 8 worker threads, meaning a single client and server using default configs can cause the above server DOS issue. e.g on a bleeding edge debian distro install: $ head -2 /etc/default/nfs-kernel-server # Number of servers to start up RPCNFSDCOUNT=8 $ So, yeah, distros still only configure the nfs server with 8 worker thread by default. If it's a dedicated NFS server, then I'd be using somewhere around 64 NFSD threads *per CPU* as a starting point for the server config... At minimum, you need to ensure that the NFS server has at least double the number of server threads as the largest client side concurrent RPC count so that a single client can't DOS the NFS server with a single blocked write stream. Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Locking problems with Linux 4.9 and 4.11 with NFSD and `fs/iomap.c` 2017-08-01 22:51 ` Dave Chinner @ 2017-08-10 14:11 ` Paul Menzel 2017-08-10 19:54 ` AW: " Markus Stockhausen 0 siblings, 1 reply; 13+ messages in thread From: Paul Menzel @ 2017-08-10 14:11 UTC (permalink / raw) To: Dave Chinner; +Cc: it+linux-nfs, Brian Foster, Christoph Hellwig Dear Dave, On 08/02/17 00:51, Dave Chinner wrote: > On Tue, Aug 01, 2017 at 07:49:50PM +0200, Paul Menzel wrote: >> On 06/27/17 13:59, Paul Menzel wrote: >> >>> Just a small update that we were hit by the problem on a different >>> machine (identical model) with Linux 4.9.32 and the exact same >>> symptoms. >>> >>> ``` >>> $ sudo cat /proc/2085/stack >>> [<ffffffff811f920c>] iomap_write_begin+0x8c/0x120 >>> [<ffffffff811f982b>] iomap_zero_range_actor+0xeb/0x210 >>> [<ffffffff811f9a82>] iomap_apply+0xa2/0x110 >>> [<ffffffff811f9c58>] iomap_zero_range+0x58/0x80 >>> [<ffffffff8133c7de>] xfs_zero_eof+0x4e/0xb0 >>> [<ffffffff8133c9dd>] xfs_file_aio_write_checks+0x19d/0x1c0 >>> [<ffffffff8133ce89>] xfs_file_buffered_aio_write+0x79/0x2d0 >>> [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 >>> [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 >>> [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 >>> [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 >>> [<ffffffffffffffff>] 0xffffffffffffffff >>> ``` >>> >>> We haven’t had time to set up a test system yet to analyze that further. >> >> Today, two systems with Linux 4.9.23 exhibited the problem of `top` >> showing that `nfsd` is at 100 %. Restarting one machine into Linux >> *4.9.38* showed the same problem. One of them with a 1 GBit/s >> network device got traffic from a 10 GBit/s system, so the >> connection was saturated. > > So the question is this: is there IO being issued here, is the page > cache growing, or is it in a tight loop doing nothing? Details of > your hardware, XFS config and NFS server config is kinda important > here, too. Could you please guide me, where I can get the information you request? The hardware ranges from slow 12 thread systems with 96 GB RAM to 80 thread 1 TB RAM machines. Often big files (up to 100 GB) are written. > For example, if the NFS server IO patterns trigger a large > speculative delayed allocation, then the client does a write at the > end of the speculative delalloc range, we will zero the entire > speculative delalloc range. That could be several GB of zeros that > need to be written here. It's sub-optimal, yes, and but large > zeroing is rare enough that we haven't needed to optimise it by > allocating unwritten extents instead. It would be really handy to > know what application the NFS client is running as that might give > insight into the trigger behaviour and whether you are hitting this > case. It ranges from simple `cp` to scripts writing FASTQ files with biological sequences in it. > Also, if the NFS client is only writing to one file, then all the > other writes that are on the wire will end up being serviced by nfsd > threads that then block waiting for the inode lock. If the client > issues more writes on the wire than the NFS server has worker > threads, the client side write will starve the NFS server of > worker threads until the zeroing completes. This is the behaviour > you are seeing - it's a common server side config error that's been > known for at least 15 years... > > FWIW, it used to be that a linux NFS client could have 16 concurrent > outstanding NFS RPCs to a server at a time - I don't know if that > limit still exists or whether it's been increased. However, the > typical knfsd default is (still) only 8 worker threads, meaning a > single client and server using default configs can cause the above > server DOS issue. e.g on a bleeding edge debian distro install: > > $ head -2 /etc/default/nfs-kernel-server > # Number of servers to start up > RPCNFSDCOUNT=8 > $ > > So, yeah, distros still only configure the nfs server with 8 worker > thread by default. If it's a dedicated NFS server, then I'd be using > somewhere around 64 NFSD threads *per CPU* as a starting point for > the server config... > > At minimum, you need to ensure that the NFS server has at least > double the number of server threads as the largest client side > concurrent RPC count so that a single client can't DOS the NFS > server with a single blocked write stream. That’s not the issue here. It’s started with 64 threads here. Also this doesn’t explain, why it works with the 4.4 series. The directory cannot be accessed at all. `ls /mounted/path` just hangs on remote systems. Kind regards, Paul ^ permalink raw reply [flat|nested] 13+ messages in thread
* AW: Locking problems with Linux 4.9 and 4.11 with NFSD and `fs/iomap.c` 2017-08-10 14:11 ` Paul Menzel @ 2017-08-10 19:54 ` Markus Stockhausen 2017-08-11 10:15 ` Christoph Hellwig 0 siblings, 1 reply; 13+ messages in thread From: Markus Stockhausen @ 2017-08-10 19:54 UTC (permalink / raw) To: 'Paul Menzel', Dave Chinner Cc: it+linux-nfs@molgen.mpg.de, Brian Foster, Christoph Hellwig [-- Attachment #1: Type: text/plain, Size: 5173 bytes --] Hi Paul et al, > Dear Dave, > > > On 08/02/17 00:51, Dave Chinner wrote: > > On Tue, Aug 01, 2017 at 07:49:50PM +0200, Paul Menzel wrote: > > >> On 06/27/17 13:59, Paul Menzel wrote: > >> > >>> Just a small update that we were hit by the problem on a different > >>> machine (identical model) with Linux 4.9.32 and the exact same > >>> symptoms. > >>> > >>> ``` > >>> $ sudo cat /proc/2085/stack > >>> [<ffffffff811f920c>] iomap_write_begin+0x8c/0x120 > >>> [<ffffffff811f982b>] iomap_zero_range_actor+0xeb/0x210 > >>> [<ffffffff811f9a82>] iomap_apply+0xa2/0x110 [<ffffffff811f9c58>] > >>> iomap_zero_range+0x58/0x80 [<ffffffff8133c7de>] > >>> xfs_zero_eof+0x4e/0xb0 [<ffffffff8133c9dd>] > >>> xfs_file_aio_write_checks+0x19d/0x1c0 > >>> [<ffffffff8133ce89>] xfs_file_buffered_aio_write+0x79/0x2d0 > >>> [<ffffffff8133d17e>] xfs_file_write_iter+0x9e/0x150 > >>> [<ffffffff81198dc0>] do_iter_readv_writev+0xa0/0xf0 > >>> [<ffffffff81199fba>] do_readv_writev+0x18a/0x230 > >>> [<ffffffff8119a2ac>] vfs_writev+0x3c/0x50 [<ffffffffffffffff>] > >>> 0xffffffffffffffff ``` > >>> > >>> We haven’t had time to set up a test system yet to analyze that further. > >> > >> Today, two systems with Linux 4.9.23 exhibited the problem of `top` > >> showing that `nfsd` is at 100 %. Restarting one machine into Linux > >> *4.9.38* showed the same problem. One of them with a 1 GBit/s network > >> device got traffic from a 10 GBit/s system, so the connection was > >> saturated. > > > > So the question is this: is there IO being issued here, is the page > > cache growing, or is it in a tight loop doing nothing? Details of your > > hardware, XFS config and NFS server config is kinda important here, > > too. Looking at 64 bit input for variable count of iomap_zero_range_actor(struct inode *inode, loff_t pos, loff_t count, void *data, struct iomap *iomap) and 32 bit internal variables unsigned offset, bytes; culprit could be bytes = min_t(unsigned, PAGE_SIZE - offset, count); Lets say you are trying to zero multiple of 4GB chunks. With bytes evaluated towards 0 this will hit an endless loop within that iomap function. That might explain your observation. If that is right a bugfix would qualify for stable 4.8+ Best regards. Markus > Could you please guide me, where I can get the information you request? > > The hardware ranges from slow 12 thread systems with 96 GB RAM to 80 thread 1 TB RAM machines. Often big files (up to 100 GB) are written. > > > For example, if the NFS server IO patterns trigger a large speculative > > delayed allocation, then the client does a write at the end of the > > speculative delalloc range, we will zero the entire speculative > > delalloc range. That could be several GB of zeros that need to be > > written here. It's sub-optimal, yes, and but large zeroing is rare > > enough that we haven't needed to optimise it by allocating unwritten > > extents instead. It would be really handy to know what application > > the NFS client is running as that might give insight into the trigger > > behaviour and whether you are hitting this case. > > It ranges from simple `cp` to scripts writing FASTQ files with biological sequences in it. > > > > Also, if the NFS client is only writing to one file, then all the > > other writes that are on the wire will end up being serviced by nfsd > > threads that then block waiting for the inode lock. If the client > > issues more writes on the wire than the NFS server has worker threads, > > the client side write will starve the NFS server of worker threads > > until the zeroing completes. This is the behaviour you are seeing - > > it's a common server side config error that's been known for at least > > 15 years... > > > > FWIW, it used to be that a linux NFS client could have 16 concurrent > > outstanding NFS RPCs to a server at a time - I don't know if that > > limit still exists or whether it's been increased. However, the > > typical knfsd default is (still) only 8 worker threads, meaning a > > single client and server using default configs can cause the above > > server DOS issue. e.g on a bleeding edge debian distro install: > > > > $ head -2 /etc/default/nfs-kernel-server # Number of servers to start > > up > > RPCNFSDCOUNT=8 > > $ > > > > So, yeah, distros still only configure the nfs server with 8 worker > > thread by default. If it's a dedicated NFS server, then I'd be using > > somewhere around 64 NFSD threads *per CPU* as a starting point for the > > server config... > > > > At minimum, you need to ensure that the NFS server has at least double > > the number of server threads as the largest client side concurrent RPC > > count so that a single client can't DOS the NFS server with a single > > blocked write stream. > > That’s not the issue here. It’s started with 64 threads here. Also this doesn’t explain, why it works with the 4.4 series. > > The directory cannot be accessed at all. `ls /mounted/path` just hangs on remote systems. > > > Kind regards, > > Paul [-- Attachment #2: InterScan_Disclaimer.txt --] [-- Type: text/plain, Size: 1650 bytes --] **************************************************************************** Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. Ãœber das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497 **************************************************************************** ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: AW: Locking problems with Linux 4.9 and 4.11 with NFSD and `fs/iomap.c` 2017-08-10 19:54 ` AW: " Markus Stockhausen @ 2017-08-11 10:15 ` Christoph Hellwig 2017-08-11 15:14 ` Paul Menzel 0 siblings, 1 reply; 13+ messages in thread From: Christoph Hellwig @ 2017-08-11 10:15 UTC (permalink / raw) To: Markus Stockhausen Cc: 'Paul Menzel', Dave Chinner, it+linux-nfs@molgen.mpg.de, Brian Foster, Christoph Hellwig, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, J. Bruce Fields, Jeff Layton On Thu, Aug 10, 2017 at 07:54:51PM +0000, Markus Stockhausen wrote: > Lets say you are trying to zero multiple of 4GB chunks. With bytes > evaluated towards 0 this will hit an endless loop within that iomap > function. That might explain your observation. If that is right a bugfix > would qualify for stable 4.8+ Yes, it seems like min_t casts arguments 2 and 3 to the type in argument 1, which could lead to incorrect truncation. Paul, please try the patch below: diff --git a/fs/iomap.c b/fs/iomap.c index 039266128b7f..59cc98ad7577 100644 --- a/fs/iomap.c +++ b/fs/iomap.c @@ -278,7 +278,7 @@ iomap_dirty_actor(struct inode *inode, loff_t pos, loff_t length, void *data, unsigned long bytes; /* Bytes to write to page */ offset = (pos & (PAGE_SIZE - 1)); - bytes = min_t(unsigned long, PAGE_SIZE - offset, length); + bytes = min_t(loff_t, PAGE_SIZE - offset, length); rpage = __iomap_read_page(inode, pos); if (IS_ERR(rpage)) @@ -373,7 +373,7 @@ iomap_zero_range_actor(struct inode *inode, loff_t pos, loff_t count, unsigned offset, bytes; offset = pos & (PAGE_SIZE - 1); /* Within page */ - bytes = min_t(unsigned, PAGE_SIZE - offset, count); + bytes = min_t(loff_t, PAGE_SIZE - offset, count); if (IS_DAX(inode)) status = iomap_dax_zero(pos, offset, bytes, iomap); ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: Locking problems with Linux 4.9 and 4.11 with NFSD and `fs/iomap.c` 2017-08-11 10:15 ` Christoph Hellwig @ 2017-08-11 15:14 ` Paul Menzel 0 siblings, 0 replies; 13+ messages in thread From: Paul Menzel @ 2017-08-11 15:14 UTC (permalink / raw) To: Christoph Hellwig, Markus Stockhausen Cc: Dave Chinner, it+linux-nfs@molgen.mpg.de, Brian Foster, linux-nfs@vger.kernel.org, linux-xfs@vger.kernel.org, J. Bruce Fields, Jeff Layton Dear Christoph, dear Markus, On 08/11/17 12:15, Christoph Hellwig wrote: > On Thu, Aug 10, 2017 at 07:54:51PM +0000, Markus Stockhausen wrote: >> Lets say you are trying to zero multiple of 4GB chunks. With bytes >> evaluated towards 0 this will hit an endless loop within that iomap >> function. That might explain your observation. If that is right a bugfix >> would qualify for stable 4.8+ > > Yes, it seems like min_t casts arguments 2 and 3 to the type in argument > 1, which could lead to incorrect truncation. > > Paul, please try the patch below: > > diff --git a/fs/iomap.c b/fs/iomap.c > index 039266128b7f..59cc98ad7577 100644 > --- a/fs/iomap.c > +++ b/fs/iomap.c > @@ -278,7 +278,7 @@ iomap_dirty_actor(struct inode *inode, loff_t pos, loff_t length, void *data, > unsigned long bytes; /* Bytes to write to page */ > > offset = (pos & (PAGE_SIZE - 1)); > - bytes = min_t(unsigned long, PAGE_SIZE - offset, length); > + bytes = min_t(loff_t, PAGE_SIZE - offset, length); > > rpage = __iomap_read_page(inode, pos); > if (IS_ERR(rpage)) > @@ -373,7 +373,7 @@ iomap_zero_range_actor(struct inode *inode, loff_t pos, loff_t count, > unsigned offset, bytes; > > offset = pos & (PAGE_SIZE - 1); /* Within page */ > - bytes = min_t(unsigned, PAGE_SIZE - offset, count); > + bytes = min_t(loff_t, PAGE_SIZE - offset, count); > > if (IS_DAX(inode)) > status = iomap_dax_zero(pos, offset, bytes, iomap); I applied this on top of Linux 4.9.41. Even when writing 40 100 GB in parallel on an NFS exported directory from different systems, and keeping the load high on the system, the NFS exported directory was always accessible from all hosts. Markus, thank you very much for looking into this and spotting the bug. A bug thank you to all people helping to analyze this issue. I’ll keep you posted, but until know. Tested-by: Paul Menzel <pmenzel@molgen.mpg.de> Kind regards, Paul ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Locking problems with Linux 4.9 with NFSD and `fs/iomap.c` 2017-05-08 13:18 ` Brian Foster 2017-05-09 9:05 ` Christoph Hellwig @ 2017-05-10 9:08 ` Paul Menzel 2017-05-10 17:23 ` J. Bruce Fields 1 sibling, 1 reply; 13+ messages in thread From: Paul Menzel @ 2017-05-10 9:08 UTC (permalink / raw) To: Brian Foster Cc: linux-nfs, linux-xfs, it+linux-nfs, Christoph Hellwig, J. Bruce Fields, Jeff Layton Dear Brian, On 05/08/17 15:18, Brian Foster wrote: > cc Christoph, who's more familiar with nfs and wrote the iomap bits. Thank you. > On Sun, May 07, 2017 at 09:09:49PM +0200, Paul Menzel wrote: >> There seems to be a regression in Linux 4.9 compared to 4.4. Maybe you have >> an idea. >> >> The system used 4.4.38 without issues, and was updated to 4.9.23 on April >> 24th. Since Friday, the NFS exports where not accessible anymore. Rebooting >> the system into 4.9.24, 4.9.24, and 4.9.25 didn’t change anything, and the >> system went into the some lock right away. Booting 4.4.38 fixed the issue >> though. > > The buffered write path was rewritten with the iomap mechanism around > 4.7 or so, so there's a pretty big functionality gap between 4.4 and > 4.9. > >> Here is more information. >> >> NFS doesn’t respond to a null call. > > What exactly is a NULL call? Sorry for not making that clear for non-NFS people. From *NFS Version 3 Protocol Specification* [1]: > Procedure NULL does not do any work. It is made available to > allow server response testing and timing. > Can this be reproduced easily? Unfortunately, we don’t know how to reproduce it. It seems to happen after heavy input/output operations though. ``` $ sudo nfsstat -s Server rpc stats: calls badcalls badclnt badauth xdrcall 15644232 0 0 0 0 Server nfs v4: null compound 1071 0% 15643006 99% Server nfs v4 operations: op0-unused op1-unused op2-future access close commit 0 0% 0 0% 0 0% 87846 0% 42798 0% 50658 0% create delegpurge delegreturn getattr getfh link 2805 0% 0 0% 16866 0% 8271204 21% 82924 0% 0 0% lock lockt locku lookup lookup_root nverify 0 0% 0 0% 0 0% 64424 0% 0 0% 0 0% open openattr open_conf open_dgrd putfh putpubfh 53848 0% 0 0% 1081 0% 20 0% 15569041 39% 0 0% putrootfh read readdir readlink remove rename 1072 0% 7187366 18% 2045 0% 73 0% 9116 0% 5836 0% renew restorefh savefh secinfo setattr setcltid 72534 0% 0 0% 5836 0% 0 0% 21817 0% 1854 0% setcltidconf verify write rellockowner bc_ctl bind_conn 1854 0% 0 0% 7794634 19% 0 0% 0 0% 0 0% exchange_id create_ses destroy_ses free_stateid getdirdeleg getdevinfo 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% getdevlist layoutcommit layoutget layoutreturn secinfononam sequence 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% set_ssv test_stateid want_deleg destroy_clid reclaim_comp 0 0% 0 0% 0 0% 0 0% 0 0% ``` > Otherwise, it's not clear to me whether you've hit a deadlock or some > kind of livelock. Have you checked syslog for any crash or hung task > messages? Please also provide the hung task output (echo w > > /proc/sysrq-trigger) once you've hit this state. It would be > particularly interesting to see whether the iomap_zero_range() path is > included in that output. Please see the Linux messages in my reply to Christoph’s message. > It may also be interesting to enable the xfs_zero_eof() tracepoint > (trace-cmd start -e 'xfs:xfs_zero_eof') and see what the last few > entries are from /sys/kernel/debug/tracing/trace_pipe. I built `trace-cmd`, and did what you asked, but there are no messages. ``` $ sudo strace ~/src/trace-cmd/trace-cmd start -e 'xfs:xfs_zero_eof' $ sudo cat /sys/kernel/tracing/events/xfs/xfs_zero_eof/enable 1 $ sudo cat /sys/kernel/debug/tracing/tracing_on 1 $ sudo cat /sys/kernel/debug/tracing/trace_pipe ``` Kind regards, Paul [1] https://www.ietf.org/rfc/rfc1813.txt ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Locking problems with Linux 4.9 with NFSD and `fs/iomap.c` 2017-05-10 9:08 ` Locking problems with Linux 4.9 " Paul Menzel @ 2017-05-10 17:23 ` J. Bruce Fields 0 siblings, 0 replies; 13+ messages in thread From: J. Bruce Fields @ 2017-05-10 17:23 UTC (permalink / raw) To: Paul Menzel Cc: Brian Foster, linux-nfs, linux-xfs, it+linux-nfs, Christoph Hellwig, Jeff Layton On Wed, May 10, 2017 at 11:08:52AM +0200, Paul Menzel wrote: > On 05/08/17 15:18, Brian Foster wrote: > >>NFS doesn’t respond to a null call. > > > >What exactly is a NULL call? > > Sorry for not making that clear for non-NFS people. From *NFS > Version 3 Protocol Specification* [1]: > > >Procedure NULL does not do any work. It is made available to > >allow server response testing and timing. NFSD has a fixed number of threads for processing requests, so if one of them hangs in the filesystem while holding a lock, all of them are likely to eventually end up waiting for that lock, and there will be none left even to answer NULL requests. So this is a pretty normal symptom of a deadlock or similar problem. --b. ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2017-08-11 15:14 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-05-07 19:09 Locking problems with Linux 4.9 with NFSD and `fs/iomap.c` Paul Menzel
2017-05-08 13:18 ` Brian Foster
2017-05-09 9:05 ` Christoph Hellwig
[not found] ` <7ae18b0d-38e3-9b12-0989-ede68956ad43@molgen.mpg.de>
[not found] ` <358037e8-6784-ebca-9fbb-ec7eef3977d6@molgen.mpg.de>
[not found] ` <20170510171757.GA10534@localhost.localdomain>
2017-06-27 11:59 ` Locking problems with Linux 4.9 and 4.11 " Paul Menzel
2017-06-28 16:41 ` Christoph Hellwig
2017-08-01 17:49 ` Paul Menzel
2017-08-01 22:51 ` Dave Chinner
2017-08-10 14:11 ` Paul Menzel
2017-08-10 19:54 ` AW: " Markus Stockhausen
2017-08-11 10:15 ` Christoph Hellwig
2017-08-11 15:14 ` Paul Menzel
2017-05-10 9:08 ` Locking problems with Linux 4.9 " Paul Menzel
2017-05-10 17:23 ` J. Bruce Fields
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).