Linux NFS development
 help / color / mirror / Atom feed
From: Bixuan Cui <cuibixuan@linux.alibaba.com>
To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org,
	linux-nfs@vger.kernel.org
Cc: cuibixuan@linux.alibaba.com, bfields@fieldses.org,
	chuck.lever@oracle.com, trond.myklebust@hammerspace.com,
	anna.schumaker@netapp.com, davem@davemloft.net, kuba@kernel.org,
	pete.wl@alibaba-inc.com, wenan.mwa@alibaba-inc.com,
	xiaoh.peixh@alibaba-inc.com, weipu.zy@alibaba-inc.com
Subject: [PATCH -next] SUNRPC: Clean XPRT_CONGESTED of xprt->state when rpc task is killed
Date: Tue, 14 Dec 2021 21:53:37 +0800	[thread overview]
Message-ID: <1639490018-128451-1-git-send-email-cuibixuan@linux.alibaba.com> (raw)

There is a problem with nfs hang on my machine, Steps to reproduce:

1.
echo "options sunrpc tcp_slot_table_entries=2" >> /etc/modprobe.d/sunrpc.conf
echo "options sunrpc tcp_max_slot_table_entries=2" >>  /etc/modprobe.d/sunrpc.conf
and remount nfsv3: sudo mount -t nfs -o vers=3,nolock,proto=tcp 127.0.0.1:/root/nfs_server/ /root/nfs_client/

2.Run case
# cat case.sh 
while true
do
    ls &
done &

while true
do
    killall ls
done &

[root@cuibixuan nfs_client]# cd /root/nfs_client/; sh case.sh

3. Run 'killall sh' to kill case after waiting 10 seconds

4. The nfs hang and xprt->state is 0x212(XPRT_CONGESTED):
[root@cuibixuan sunrpc]# cat /sys/kernel/debug/sunrpc/rpc_xprt/4/info 
netid: tcp
addr:  127.0.0.1
port:  2049
state: 0x212

[root@cuibixuan sunrpc]# cat /sys/kernel/debug/sunrpc/rpc_clnt/4/tasks 
  155 0080    -11 0x4 0x0        0 rpc_default_ops [sunrpc] nfsv3 GETATTR a:call_reserveresult [sunrpc] q:xprt_backlog
15418 0080    -11 0x4 0x0        0 rpc_default_ops [sunrpc] nfsv3 GETATTR a:call_reserveresult [sunrpc] q:xprt_backlog

5. Enable ftrace
 # echo 1 > /sys/kernel/debug/tracing/events/sunrpc/enable
 # cat /sys/kernel/debug/tracing/trace
	      ...
              ls-18792 [001] ....   668.541062: rpc_task_run_action: task:28885@4 flags=0080 state=0005 status=0 action=call_start [sunrpc]
              ls-18792 [001] ....   668.541063: rpc_request: task:28885@4 nfsv3 GETATTR (sync)
              ls-18792 [001] ....   668.541063: rpc_task_run_action: task:28885@4 flags=0080 state=0005 status=0 action=call_reserve [sunrpc]
              ls-18792 [001] ....   668.541065: rpc_task_sleep: task:28885@4 flags=0080 state=0005 status=-11 timeout=0 queue=xprt_backlog
              ls-18792 [000] ....   670.815066: rpc_task_wakeup: task:28885@4 flags=0180 state=0006 status=-512 timeout=0 queue=xprt_backlog
              ls-18792 [000] ....   670.815069: rpc_task_run_action: task:28885@4 flags=0180 state=0005 status=-512 action=rpc_exit_task [sunrpc]
              ls-18846 [001] ....   682.302970: rpc_task_begin: task:28886@4 flags=0080 state=0004 status=0 action=          (null)
              ls-18846 [001] ....   682.302972: rpc_task_run_action: task:28886@4 flags=0080 state=0005 status=0 action=call_start [sunrpc]
              ls-18846 [001] ....   682.302973: rpc_request: task:28886@4 nfsv3 GETATTR (sync)
              ls-18846 [001] ....   682.302973: rpc_task_run_action: task:28886@4 flags=0080 state=0005 status=0 action=call_reserve [sunrpc]
              ls-18846 [001] ....   682.302974: rpc_task_sleep: task:28886@4 flags=0080 state=0005 status=-11 timeout=0 queue=xprt_backlog
              ls-18846 [000] ....   684.690924: rpc_task_wakeup: task:28886@4 flags=0180 state=0006 status=-512 timeout=0 queue=xprt_backlog
              ls-18846 [000] ....   684.690928: rpc_task_run_action: task:28886@4 flags=0180 state=0005 status=-512 action=rpc_exit_task [sunrpc]

The kernel version of my machine is 4.19.91, and it can be fixed with
the following patch (in v4.19):

diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index 793f595..2656048 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -1358,6 +1358,7 @@ void xprt_release(struct rpc_task *task)
        if (req == NULL) {
                if (task->tk_client) {
                        xprt = task->tk_xprt;
+                       xprt_wake_up_backlog(xprt);
                        if (xprt->snd_task == task)
                                xprt_release_write(xprt, task);
                }

I checked the -next code, there should be the same problem.

Bixuan Cui (1):
  SUNRPC: Clean XPRT_CONGESTED of xprt->state when rpc task is killed

 net/sunrpc/xprt.c | 1 +
 1 file changed, 1 insertion(+)

-- 
1.8.3.1


             reply	other threads:[~2021-12-14 13:53 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-12-14 13:53 Bixuan Cui [this message]
2021-12-14 13:53 ` [PATCH -next] SUNRPC: Clean XPRT_CONGESTED of xprt->state when rpc task is killed Bixuan Cui
2021-12-20  3:39   ` Bixuan Cui
2021-12-20 18:22     ` Trond Myklebust
     [not found]       ` <efbf73f3-c6cd-90f6-ef22-bde14be708cc@linux.alibaba.com>
2021-12-22 15:02         ` Trond Myklebust
2021-12-31  2:00           ` Bixuan Cui

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1639490018-128451-1-git-send-email-cuibixuan@linux.alibaba.com \
    --to=cuibixuan@linux.alibaba.com \
    --cc=anna.schumaker@netapp.com \
    --cc=bfields@fieldses.org \
    --cc=chuck.lever@oracle.com \
    --cc=davem@davemloft.net \
    --cc=kuba@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-nfs@vger.kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=pete.wl@alibaba-inc.com \
    --cc=trond.myklebust@hammerspace.com \
    --cc=weipu.zy@alibaba-inc.com \
    --cc=wenan.mwa@alibaba-inc.com \
    --cc=xiaoh.peixh@alibaba-inc.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox