From: Simon Kirby <sim@hostway.ca>
To: Yan-Pai Chen <yanpai.chen@gmail.com>
Cc: linux-nfs@vger.kernel.org
Subject: Re: [3.2.5] NFSv3 CLOSE_WAIT hang
Date: Tue, 11 Sep 2012 12:40:51 -0700 [thread overview]
Message-ID: <20120911194051.GB11160@hostway.ca> (raw)
In-Reply-To: <loom.20120910T105944-416@post.gmane.org>
On Mon, Sep 10, 2012 at 09:00:37AM +0000, Yan-Pai Chen wrote:
> Hi Trond,
>
> Apologies for my late response.
> Upgrading to kernel 3.5 requires some effort. I am still working on it.
>
> After applying your patch on 3.3 kernel, the problem is gone when using UDP
> mounts.
> But it remains hang in the case of NFS over TCP mounts.
>
> I reproduced the problem by executing mm/mtest06_3 (i.e. mmap3) in the LTP test
> suite repeatedly.
> About less than 200 times, it eventually ran into the CLOSE_WAIT hang.
> I got the following messages after enabling rpc_debug & nfs_debug:
>
> 47991 0001 -11 cf2910e0 (null) 0 c0243f40 nfsv3 WRITE
> a:call_reserveresult q:xprt_sending
> 47992 0001 -11 cf2910e0 (null) 0 c0243f40 nfsv3 WRITE
> a:call_reserveresult q:xprt_sending
> 47993 0001 -11 cf2910e0 (null) 0 c0243f40 nfsv3 WRITE
> a:call_reserveresult q:xprt_sending
> 47994 0001 -11 cf2910e0 (null) 0 c0243f40 nfsv3 WRITE
> a:call_reserveresult q:xprt_sending
> 47995 0001 -11 cf2910e0 (null) 0 c0243f40 nfsv3 WRITE
> a:call_reserveresult q:xprt_sending
> ...
Hello!
This problem still bites us rarely, and we've been using TCP NFS for some
time. However, our case seems to be narrowed it down to a very long
storage hang on the knfsd side. If storage never has any problems, we
don't see the NFS client hang. I was going to try to make a test-case by
forcing the server to hang, but I never got around to this. Meanwhile,
I've been running the clients with the debugging patches I posted
earlier, and it always prints the 'xprt_force_disconnect(): setting
XPRT_CLOSE_WAIT" warning before hanging. If Apache is in sendfile() at
the time, it seems to get stuck forever; otherwise, it might recover.
http://www.spinics.net/lists/linux-nfs/msg29495.html
http://0x.ca/sim/ref/3.2.10/dmesg
I suppose we could try 3.5 at this point.
Simon-
next prev parent reply other threads:[~2012-09-11 19:40 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-09-09 19:45 [3.1-rc4] NFSv3 client hang Simon Kirby
2011-09-09 23:18 ` Trond Myklebust
2011-10-20 19:03 ` Simon Kirby
2012-03-01 22:55 ` Simon Kirby
2012-03-02 0:25 ` Simon Kirby
2012-03-02 18:49 ` [3.2.5] NFSv3 CLOSE_WAIT hang Simon Kirby
2012-09-05 7:49 ` Yan-Pai Chen
2012-09-05 15:09 ` Myklebust, Trond
2012-09-07 13:57 ` Dick Streefland, rnews
2012-09-07 14:13 ` Myklebust, Trond
2012-09-07 14:33 ` Dick Streefland, rnews
2012-09-07 15:46 ` Myklebust, Trond
2012-09-08 19:32 ` Dick Streefland, rnews
2012-09-10 9:00 ` Yan-Pai Chen
2012-09-11 19:40 ` Simon Kirby [this message]
2012-09-11 22:17 ` Myklebust, Trond
2012-09-13 5:22 ` Yan-Pai Chen
2012-09-13 13:32 ` Myklebust, Trond
2012-09-21 7:30 ` Yan-Pai Chen
[not found] ` <1347401844.15208.17.camel@lade.trondhjem.org>
2012-09-12 20:54 ` Myklebust, Trond
2012-09-19 22:01 ` Simon Kirby
2012-09-19 22:11 ` Myklebust, Trond
2012-10-12 8:15 ` Simon Kirby
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20120911194051.GB11160@hostway.ca \
--to=sim@hostway.ca \
--cc=linux-nfs@vger.kernel.org \
--cc=yanpai.chen@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).