From: Nikolaus Rath <Nikolaus@rath.org>
To: "J. Bruce Fields" <bfields@fieldses.org>
Cc: linux-nfs@vger.kernel.org
Subject: Re: NFS4 over VPN hangs when connecting > 2 clients
Date: Mon, 12 Mar 2012 15:45:05 -0400 [thread overview]
Message-ID: <4F5E5241.7070008@rath.org> (raw)
In-Reply-To: <20120312193115.GA7203@fieldses.org>
On 03/12/2012 03:31 PM, J. Bruce Fields wrote:
> On Mon, Mar 12, 2012 at 12:20:17PM -0400, Nikolaus Rath wrote:
>> Nikolaus Rath <Nikolaus-BTH8mxji4b0@public.gmane.org> writes:
>>> The problem is that as soon as more than three clients are accessing the
>>> NFS shares, any operations on the NFS mountpoints by the clients hang.
>>> At the same time, CPU usage of the VPN processes becomes very high. If I
>>> run the VPN in debug mode, all I can see is that it is busy forwarding
>>> lots of packets. I also ran a packet sniffer which showed me that 90% of
>>> the packets were NFS related, but I am not familiar enough with NFS to
>>> be able to tell anything from the packets themselves. I can provide an
>>> example of the dump if that helps.
>>
>> I have put a screenshot of the dump on
>> http://www.rath.org/res/wireshark.png (the full dump is 18 MB, and I'm
>> not sure which parts are important).
>
> Looks like they're doing SETCLIENTID, SETCLIENTID_CONFIRM, OPEN,
> OPEN_CONFIRM repeatedly.
>
>> Any suggestions how I could further debug this?
>
> Could the clients be stepping on each others' state if they all think
> they have the same IP address (because of something to do with the VPN
> networking?)
That sounds like promising path of investigation. What determines the IP
of a client as far as NFS is concerned? ifconfig on the clients reports
different IPs, e.g.
hbt Link encap:UNSPEC HWaddr
00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:192.168.1.20 P-t-P:192.168.1.20 Mask:255.255.255.0
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1
RX packets:6285 errors:0 dropped:0 overruns:0 frame:0
TX packets:6851 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:2749422 (2.7 MB) TX bytes:1860654 (1.8 MB)
hbt Link encap:UNSPEC HWaddr
00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:192.168.1.19 P-t-P:192.168.1.19 Mask:255.255.255.0
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1
RX packets:2306277 errors:0 dropped:0 overruns:0 frame:0
TX packets:1761902 errors:0 dropped:0 overruns:33054 carrier:0
collisions:0 txqueuelen:500
RX bytes:2534582154 (2.5 GB) TX bytes:888058175 (888.0 MB)
> It'd be interesting to know the fields of the setclientid call, and the
> errors that the server is responding with to these calls. If you look
> at the packet details you'll probably see the same thing happening
> over and over again.
>
> Filtering to look at traffic between server and one client at a time
> might help to see the pattern.
Hmm. I'm looking at the fields, but I just have no idea what any of
those mean. Would you possibly be willing to take a look? I uploaded a
pcap dump of a few packets to http://www.rath.org/res/sample.pcap.
Best,
-Nikolaus
--
»Time flies like an arrow, fruit flies like a Banana.«
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C
next prev parent reply other threads:[~2012-03-12 19:45 UTC|newest]
Thread overview: 46+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-03-11 1:34 NFS4 over VPN hangs when connecting > 2 clients Nikolaus Rath
2012-03-12 16:20 ` Nikolaus Rath
2012-03-12 19:31 ` J. Bruce Fields
2012-03-12 19:45 ` Nikolaus Rath [this message]
2012-03-12 20:15 ` J. Bruce Fields
2012-03-12 20:30 ` Nikolaus Rath
2012-03-12 20:42 ` J. Bruce Fields
2012-03-12 20:49 ` Chuck Lever
2012-03-12 21:04 ` J. Bruce Fields
2012-03-12 21:14 ` Chuck Lever
2012-03-12 21:27 ` J. Bruce Fields
2012-03-19 16:28 ` J. Bruce Fields
2012-03-19 16:44 ` [nfsv4] " Rick Macklem
2012-03-19 17:06 ` Rick Macklem
2012-03-19 17:36 ` J. Bruce Fields
2012-03-19 17:47 ` Chuck Lever
2012-03-19 18:24 ` Myklebust, Trond
2012-03-19 18:27 ` J. Bruce Fields
2012-03-19 18:29 ` Chuck Lever
2012-03-19 18:39 ` J. Bruce Fields
2012-03-19 18:42 ` Chuck Lever
2012-03-19 18:54 ` J. Bruce Fields
2012-03-19 19:00 ` Chuck Lever
2012-03-19 19:08 ` J. Bruce Fields
2012-03-19 18:43 ` Nikolaus Rath
2012-03-19 22:25 ` Rick Macklem
2012-03-20 13:29 ` Nikolaus Rath
2012-03-20 13:55 ` Myklebust, Trond
2012-03-20 14:36 ` Nikolaus Rath
2012-03-20 16:49 ` Myklebust, Trond
2012-03-20 14:01 ` Chuck Lever
2012-03-20 14:38 ` Nikolaus Rath
2012-03-20 15:53 ` Chuck Lever
2012-03-19 18:51 ` Nikolaus Rath
2012-03-19 18:56 ` J. Bruce Fields
2012-03-19 22:31 ` Rick Macklem
2012-03-19 18:26 ` Myklebust, Trond
2012-03-12 21:24 ` Nikolaus Rath
2012-03-12 21:27 ` Chuck Lever
2012-03-12 21:38 ` Nikolaus Rath
2012-03-12 21:46 ` Chuck Lever
2012-03-12 21:54 ` Chuck Lever
2012-03-12 21:54 ` Nikolaus Rath
2012-03-12 21:57 ` Myklebust, Trond
2012-03-13 13:23 ` Nikolaus Rath
2012-03-13 14:50 ` Myklebust, Trond
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4F5E5241.7070008@rath.org \
--to=nikolaus@rath.org \
--cc=bfields@fieldses.org \
--cc=linux-nfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox