From: Nikolaus Rath <Nikolaus@rath.org>
To: "J. Bruce Fields" <bfields@fieldses.org>
Cc: linux-nfs@vger.kernel.org
Subject: Re: NFS4 over VPN hangs when connecting > 2 clients
Date: Mon, 12 Mar 2012 15:45:05 -0400 [thread overview]
Message-ID: <4F5E5241.7070008@rath.org> (raw)
In-Reply-To: <20120312193115.GA7203@fieldses.org>
On 03/12/2012 03:31 PM, J. Bruce Fields wrote:
> On Mon, Mar 12, 2012 at 12:20:17PM -0400, Nikolaus Rath wrote:
>> Nikolaus Rath <Nikolaus-BTH8mxji4b0@public.gmane.org> writes:
>>> The problem is that as soon as more than three clients are accessing the
>>> NFS shares, any operations on the NFS mountpoints by the clients hang.
>>> At the same time, CPU usage of the VPN processes becomes very high. If I
>>> run the VPN in debug mode, all I can see is that it is busy forwarding
>>> lots of packets. I also ran a packet sniffer which showed me that 90% of
>>> the packets were NFS related, but I am not familiar enough with NFS to
>>> be able to tell anything from the packets themselves. I can provide an
>>> example of the dump if that helps.
>>
>> I have put a screenshot of the dump on
>> http://www.rath.org/res/wireshark.png (the full dump is 18 MB, and I'm
>> not sure which parts are important).
>
> Looks like they're doing SETCLIENTID, SETCLIENTID_CONFIRM, OPEN,
> OPEN_CONFIRM repeatedly.
>
>> Any suggestions how I could further debug this?
>
> Could the clients be stepping on each others' state if they all think
> they have the same IP address (because of something to do with the VPN
> networking?)
That sounds like promising path of investigation. What determines the IP
of a client as far as NFS is concerned? ifconfig on the clients reports
different IPs, e.g.
hbt Link encap:UNSPEC HWaddr
00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:192.168.1.20 P-t-P:192.168.1.20 Mask:255.255.255.0
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1
RX packets:6285 errors:0 dropped:0 overruns:0 frame:0
TX packets:6851 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:2749422 (2.7 MB) TX bytes:1860654 (1.8 MB)
hbt Link encap:UNSPEC HWaddr
00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:192.168.1.19 P-t-P:192.168.1.19 Mask:255.255.255.0
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1
RX packets:2306277 errors:0 dropped:0 overruns:0 frame:0
TX packets:1761902 errors:0 dropped:0 overruns:33054 carrier:0
collisions:0 txqueuelen:500
RX bytes:2534582154 (2.5 GB) TX bytes:888058175 (888.0 MB)
> It'd be interesting to know the fields of the setclientid call, and the
> errors that the server is responding with to these calls. If you look
> at the packet details you'll probably see the same thing happening
> over and over again.
>
> Filtering to look at traffic between server and one client at a time
> might help to see the pattern.
Hmm. I'm looking at the fields, but I just have no idea what any of
those mean. Would you possibly be willing to take a look? I uploaded a
pcap dump of a few packets to http://www.rath.org/res/sample.pcap.
Best,
-Nikolaus
--
»Time flies like an arrow, fruit flies like a Banana.«
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C
next prev parent reply other threads:[~2012-03-12 19:45 UTC|newest]
Thread overview: 46+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-03-11 1:34 NFS4 over VPN hangs when connecting > 2 clients Nikolaus Rath
2012-03-12 16:20 ` Nikolaus Rath
2012-03-12 19:31 ` J. Bruce Fields
2012-03-12 19:45 ` Nikolaus Rath [this message]
2012-03-12 20:15 ` J. Bruce Fields
2012-03-12 20:30 ` Nikolaus Rath
2012-03-12 20:42 ` J. Bruce Fields
2012-03-12 20:49 ` Chuck Lever
2012-03-12 21:04 ` J. Bruce Fields
2012-03-12 21:14 ` Chuck Lever
2012-03-12 21:27 ` J. Bruce Fields
2012-03-19 16:28 ` J. Bruce Fields
2012-03-19 16:44 ` [nfsv4] " Rick Macklem
2012-03-19 17:06 ` Rick Macklem
2012-03-19 17:36 ` J. Bruce Fields
2012-03-19 17:47 ` Chuck Lever
2012-03-19 18:24 ` Myklebust, Trond
2012-03-19 18:27 ` J. Bruce Fields
2012-03-19 18:29 ` Chuck Lever
2012-03-19 18:39 ` J. Bruce Fields
2012-03-19 18:42 ` Chuck Lever
2012-03-19 18:54 ` J. Bruce Fields
2012-03-19 19:00 ` Chuck Lever
2012-03-19 19:08 ` J. Bruce Fields
2012-03-19 18:43 ` Nikolaus Rath
2012-03-19 22:25 ` Rick Macklem
2012-03-20 13:29 ` Nikolaus Rath
2012-03-20 13:55 ` Myklebust, Trond
2012-03-20 14:36 ` Nikolaus Rath
2012-03-20 16:49 ` Myklebust, Trond
2012-03-20 14:01 ` Chuck Lever
2012-03-20 14:38 ` Nikolaus Rath
2012-03-20 15:53 ` Chuck Lever
2012-03-19 18:51 ` Nikolaus Rath
2012-03-19 18:56 ` J. Bruce Fields
2012-03-19 22:31 ` Rick Macklem
2012-03-19 18:26 ` Myklebust, Trond
2012-03-12 21:24 ` Nikolaus Rath
2012-03-12 21:27 ` Chuck Lever
2012-03-12 21:38 ` Nikolaus Rath
2012-03-12 21:46 ` Chuck Lever
2012-03-12 21:54 ` Chuck Lever
2012-03-12 21:54 ` Nikolaus Rath
2012-03-12 21:57 ` Myklebust, Trond
2012-03-13 13:23 ` Nikolaus Rath
2012-03-13 14:50 ` Myklebust, Trond
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4F5E5241.7070008@rath.org \
--to=nikolaus@rath.org \
--cc=bfields@fieldses.org \
--cc=linux-nfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.