* Re: 2.4.18: NFS_ALL patch greatly hurting UDP speed
[not found] ` <shs7kollatu.fsf@charged.uio.no>
@ 2002-03-20 15:39 ` I Lee Hetherington
2002-03-20 15:45 ` Trond Myklebust
0 siblings, 1 reply; 6+ messages in thread
From: I Lee Hetherington @ 2002-03-20 15:39 UTC (permalink / raw)
To: Trond Myklebust; +Cc: nfs
Did you ever get a chance to look at the tcpdumps I sent you?
Given that UDP is now *100* times slower with NFS_ALL vs. without,
something is fishy. We have always gotten good performance with UDP in
our network, even with the 1Gb/s to 100Mb/s switching. Maybe our
switches have big enough buffers to deal with the peaks.
--Lee Hetherington
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 2.4.18: NFS_ALL patch greatly hurting UDP speed
2002-03-20 15:39 ` 2.4.18: NFS_ALL patch greatly hurting UDP speed I Lee Hetherington
@ 2002-03-20 15:45 ` Trond Myklebust
2002-03-20 15:50 ` Trond Myklebust
0 siblings, 1 reply; 6+ messages in thread
From: Trond Myklebust @ 2002-03-20 15:45 UTC (permalink / raw)
To: I Lee Hetherington; +Cc: nfs
>>>>> " " == I Lee Hetherington <I> writes:
> Did you ever get a chance to look at the tcpdumps I sent you?
> Given that UDP is now *100* times slower with NFS_ALL
> vs. without, something is fishy. We have always gotten good
> performance with UDP in our network, even with the 1Gb/s to
> 100Mb/s switching. Maybe our switches have big enough buffers
> to deal with the peaks.
> --Lee Hetherington
Yes. I already sent you a breakdown based on those tcpdumps. Here it is
again.
Cheers,
Trond
----
Interesting... On the 2.2.18 vanilla, the system is settling in a routine
where the client only sends 1 request at a time, and then waits for the
answer.
On 2.2.18 w/ NFS_ALL, the client attempts to send off 6 requests, but only
gets answered *very* slowly (one request at a time). Eventually (line number
113 in the tcpdump sequence) the replies start to miss fragments, and this is
where the problem starts...
My guess is that while you are able to send off 1 request at a time, the GigE
-> 100Mbit bridge is able to cache the replies. As the the number of
'simultaneous' replies being sent off by the server increases, though, the
bridge is no longer able to cache enough, and so it starts to truncate the
messages (and the NFS read requests have to time out and retry).
Another sign of this is the fact that in the NFS_ALL tcpdump, there are
assorted 'Time-to-live exceeded' ICMP messages littered around the place. The
first one comes just after the loss of fragments, and is accompanied by a 2
second delay, during which all the reads that are sent time out without
receiving a single reply...
Basically, as I said before, you are in a situation where TCP should always
be able to outperform UDP. I'm a bit surprised that this is not the case for
'vanilla' 2.2.18.
Cheers,
Trond
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 2.4.18: NFS_ALL patch greatly hurting UDP speed
2002-03-20 15:45 ` Trond Myklebust
@ 2002-03-20 15:50 ` Trond Myklebust
2002-03-20 18:23 ` Eric Whiting
0 siblings, 1 reply; 6+ messages in thread
From: Trond Myklebust @ 2002-03-20 15:50 UTC (permalink / raw)
To: I Lee Hetherington; +Cc: nfs
>>>>> " " == Trond Myklebust <trond.myklebust@fys.uio.no> writes:
> exceeded' ICMP messages littered around the place. The first
> one comes just after the loss of fragments, and is accompanied
> by a 2 second delay, during which all the reads that are sent
> time out without receiving a single reply...
Note: this 2 second period of silence appears to be what is really
causing the *100 slowdown. I've no idea what the switch is engaging in
during that time, but you might want to take a look to see if those
messages being sent during that period are indeed being received on
the server.
Cheers,
Trond
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 2.4.18: NFS_ALL patch greatly hurting UDP speed
2002-03-20 15:50 ` Trond Myklebust
@ 2002-03-20 18:23 ` Eric Whiting
2002-03-21 0:20 ` Ion Badulescu
0 siblings, 1 reply; 6+ messages in thread
From: Eric Whiting @ 2002-03-20 18:23 UTC (permalink / raw)
To: trond.myklebust; +Cc: I Lee Hetherington, nfs
Trond,
Some more data from some 2.4.18 tests I just ran...
==FIRST TEST
CLIENT: CLEAN 2.4.18 kernel with NFSV3 enabled
SERVER: Solaris 8 server (E280R)
100M switched network (1G backbone connecting these two boxes -- but
results the same when staying on local switch)
Writing with putc()... done: 5941 kB/s 59.3 %CPU
Rewriting... done: 5254 kB/s 6.7 %CPU
Writing intelligently... done: 5605 kB/s 4.1 %CPU
Reading with getc()... done: 9028 kB/s 89.3 %CPU
Reading intelligently... done: 197558 kB/s 100.3 %CPU
==SECOND TEST
CLIENT: Same setup as the first but with linux-2.4.18-NFS_ALL.dif
applied. Same .config file.
SERVER: Same
Writing with putc()... done: 5977 kB/s 63.5 %CPU
Rewriting... done: 3494 kB/s 5.7 %CPU
Writing intelligently... done: 734 kB/s 0.7 %CPU
Reading with getc()... done: 9255 kB/s 72.8 %CPU
Reading intelligently... done: 195361 kB/s 34.3 %CPU
PROBLEM: the intelligent writes look bad.
==THIRD TEST
Same setup as second test, but using dd instead of bonnie
mohawk/test> time dd if=/dev/zero of=file bs=1024k count=1
1+0 records in
1+0 records out
0.000u 0.030s 0:05.06 0.5% 0+0k 0+0io 136pf+0w
About 200k/s on a 100Mbit network -- not very good.
I grabbed a simple tcpdump of that whole dd session. I'll send a
off-list copy of the dump to Trond.
eric
Trond Myklebust wrote:
>
> >>>>> " " == Trond Myklebust <trond.myklebust@fys.uio.no> writes:
>
> > exceeded' ICMP messages littered around the place. The first
> > one comes just after the loss of fragments, and is accompanied
> > by a 2 second delay, during which all the reads that are sent
> > time out without receiving a single reply...
>
> Note: this 2 second period of silence appears to be what is really
> causing the *100 slowdown. I've no idea what the switch is engaging in
> during that time, but you might want to take a look to see if those
> messages being sent during that period are indeed being received on
> the server.
>
> Cheers,
> Trond
>
> _______________________________________________
> NFS maillist - NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 2.4.18: NFS_ALL patch greatly hurting UDP speed
2002-03-20 18:23 ` Eric Whiting
@ 2002-03-21 0:20 ` Ion Badulescu
2002-03-21 0:33 ` Eric Whiting
0 siblings, 1 reply; 6+ messages in thread
From: Ion Badulescu @ 2002-03-21 0:20 UTC (permalink / raw)
To: Eric Whiting; +Cc: nfs
On Wed, 20 Mar 2002 11:23:43 -0700, Eric Whiting <ewhiting@amis.com> wrote:
> ==THIRD TEST
> Same setup as second test, but using dd instead of bonnie
>
> mohawk/test> time dd if=/dev/zero of=file bs=1024k count=1
> 1+0 records in
> 1+0 records out
> 0.000u 0.030s 0:05.06 0.5% 0+0k 0+0io 136pf+0w
>
> About 200k/s on a 100Mbit network -- not very good.
Umm... no. I don't recall your setup (and you didn't specify it
in your email), but if this is the same setup as Lee's (GigE
bridged into 100Mbit), then you don't really have a 100Mbit network.
What you have is a GigE network with 90% packet loss. 200k/s is
actually pretty good in that case.
This is what Trond has been trying to say all along: UDP is a losing
proposition for this kind of setup. The only way you can make it work
sort of ok is by slowing it down and thus penalizing the rest of us
with sane setups.
Just use TCP and be happy.
Ion
--
It is better to keep your mouth shut and be thought a fool,
than to open it and remove all doubt.
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 2.4.18: NFS_ALL patch greatly hurting UDP speed
2002-03-21 0:20 ` Ion Badulescu
@ 2002-03-21 0:33 ` Eric Whiting
0 siblings, 0 replies; 6+ messages in thread
From: Eric Whiting @ 2002-03-21 0:33 UTC (permalink / raw)
To: Ion Badulescu; +Cc: nfs
Ion Badulescu wrote:
>
> On Wed, 20 Mar 2002 11:23:43 -0700, Eric Whiting <ewhiting@amis.com> wrote:
>
> > ==THIRD TEST
> > Same setup as second test, but using dd instead of bonnie
> >
> > mohawk/test> time dd if=/dev/zero of=file bs=1024k count=1
> > 1+0 records in
> > 1+0 records out
> > 0.000u 0.030s 0:05.06 0.5% 0+0k 0+0io 136pf+0w
> >
> > About 200k/s on a 100Mbit network -- not very good.
>
> Umm... no. I don't recall your setup (and you didn't specify it
> in your email), but if this is the same setup as Lee's (GigE
> bridged into 100Mbit), then you don't really have a 100Mbit network.
> What you have is a GigE network with 90% packet loss. 200k/s is
> actually pretty good in that case.
The only variable I change is the kernel on the linux client. The
network is 100M at both the server and the client. (gigE inbetween --
but I've tested with suns on the local switch and found the same
results)
The default kernel runs 5-10M/s on writes. Reboot to a 2.4.18 with
NFS_ALL and you see 200K/s on writes. Same network. Same UDP transport.
Just something goofy going on.
>
> This is what Trond has been trying to say all along: UDP is a losing
> proposition for this kind of setup. The only way you can make it work
> sort of ok is by slowing it down and thus penalizing the rest of us
> with sane setups.
>
> Just use TCP and be happy.
Yes.. That is what Sun does. I'm starting to do that more and more. I've
been using Neil's knfsd and tcp patches on some boxes. (but I'm not
using those patches on these recent tests that I've been reporting)
eric
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2002-03-21 0:34 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <3C88F8EE.86058BD5@sls.lcs.mit.edu>
[not found] ` <shs7kollatu.fsf@charged.uio.no>
2002-03-20 15:39 ` 2.4.18: NFS_ALL patch greatly hurting UDP speed I Lee Hetherington
2002-03-20 15:45 ` Trond Myklebust
2002-03-20 15:50 ` Trond Myklebust
2002-03-20 18:23 ` Eric Whiting
2002-03-21 0:20 ` Ion Badulescu
2002-03-21 0:33 ` Eric Whiting
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.