* Re: 2.4.18: NFS_ALL patch greatly hurting UDP speed [not found] ` <shs7kollatu.fsf@charged.uio.no> @ 2002-03-20 15:39 ` I Lee Hetherington 2002-03-20 15:45 ` Trond Myklebust 0 siblings, 1 reply; 6+ messages in thread From: I Lee Hetherington @ 2002-03-20 15:39 UTC (permalink / raw) To: Trond Myklebust; +Cc: nfs Did you ever get a chance to look at the tcpdumps I sent you? Given that UDP is now *100* times slower with NFS_ALL vs. without, something is fishy. We have always gotten good performance with UDP in our network, even with the 1Gb/s to 100Mb/s switching. Maybe our switches have big enough buffers to deal with the peaks. --Lee Hetherington _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 2.4.18: NFS_ALL patch greatly hurting UDP speed 2002-03-20 15:39 ` 2.4.18: NFS_ALL patch greatly hurting UDP speed I Lee Hetherington @ 2002-03-20 15:45 ` Trond Myklebust 2002-03-20 15:50 ` Trond Myklebust 0 siblings, 1 reply; 6+ messages in thread From: Trond Myklebust @ 2002-03-20 15:45 UTC (permalink / raw) To: I Lee Hetherington; +Cc: nfs >>>>> " " == I Lee Hetherington <I> writes: > Did you ever get a chance to look at the tcpdumps I sent you? > Given that UDP is now *100* times slower with NFS_ALL > vs. without, something is fishy. We have always gotten good > performance with UDP in our network, even with the 1Gb/s to > 100Mb/s switching. Maybe our switches have big enough buffers > to deal with the peaks. > --Lee Hetherington Yes. I already sent you a breakdown based on those tcpdumps. Here it is again. Cheers, Trond ---- Interesting... On the 2.2.18 vanilla, the system is settling in a routine where the client only sends 1 request at a time, and then waits for the answer. On 2.2.18 w/ NFS_ALL, the client attempts to send off 6 requests, but only gets answered *very* slowly (one request at a time). Eventually (line number 113 in the tcpdump sequence) the replies start to miss fragments, and this is where the problem starts... My guess is that while you are able to send off 1 request at a time, the GigE -> 100Mbit bridge is able to cache the replies. As the the number of 'simultaneous' replies being sent off by the server increases, though, the bridge is no longer able to cache enough, and so it starts to truncate the messages (and the NFS read requests have to time out and retry). Another sign of this is the fact that in the NFS_ALL tcpdump, there are assorted 'Time-to-live exceeded' ICMP messages littered around the place. The first one comes just after the loss of fragments, and is accompanied by a 2 second delay, during which all the reads that are sent time out without receiving a single reply... Basically, as I said before, you are in a situation where TCP should always be able to outperform UDP. I'm a bit surprised that this is not the case for 'vanilla' 2.2.18. Cheers, Trond _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 2.4.18: NFS_ALL patch greatly hurting UDP speed 2002-03-20 15:45 ` Trond Myklebust @ 2002-03-20 15:50 ` Trond Myklebust 2002-03-20 18:23 ` Eric Whiting 0 siblings, 1 reply; 6+ messages in thread From: Trond Myklebust @ 2002-03-20 15:50 UTC (permalink / raw) To: I Lee Hetherington; +Cc: nfs >>>>> " " == Trond Myklebust <trond.myklebust@fys.uio.no> writes: > exceeded' ICMP messages littered around the place. The first > one comes just after the loss of fragments, and is accompanied > by a 2 second delay, during which all the reads that are sent > time out without receiving a single reply... Note: this 2 second period of silence appears to be what is really causing the *100 slowdown. I've no idea what the switch is engaging in during that time, but you might want to take a look to see if those messages being sent during that period are indeed being received on the server. Cheers, Trond _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 2.4.18: NFS_ALL patch greatly hurting UDP speed 2002-03-20 15:50 ` Trond Myklebust @ 2002-03-20 18:23 ` Eric Whiting 2002-03-21 0:20 ` Ion Badulescu 0 siblings, 1 reply; 6+ messages in thread From: Eric Whiting @ 2002-03-20 18:23 UTC (permalink / raw) To: trond.myklebust; +Cc: I Lee Hetherington, nfs Trond, Some more data from some 2.4.18 tests I just ran... ==FIRST TEST CLIENT: CLEAN 2.4.18 kernel with NFSV3 enabled SERVER: Solaris 8 server (E280R) 100M switched network (1G backbone connecting these two boxes -- but results the same when staying on local switch) Writing with putc()... done: 5941 kB/s 59.3 %CPU Rewriting... done: 5254 kB/s 6.7 %CPU Writing intelligently... done: 5605 kB/s 4.1 %CPU Reading with getc()... done: 9028 kB/s 89.3 %CPU Reading intelligently... done: 197558 kB/s 100.3 %CPU ==SECOND TEST CLIENT: Same setup as the first but with linux-2.4.18-NFS_ALL.dif applied. Same .config file. SERVER: Same Writing with putc()... done: 5977 kB/s 63.5 %CPU Rewriting... done: 3494 kB/s 5.7 %CPU Writing intelligently... done: 734 kB/s 0.7 %CPU Reading with getc()... done: 9255 kB/s 72.8 %CPU Reading intelligently... done: 195361 kB/s 34.3 %CPU PROBLEM: the intelligent writes look bad. ==THIRD TEST Same setup as second test, but using dd instead of bonnie mohawk/test> time dd if=/dev/zero of=file bs=1024k count=1 1+0 records in 1+0 records out 0.000u 0.030s 0:05.06 0.5% 0+0k 0+0io 136pf+0w About 200k/s on a 100Mbit network -- not very good. I grabbed a simple tcpdump of that whole dd session. I'll send a off-list copy of the dump to Trond. eric Trond Myklebust wrote: > > >>>>> " " == Trond Myklebust <trond.myklebust@fys.uio.no> writes: > > > exceeded' ICMP messages littered around the place. The first > > one comes just after the loss of fragments, and is accompanied > > by a 2 second delay, during which all the reads that are sent > > time out without receiving a single reply... > > Note: this 2 second period of silence appears to be what is really > causing the *100 slowdown. I've no idea what the switch is engaging in > during that time, but you might want to take a look to see if those > messages being sent during that period are indeed being received on > the server. > > Cheers, > Trond > > _______________________________________________ > NFS maillist - NFS@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 2.4.18: NFS_ALL patch greatly hurting UDP speed 2002-03-20 18:23 ` Eric Whiting @ 2002-03-21 0:20 ` Ion Badulescu 2002-03-21 0:33 ` Eric Whiting 0 siblings, 1 reply; 6+ messages in thread From: Ion Badulescu @ 2002-03-21 0:20 UTC (permalink / raw) To: Eric Whiting; +Cc: nfs On Wed, 20 Mar 2002 11:23:43 -0700, Eric Whiting <ewhiting@amis.com> wrote: > ==THIRD TEST > Same setup as second test, but using dd instead of bonnie > > mohawk/test> time dd if=/dev/zero of=file bs=1024k count=1 > 1+0 records in > 1+0 records out > 0.000u 0.030s 0:05.06 0.5% 0+0k 0+0io 136pf+0w > > About 200k/s on a 100Mbit network -- not very good. Umm... no. I don't recall your setup (and you didn't specify it in your email), but if this is the same setup as Lee's (GigE bridged into 100Mbit), then you don't really have a 100Mbit network. What you have is a GigE network with 90% packet loss. 200k/s is actually pretty good in that case. This is what Trond has been trying to say all along: UDP is a losing proposition for this kind of setup. The only way you can make it work sort of ok is by slowing it down and thus penalizing the rest of us with sane setups. Just use TCP and be happy. Ion -- It is better to keep your mouth shut and be thought a fool, than to open it and remove all doubt. _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 2.4.18: NFS_ALL patch greatly hurting UDP speed 2002-03-21 0:20 ` Ion Badulescu @ 2002-03-21 0:33 ` Eric Whiting 0 siblings, 0 replies; 6+ messages in thread From: Eric Whiting @ 2002-03-21 0:33 UTC (permalink / raw) To: Ion Badulescu; +Cc: nfs Ion Badulescu wrote: > > On Wed, 20 Mar 2002 11:23:43 -0700, Eric Whiting <ewhiting@amis.com> wrote: > > > ==THIRD TEST > > Same setup as second test, but using dd instead of bonnie > > > > mohawk/test> time dd if=/dev/zero of=file bs=1024k count=1 > > 1+0 records in > > 1+0 records out > > 0.000u 0.030s 0:05.06 0.5% 0+0k 0+0io 136pf+0w > > > > About 200k/s on a 100Mbit network -- not very good. > > Umm... no. I don't recall your setup (and you didn't specify it > in your email), but if this is the same setup as Lee's (GigE > bridged into 100Mbit), then you don't really have a 100Mbit network. > What you have is a GigE network with 90% packet loss. 200k/s is > actually pretty good in that case. The only variable I change is the kernel on the linux client. The network is 100M at both the server and the client. (gigE inbetween -- but I've tested with suns on the local switch and found the same results) The default kernel runs 5-10M/s on writes. Reboot to a 2.4.18 with NFS_ALL and you see 200K/s on writes. Same network. Same UDP transport. Just something goofy going on. > > This is what Trond has been trying to say all along: UDP is a losing > proposition for this kind of setup. The only way you can make it work > sort of ok is by slowing it down and thus penalizing the rest of us > with sane setups. > > Just use TCP and be happy. Yes.. That is what Sun does. I'm starting to do that more and more. I've been using Neil's knfsd and tcp patches on some boxes. (but I'm not using those patches on these recent tests that I've been reporting) eric _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2002-03-21 0:34 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <3C88F8EE.86058BD5@sls.lcs.mit.edu>
[not found] ` <shs7kollatu.fsf@charged.uio.no>
2002-03-20 15:39 ` 2.4.18: NFS_ALL patch greatly hurting UDP speed I Lee Hetherington
2002-03-20 15:45 ` Trond Myklebust
2002-03-20 15:50 ` Trond Myklebust
2002-03-20 18:23 ` Eric Whiting
2002-03-21 0:20 ` Ion Badulescu
2002-03-21 0:33 ` Eric Whiting
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox