* NFS performance issue - P3 vs P4
@ 2002-10-11 19:43 Andrei Maslennikov
2002-10-11 21:01 ` Trond Myklebust
0 siblings, 1 reply; 4+ messages in thread
From: Andrei Maslennikov @ 2002-10-11 19:43 UTC (permalink / raw)
To: nfs
We've ran into a funny problem with NFS performance and
since a couple of days are trying to overcome it. We
will be digging further in, but maybe somebody could give
us a valid advice..
NetApp F760 (NFS server 1)
/ | | \
/ | | \
18 MB/s 18 MB/s 18 MB/s 18 MB/s
| | | |
| | | |
Clients>> P3 P4 ALPHA IBM
| | | |
| | | |
34 MB/s 2 MB/s 22 MB/s 22 MB/s
\ | | /
\ | | /
Linux NFS Server on P3 (server 2)
As you may see here, we are looking at the performance of two
different NFS servers. One is a 3-year old commercial NAS system
(Network Appliance Filer F760), another is a P3 machine
(biprocessor Pentium III at 1 Ghz).
Then, we have 4 NFS clients (one ALPHA quadriprocessor at 600
Mhz, one IBM quadriprocessor at 375 Mhz, one P3 biprocessor
at 1 Ghz and one P4 biprocessor at 2 Ghz). Alpha client is
running Tru 64 5.1, IBM client - AIX 5.1D. All Pentiums
including the P3 server run exactly the same version of OS:
Red Hat Linux 7.3 with all the recent patches. On clients, we
use RH kernel 2.4.18-10smp. And on server - 2.4.18 from
kernel.org + Alan Cox patch 3 + EVMS suport patches for
volume handling and snapshotting.
All machines are sitting on the same switched non-blocking
Gigabit network.
What we observe, is *very* strange. NetApp is uniformly
delivering 18 MB/sec write speed to all 4 different clients.
P3 server does visibly better for ALPHA, IBM and P3 clients
(in case of P3 client, we observe almost a factor of two in
performance, 34 MB/s compared with the 18 MB/sec of NetApp).
However, when the same P3 server is used from P4 client, we
see an enormous drop in performance. We can write basically
only at 2 MB/s!
We first suspected the network was responsible for this
behaviour. In fact, the retrans rate on P4 as seen with
"nfsstat -r" between P4 client and P3 server was ridicolously
high. However, when we checked the P3srv-P4cli link with ttcp,
we have seen a perfect speed of memory-to-memory transfer of
more than 100 MB/s (both udp and tcp). Ftp disk-to-disk between
these two machines runs at more than 30 MB/s, exactly as one
would expect. Server runs with 64 nfsd's, and we played a lot
with timeo, rsize and wsize, nfsvers etc to arrive at most to
5 MB/s. Never we got more than that..
At this point, we have checked the cables: used on
P4 client the cable that was connecting before the
extra-performant P3 client. Exactly the same behaviour. We
then added another Gigabit card of different make to the P4
client - same lousy performance. We then jumped to yet
another P4 machine with exactly the same hw and sw
configuration, and obtained the same identical result...
The worst point about it all is the fact that the same P4 client
nicely operates at 18 MB/s when NetApp is used. If this would be
a P4 client issue, than its performance should be equally bad for
both P3 and NetApp servers. And if it would be a P3 server issue,
then why the latter is able to deliver a good performance to
ALPHA, IBM and P3 clients? And this is definitively *not* a
hardware problem, as we are able to reproduce this behaviour on
another equivalent hardware.
What else we could look at? If anybody has any idea, please let
us know.
Thanks ahead - Andrei.
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: NFS performance issue - P3 vs P4
2002-10-11 19:43 NFS performance issue - P3 vs P4 Andrei Maslennikov
@ 2002-10-11 21:01 ` Trond Myklebust
2002-10-11 22:42 ` Andrei Maslennikov
0 siblings, 1 reply; 4+ messages in thread
From: Trond Myklebust @ 2002-10-11 21:01 UTC (permalink / raw)
To: Andrei Maslennikov; +Cc: nfs
>>>>> " " == Andrei Maslennikov <andrei@caspur.it> writes:
> with all the recent patches. On clients, we use RH kernel
> 2.4.18-10smp.
Has a known UDP bug that can cause it to flood servers. Netapp over
Gigabit has previously been seen to be vulnerable. Please either use a
TCP connection to the Netapp, or try the appended patch.
Cheers,
Trond
# This is a BitKeeper generated patch for the following project:
# Project Name: Linux kernel tree
# This patch format is intended for GNU patch command version 2.5 or higher.
# This patch includes the following deltas:
# ChangeSet 1.555.13.1 -> 1.555.13.2
# net/ipv4/ip_output.c 1.10 -> 1.11
#
# The following is the BitKeeper ChangeSet Log
# --------------------------------------------
# 02/08/03 davem@nuts.ninka.net 1.555.13.2
# IPv4: Fix MSG_DONTWAIT behavior on output fragmentation.
# --------------------------------------------
#
diff -Nru a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
--- a/net/ipv4/ip_output.c Fri Oct 11 22:59:13 2002
+++ b/net/ipv4/ip_output.c Fri Oct 11 22:59:13 2002
@@ -520,8 +520,18 @@
/*
* Get the memory we require with some space left for alignment.
*/
-
- skb = sock_alloc_send_skb(sk, fraglen+hh_len+15, flags&MSG_DONTWAIT, &err);
+ if (!(flags & MSG_DONTWAIT) || nfrags == 0) {
+ skb = sock_alloc_send_skb(sk, fraglen + hh_len + 15,
+ (flags & MSG_DONTWAIT), &err);
+ } else {
+ /* On a non-blocking write, we check for send buffer
+ * usage on the first fragment only.
+ */
+ skb = sock_wmalloc(sk, fraglen + hh_len + 15, 1,
+ sk->allocation);
+ if (!skb)
+ err = -ENOBUFS;
+ }
if (skb == NULL)
goto error;
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: NFS performance issue - P3 vs P4
2002-10-11 21:01 ` Trond Myklebust
@ 2002-10-11 22:42 ` Andrei Maslennikov
2002-10-11 23:40 ` Trond Myklebust
0 siblings, 1 reply; 4+ messages in thread
From: Andrei Maslennikov @ 2002-10-11 22:42 UTC (permalink / raw)
To: Trond Myklebust; +Cc: nfs, g.palumbo
On 11 Oct 2002, Trond Myklebust wrote:
> >>>>> " " == Andrei Maslennikov <andrei@caspur.it> writes:
>
> > with all the recent patches. On clients, we use RH kernel
> > 2.4.18-10smp.
>
> Has a known UDP bug that can cause it to flood servers. Netapp over
> Gigabit has previously been seen to be vulnerable. Please either use a
> TCP connection to the Netapp, or try the appended patch.
Hi Trond, and thanks for the hint.
I have just checked that the patch you mentioned was applied
in the Red Hat's 2.4.18-14 kernel released with Red Hat 8.0
("linux-2.4.18-udp-fix.patch"). I have then upgraded to 2.4.18-14
on P4 client and - bingo: I see a stable 34 MB/sec as in case of P3!
Lots of thanks!
We'll also mount "tcp" from NetApp on older clients that still
may have that bug - thanks again for your suggestion. Actually,
I now understand that we have seen indeed several server flooding
cases with F760 on Gigabit (and we were looking for the problem
elsewhere...).
Best regards - Andrei.
PS One point still remains unclear - how then it could be explained that
P3 client was not manifesting any problem with P3 server? It had
exactly the buggy 2.4.18-10smp, like P4. Maybe it may be attributed
to a higher (2 Ghz against 1 Ghz of P3's) CPU clock on P4 client?
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: NFS performance issue - P3 vs P4
2002-10-11 22:42 ` Andrei Maslennikov
@ 2002-10-11 23:40 ` Trond Myklebust
0 siblings, 0 replies; 4+ messages in thread
From: Trond Myklebust @ 2002-10-11 23:40 UTC (permalink / raw)
To: Andrei Maslennikov; +Cc: nfs, g.palumbo
>>>>> " " == Andrei Maslennikov <andrei@caspur.it> writes:
> PS One point still remains unclear - how then it could be
> explained that
> P3 client was not manifesting any problem with P3 server? It
> had exactly the buggy 2.4.18-10smp, like P4. Maybe it may be
> attributed to a higher (2 Ghz against 1 Ghz of P3's) CPU
> clock on P4 client?
Yep: I would guess it would be a timing thing. The flooding only
occurs when the socket buffer is (almost) full. If the socket buffer
does not get filled (because the NIC is able to send off requests
faster than the P3 can generate them) then you are safe...
Cheers,
Trond
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2002-10-11 23:40 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2002-10-11 19:43 NFS performance issue - P3 vs P4 Andrei Maslennikov
2002-10-11 21:01 ` Trond Myklebust
2002-10-11 22:42 ` Andrei Maslennikov
2002-10-11 23:40 ` Trond Myklebust
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.