RE: NFS Performance Between SGI Servers and Linux Clients

RE: NFS Performance Between SGI Servers and Linux Clients

[Lever, Charles]

> -----Original Message-----
> From: Trond Myklebust [mailto:trond.myklebust@fys.uio.no]
> Sent: Thursday, May 22, 2003 5:13 PM
> To: Iain Irwin-Powell
> Cc: nfs@lists.sourceforge.net
> Subject: Re: [NFS] NFS Performance Between SGI Servers and=20
> Linux Clients
>=20
>=20
> >>>>> " " =3D=3D Iain Irwin-Powell <iain@cinesite.co.uk> writes:
>=20
>      > Looking a bit deeper in a packet trace when the grep is
>      > running, Linux seems to issue 16 read requests (16KB) at a time
>      > whilst an SGI will only issue between 2 and 4. May or may not
>      > be relevant.
>=20
> ... and this is a problem because .... ???
>=20
> The Linux client tries to read ahead as much as it can and as quickly
> as it can. Why should it doing anything else?

well, this is one reason why NFS O_DIRECT is a desirable
feature.  no readahead.  certain applications don't need
it because the workload they generate is entirely random
small reads.  in these cases, read ahead wastes I/O
bandwidth and memory space with data that is never used
by an application.

trond, isn't there a read ahead value in the client that
can be tweaked?  he could trim down vm_max_readahead,=20
unless there is a max_readahead[] entry for anonymous
file systems.

either that will help iain, or prove that the problem
is not a result of read ahead.


-------------------------------------------------------
This SF.net email is sponsored by: ObjectStore.
If flattening out C++ or Java code to make your application fit in a
relational database is painful, don't do it! Check out ObjectStore.
Now part of Progress Software. http://www.objectstore.net/sourceforge
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: NFS Performance Between SGI Servers and Linux Clients

[Trond Myklebust]

>>>>> " " == Charles Lever <Lever> writes:

     > trond, isn't there a read ahead value in the client that can be
     > tweaked?  he could trim down vm_max_readahead, unless there is
     > a max_readahead[] entry for anonymous file systems.

That should partly help to trim it down unless the client is reading
from > 1 file at a time. In that case the RPC layer will still try to
issue more reads (up to MIN(16,vm_max_readahead) requests per process)
if the network and server permits it.

Beware, though, that vm_max_readahead will effect not only
NFS. Performance on other tasks may suffer.

Cheers,
  Trond


-------------------------------------------------------
This SF.net email is sponsored by: ObjectStore.
If flattening out C++ or Java code to make your application fit in a
relational database is painful, don't do it! Check out ObjectStore.
Now part of Progress Software. http://www.objectstore.net/sourceforge
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS Performance Between SGI Servers and Linux Clients

[Danny Smith]

Trond Myklebust wrote:

>>>>>>" " == Charles Lever <Lever> writes:
>>>>>>            
>>>>>>
>
>     > trond, isn't there a read ahead value in the client that can be
>     > tweaked?  he could trim down vm_max_readahead, unless there is
>     > a max_readahead[] entry for anonymous file systems.
>
>That should partly help to trim it down unless the client is reading
>from > 1 file at a time. In that case the RPC layer will still try to
>issue more reads (up to MIN(16,vm_max_readahead) requests per process)
>if the network and server permits it.
>
>Beware, though, that vm_max_readahead will effect not only
>NFS. Performance on other tasks may suffer.
>  
>
I will give this a try in Iain's absence. In the real world we *are* likely
to be reading multiple files, however for testing purposes we don't have to.

Danny

-- 
Danny Smith
Senior Systems Administrator, Cinesite (Europe) Ltd
020 7973 4000 - x4055    /    dannys@cinesite.co.uk




-------------------------------------------------------
This SF.net email is sponsored by: ObjectStore.
If flattening out C++ or Java code to make your application fit in a
relational database is painful, don't do it! Check out ObjectStore.
Now part of Progress Software. http://www.objectstore.net/sourceforge
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: NFS Performance Between SGI Servers and Linux Clients

[Iain Irwin-Powell]

Managed to get some quiet time to test the vm.max-readahead tweaking.

This definitely makes a difference to the load average on the SGI server
end. On this test I was only opening 1 file at a time.

Tuning vm.max-readahead down to 8 provided the best combination of
server load to client throughput, although the client throughput was
compromised by the down tuning (no real surprise there).

What I am waiting to get is a 'real world' situation so that I can tune
whilst I am seeing the problem to see if we can make it go away or at
least alleviate the symptoms.

The implication here (to me at least) is that the SGI is having problems
servicing all the requests from the Linux clients. I will feed this back
to SGI and see what they can come up with.

More when I know.

Iain



******************************************************************
Iain Irwin-Powell (AKA Iain Powell)
Senior Systems Administrator
Cinesite Europe Limited
9 Carlisle Street
London
W1D 3BP
 
T: +442079734000
DDI: +442079734053
*******************************************************************
It's not broken, it just doesn't work the way you expected.
*******************************************************************

> -----Original Message-----
> From: Danny Smith [mailto:dannys@cinesite.co.uk]
> Sent: 23 May 2003 13:12
> To: trond.myklebust@fys.uio.no
> Cc: Lever, Charles; Iain Irwin-Powell; nfs@lists.sourceforge.net
> Subject: Re: [NFS] NFS Performance Between SGI Servers and Linux
Clients
> 
> Trond Myklebust wrote:
> 
> >>>>>>" " == Charles Lever <Lever> writes:
> >>>>>>
> >>>>>>
> >
> >     > trond, isn't there a read ahead value in the client that can
be
> >     > tweaked?  he could trim down vm_max_readahead, unless there is
> >     > a max_readahead[] entry for anonymous file systems.
> >
> >That should partly help to trim it down unless the client is reading
> >from > 1 file at a time. In that case the RPC layer will still try to
> >issue more reads (up to MIN(16,vm_max_readahead) requests per
process)
> >if the network and server permits it.
> >
> >Beware, though, that vm_max_readahead will effect not only
> >NFS. Performance on other tasks may suffer.
> >
> >
> I will give this a try in Iain's absence. In the real world we *are*
> likely
> to be reading multiple files, however for testing purposes we don't
have
> to.
> 
> Danny
> 
> --
> Danny Smith
> Senior Systems Administrator, Cinesite (Europe) Ltd
> 020 7973 4000 - x4055    /    dannys@cinesite.co.uk
> 
> 




-------------------------------------------------------
This SF.net email is sponsored by: ObjectStore.
If flattening out C++ or Java code to make your application fit in a
relational database is painful, don't do it! Check out ObjectStore.
Now part of Progress Software. http://www.objectstore.net/sourceforge
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: NFS Performance Between SGI Servers and Linux Clients

[Kiernan, Michael]

Iain,

6.5.19 (and 6.5.17/18/20 for that matter) have a number of known
fundamental problems [we've seen some nasty client side problems,
though I've obviously not been involved in server side issues]

NFS patch (5021 or 5022 [don't ask me which one is for 6.5.19,=20
I'm not *that* into irix!] contains a number of important fixes.
What you're describing is possibly, or is closely related to the
buf_relse_age bug 886112. you might try the workaround online:
sgi-hd 2# systune buf_relse_age 10
        buf_relse_age =3D 0 (0x0)
        Do you really want to change buf_relse_age to 10 (0xa)? (y/n)  y
sgi-hd 3# systune buf_relse_age
        buf_relse_age =3D 10 (0xa)

- Mike

-----Original Message-----
From: Iain Irwin-Powell [mailto:iain@cinesite.co.uk]
Sent: 22 May 2003 18:32
To: nfs@lists.sourceforge.net
Subject: [NFS] NFS Performance Between SGI Servers and Linux Clients



Does anyone have any ideas about this?

We have been experiencing a number of issues regarding NFS serving on
SGI IRIX (6.5.19) servers which now appear to point back to Linux NFS
clients.

The server is an Origin 2000, quad processors, 1.5GB memory, serving
over Gigabit. This can be reproduced on connections whether or not they
are using jumbo frames. The disks that are served are hardware RAID3
capable of around 80-90MB/s (tested and optimised not theoretical).=20

Under certain circumstances will get a load average of 8 (or how ever
many nfsd's are running). The server itself is not that busy. Most of
the nfsd's show as sleeping but they are obviously waiting for
something.

Further investigation revealed that this situation arises when a Linux
client opens many files in quick succession (doing
open-read-close,open-read-close). We tend to use files in the range
3MB-13MB in size generally.

If we do the same test on an SGI client there is very little impact on
the load average.

We can recreate this by doing (on the client);

find . -type -f -print -exec grep jsdhgkhjfdk {} \;

The same effect appears to happen on Linux servers as well. A single
Linux client doing this will make the load average creep up towards 2.5.

Linux Version 2.4.20
nfs-utils-1.0.1-1
exports are exported with -32bitclients

Looking a bit deeper in a packet trace when the grep is running, Linux
seems to issue 16 read requests (16KB) at a time whilst an SGI will only
issue between 2 and 4. May or may not be relevant.

This information has also been passed back to SGI who are looking at it.

Packet traces and more detail is available if anyone wants them.

Any ideas or suggestions would be welcome.

******************************************************************
Iain Irwin-Powell (AKA Iain Powell)
Senior Systems Administrator
Cinesite Europe Limited
9 Carlisle Street
London
W1D 3BP
=20
T: +442079734000
DDI: +442079734053
*******************************************************************
It's not broken, it just doesn't work the way you expected.
*******************************************************************




-------------------------------------------------------
This SF.net email is sponsored by: ObjectStore.
If flattening out C++ or Java code to make your application fit in a
relational database is painful, don't do it! Check out ObjectStore.
Now part of Progress Software. http://www.objectstore.net/sourceforge
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs


-------------------------------------------------------
This SF.net email is sponsored by: ObjectStore.
If flattening out C++ or Java code to make your application fit in a
relational database is painful, don't do it! Check out ObjectStore.
Now part of Progress Software. http://www.objectstore.net/sourceforge
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS Performance Between SGI Servers and Linux Clients

[Danny Smith]

People,

We have patch 5067, which appears from the release notes to obsolete 5022.
We've also got the buf_relse_age systune in place, and haven't seen an
appreciable difference.

Not sure if this is relevant here or not, but we also get infrequent
messages like the one below:

May 23 10:50:57 animal16 kernel: nfs_stat_to_errno: bad nfs status 
return value: 11

My research in the list archives seems to indicate this is a pure server 
issue.
Can anyone explain it in more detail? It would help us greatly in 
feeding this problem
back to SGI.

Any help appreciated.

Danny

Kiernan, Michael wrote:

>Iain,
>
>6.5.19 (and 6.5.17/18/20 for that matter) have a number of known
>fundamental problems [we've seen some nasty client side problems,
>though I've obviously not been involved in server side issues]
>
>NFS patch (5021 or 5022 [don't ask me which one is for 6.5.19, 
>I'm not *that* into irix!] contains a number of important fixes.
>What you're describing is possibly, or is closely related to the
>buf_relse_age bug 886112. you might try the workaround online:
>sgi-hd 2# systune buf_relse_age 10
>        buf_relse_age = 0 (0x0)
>        Do you really want to change buf_relse_age to 10 (0xa)? (y/n)  y
>sgi-hd 3# systune buf_relse_age
>        buf_relse_age = 10 (0xa)
>
>- Mike
>
>-----Original Message-----
>From: Iain Irwin-Powell [mailto:iain@cinesite.co.uk]
>Sent: 22 May 2003 18:32
>To: nfs@lists.sourceforge.net
>Subject: [NFS] NFS Performance Between SGI Servers and Linux Clients
>
>
>
>Does anyone have any ideas about this?
>
>We have been experiencing a number of issues regarding NFS serving on
>SGI IRIX (6.5.19) servers which now appear to point back to Linux NFS
>clients.
>
>The server is an Origin 2000, quad processors, 1.5GB memory, serving
>over Gigabit. This can be reproduced on connections whether or not they
>are using jumbo frames. The disks that are served are hardware RAID3
>capable of around 80-90MB/s (tested and optimised not theoretical). 
>
>Under certain circumstances will get a load average of 8 (or how ever
>many nfsd's are running). The server itself is not that busy. Most of
>the nfsd's show as sleeping but they are obviously waiting for
>something.
>
>Further investigation revealed that this situation arises when a Linux
>client opens many files in quick succession (doing
>open-read-close,open-read-close). We tend to use files in the range
>3MB-13MB in size generally.
>
>If we do the same test on an SGI client there is very little impact on
>the load average.
>
>We can recreate this by doing (on the client);
>
>find . -type -f -print -exec grep jsdhgkhjfdk {} \;
>
>The same effect appears to happen on Linux servers as well. A single
>Linux client doing this will make the load average creep up towards 2.5.
>
>Linux Version 2.4.20
>nfs-utils-1.0.1-1
>exports are exported with -32bitclients
>
>Looking a bit deeper in a packet trace when the grep is running, Linux
>seems to issue 16 read requests (16KB) at a time whilst an SGI will only
>issue between 2 and 4. May or may not be relevant.
>
>This information has also been passed back to SGI who are looking at it.
>
>Packet traces and more detail is available if anyone wants them.
>
>Any ideas or suggestions would be welcome.
>
>******************************************************************
>Iain Irwin-Powell (AKA Iain Powell)
>Senior Systems Administrator
>Cinesite Europe Limited
>9 Carlisle Street
>London
>W1D 3BP
> 
>T: +442079734000
>DDI: +442079734053
>*******************************************************************
>It's not broken, it just doesn't work the way you expected.
>*******************************************************************
>
>
>
>
>-------------------------------------------------------
>This SF.net email is sponsored by: ObjectStore.
>If flattening out C++ or Java code to make your application fit in a
>relational database is painful, don't do it! Check out ObjectStore.
>Now part of Progress Software. http://www.objectstore.net/sourceforge
>_______________________________________________
>NFS maillist  -  NFS@lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/nfs
>
>
>-------------------------------------------------------
>This SF.net email is sponsored by: ObjectStore.
>If flattening out C++ or Java code to make your application fit in a
>relational database is painful, don't do it! Check out ObjectStore.
>Now part of Progress Software. http://www.objectstore.net/sourceforge
>_______________________________________________
>NFS maillist  -  NFS@lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/nfs
>
>  
>


-- 
Danny Smith
Senior Systems Administrator, Cinesite (Europe) Ltd
020 7973 4000 - x4055    /    dannys@cinesite.co.uk




-------------------------------------------------------
This SF.net email is sponsored by: ObjectStore.
If flattening out C++ or Java code to make your application fit in a
relational database is painful, don't do it! Check out ObjectStore.
Now part of Progress Software. http://www.objectstore.net/sourceforge
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

NFS Performance Between SGI Servers and Linux Clients

[Iain Irwin-Powell]

Does anyone have any ideas about this?

We have been experiencing a number of issues regarding NFS serving on
SGI IRIX (6.5.19) servers which now appear to point back to Linux NFS
clients.

The server is an Origin 2000, quad processors, 1.5GB memory, serving
over Gigabit. This can be reproduced on connections whether or not they
are using jumbo frames. The disks that are served are hardware RAID3
capable of around 80-90MB/s (tested and optimised not theoretical). 

Under certain circumstances will get a load average of 8 (or how ever
many nfsd's are running). The server itself is not that busy. Most of
the nfsd's show as sleeping but they are obviously waiting for
something.

Further investigation revealed that this situation arises when a Linux
client opens many files in quick succession (doing
open-read-close,open-read-close). We tend to use files in the range
3MB-13MB in size generally.

If we do the same test on an SGI client there is very little impact on
the load average.

We can recreate this by doing (on the client);

find . -type -f -print -exec grep jsdhgkhjfdk {} \;

The same effect appears to happen on Linux servers as well. A single
Linux client doing this will make the load average creep up towards 2.5.

Linux Version 2.4.20
nfs-utils-1.0.1-1
exports are exported with -32bitclients

Looking a bit deeper in a packet trace when the grep is running, Linux
seems to issue 16 read requests (16KB) at a time whilst an SGI will only
issue between 2 and 4. May or may not be relevant.

This information has also been passed back to SGI who are looking at it.

Packet traces and more detail is available if anyone wants them.

Any ideas or suggestions would be welcome.

******************************************************************
Iain Irwin-Powell (AKA Iain Powell)
Senior Systems Administrator
Cinesite Europe Limited
9 Carlisle Street
London
W1D 3BP
 
T: +442079734000
DDI: +442079734053
*******************************************************************
It's not broken, it just doesn't work the way you expected.
*******************************************************************




-------------------------------------------------------
This SF.net email is sponsored by: ObjectStore.
If flattening out C++ or Java code to make your application fit in a
relational database is painful, don't do it! Check out ObjectStore.
Now part of Progress Software. http://www.objectstore.net/sourceforge
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS Performance Between SGI Servers and Linux Clients

[Trond Myklebust]

>>>>> " " == Iain Irwin-Powell <iain@cinesite.co.uk> writes:

     > Looking a bit deeper in a packet trace when the grep is
     > running, Linux seems to issue 16 read requests (16KB) at a time
     > whilst an SGI will only issue between 2 and 4. May or may not
     > be relevant.

... and this is a problem because .... ???

The Linux client tries to read ahead as much as it can and as quickly
as it can. Why should it doing anything else?

Cheers,
  Trond


-------------------------------------------------------
This SF.net email is sponsored by: ObjectStore.
If flattening out C++ or Java code to make your application fit in a
relational database is painful, don't do it! Check out ObjectStore.
Now part of Progress Software. http://www.objectstore.net/sourceforge
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs