2.4.19 NFSALL performance oddity

2.4.19 NFSALL performance oddity

[Heflin, Roger A.]

I am running some tests using 2.4.19 with all posted nfs patchs, =
everything is
being run sync.   All runs are with NFS3 over UDP.

I have noticed that the IO rates (writes) start slowly (about =
1.5MB/second) and
slowly build up to a peak of 2.5MB/second over several minutes, with the
test averaging 2.32 MB/second over a 2GB file write.  This is=20
with a server with Gigabit copper and the client with 100BT ethernet.  =20
No packets are being lost from what I can tell, on either the server or=20
the client, both machines are dual cpu with 2GB of ram and otherwise=20
unloaded.

The 2.2.19 client (with almost no NFS patches) runs about 3.98MB/second =
over
a 2GB file write, against the same 2.4.19 NFS server, and hits that rate =

almost immediately.

Using an older 2.4.19 NFS server (that I don't believe has any extra NFS
patches) gets a rate of 6.50 average over a 2GB file from
a 2.2.19 client and gets 2.00  MB/second average from one of the
original 2.4 test clients.

Both rsize and wsize are 8192, I have tried it with 32768 on at least
the 2.4 client side, but it does not appear to affect the above results.
I am tried with 32 and 64 nfsd and this did not appear to change
any of the results.   The rmem/wmem parameters have been upped
to 256K and did not appear to make any difference on the server.

The disk is locally capable of 20MB/second writes over a 4GB file, using
the same test as was used above.

Is there a way to make the IO start fast and stay fast?   What code =
could
I look at to see if I can affect the way this works?   Any parameters =
that
I could try changing that might improve things?

It appears to partially be client issue since the 2.2.19 NFS client is =
noticeably
faster, but since the older 2.4.19 server and the 2.2.19 client is even =
faster it
appears to also be a server issue.

So in summary:

2.2.19 -> 2.4.19 no patches -> 6.5MB/second
2.4.19nfsall -> 2.4.19 no patches ->  2.00 MB/second
2.2.19 -> 2.4.19 nfsall patch -> 3.98MB/second
2.4.19nfsall -> 2.4.19 nfsall patch -> 2.32 MB/second


				Roger


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4.19 NFSALL performance oddity

[Trond Myklebust]

>>>>> " " == Roger A Heflin <Heflin> writes:

     > I have noticed that the IO rates (writes) start slowly (about
     > 1.5MB/second) and slowly build up to a peak of 2.5MB/second
     > over several minutes, with the test averaging 2.32 MB/second
     > over a 2GB file write.  This is with a server with Gigabit
     > copper and the client with 100BT ethernet.  No packets are
     > being lost from what I can tell, on either the server or the
     > client, both machines are dual cpu with 2GB of ram and
     > otherwise unloaded.

Have you enabled CONFIG_HIGHMEM? If so, could you please try disabling
it.

Cheers,
  Trond


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4.19 NFSALL performance oddity

[Bryan O'Sullivan]

On Fri, 2002-10-04 at 16:46, Trond Myklebust wrote:

> Have you enabled CONFIG_HIGHMEM? If so, could you please try disabling
> it.

What's the issue with CONFIG_HIGHMEM, and on which machine should it be
expected to make a difference?

	<b


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4.19 NFSALL performance oddity

[Trond Myklebust]

>>>>> " " == Bryan O'Sullivan <bos@serpentine.com> writes:

     > On Fri, 2002-10-04 at 16:46, Trond Myklebust wrote:
    >> Have you enabled CONFIG_HIGHMEM? If so, could you please try
    >> disabling it.

     > What's the issue with CONFIG_HIGHMEM, and on which machine
     > should it be expected to make a difference?

I'm seeing major slowdowns, mainly on NFS reads, when I enable
CONFIG_HIGHMEM. I haven't yet had time to investigate this.

Cheers,
  Trond


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4.19 NFSALL performance oddity

[Bryan O'Sullivan]

On Sat, 2002-10-05 at 09:18, Trond Myklebust wrote:

> I'm seeing major slowdowns, mainly on NFS reads, when I enable
> CONFIG_HIGHMEM. I haven't yet had time to investigate this.

I'll try to take a look on a local system in the next few days.  Are you
or Roger using Jens Axboe's block-highmem patch or any of the drivers it
touches?  Knowing this should help control for at least one variable.

	<b



-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4.19 NFSALL performance oddity

[Trond Myklebust]

>>>>> " " == Bryan O'Sullivan <bos@serpentine.com> writes:

     > I'll try to take a look on a local system in the next few days.
     > Are you or Roger using Jens Axboe's block-highmem patch or any
     > of the drivers it touches?  Knowing this should help control
     > for at least one variable.

I've seen the behaviour on stock 2.4.19 and 2.4.20-preX. I haven't
tested any of Jens' patches...

Cheers,
  Trond


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4.19 NFSALL performance oddity

[H. J. Lu]

On Sat, Oct 05, 2002 at 11:51:29PM +0200, Trond Myklebust wrote:
> >>>>> " " == Bryan O'Sullivan <bos@serpentine.com> writes:
> 
>      > I'll try to take a look on a local system in the next few days.
>      > Are you or Roger using Jens Axboe's block-highmem patch or any
>      > of the drivers it touches?  Knowing this should help control
>      > for at least one variable.
> 
> I've seen the behaviour on stock 2.4.19 and 2.4.20-preX. I haven't
> tested any of Jens' patches...
> 

It must be a new bug. I used to have Quad P/II Xeon with 8GB RAM.
I had no problem with its NFS server performance. Also my modified
kernel based on RedHat kernel 2.4.18-14 has no NFS server problem
with 1.5GB RAM.


H.J.


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: 2.4.19 NFSALL performance oddity

[Heflin, Roger A.]

The only extra patches to 2.4.19 are the NFS patches, so
I don't believe so unless it was included in 2.4.19.

We are using config_himem, because at least some of the
machines we have will have a bit too much memory.

I will try turning that off and see if it makes a difference, is
this on the client or the server or both?   I am pretty sure
we have it enabled on all of the nfsall patched 2.4.19 builts.

				Roger

> -----Original Message-----
> From:	Bryan O'Sullivan [SMTP:bos@serpentine.com]
> Sent:	Saturday, October 05, 2002 3:55 PM
> To:	Trond Myklebust
> Cc:	Heflin, Roger A.; nfs@lists.sourceforge.net
> Subject:	Re: [NFS] 2.4.19 NFSALL performance oddity
>=20
> On Sat, 2002-10-05 at 09:18, Trond Myklebust wrote:
>=20
> > I'm seeing major slowdowns, mainly on NFS reads, when I enable
> > CONFIG_HIGHMEM. I haven't yet had time to investigate this.
>=20
> I'll try to take a look on a local system in the next few days.  Are =
you
> or Roger using Jens Axboe's block-highmem patch or any of the drivers =
it
> touches?  Knowing this should help control for at least one variable.
>=20
> 	<b
>=20


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4.19 NFSALL performance oddity

[Bryan O'Sullivan]

On Sat, 2002-10-05 at 16:41, H. J. Lu wrote:

> It must be a new bug. I used to have Quad P/II Xeon with 8GB RAM.
> I had no problem with its NFS server performance. Also my modified
> kernel based on RedHat kernel 2.4.18-14 has no NFS server problem
> with 1.5GB RAM.

You're not using Trond's NFS_ALL patch, though, right?  That has some
substantial modifications to the source.

	<b


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4.19 NFSALL performance oddity

[H. J. Lu]

On Sat, Oct 05, 2002 at 09:01:02PM -0700, Bryan O'Sullivan wrote:
> On Sat, 2002-10-05 at 16:41, H. J. Lu wrote:
> 
> > It must be a new bug. I used to have Quad P/II Xeon with 8GB RAM.
> > I had no problem with its NFS server performance. Also my modified
> > kernel based on RedHat kernel 2.4.18-14 has no NFS server problem
> > with 1.5GB RAM.
> 
> You're not using Trond's NFS_ALL patch, though, right?  That has some
> substantial modifications to the source.
> 

RedHat 2.4.18-14 is kernel 2.4.18 + tons of patches. I don't think
it has straight Trond's NFS_ALL patch. But it does have some NFS
changes. The only NFS related complaint I have is they default the
NFS V3 block size to 4096 which kills the NFS client performance. I
believe they put that in to avoid crashing NetApp. I changed it to
8192 and I am quite happy with its NFS performance.


H.J.


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4.19 NFSALL performance oddity

[Bryan O'Sullivan]

On Sat, 2002-10-05 at 14:51, Trond Myklebust wrote:

> I've seen the behaviour on stock 2.4.19 and 2.4.20-preX.

That tends to rule out the block-highmem stuff, at least, since it's
been in since 2.4.20-pre2.

	<b


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: 2.4.19 NFSALL performance oddity

[Heflin, Roger A.]

Update on the issue:

And given that I am using MS outlook, who knows
what this actual message will look like.

Original Test:
> 2.2.19 -> 2.4.19 no patches -> 6.5MB/second
> 2.4.19nfsall -> 2.4.19 no patches ->  2.00 MB/second
> 2.2.19 -> 2.4.19 nfsall patch -> 3.98MB/second
> 2.4.19nfsall -> 2.4.19 nfsall patch -> 2.32 MB/second
>=20
	All 2.4.19 kernels have at least 4GB himem enabled.
	The 2.4.19 kernels with nfsall have 64GB himem enabled.

	New tests:

	The write is a sequential write of a 2GB file,=20
	the read is a sequentail read of a 2GB file, the=20
	machines memory is just under 2GB, and it appears that
	the file is just large enough on the read test to make=20
	sure little or none of  the file is ever read straight=20
	from the cache.

	From the results, it looks like something in nfsall
	slows down both the client and the server.   The=20
	picture all of this data paints is rather unclear to me.
	If anyone sees any other test that would be possibly
	useful, suggest them.

	64GB/4GB/none does not appear to make a=20
	difference, for either the client or the server.   =20
	I did get some wide variations
	in some limited testing by adjusting the rmem/wmem
	and the number of nfsds, generally always slowing=20
	things down by a large amount, none of the numbers=20
	below are any of the especially slow cases.

	Tests with nohimem on NFSALL 2.4.19 server (32K blocks):
		nfsd=3D8, mem=3D65536, nohimem
			Client nohimem
				write: 2.312MB
				read: 8.938MB
			Client himem (64GB)
				write: 2.312MB
				read: 8.938MB=09

		nfsd=3D8, mem=3D262144, nohimem
			Client nohimem:=20
				write:  2.250MB
				read:   8.875MB
			Client himem (64GB)
				write: 2.312MB=20
				read:  8.812MB
	=09
	Tests with 4GB himem on nfsall 2.4.19 server (32k blocks)
			Client nohimem:
				write: 2.312MB
				read: 8.932MB
			Client 2.2.19:
				write: 4.50MB
				read:  8.812MB

	Tests with himem on NFSALL 2.4.19 server (8k blocks, different machine =
that 32k blocksize))
		nfsd=3D8, mem=3D65536, himem
			Client nohimem:
				write:	1.875MB
				read:	8.688MB
			Client himem (64GB)
				write:	1.875MB
				read:	8.625MB

	Tests with 4GB himem server, 2.4.19 server nohimem (8k blocks, =
different machine than other servers)
		nfsd=3D8, mem=3D65536
			Client nohimem, 2.4.19 nfsall
				write: 2.00 MB
				read:  2.00 MB
			Client 2.2.19:
				write:  6.5MB
				read:  10.0MB
			Client 2.4.19 no nfs patches, 64GB himem:
				write: not yet done
				read:

	Test with 2.2.19 server - these results seem to be unclear on their =
meaning:
			Client 2.4.19 64GB himem, nfsall:
				write: 3.875MB
				read:  4.875MB=09
			Client 2.2.19:
				write: 4.00 MB
				read: 8.188MB

> 				Roger


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: 2.4.19 NFSALL performance oddity

[Trond Myklebust]

FYI: I updated 2.4.19-NFS_ALL on Saturday with a couple of
bugfixes. Among them was one which fixes a queueing bottleneck when a
timeout+resend occurs.

Cheers,
  Trond


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: 2.4.19 NFSALL performance oddity

[Heflin, Roger A.]

Ok,

I applied the new patch, it does not appear to have made any difference=20
in my setup and tests, it also does not appear to break anything.
I am still analyzing the results, but not finding=20
anything obvious about what is going on.  2.2.19 appears to make alot
better client to a 2.4.19 server than 2.4.1[89][no patch/nfsall]
does.   And 2.4.19 with no patches appears to make a better
server than 2.4.19 (for the writes) with the NFS patches
(using 2.2.19 as a client).

Using 2.4.xx as a client, a 2.4.1[89][nfsall/nopatch] server is=20
about the same on writes, but 2.4.19 nfsall is much better
on the reads than the previous 2.4.1[89] without
the nfsall patch.

I guess the read numbers with NFSall look pretty good,=20
the write numbers probably need to be better, though the write=20
numbers do look ok  (2.5x larger) with 2.2.19 as
a client, they don't look good with 2.4.19 nfsall as a client.  =20
And from 2.4.19 to 2.4.19 nfsall the write numbers got a bit worse=20
for a 2.2.19 client, but did not appear to change a large amount
when using a 2.4.19  client.

I have a excel spreadsheet of the various results, that try to
summarize up all of the recent emails.  If anyone wants this=20
spreadsheet I will sent it out.


2.4.19 NFSALLnew Server:
	2.4.19 NFSALLnew Client
		write: 2.312MB
		read: 8.875MB
	2.4.18 Client:
		write: 2.125
		read: 8.125

2.4.19 NFSALLold Server:
	2.4.19 NFSALLnew Client:
		write: 2.312 MB
		read:  8.688 MB

2.4.18 no patches Server:
	2.4.19 NFSALL Client:
		write: 2.00 MB
		read:  2.50 MB
	2.4.19 nopatches client:
		write: 2.0MB
		read: 3.0MB
	2.2.19 stock client:
		write: 6.00 MB
		read: 10.0 MB



=20
			Roger
> -----Original Message-----
> From:	Trond Myklebust [SMTP:trond.myklebust@fys.uio.no]
> Sent:	Monday, October 07, 2002 5:21 PM
> To:	Heflin, Roger A.
> Cc:	nfs@lists.sourceforge.net
> Subject:	RE: 2.4.19 NFSALL performance oddity
>=20
>=20
> FYI: I updated 2.4.19-NFS_ALL on Saturday with a couple of
> bugfixes. Among them was one which fixes a queueing bottleneck when a
> timeout+resend occurs.
>=20
> Cheers,
>   Trond


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs