Poor NFS client performance on 2.4.18?

Poor NFS client performance on 2.4.18?

[Dan Yocum]

Trond, et al.

I'm getting poor NFS performance (~250KBps read and write) on 2.4.18 and am
wondering if I'm the only one.  There is no performance drop under other OSs
or other kernel versions, so I don't think it's the server.

Here's the the details:

2.4.18 patched with:
	NFS client patches (linux-2.4.18-NFS_ALL.dif) 
	xfs-1.1-PR1-2.4.18-all.patch
	Ingo's Foster IRQ patch (these are dual Xeons)

If you need any more details, let me know.

Thanks,
Dan


-- 
Dan Yocum
Sloan Digital Sky Survey, Fermilab  630.840.6509
yocum@fnal.gov, http://www.sdss.org
SDSS.  Mapping the Universe.

Re: Poor NFS client performance on 2.4.18?

[Trond Myklebust]

>>>>> " " == Dan Yocum <yocum@fnal.gov> writes:

     > Trond, et al.  I'm getting poor NFS performance (~250KBps read
     > and write) on 2.4.18 and am wondering if I'm the only one.
     > There is no performance drop under other OSs or other kernel
     > versions, so I don't think it's the server.

     > Here's the the details:

     > 2.4.18 patched with:
     > 	NFS client patches (linux-2.4.18-NFS_ALL.dif)
     > 	xfs-1.1-PR1-2.4.18-all.patch Ingo's Foster IRQ patch (these
     > 	are dual Xeons)

     > If you need any more details, let me know.

The latest NFS_ALL patches include experimental code that changes the
UDP congestion control. I'm basically trying to relax the algorithm to
what is standard on *BSD (i.e. we follow the standard Van Jacobson).

This would mean that we don't wait for the reply from the server
before we send off the next request. Unfortunately, there appears to
be a lot of setups out there that start to drop packets when this
occurs, and I haven't yet finished determining the root cause.

If I can manage to get my laptop to work again, I'll try to
investigate a bit more this weekend...

Cheers,
  Trond

Re: Poor NFS client performance on 2.4.18?

[Dan Yocum]

Trond,

So, would I be correct in assuming that backing out the
linux-2.4.18-ping.dif patch would solve the problem in the short term?

Thanks,
Dan

Trond Myklebust wrote:
> 
> >>>>> " " == Dan Yocum <yocum@fnal.gov> writes:
> 
>      > Trond, et al.  I'm getting poor NFS performance (~250KBps read
>      > and write) on 2.4.18 and am wondering if I'm the only one.
>      > There is no performance drop under other OSs or other kernel
>      > versions, so I don't think it's the server.
> 
>      > Here's the the details:
> 
>      > 2.4.18 patched with:
>      >  NFS client patches (linux-2.4.18-NFS_ALL.dif)
>      >  xfs-1.1-PR1-2.4.18-all.patch Ingo's Foster IRQ patch (these
>      >  are dual Xeons)
> 
>      > If you need any more details, let me know.
> 
> The latest NFS_ALL patches include experimental code that changes the
> UDP congestion control. I'm basically trying to relax the algorithm to
> what is standard on *BSD (i.e. we follow the standard Van Jacobson).
> 
> This would mean that we don't wait for the reply from the server
> before we send off the next request. Unfortunately, there appears to
> be a lot of setups out there that start to drop packets when this
> occurs, and I haven't yet finished determining the root cause.
> 
> If I can manage to get my laptop to work again, I'll try to
> investigate a bit more this weekend...
> 
> Cheers,
>   Trond



-- 
Dan Yocum
Sloan Digital Sky Survey, Fermilab  630.840.6509
yocum@fnal.gov, http://www.sdss.org
SDSS.  Mapping the Universe.

Re: Poor NFS client performance on 2.4.18?

[Trond Myklebust]

>>>>> " " == Dan Yocum <yocum@fnal.gov> writes:

     > Trond, So, would I be correct in assuming that backing out the
     > linux-2.4.18-ping.dif patch would solve the problem in the
     > short term?

Not ping, but linux-2.4.18-rpc_tweaks.dif...

Cheers,
  Trond

Re: Poor NFS client performance on 2.4.18?

[Dan Yocum]

Trond,

OK, so backing out the rpc_tweaks dif fixed the performance problem,
however, seems to have introduced another problem that appears to be
stemming from the seekdir.dif.  Attempting to run an app from an IRIX client
(that has the 32bitclients option set) freezes the NFS volume - one can't
access it from the Linux side, anymore.

You can read and write to the NFS volume *before* trying to run something
from there, but not after.

Ideas?

Thanks,
Dan




Trond Myklebust wrote:
> 
> >>>>> " " == Dan Yocum <yocum@fnal.gov> writes:
> 
>      > Trond, et al.  I'm getting poor NFS performance (~250KBps read
>      > and write) on 2.4.18 and am wondering if I'm the only one.
>      > There is no performance drop under other OSs or other kernel
>      > versions, so I don't think it's the server.
> 
>      > Here's the the details:
> 
>      > 2.4.18 patched with:
>      >  NFS client patches (linux-2.4.18-NFS_ALL.dif)
>      >  xfs-1.1-PR1-2.4.18-all.patch Ingo's Foster IRQ patch (these
>      >  are dual Xeons)
> 
>      > If you need any more details, let me know.
> 
> The latest NFS_ALL patches include experimental code that changes the
> UDP congestion control. I'm basically trying to relax the algorithm to
> what is standard on *BSD (i.e. we follow the standard Van Jacobson).
> 
> This would mean that we don't wait for the reply from the server
> before we send off the next request. Unfortunately, there appears to
> be a lot of setups out there that start to drop packets when this
> occurs, and I haven't yet finished determining the root cause.
> 
> If I can manage to get my laptop to work again, I'll try to
> investigate a bit more this weekend...
> 
> Cheers,
>   Trond




-- 
Dan Yocum
Sloan Digital Sky Survey, Fermilab  630.840.6509
yocum@fnal.gov, http://www.sdss.org
SDSS.  Mapping the Universe.

Re: Poor NFS client performance on 2.4.18?

[Trond Myklebust]

On Tuesday 7. May 2002 00:05, Dan Yocum wrote:
> Trond,
>
> OK, so backing out the rpc_tweaks dif fixed the performance problem,
> however, seems to have introduced another problem that appears to be
> stemming from the seekdir.dif.  Attempting to run an app from an IRIX
> client (that has the 32bitclients option set) freezes the NFS volume - one
> can't access it from the Linux side, anymore.
>
> You can read and write to the NFS volume *before* trying to run something
> from there, but not after.
>
> Ideas?

That smells like another network driver bug. Have you tcpdumped the traffic 
between client and server?

Cheers,
  Trond

Re: Poor NFS client performance on 2.4.18?

[Dan Yocum]

Trond Myklebust wrote:
> 
> On Tuesday 7. May 2002 00:05, Dan Yocum wrote:
> > Trond,
> >
> > OK, so backing out the rpc_tweaks dif fixed the performance problem,
> > however, seems to have introduced another problem that appears to be
> > stemming from the seekdir.dif.  Attempting to run an app from an IRIX
> > client (that has the 32bitclients option set) freezes the NFS volume - one
> > can't access it from the Linux side, anymore.
> >
> > You can read and write to the NFS volume *before* trying to run something
> > from there, but not after.
> >
> > Ideas?
> 
> That smells like another network driver bug. Have you tcpdumped the traffic
> between client and server?


Ah, that may be the case - the problem also exists with a Linux server as
well... let me check, and I'll let you know.

Thanks,
Dan


-- 
Dan Yocum
Sloan Digital Sky Survey, Fermilab  630.840.6509
yocum@fnal.gov, http://www.sdss.org
SDSS.  Mapping the Universe.

Re: Poor NFS client performance on 2.4.18?

[Dan Yocum]

Dan Yocum wrote:
> 
> Trond Myklebust wrote:
> >
> > On Tuesday 7. May 2002 00:05, Dan Yocum wrote:
> > > Trond,
> > >
> > > OK, so backing out the rpc_tweaks dif fixed the performance problem,
> > > however, seems to have introduced another problem that appears to be
> > > stemming from the seekdir.dif.  Attempting to run an app from an IRIX
> > > client (that has the 32bitclients option set) freezes the NFS volume - one
> > > can't access it from the Linux side, anymore.
> > >
> > > You can read and write to the NFS volume *before* trying to run something
> > > from there, but not after.
> > >
> > > Ideas?
> >
> > That smells like another network driver bug. Have you tcpdumped the traffic
> > between client and server?
> 
> Ah, that may be the case - the problem also exists with a Linux server as
> well... let me check, and I'll let you know.


I take that back - it's only hanging on the Linux server when the IRIX
server is already hung.



-- 
Dan Yocum
Sloan Digital Sky Survey, Fermilab  630.840.6509
yocum@fnal.gov, http://www.sdss.org
SDSS.  Mapping the Universe.

ns83820 bug. [was Re: Poor NFS client performance on 2.4.18?]

[Dan Yocum]

Trond, et al.

You're right, it's a driver (ns83820) issue.  Strange that it only shows up
when trying to execute an app that's mounted via NFS, but, whatever. 
Running apps from the the NFS volumes with the eepro100 adapter that's on
the machine works fine with the updated NFS_all patch applied.

Thanks, again,
Dan


Dan Yocum wrote:
> 
> Dan Yocum wrote:
> >
> > Trond Myklebust wrote:
> > >
> > > On Tuesday 7. May 2002 00:05, Dan Yocum wrote:
> > > > Trond,
> > > >
> > > > OK, so backing out the rpc_tweaks dif fixed the performance problem,
> > > > however, seems to have introduced another problem that appears to be
> > > > stemming from the seekdir.dif.  Attempting to run an app from an IRIX
> > > > client (that has the 32bitclients option set) freezes the NFS volume - one
> > > > can't access it from the Linux side, anymore.
> > > >
> > > > You can read and write to the NFS volume *before* trying to run something
> > > > from there, but not after.
> > > >
> > > > Ideas?
> > >
> > > That smells like another network driver bug. Have you tcpdumped the traffic
> > > between client and server?
> >
> > Ah, that may be the case - the problem also exists with a Linux server as
> > well... let me check, and I'll let you know.
> 
> I take that back - it's only hanging on the Linux server when the IRIX
> server is already hung.


-- 
Dan Yocum
Sloan Digital Sky Survey, Fermilab  630.840.6509
yocum@fnal.gov, http://www.sdss.org
SDSS.  Mapping the Universe.

Re: ns83820 bug. [was Re: Poor NFS client performance on 2.4.18?]

[Benjamin LaHaise]

Upgrade to 0.17 (which is in 2.4.19-pre5 or so and later) and you should 
find the issue resolved.

		-ben

On Wed, May 08, 2002 at 03:19:03PM -0500, Dan Yocum wrote:
> Trond, et al.
> 
> You're right, it's a driver (ns83820) issue.  Strange that it only shows up
> when trying to execute an app that's mounted via NFS, but, whatever. 
> Running apps from the the NFS volumes with the eepro100 adapter that's on
> the machine works fine with the updated NFS_all patch applied.
> 
> Thanks, again,
> Dan
> 
> 
> Dan Yocum wrote:
> > 
> > Dan Yocum wrote:
> > >
> > > Trond Myklebust wrote:
> > > >
> > > > On Tuesday 7. May 2002 00:05, Dan Yocum wrote:
> > > > > Trond,
> > > > >
> > > > > OK, so backing out the rpc_tweaks dif fixed the performance problem,
> > > > > however, seems to have introduced another problem that appears to be
> > > > > stemming from the seekdir.dif.  Attempting to run an app from an IRIX
> > > > > client (that has the 32bitclients option set) freezes the NFS volume - one
> > > > > can't access it from the Linux side, anymore.
> > > > >
> > > > > You can read and write to the NFS volume *before* trying to run something
> > > > > from there, but not after.
> > > > >
> > > > > Ideas?
> > > >
> > > > That smells like another network driver bug. Have you tcpdumped the traffic
> > > > between client and server?
> > >
> > > Ah, that may be the case - the problem also exists with a Linux server as
> > > well... let me check, and I'll let you know.
> > 
> > I take that back - it's only hanging on the Linux server when the IRIX
> > server is already hung.
> 
> 
> -- 
> Dan Yocum
> Sloan Digital Sky Survey, Fermilab  630.840.6509
> yocum@fnal.gov, http://www.sdss.org
> SDSS.  Mapping the Universe.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
"You will be reincarnated as a toad; and you will be much happier."

Re: ns83820 bug. [was Re: Poor NFS client performance on 2.4.18?]

[Dan Yocum]

Ben, Trond,

Ugh.  It's been a bad week in general.  Anyway, yes, you're right, the 0.17
upgrade did the trick - I had actually tried that, but had accidentally
re-applied the rpc_tweaks dif which knocked the performance down again. 
Backing out that patch, updating to v 0.17 of the ns83820 driver and all is
well, again.  

So, it's back in Trond's court, with the rpc_tweaks causing the slow
read/write over NFS, problem.


Trond, if all I want is 32k block transfers from that dif, can I just apply
the following, or is there more to it:


diff -u --recursive --new-file
linux-2.4.18-svc_tcp/include/linux/nfsd/const.h
linux-2.4.18-rpc_cong/include/linux/nfsd/const.h
--- linux-2.4.18-svc_tcp/include/linux/nfsd/const.h     Sat Apr  1 18:04:27
2000+++ linux-2.4.18-rpc_cong/include/linux/nfsd/const.h    Wed Feb 20
17:20:45 2002@@ -21,7 +21,7 @@
 /*
  * Maximum blocksize supported by daemon currently at 8K
  */
-#define NFSSVC_MAXBLKSIZE      8192
+#define NFSSVC_MAXBLKSIZE      (32*1024)
 
 #ifdef __KERNEL__
 

Thanks,
Dan



Benjamin LaHaise wrote:
> 
> Upgrade to 0.17 (which is in 2.4.19-pre5 or so and later) and you should
> find the issue resolved.
> 
>                 -ben
> 
> On Wed, May 08, 2002 at 03:19:03PM -0500, Dan Yocum wrote:
> > Trond, et al.
> >
> > You're right, it's a driver (ns83820) issue.  Strange that it only shows up
> > when trying to execute an app that's mounted via NFS, but, whatever.
> > Running apps from the the NFS volumes with the eepro100 adapter that's on
> > the machine works fine with the updated NFS_all patch applied.
> >
> > Thanks, again,
> > Dan
> >
> >
> > Dan Yocum wrote:
> > >
> > > Dan Yocum wrote:
> > > >
> > > > Trond Myklebust wrote:
> > > > >
> > > > > On Tuesday 7. May 2002 00:05, Dan Yocum wrote:
> > > > > > Trond,
> > > > > >
> > > > > > OK, so backing out the rpc_tweaks dif fixed the performance problem,
> > > > > > however, seems to have introduced another problem that appears to be
> > > > > > stemming from the seekdir.dif.  Attempting to run an app from an IRIX
> > > > > > client (that has the 32bitclients option set) freezes the NFS volume - one
> > > > > > can't access it from the Linux side, anymore.
> > > > > >
> > > > > > You can read and write to the NFS volume *before* trying to run something
> > > > > > from there, but not after.
> > > > > >
> > > > > > Ideas?
> > > > >
> > > > > That smells like another network driver bug. Have you tcpdumped the traffic
> > > > > between client and server?
> > > >
> > > > Ah, that may be the case - the problem also exists with a Linux server as
> > > > well... let me check, and I'll let you know.
> > >
> > > I take that back - it's only hanging on the Linux server when the IRIX
> > > server is already hung.
> >
> >
> > --
> > Dan Yocum
> > Sloan Digital Sky Survey, Fermilab  630.840.6509
> > yocum@fnal.gov, http://www.sdss.org
> > SDSS.  Mapping the Universe.
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> 
> --
> "You will be reincarnated as a toad; and you will be much happier."

-- 
Dan Yocum
Sloan Digital Sky Survey, Fermilab  630.840.6509
yocum@fnal.gov, http://www.sdss.org
SDSS.  Mapping the Universe.