Getting lousy NFS + tar-pipe throughput on 2.4.20

Getting lousy NFS + tar-pipe throughput on 2.4.20

[Timothy Miller]

I'm running a fresh install of RH9 (kernel 2.4.20-something) on a 
workstation.  The workstation is an Athlon 3200+ with 512 megs of RAM on 
an ABIT KV7 (KT600 chipset).  The ethernet controller built into the KV7 
is "VIA RhineII".  The file system is ext3.


We are mounting an NFS filesystem from a Sun box using automount, and 
we're using a tar-pipe to move data from the server to the workstation. 
  Both tars of the tar-pipe are running on the workstation, so the 
network traffic is all NFS.

(1) We have verified that the disk load on the server is very low.  The 
disk is not being saturated.

(2) We have verified that the ethernet on the server is not being saturated.

(3) The workstation is connected to the server through a switch, so it's 
not competing for bandwidth with anything else.


In theory, we should get about 10 megabytes/sec throughput, but what 
we're measuring is about 1 to 2 megs/sec.


The workstation is using a single 120 gig WD IDE drive (WD1200JB), which 
as I was talking about in other emails should be able to do up to 30 
megs/sec for writes.


While this tar-pipe is going on, the workstation is very unresponsive. 
"top" reports that kernel CPU usage is anywhere from 30% to 70%, but 
mostly around 40%.  User space is using about 10%; that varies also. 
Despite the fact that there is some amount of idle time, the X cursor 
jumps about badly.

We're not compressing or anything.  We're just doing the tar-pipe. 
Therefore, the workstation should be experiencing very little load while 
it transfers a mere 10 megs/sec to disk.  Buffering in RAM should also 
allow the kernel to order writes efficiently.

Since the source tar process is talking to an NFS volume, the overhead 
of opening, reading, and closing small files could hurt throughput 
(would have been better to rsh the source tar so that the tar data is 
what was going over ethernet through a single socket).  But that should 
_reduce_ the amount of I/O that is being accomplished, thereby reducing 
the work being done by the workstation.  It would just WAIT more.  It 
should not be unresponsive.


I would like to investigate this performance issue, but I don't know 
what tools I should run to investigate.  If anyone could please give me 
some tips on it, I would be most appreciative.

Thanks!

IDE DMA problem [WAS: Re: Getting lousy NFS + tar-pipe throughput on 2.4.20]

[Timothy Miller]

I have discovered one major problem that I didn't think to check 
before.  I have never before had trouble using DMA with WD drives, VIA 
chipsets, or RH9, so I didn't think to check before, but hdparm reports 
that DMA is disabled for the disk on the workstation I mentioned in my 
earlier post.

So, I tried this:
    hdparm -d1 /dev/hda

And I got this result:
/dev/hda:
 Setting using_dma to 1 (on)
 HDIO_SET_DMA failed: Operation not permitted
 using_dma = 0 (off)

How do I fix this?  Do I have to unmount the filesystem before I can 
change the dma setting?  Why would it be off to begin with?  I know that 
the BIOS is set up right.  Does 2.4.20 not work well with the KT600 chipset?

Thanks.


Timothy Miller wrote:
 > [snip everything about slow NFS performance]

Re: IDE DMA problem [WAS: Re: Getting lousy NFS + tar-pipe throughput on 2.4.20]

[Bartlomiej Zolnierkiewicz]

On Saturday 14 of February 2004 01:16, Timothy Miller wrote:
> I have discovered one major problem that I didn't think to check
> before.  I have never before had trouble using DMA with WD drives, VIA
> chipsets, or RH9, so I didn't think to check before, but hdparm reports
> that DMA is disabled for the disk on the workstation I mentioned in my
> earlier post.
>
> So, I tried this:
>     hdparm -d1 /dev/hda
>
> And I got this result:
> /dev/hda:
>  Setting using_dma to 1 (on)
>  HDIO_SET_DMA failed: Operation not permitted
>  using_dma = 0 (off)
>
> How do I fix this?  Do I have to unmount the filesystem before I can
> change the dma setting?  Why would it be off to begin with?  I know that
> the BIOS is set up right.  Does 2.4.20 not work well with the KT600
> chipset?

It doesn't, upgrade to 2.4.24.

2.4.24 problems [WAS: Re: IDE DMA problem [WAS: Re: Getting lousy NFS + tar-pipe throughput on 2.4.20]]

[Timothy Miller]

Bartlomiej Zolnierkiewicz wrote:
> On Saturday 14 of February 2004 01:16, Timothy Miller wrote:
> 
>   Does 2.4.20 not work well with the KT600
>>chipset?
> 
> 
> It doesn't, upgrade to 2.4.24.


Ok, so we tried that.  That caused all sorts of havoc, most of which we 
might be able to figure out.  It seems that RedHat's utilities and stuff 
don't get along well with 2.4.24 if all you do is just boot the new 
kernel.  Seems a bunch of other stuff needs to be upgraded.

Also, 'root=LABEL=/' doesn't work anymore in grub.conf, and there seems 
to be no way to enable it.  Is this a RedHat only thing?

In any event, when we managed to get it to boot, it absolutely did NOT 
fix the DMA problem with the KT600.  Trying to enable it with hdparm 
still says "Operation not permitted".  We spend a LOT of time trying to 
make sure we got the kernel configured right with all of the right 
options, etc., but we're still without IDE DMA.

Any further suggestions?

Thanks.

Re: 2.4.24 problems [WAS: Re: IDE DMA problem [WAS: Re: Getting lousy NFS + tar-pipe throughput on 2.4.20]]

[Bartlomiej Zolnierkiewicz]

On Monday 16 of February 2004 18:09, Timothy Miller wrote:
> Bartlomiej Zolnierkiewicz wrote:
> > On Saturday 14 of February 2004 01:16, Timothy Miller wrote:
> >
> >   Does 2.4.20 not work well with the KT600
> >
> >>chipset?
> >
> > It doesn't, upgrade to 2.4.24.
>
> Ok, so we tried that.  That caused all sorts of havoc, most of which we
> might be able to figure out.  It seems that RedHat's utilities and stuff
> don't get along well with 2.4.24 if all you do is just boot the new
> kernel.  Seems a bunch of other stuff needs to be upgraded.
>
> Also, 'root=LABEL=/' doesn't work anymore in grub.conf, and there seems
> to be no way to enable it.  Is this a RedHat only thing?

This feature is not present in vanilla kernels.

> In any event, when we managed to get it to boot, it absolutely did NOT
> fix the DMA problem with the KT600.  Trying to enable it with hdparm
> still says "Operation not permitted".  We spend a LOT of time trying to
> make sure we got the kernel configured right with all of the right
> options, etc., but we're still without IDE DMA.
>
> Any further suggestions?

Do you have VIA IDE driver compiled in?
CONFIG_BLK_DEV_VIA82CXXX=y

If so, please send output of 'dmesg' command.

--bart

Re: 2.4.24 problems [WAS: Re: IDE DMA problem [WAS: Re: Getting lousy NFS + tar-pipe throughput on 2.4.20]]

[Jeff Garzik]

Bartlomiej Zolnierkiewicz wrote:
> On Monday 16 of February 2004 18:09, Timothy Miller wrote:
>>Also, 'root=LABEL=/' doesn't work anymore in grub.conf, and there seems
>>to be no way to enable it.  Is this a RedHat only thing?

> This feature is not present in vanilla kernels.


It's not a kernel feature.

Red Hat's initrd image makes "root=LABEL=/" work.  The user probably 
needs to regen his initrd image...

	Jeff