Interesting disk throughput performance problem

Interesting disk throughput performance problem

[Jimmie Mayfield]

Hi.  I'm running into some disk throughput issues that I can't explain.
Hopefully someone reading this can offer an explanation.

One of my machines is running 2.4.5 and has 2 hard drives: a 7200 rpm
ATA100 Maxtor and a 5400 rpm ATA33 IBM.  Each drive is a master on its own
controller (AMI CMD649 as found on the IWill KT266-R).  Both drives contain
reiserfs 3.6x filesystems.  

By all local benchmarks, the 7200 rpm drive is the faster drive.  But this 
doesn't seem to be the case for large files originating from remote clients.  
Witness:

My crude test involves scp'ing a 100MB file from another machine on my home 
network over 100bT ethernet.

1)  scp to the 5400rpm drive:  roughly 10MB/sec.  
2)  scp to the 7200rpm drive:  roughly 2MB/sec.  

I've tried 'tail' and 'notail' mount options with no change (as expected since 
this is a single large file).  I suspect that the machine would become CPU-bound 
somewhere in the 20MB/sec range (see below for my reasoning).

I see the same sort of behavior using Samba though not nearly as 
pronounced (the 5400rpm drive is merely 2x as fast as the 7200rpm drive).

Okay.  Since the test involved 2 separate drives with different geometries,
I figured this might be due to physical block location.  Perhaps the file
is getting allocated to the fastest cylinders on the 5400 rpm drive and
the slowest cylinders on the 7200 rpm drive.  Or it could be a fragmentation
issue.

So I tried the test locally:  with the file stored on the 5400rpm drive, 
scp it to localhost and write it to the 7200rpm drive.  Results were a little 
below 10MB/sec (CPU near 100% presumably due to encrypting/decrypting on 
the fly).

Any ideas why the 7200rpm drive performs so poorly for remote clients but 
performs wonderfully well when those same operations are performed locally?

Jimmie

Re: Interesting disk throughput performance problem

[Hans Reiser]

I'm just guessing here, but is write caching active on one but not the other?

Hans


Jimmie Mayfield wrote:
> 
> Hi.  I'm running into some disk throughput issues that I can't explain.
> Hopefully someone reading this can offer an explanation.
> 
> One of my machines is running 2.4.5 and has 2 hard drives: a 7200 rpm
> ATA100 Maxtor and a 5400 rpm ATA33 IBM.  Each drive is a master on its own
> controller (AMI CMD649 as found on the IWill KT266-R).  Both drives contain
> reiserfs 3.6x filesystems.
> 
> By all local benchmarks, the 7200 rpm drive is the faster drive.  But this
> doesn't seem to be the case for large files originating from remote clients.
> Witness:
> 
> My crude test involves scp'ing a 100MB file from another machine on my home
> network over 100bT ethernet.
> 
> 1)  scp to the 5400rpm drive:  roughly 10MB/sec.
> 2)  scp to the 7200rpm drive:  roughly 2MB/sec.
> 
> I've tried 'tail' and 'notail' mount options with no change (as expected since
> this is a single large file).  I suspect that the machine would become CPU-bound
> somewhere in the 20MB/sec range (see below for my reasoning).
> 
> I see the same sort of behavior using Samba though not nearly as
> pronounced (the 5400rpm drive is merely 2x as fast as the 7200rpm drive).
> 
> Okay.  Since the test involved 2 separate drives with different geometries,
> I figured this might be due to physical block location.  Perhaps the file
> is getting allocated to the fastest cylinders on the 5400 rpm drive and
> the slowest cylinders on the 7200 rpm drive.  Or it could be a fragmentation
> issue.
> 
> So I tried the test locally:  with the file stored on the 5400rpm drive,
> scp it to localhost and write it to the 7200rpm drive.  Results were a little
> below 10MB/sec (CPU near 100% presumably due to encrypting/decrypting on
> the fly).
> 
> Any ideas why the 7200rpm drive performs so poorly for remote clients but
> performs wonderfully well when those same operations are performed locally?
> 
> Jimmie
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Interesting disk throughput performance problem

[toon]

On Sun, Jul 22, 2001 at 01:20:24PM +0400, Hans Reiser wrote:
> 
> I'm just guessing here, but is write caching active on one but not the other?

What do you mean?
That he should activate the write caching in all his drives?
I thought that was plain stupid and wrong, because the filesystem
expects journal data to hit the disk immedialtely. The journal
is written synchronously, isn't it?
So I would expect you to advise everybody to deactivate any
caching in drives and controllers. An the we are back with
Jimmie's question: the throughput performance of his drives
is bad, and the (theoretically) fastest drive is worst.

Regards,
Toon (running a newsserver with reiserfs-3.5.32 on top of LVM-0.9.1-beta7 on
top of DAC960 hardware RAID5 with write-caching turned off, using linux-2.2.19)

> Jimmie Mayfield wrote:
> > 
> > Hi.  I'm running into some disk throughput issues that I can't explain.
> > Hopefully someone reading this can offer an explanation.
> > 
> > One of my machines is running 2.4.5 and has 2 hard drives: a 7200 rpm
> > ATA100 Maxtor and a 5400 rpm ATA33 IBM.  Each drive is a master on its own
> > controller (AMI CMD649 as found on the IWill KT266-R).  Both drives contain
> > reiserfs 3.6x filesystems.
> > 
> > By all local benchmarks, the 7200 rpm drive is the faster drive.  But this
> > doesn't seem to be the case for large files originating from remote clients.
> > Witness:
> > 
> > My crude test involves scp'ing a 100MB file from another machine on my home
> > network over 100bT ethernet.
> > 
> > 1)  scp to the 5400rpm drive:  roughly 10MB/sec.
> > 2)  scp to the 7200rpm drive:  roughly 2MB/sec.
> > 
> > I've tried 'tail' and 'notail' mount options with no change (as expected since
> > this is a single large file).  I suspect that the machine would become CPU-bound
> > somewhere in the 20MB/sec range (see below for my reasoning).
> > 
> > I see the same sort of behavior using Samba though not nearly as
> > pronounced (the 5400rpm drive is merely 2x as fast as the 7200rpm drive).
> > 
> > Okay.  Since the test involved 2 separate drives with different geometries,
> > I figured this might be due to physical block location.  Perhaps the file
> > is getting allocated to the fastest cylinders on the 5400 rpm drive and
> > the slowest cylinders on the 7200 rpm drive.  Or it could be a fragmentation
> > issue.
> > 
> > So I tried the test locally:  with the file stored on the 5400rpm drive,
> > scp it to localhost and write it to the 7200rpm drive.  Results were a little
> > below 10MB/sec (CPU near 100% presumably due to encrypting/decrypting on
> > the fly).
> > 
> > Any ideas why the 7200rpm drive performs so poorly for remote clients but
> > performs wonderfully well when those same operations are performed locally?
> > 
> > Jimmie
> > 
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Interesting disk throughput performance problem

[Hans Reiser]

toon@vdpas.hobby.nl wrote:
> 
> On Sun, Jul 22, 2001 at 01:20:24PM +0400, Hans Reiser wrote:
> >
> > I'm just guessing here, but is write caching active on one but not the other?
> 
> What do you mean?
> That he should activate the write caching in all his drives?

Didn't say that, I simply asked a question to collect data before formulating any opinion.

> I thought that was plain stupid and wrong, because the filesystem
> expects journal data to hit the disk immedialtely. The journal
> is written synchronously, isn't it?
> So I would expect you to advise everybody to deactivate any
> caching in drives and controllers. An the we are back with
> Jimmie's question: the throughput performance of his drives
> is bad, and the (theoretically) fastest drive is worst.

A difference that large in throughput is not what I would first guess.

> 
> Regards,
> Toon (running a newsserver with reiserfs-3.5.32 on top of LVM-0.9.1-beta7 on
> top of DAC960 hardware RAID5 with write-caching turned off, using linux-2.2.19)
> 
> > Jimmie Mayfield wrote:
> > >
> > > Hi.  I'm running into some disk throughput issues that I can't explain.
> > > Hopefully someone reading this can offer an explanation.
> > >
> > > One of my machines is running 2.4.5 and has 2 hard drives: a 7200 rpm
> > > ATA100 Maxtor and a 5400 rpm ATA33 IBM.  Each drive is a master on its own
> > > controller (AMI CMD649 as found on the IWill KT266-R).  Both drives contain
> > > reiserfs 3.6x filesystems.
> > >
> > > By all local benchmarks, the 7200 rpm drive is the faster drive.  But this
> > > doesn't seem to be the case for large files originating from remote clients.
> > > Witness:
> > >
> > > My crude test involves scp'ing a 100MB file from another machine on my home
> > > network over 100bT ethernet.
> > >
> > > 1)  scp to the 5400rpm drive:  roughly 10MB/sec.
> > > 2)  scp to the 7200rpm drive:  roughly 2MB/sec.
> > >
> > > I've tried 'tail' and 'notail' mount options with no change (as expected since
> > > this is a single large file).  I suspect that the machine would become CPU-bound
> > > somewhere in the 20MB/sec range (see below for my reasoning).
> > >
> > > I see the same sort of behavior using Samba though not nearly as
> > > pronounced (the 5400rpm drive is merely 2x as fast as the 7200rpm drive).
> > >
> > > Okay.  Since the test involved 2 separate drives with different geometries,
> > > I figured this might be due to physical block location.  Perhaps the file
> > > is getting allocated to the fastest cylinders on the 5400 rpm drive and
> > > the slowest cylinders on the 7200 rpm drive.  Or it could be a fragmentation
> > > issue.
> > >
> > > So I tried the test locally:  with the file stored on the 5400rpm drive,
> > > scp it to localhost and write it to the 7200rpm drive.  Results were a little
> > > below 10MB/sec (CPU near 100% presumably due to encrypting/decrypting on
> > > the fly).
> > >
> > > Any ideas why the 7200rpm drive performs so poorly for remote clients but
> > > performs wonderfully well when those same operations are performed locally?
> > >
> > > Jimmie
> > >
> > > -
> > > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > Please read the FAQ at  http://www.tux.org/lkml/
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/

Re: Interesting disk throughput performance problem

[Jakob Østergaard]

On Sat, Jul 21, 2001 at 11:33:13PM -0400, Jimmie Mayfield wrote:
> Hi.  I'm running into some disk throughput issues that I can't explain.
> Hopefully someone reading this can offer an explanation.
> 
> One of my machines is running 2.4.5 and has 2 hard drives: a 7200 rpm
> ATA100 Maxtor and a 5400 rpm ATA33 IBM.  Each drive is a master on its own
> controller (AMI CMD649 as found on the IWill KT266-R).  Both drives contain
> reiserfs 3.6x filesystems.  
> 
> By all local benchmarks, the 7200 rpm drive is the faster drive.  But this 
> doesn't seem to be the case for large files originating from remote clients.  
> Witness:
....
> So I tried the test locally:  with the file stored on the 5400rpm drive, 
> scp it to localhost and write it to the 7200rpm drive.  Results were a little 
> below 10MB/sec (CPU near 100% presumably due to encrypting/decrypting on 
> the fly).
> 
> Any ideas why the 7200rpm drive performs so poorly for remote clients but 
> performs wonderfully well when those same operations are performed locally?

This is a wild guess:

Try cat /proc/interrupts

Would the 7200 rpm drive controller happen to share an IRQ with your NIC ?

If so, something is horribly wrong since that shouldn't give that kind of
performance penalty.  But it's my best guess   :)

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: Interesting disk throughput performance problem

[Mike Black]

Not enough info (plenty for guessing though :-)

First off show "hdparm -i /dev/hd_" and "hdparm /dev/hd_" -- this will
ensure both drives have things like DMA, etc.
Next -- you didn't say what benchmarks you're using locally.
And as the previous poster said provide "cat /proc/interrupts".

----- Original Message -----
From: "Jimmie Mayfield" <mayfield+usenet@sackheads.org>
To: <linux-kernel@vger.kernel.org>
Sent: Saturday, July 21, 2001 11:33 PM
Subject: Interesting disk throughput performance problem


> Hi.  I'm running into some disk throughput issues that I can't explain.
> Hopefully someone reading this can offer an explanation.
>
> One of my machines is running 2.4.5 and has 2 hard drives: a 7200 rpm
> ATA100 Maxtor and a 5400 rpm ATA33 IBM.  Each drive is a master on its own
> controller (AMI CMD649 as found on the IWill KT266-R).  Both drives
contain
> reiserfs 3.6x filesystems.
>
> By all local benchmarks, the 7200 rpm drive is the faster drive.  But this
> doesn't seem to be the case for large files originating from remote
clients.
> Witness:
>
> My crude test involves scp'ing a 100MB file from another machine on my
home
> network over 100bT ethernet.
>
> 1)  scp to the 5400rpm drive:  roughly 10MB/sec.
> 2)  scp to the 7200rpm drive:  roughly 2MB/sec.
>
> I've tried 'tail' and 'notail' mount options with no change (as expected
since
> this is a single large file).  I suspect that the machine would become
CPU-bound
> somewhere in the 20MB/sec range (see below for my reasoning).
>
> I see the same sort of behavior using Samba though not nearly as
> pronounced (the 5400rpm drive is merely 2x as fast as the 7200rpm drive).
>
> Okay.  Since the test involved 2 separate drives with different
geometries,
> I figured this might be due to physical block location.  Perhaps the file
> is getting allocated to the fastest cylinders on the 5400 rpm drive and
> the slowest cylinders on the 7200 rpm drive.  Or it could be a
fragmentation
> issue.
>
> So I tried the test locally:  with the file stored on the 5400rpm drive,
> scp it to localhost and write it to the 7200rpm drive.  Results were a
little
> below 10MB/sec (CPU near 100% presumably due to encrypting/decrypting on
> the fly).
>
> Any ideas why the 7200rpm drive performs so poorly for remote clients but
> performs wonderfully well when those same operations are performed
locally?
>
> Jimmie
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Interesting disk throughput performance problem

[Jimmie Mayfield]

On Sun, Jul 22, 2001 at 06:44:38AM -0400, Mike Black wrote:
> Not enough info (plenty for guessing though :-)
> 
> First off show "hdparm -i /dev/hd_" and "hdparm /dev/hd_" -- this will
> ensure both drives have things like DMA, etc.
> Next -- you didn't say what benchmarks you're using locally.
> And as the previous poster said provide "cat /proc/interrupts".

/proc/interrupts looks like this:
           CPU0       
  0:   17319250          XT-PIC  timer
  1:      85980          XT-PIC  keyboard
  2:          0          XT-PIC  cascade
  5:    1084144          XT-PIC  ide3, usb-uhci, usb-uhci
  6:        811          XT-PIC  floppy
  9:    1284463          XT-PIC  mga@PCI:1:0:0
 10:     142416          XT-PIC  eth0
 11:    8296678          XT-PIC  eth1, C-Media PCI CM8738
 12:     385915          XT-PIC  PS/2 Mouse
 15:          7          XT-PIC  ide1
NMI:          0 
LOC:   17319087 
ERR:       3022
MIS:          0


I would like to make a correction to my original post.  In that post I said 
that the drives are "masters" on their own controller.  This was false.  
They share a controller (CMD 649) with the Maxtor drive being "master" and 
the IBM drive being "slave".  To test if this was the problem, I reran the
tests (see URL below) with the IBM drive completely disconnected.  I didn't 
notice any difference.

I collected my benchmarks and tests into a simple webpage to avoid cluttering
this list.  Hopefully someone will see something obvious that I've 
misconfigured.

http://sackheads.org/~mayfield/dp.html


Jimmie

-- 
Jimmie Mayfield  
http://www.sackheads.org/mayfield       email: mayfield+usenet@sackheads.org
My mail provider does not welcome UCE -- http://www.sackheads.org/uce

Re: Interesting disk throughput performance problem

[Tim Schmielau]

Did you monitor network traffic for lost packets?

Just another wild guess: Considering the weakness of the VIA chipset,
maybe the NIC cannot do enough DMA during UDMA5 transfers.

Tim

> Hi. I'm running into some disk throughput issues that I can't explain.
> Hopefully someone reading this can offer an explanation.
>
> One of my machines is running 2.4.5 and has 2 hard drives: a 7200 rpm
> ATA100 Maxtor
> and a 5400 rpm ATA33 IBM. Each drive is a master on its own
> controller (AMI CMD649 as found on the IWill KT266-R). Both drives
> contain reiserfs 3.6x filesystems.
>
> By all local benchmarks, the 7200 rpm drive is the faster drive. But
> this doesn't seem to be the case for large files originating from remote
> clients.