strange performance of raid0

strange performance of raid0

[Keld Jørn Simonsen]

Hi

I have some strange performance results on a raid0
I have 4 IDE disks on two controllers, the one on the 
motherboard of the duron 1 GHz mackine, the other a promise TX2 plus 
SATA + PATA controller. I run kernel 2.4.22

The disks and hdparm -t on each of them

/dev/hdc1 seagate 80 GB                   16 MB/s
/dev/sda1 maxtor sata 200 GB              50 MB/s
/dev/sdb7 maxtor 160 GB                   54 MB/s

The partitions are al about 5 GB each.

If I make a raid0 device of all of them I get a thruput of 45 MB/s
IIf I exclude the hdc1 partition, I get around 75 MB/s.
The system is a little loaded - but that would be normal operating
conditions. CPU is 90 % idle. I have about 100 MB free RAM.

Why is it slower with 3 disks in the raid0 than
2 disks? Why don't I get approx 100 MB/s out of just the two?

Best regards
Keld

Re: strange performance of raid0

[Matt Thrailkill]

The you don't get as good performance because hdc1 is slow enough to
really drag it down.  Think about it, if reads are all going about
sequentially, it has to spend alot of time waiting on that 16mb/s drive
compared to the others.

You probably can't get 100mb/s with the two because that would be pretty
efficient, and there's probably more overhead than that.

Shouldn't the 80gb drive be going faster than 16mb/s though?  Have you
checked hdparm to make sure dma and all the goodies are turned on for
it?

On Sun, 2004-02-15 at 02:04, Keld Jørn Simonsen wrote:
> Hi
> 
> I have some strange performance results on a raid0
> I have 4 IDE disks on two controllers, the one on the 
> motherboard of the duron 1 GHz mackine, the other a promise TX2 plus 
> SATA + PATA controller. I run kernel 2.4.22
> 
> The disks and hdparm -t on each of them
> 
> /dev/hdc1 seagate 80 GB                   16 MB/s
> /dev/sda1 maxtor sata 200 GB              50 MB/s
> /dev/sdb7 maxtor 160 GB                   54 MB/s
> 
> The partitions are al about 5 GB each.
> 
> If I make a raid0 device of all of them I get a thruput of 45 MB/s
> IIf I exclude the hdc1 partition, I get around 75 MB/s.
> The system is a little loaded - but that would be normal operating
> conditions. CPU is 90 % idle. I have about 100 MB free RAM.
> 
> Why is it slower with 3 disks in the raid0 than
> 2 disks? Why don't I get approx 100 MB/s out of just the two?
> 
> Best regards
> Keld
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strange performance of raid0

[Keld Jørn Simonsen]

On Sun, Feb 15, 2004 at 01:54:11AM -0800, Matt Thrailkill wrote:
> The you don't get as good performance because hdc1 is slow enough to
> really drag it down.  Think about it, if reads are all going about
> sequentially, it has to spend alot of time waiting on that 16mb/s drive
> compared to the others.

I would have thought the disks would have worked towards gaining
performance enhancements in some cumulative way, is that not so, in theory?

So why is 3 disks slower than 2?

> You probably can't get 100mb/s with the two because that would be pretty
> efficient, and there's probably more overhead than that.
> 
> Shouldn't the 80gb drive be going faster than 16mb/s though?  Have you
> checked hdparm to make sure dma and all the goodies are turned on for
> it?

I have a 40 GB segate disk also, on the same motherboard IDE
controller, it runs about 40 MB/s. So yes, the hdc1
should go faster, and I have measured something like 40 MB/s
on it in idler times. I think it is because it is running other 
processes. The machine is a ftp mirror and the disk has RedHat
and Fedora ISO images, so it is quite popular.

> On Sun, 2004-02-15 at 02:04, Keld Jørn Simonsen wrote:
> > Hi
> > 
> > I have some strange performance results on a raid0
> > I have 4 IDE disks on two controllers, the one on the 
> > motherboard of the duron 1 GHz mackine, the other a promise TX2 plus 
> > SATA + PATA controller. I run kernel 2.4.22
> > 
> > The disks and hdparm -t on each of them
> > 
> > /dev/hdc1 seagate 80 GB                   16 MB/s
> > /dev/sda1 maxtor sata 200 GB              50 MB/s
> > /dev/sdb7 maxtor 160 GB                   54 MB/s
> > 
> > The partitions are al about 5 GB each.
> > 
> > If I make a raid0 device of all of them I get a thruput of 45 MB/s
> > IIf I exclude the hdc1 partition, I get around 75 MB/s.
> > The system is a little loaded - but that would be normal operating
> > conditions. CPU is 90 % idle. I have about 100 MB free RAM.
> > 
> > Why is it slower with 3 disks in the raid0 than
> > 2 disks? Why don't I get approx 100 MB/s out of just the two?
> > 
> > Best regards
> > Keld
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strange performance of raid0

[Matt Thrailkill]

On Sun, 2004-02-15 at 04:05, Keld Jørn Simonsen wrote:
> I would have thought the disks would have worked towards gaining
> performance enhancements in some cumulative way, is that not so, in theory?
> 
> So why is 3 disks slower than 2?

Well, I'm not a complete guru so maybe I'm wrong, but in a stripe the
point is that you are pressing more platters into simultaneous service. 
There's a certain amount of synchronization because the things your
computer needs at higher levels are spread across multiple disks, so it
wants them at about the same time.  If the first two disks provide the
data quickly and the third doesn't, the system probably will have to
wait for that third one before it can move on.

Thats my suspicion anyways.

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strange performance of raid0

[Keld Jørn Simonsen]

On Sun, Feb 15, 2004 at 01:05:25PM +0100, Keld Jørn Simonsen wrote:
> On Sun, Feb 15, 2004 at 01:54:11AM -0800, Matt Thrailkill wrote:
> > The you don't get as good performance because hdc1 is slow enough to
> > really drag it down.  Think about it, if reads are all going about
> > sequentially, it has to spend alot of time waiting on that 16mb/s drive
> > compared to the others.
> 
> I would have thought the disks would have worked towards gaining
> performance enhancements in some cumulative way, is that not so, in theory?
> 
> So why is 3 disks slower than 2?
> 
> > You probably can't get 100mb/s with the two because that would be pretty
> > efficient, and there's probably more overhead than that.
> > 
> > Shouldn't the 80gb drive be going faster than 16mb/s though?  Have you
> > checked hdparm to make sure dma and all the goodies are turned on for
> > it?
> 
> I have a 40 GB segate disk also, on the same motherboard IDE
> controller, it runs about 40 MB/s. So yes, the hdc1
> should go faster, and I have measured something like 40 MB/s
> on it in idler times. I think it is because it is running other 
> processes. The machine is a ftp mirror and the disk has RedHat
> and Fedora ISO images, so it is quite popular.

Hmm, I have ext3 filesystems, and they are updating the atime in the
inodes. Could that be it? inode flushing obstructing the striping?

Best regards
Keld
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strange performance of raid0

[Mark Hahn]

> > The you don't get as good performance because hdc1 is slow enough to
> > really drag it down.

right.

> I would have thought the disks would have worked towards gaining
> performance enhancements in some cumulative way, is that not so, in theory?

raid0 stripes IO across devices, so more devices help, in general.
but suppose your config stores 1k per disk.  a 3k write would 
touch all three disks.  when would that write complete?  according
to your numbers, hdc1 takes much longer than the other two.
that means that a large upper-level IO request (all three disks)
will complete at about 3x16 MB/s (ignoring overhead, etc).

> So why is 3 disks slower than 2?

because two of the disks are sometimes idle while the slow one completes.
since striping is always parallel across all disks, the system can't 
somehow find other IO to do on the fast disks while waiting for the slow one.

> > You probably can't get 100mb/s with the two because that would be pretty
> > efficient, and there's probably more overhead than that.

100 MB/s is a pretty agressive goal for a duron: the speed of the CPU and
memory bandwidth *DO* have a significant effect on how well raid0 scales,
and where it tops out.  hdparm -t is also quite cpu-inefficient, as
benchmarks go.

> > Shouldn't the 80gb drive be going faster than 16mb/s though?  Have you
> > checked hdparm to make sure dma and all the goodies are turned on for
> > it?

16 MB/s is a clear config or measurement mistake.  16 MB/s is the speed of
disks from 5+ years ago, and >= 80G didn't exist back then.

regards, mark hahn.

Re: strange performance of raid0

[Keld Jørn Simonsen]

On Sun, Feb 15, 2004 at 03:35:32PM -0500, Mark Hahn wrote:
> > > The you don't get as good performance because hdc1 is slow enough to
> > > really drag it down.
> 
> right.
> 
> > I would have thought the disks would have worked towards gaining
> > performance enhancements in some cumulative way, is that not so, in theory?
> 
> raid0 stripes IO across devices, so more devices help, in general.
> but suppose your config stores 1k per disk.  a 3k write would 
> touch all three disks.  when would that write complete?  according
> to your numbers, hdc1 takes much longer than the other two.
> that means that a large upper-level IO request (all three disks)
> will complete at about 3x16 MB/s (ignoring overhead, etc).

So max thruput in sequential read would never be faster than the slowest
disk times the total number of drives. Sounds right there, then
3 * 16 is 48, close to the observed value of 45.

> > > You probably can't get 100mb/s with the two because that would be pretty
> > > efficient, and there's probably more overhead than that.
> 
> 100 MB/s is a pretty agressive goal for a duron: the speed of the CPU and
> memory bandwidth *DO* have a significant effect on how well raid0 scales,
> and where it tops out.  hdparm -t is also quite cpu-inefficient, as
> benchmarks go.

Why should the duron CPU speed be a problem? Or the RAM?

> > > Shouldn't the 80gb drive be going faster than 16mb/s though?  Have you
> > > checked hdparm to make sure dma and all the goodies are turned on for
> > > it?
> 
> 16 MB/s is a clear config or measurement mistake.  16 MB/s is the speed of
> disks from 5+ years ago, and >= 80G didn't exist back then.

yeah, something is wrong there. I observed a going of about 40 MB/s
a couple of days ago on the same disk.

keld

Re: strange performance of raid0

[Keld Jørn Simonsen]

On Sun, Feb 15, 2004 at 11:04:32AM +0100, Keld Jørn Simonsen wrote:
> Hi
> 
> I have some strange performance results on a raid0
> I have 4 IDE disks on two controllers, the one on the 
> motherboard of the duron 1 GHz mackine, the other a promise TX2 plus 
> SATA + PATA controller. I run kernel 2.4.22
> 
> The disks and hdparm -t on each of them
> 
> /dev/hdc1 seagate 80 GB                   16 MB/s
> /dev/sda1 maxtor sata 200 GB              50 MB/s
> /dev/sdb7 maxtor 160 GB                   54 MB/s

Changed my hdc1 disk with my hda1 disk, a seagate 40 GB
disk running at something like 40 GB/s.

On a cat Mandrake...iso >/dev/null  681 MB, I now got 102 MB/s
on the raid0. I am happy.

Keld
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strange performance of raid0

[Gregory Leblanc]

Keld Jørn Simonsen wrote:

> Hi
> 
> I have some strange performance results on a raid0
> I have 4 IDE disks on two controllers, the one on the 
> motherboard of the duron 1 GHz mackine, the other a promise TX2 plus 
> SATA + PATA controller. I run kernel 2.4.22
> 
> The disks and hdparm -t on each of them
> 
> /dev/hdc1 seagate 80 GB                   16 MB/s
> /dev/sda1 maxtor sata 200 GB              50 MB/s
> /dev/sdb7 maxtor 160 GB                   54 MB/s

Well, if you're lucky, these might be useful as comparative numbers 
between the different drives on the same system.  Just as likely not, 
though, hdparm rather sucks as a benchmark.

> The partitions are al about 5 GB each.
> 
> If I make a raid0 device of all of them I get a thruput of 45 MB/s
> IIf I exclude the hdc1 partition, I get around 75 MB/s.
> The system is a little loaded - but that would be normal operating
> conditions. CPU is 90 % idle. I have about 100 MB free RAM.

Let's assume that the above numbers have a basis in reality.  :)  If 
you've got disks with widely varying speeds, then the best performance 
can often be hand from setting up a linear RAID volume, rather than a 
RAID0.  RAID 0 is really designed to have matching disks, as it 
distributes data evenly across them.  With Linear, and ext2 (erm, I'm 
assuming 3 as well, I haven't heard anything different), you can 
sometimes get better performance with smaller writes, because ext2 
"scatters" data around the filesystem, in order to avoid fragmentation. 
  HTH,
	Greg
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strange performance of raid0

[Keld Jørn Simonsen]

On Tue, Feb 17, 2004 at 09:50:21AM -0800, Gregory Leblanc wrote:
> Keld Jørn Simonsen wrote:
> 
> >Hi
> >
> >I have some strange performance results on a raid0
> >I have 4 IDE disks on two controllers, the one on the 
> >motherboard of the duron 1 GHz mackine, the other a promise TX2 plus 
> >SATA + PATA controller. I run kernel 2.4.22
> >
> >The disks and hdparm -t on each of them
> >
> >/dev/hdc1 seagate 80 GB                   16 MB/s
> >/dev/sda1 maxtor sata 200 GB              50 MB/s
> >/dev/sdb7 maxtor 160 GB                   54 MB/s
> 
> Well, if you're lucky, these might be useful as comparative numbers 
> between the different drives on the same system.  Just as likely not, 
> though, hdparm rather sucks as a benchmark.

Yes, I only use them as crude benchmark measuers, but they seem
indicative.

> >The partitions are al about 5 GB each.
> >
> >If I make a raid0 device of all of them I get a thruput of 45 MB/s
> >IIf I exclude the hdc1 partition, I get around 75 MB/s.
> >The system is a little loaded - but that would be normal operating
> >conditions. CPU is 90 % idle. I have about 100 MB free RAM.
> 
> Let's assume that the above numbers have a basis in reality.  :)  If 
> you've got disks with widely varying speeds, then the best performance 
> can often be hand from setting up a linear RAID volume, rather than a 
> RAID0.  RAID 0 is really designed to have matching disks, as it 
> distributes data evenly across them.  With Linear, and ext2 (erm, I'm 
> assuming 3 as well, I haven't heard anything different), you can 
> sometimes get better performance with smaller writes, because ext2 
> "scatters" data around the filesystem, in order to avoid fragmentation. 

Yes, that would be an idea. Anyway I replaced the hdc1 with hda1 on
a seagate 40 GB disk, which hdparm said could do about 40 MB/s,
and then - when I was lucky, I could actually get about 120 MB/s
thruput on a "cat file >/dev/null", 681 MB in 5.63 secs. 
This on a mildly loaded production system. Not bad!
Then I am beginning to hit the 1 Gbit/s limit on the PCI bus.

Best regards
Keld
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html