Raid-0 speed

Raid-0 speed

[Brad Campbell]

G'day all,


As you would see I have a problem with libata drivers and raid-0.
In the process of debugging this I have been doing some speed/throughput tests.

On my machine, running 2 instances of dd in 2 terminals
dd if=/dev/sdk of=/dev/null &
dd if=/dev/sdl of=/dev/null

I see over 100MB/s read from VMSTAT.
If I pass the dd via pipebench to /dev/null I see between 50-60MB/s read speed from each drive 
simultaneously.

With the raid-0 running, doing a dd if=/dev/md1 of=/dev/null I get about 55MB/s read speed.

I can't quite figure out why the md is so slow. Given its using streaming reads with no seeking it 
should be over 100MB/s.

Raid-0 created with
mdadm --create --chunk=128 --level=0 --raid-devices=2 /dev/md1 
/dev/scsi/host12/bus0/target0/lun0/disc /dev/scsi/host13/bus0/target0/lun0/disc

Am I doing something obviously silly?

Also, doing a dd from 4 drives maxes out my read speed as per mdstat at 160MB/s (How? PCI is 133MB/s 
and according to lspci all my controllers are on the same bus). Does vmstat lie?

Regards,
Brad

Re: Raid-0 speed

[Daniel Pittman]

On 25 Aug 2004, Brad Campbell wrote:
> As you would see I have a problem with libata drivers and raid-0.
> In the process of debugging this I have been doing some speed/throughput
> 	tests.
>
> On my machine, running 2 instances of dd in 2 terminals
> dd if=/dev/sdk of=/dev/null &
> dd if=/dev/sdl of=/dev/null
>
> I see over 100MB/s read from VMSTAT.
> If I pass the dd via pipebench to /dev/null I see between 50-60MB/s read speed
> 	from each drive
> simultaneously.
>
> With the raid-0 running, doing a dd if=/dev/md1 of=/dev/null I get about
> 	55MB/s read speed.
>
> I can't quite figure out why the md is so slow. Given its using
> streaming reads with no seeking it should be over 100MB/s.

Your chunk size is way to low to give good streaming reads.  The kernel
needs to switch between devices every 128 kilobytes, while is rather
low.

Try slowly pushing the chunk size to 16384 and see when your performance
gets back up to your expected level.

     Daniel
-- 
The propagandist's purpose is to make one set of people forget
that certain other sets of people are human.
        -- Aldous Huxley

Re: Raid-0 speed

[Brad Campbell]

Daniel Pittman wrote:

>>With the raid-0 running, doing a dd if=/dev/md1 of=/dev/null I get about
>>	55MB/s read speed.
>>
>>I can't quite figure out why the md is so slow. Given its using
>>streaming reads with no seeking it should be over 100MB/s.
> 
> 
> Your chunk size is way to low to give good streaming reads.  The kernel
> needs to switch between devices every 128 kilobytes, while is rather
> low.
> 
> Try slowly pushing the chunk size to 16384 and see when your performance
> gets back up to your expected level.

Nope.. Any chunk size > 4096 gives me
too big chunk_size: 8388608 > 4194303.

And the performance difference between chunk=16 and chunk=4096 is less than 5MB/s

--chunk=16 gives me   48.43 MB/s
--chunk=4096 gives me 51.52 MB/s

Regards,
Brad

RE: Raid-0 speed

[Guy]

Based on the error message, 4096 should also fail.

Guy

-----Original Message-----
From: linux-raid-owner@vger.kernel.org
[mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Brad Campbell
Sent: Thursday, August 26, 2004 5:55 AM
To: Daniel Pittman
Cc: linux-raid@vger.kernel.org
Subject: Re: Raid-0 speed

Daniel Pittman wrote:

>>With the raid-0 running, doing a dd if=/dev/md1 of=/dev/null I get about
>>	55MB/s read speed.
>>
>>I can't quite figure out why the md is so slow. Given its using
>>streaming reads with no seeking it should be over 100MB/s.
> 
> 
> Your chunk size is way to low to give good streaming reads.  The kernel
> needs to switch between devices every 128 kilobytes, while is rather
> low.
> 
> Try slowly pushing the chunk size to 16384 and see when your performance
> gets back up to your expected level.

Nope.. Any chunk size > 4096 gives me
too big chunk_size: 8388608 > 4194303.

And the performance difference between chunk=16 and chunk=4096 is less than
5MB/s

--chunk=16 gives me   48.43 MB/s
--chunk=4096 gives me 51.52 MB/s

Regards,
Brad
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Raid-0 speed

[Brad Campbell]

Guy wrote:
> Based on the error message, 4096 should also fail.

You would think that wouldn't you. It does work though.

Regards,
Brad

Re: Raid-0 speed

[Vladimir I. Umnov]

> Also, doing a dd from 4 drives maxes out my read speed as per mdstat
> at 160MB/s (How? PCI is 133MB/s and according to lspci all my
> controllers are on the same bus). Does vmstat lie?
NO! PCI 2.3 can about 512 MB/s

Re: Raid-0 speed

[Brad Campbell]

Vladimir I. Umnov wrote:
>>Also, doing a dd from 4 drives maxes out my read speed as per mdstat
>>at 160MB/s (How? PCI is 133MB/s and according to lspci all my
>>controllers are on the same bus). Does vmstat lie?
> 
> NO! PCI 2.3 can about 512 MB/s

Ok then. Umm so for a 32 bit bus at 33Mhz I'm maxing out my PCI bandwidth.
An Asus A7V600 motherboard does not do PCI 2.3

RE: Raid-0 speed

[Guy]

The PCI bus vs slot confuses me.

My 32 bit, 33 Mhz cards can process 132 Mbytes/sec.
As I understand it, more than 1 PCI slot shares a single PCI bus.
How fast is the PCI bus?

This is from "lspci":
00:00.0 Host bridge: Intel Corp. 440BX/ZX/DX - 82443BX/ZX/DX Host bridge
(rev 03)
00:01.0 PCI bridge: Intel Corp. 440BX/ZX/DX - 82443BX/ZX/DX AGP bridge (rev
03)
00:07.0 ISA bridge: Intel Corp. 82371AB/EB/MB PIIX4 ISA (rev 02)
00:07.1 IDE interface: Intel Corp. 82371AB/EB/MB PIIX4 IDE (rev 01)
00:07.2 USB Controller: Intel Corp. 82371AB/EB/MB PIIX4 USB (rev 01)
00:07.3 Bridge: Intel Corp. 82371AB/EB/MB PIIX4 ACPI (rev 02)
00:0d.0 SCSI storage controller: Adaptec AHA-2940U2/U2W
00:0e.0 SCSI storage controller: Adaptec AHA-2940U2/U2W
00:11.0 Ethernet controller: 3Com Corporation 3c905B 100BaseTX [Cyclone]
00:13.0 PCI bridge: Digital Equipment Corporation DECchip 21152 (rev 03)
01:00.0 VGA compatible controller: Texas Instruments TVP4020 [Permedia 2]
(rev 01)
02:0a.0 SCSI storage controller: Adaptec AHA-2940U2/U2W / 7890/7891
02:0e.0 SCSI storage controller: Adaptec AIC-7880U (rev 01)

Do I have 3 PCI buses?

Thanks,
Guy

-----Original Message-----
From: linux-raid-owner@vger.kernel.org
[mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Vladimir I. Umnov
Sent: Wednesday, August 25, 2004 1:12 PM
To: linux-raid@vger.kernel.org
Subject: Re: Raid-0 speed

> Also, doing a dd from 4 drives maxes out my read speed as per mdstat
> at 160MB/s (How? PCI is 133MB/s and according to lspci all my
> controllers are on the same bus). Does vmstat lie?
NO! PCI 2.3 can about 512 MB/s


-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Raid-0 speed

[Neil Brown]

On Friday August 27, bugzilla@watkins-home.com wrote:
> The PCI bus vs slot confuses me.
> 
> My 32 bit, 33 Mhz cards can process 132 Mbytes/sec.
> As I understand it, more than 1 PCI slot shares a single PCI bus.
> How fast is the PCI bus?

Only one card can be active on any one buss at any one time.
However, a buss can (I believe) handle different speed cards and will
adjust it's speed to match the features of the card.

> 
> This is from "lspci":
> 00:00.0 Host bridge: Intel Corp. 440BX/ZX/DX - 82443BX/ZX/DX Host bridge
> (rev 03)
> 00:01.0 PCI bridge: Intel Corp. 440BX/ZX/DX - 82443BX/ZX/DX AGP bridge (rev
> 03)
> 00:07.0 ISA bridge: Intel Corp. 82371AB/EB/MB PIIX4 ISA (rev 02)
> 00:07.1 IDE interface: Intel Corp. 82371AB/EB/MB PIIX4 IDE (rev 01)
> 00:07.2 USB Controller: Intel Corp. 82371AB/EB/MB PIIX4 USB (rev 01)
> 00:07.3 Bridge: Intel Corp. 82371AB/EB/MB PIIX4 ACPI (rev 02)
> 00:0d.0 SCSI storage controller: Adaptec AHA-2940U2/U2W
> 00:0e.0 SCSI storage controller: Adaptec AHA-2940U2/U2W
> 00:11.0 Ethernet controller: 3Com Corporation 3c905B 100BaseTX [Cyclone]
> 00:13.0 PCI bridge: Digital Equipment Corporation DECchip 21152 (rev 03)
> 01:00.0 VGA compatible controller: Texas Instruments TVP4020 [Permedia 2]
> (rev 01)
> 02:0a.0 SCSI storage controller: Adaptec AHA-2940U2/U2W / 7890/7891
> 02:0e.0 SCSI storage controller: Adaptec AIC-7880U (rev 01)
> 
> Do I have 3 PCI buses?

Yes.  One with IDE/USB/scsi/ethernet and a couple of bridges
One for the VGA controller (possibly AGP rather than PCI)
One for two more SCSI controllers.

NeilBrown

RE: Raid-0 speed

[Guy]

Cool, thanks.

These 2 cards have the most usage:
00:0d.0 SCSI storage controller: Adaptec AHA-2940U2/U2W
00:0e.0 SCSI storage controller: Adaptec AHA-2940U2/U2W
7 disks each.

I will move 1 of my SCSI cards to test the other PCI slots.  Maybe I can
split them on to 2 different buses.

Not that I need any more speed. :)
Just want to learn what I can.
The only time these SCSI cards are working hard is during a re-build of my
monster 14 disk, 200Gig RAID5.  Stop laughing!

Thanks,
Guy

-----Original Message-----
From: linux-raid-owner@vger.kernel.org
[mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Neil Brown
Sent: Friday, August 27, 2004 8:05 PM
To: Guy
Cc: linux-raid@vger.kernel.org
Subject: RE: Raid-0 speed

On Friday August 27, bugzilla@watkins-home.com wrote:
> The PCI bus vs slot confuses me.
> 
> My 32 bit, 33 Mhz cards can process 132 Mbytes/sec.
> As I understand it, more than 1 PCI slot shares a single PCI bus.
> How fast is the PCI bus?

Only one card can be active on any one buss at any one time.
However, a buss can (I believe) handle different speed cards and will
adjust it's speed to match the features of the card.

> 
> This is from "lspci":
> 00:00.0 Host bridge: Intel Corp. 440BX/ZX/DX - 82443BX/ZX/DX Host bridge
> (rev 03)
> 00:01.0 PCI bridge: Intel Corp. 440BX/ZX/DX - 82443BX/ZX/DX AGP bridge
(rev
> 03)
> 00:07.0 ISA bridge: Intel Corp. 82371AB/EB/MB PIIX4 ISA (rev 02)
> 00:07.1 IDE interface: Intel Corp. 82371AB/EB/MB PIIX4 IDE (rev 01)
> 00:07.2 USB Controller: Intel Corp. 82371AB/EB/MB PIIX4 USB (rev 01)
> 00:07.3 Bridge: Intel Corp. 82371AB/EB/MB PIIX4 ACPI (rev 02)
> 00:0d.0 SCSI storage controller: Adaptec AHA-2940U2/U2W
> 00:0e.0 SCSI storage controller: Adaptec AHA-2940U2/U2W
> 00:11.0 Ethernet controller: 3Com Corporation 3c905B 100BaseTX [Cyclone]
> 00:13.0 PCI bridge: Digital Equipment Corporation DECchip 21152 (rev 03)
> 01:00.0 VGA compatible controller: Texas Instruments TVP4020 [Permedia 2]
> (rev 01)
> 02:0a.0 SCSI storage controller: Adaptec AHA-2940U2/U2W / 7890/7891
> 02:0e.0 SCSI storage controller: Adaptec AIC-7880U (rev 01)
> 
> Do I have 3 PCI buses?

Yes.  One with IDE/USB/scsi/ethernet and a couple of bridges
One for the VGA controller (possibly AGP rather than PCI)
One for two more SCSI controllers.

NeilBrown
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Raid-0 speed

[Mikael Abrahamsson]

On Fri, 27 Aug 2004, Guy wrote:

> Do I have 3 PCI buses?

"cat /proc/pci" will tell you if you have multiple busses.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se