software raid does not do parallel reads under 2.4?

software raid does not do parallel reads under 2.4?

[Lehmann]

It seems that the md driver in striped mode does not parallelize any reads
under 2.4. The scenario is that I have 3 disks on different controllers
(different pci slots), so there should not be any ide bus contention.

When I read any single disk (e.g. using hdparm or dd), I get 32MB/s. When
I read two of them at the same time, I get about 28MB/s for each disk.

Under linux-2.2 using md and striping I get about 40-50MB/s, whereas, under
2.4, the same raid gives about 30MB/s.

I then reformatted the raid to have 2MB chunksize. If I look at the disk
LED's while reading from them (e.g. dd if=/dev/md3 bs=1024x1024x8), I see
that each disk is read in turn, while the other two disks are idle.

so it looks as if md under 2.4 only reads disks in turn, which makes
striping useless as a performance tool.

hdparm for all drives looks like this:

cerebro:/tmp# hdparm /dev/hde
/dev/hde:
 multcount    = 16 (on)
 I/O support  =  1 (32-bit)
 unmaskirq    =  1 (on)
 using_dma    =  1 (on)
 keepsettings =  1 (on)
 nowerr       =  0 (off)
 readonly     =  0 (off)
 readahead    =  8 (on)
 geometry     = 23988/16/63, sectors = 156301487, start = 0

two of the disks use a promise ultra tx2 controller, one the older ultra
100, and all run at udma 100.


-- 
      -----==-                                             |
      ----==-- _                                           |
      ---==---(_)__  __ ____  __       Marc Lehmann      +--
      --==---/ / _ \/ // /\ \/ /       pcg@goof.com      |e|
      -=====/_/_//_/\_,_/ /_/\_\       XX11-RIPE         --+
    The choice of a GNU generation                       |
                                                         |

Re: software raid does not do parallel reads under 2.4?

[Neil Brown]

On Thursday August 23, pcg( Marc)@goof(A.).(Lehmann )com wrote:
> It seems that the md driver in striped mode does not parallelize any reads
> under 2.4. The scenario is that I have 3 disks on different controllers
> (different pci slots), so there should not be any ide bus contention.
> 
> When I read any single disk (e.g. using hdparm or dd), I get 32MB/s. When
> I read two of them at the same time, I get about 28MB/s for each disk.
> 
> Under linux-2.2 using md and striping I get about 40-50MB/s, whereas, under
> 2.4, the same raid gives about 30MB/s.
> 
> I then reformatted the raid to have 2MB chunksize. If I look at the disk
> LED's while reading from them (e.g. dd if=/dev/md3 bs=1024x1024x8), I see
> that each disk is read in turn, while the other two disks are idle.
> 
> so it looks as if md under 2.4 only reads disks in turn, which makes
> striping useless as a performance tool.

For raid0, the md driver just redirects requests to the right drive.
It doesn't explicitly serialise or parallelise anything. 2.4 works in
exactly the same was as 2.2.

With a 2MB chunksize, I would expect a linear read to touch just one
drive at a time.
With a 4K chunk size, I suspect that an linear read would read from
all the drives in parallel.
You say that you reformatted to 2MB chunksize.  What did you reformat
from?

NeilBrown

Re: software raid does not do parallel reads under 2.4?

[Wilfried Weissmann]

Neil Brown wrote:
> 
> On Thursday August 23, pcg( Marc)@goof(A.).(Lehmann )com wrote:
> > It seems that the md driver in striped mode does not parallelize any reads
> > under 2.4. The scenario is that I have 3 disks on different controllers
> > (different pci slots), so there should not be any ide bus contention.
> >
> > When I read any single disk (e.g. using hdparm or dd), I get 32MB/s. When
> > I read two of them at the same time, I get about 28MB/s for each disk.
> >
> > Under linux-2.2 using md and striping I get about 40-50MB/s, whereas, under
> > 2.4, the same raid gives about 30MB/s.
> >
> > I then reformatted the raid to have 2MB chunksize. If I look at the disk
> > LED's while reading from them (e.g. dd if=/dev/md3 bs=1024x1024x8), I see
> > that each disk is read in turn, while the other two disks are idle.
> >
> > so it looks as if md under 2.4 only reads disks in turn, which makes
> > striping useless as a performance tool.
> 
> For raid0, the md driver just redirects requests to the right drive.
> It doesn't explicitly serialise or parallelise anything. 2.4 works in
> exactly the same was as 2.2.
> 
> With a 2MB chunksize, I would expect a linear read to touch just one
> drive at a time.
> With a 4K chunk size, I suspect that an linear read would read from
> all the drives in parallel.
> You say that you reformatted to 2MB chunksize.  What did you reformat
> from?
> 
> NeilBrown

I experienced some performance loss when moving from kernel 2.4.4 to
2.4.7-ac3 regarding ide harddisks. I am useing the hpt ataraid driver
which does pretty much the same thing as the md disk striping driver.
I/O speed of the raid volume is about as fast as accessing a single
drive.
There were a lot of ide reports some time ago. Maybe they where problems
with concurrent I/O operations...?

Wilfried

Re: software raid does not do parallel reads under 2.4?

[Daryll Strauss]

On Fri, Aug 24, 2001 at 07:01:40PM +0200, Wilfried Weissmann wrote:

> I experienced some performance loss when moving from kernel 2.4.4 to
> 2.4.7-ac3 regarding ide harddisks. I am useing the hpt ataraid driver
> which does pretty much the same thing as the md disk striping driver.
> I/O speed of the raid volume is about as fast as accessing a single
> drive.
> There were a lot of ide reports some time ago. Maybe they where problems
> with concurrent I/O operations...?

I'm seeing similar behavior with SCSI. I've got two SCSI channels. If I
run two dd's each talking to disks on different channels, I get 2x disk
bandwidth. If I run two dd's talking to disks on the same channel I get
1x disk bandwidth. This is with 2.4.9, but I haven't isolated which
kernel versions cause this.

This system has 8 U160 disks spread over 2 U160 channels and it hasn't
been deployed yet, so if I can help with testing, let me know.

						- |Daryll

Re: software raid does not do parallel reads under 2.4?

[Lehmann]

On Fri, Aug 24, 2001 at 10:14:39AM -0700, Daryll Strauss <daryll@valinux.com> wrote:
> > There were a lot of ide reports some time ago. Maybe they where problems
> > with concurrent I/O operations...?
> 
> I'm seeing similar behavior with SCSI. I've got two SCSI channels. If I
> run two dd's each talking to disks on different channels, I get 2x disk

similar, but quite different: the difefrence is that i get full speed when
accessing drives as /dev/hdx, but slow speed when doping the same opertaion
using md.

-- 
      -----==-                                             |
      ----==-- _                                           |
      ---==---(_)__  __ ____  __       Marc Lehmann      +--
      --==---/ / _ \/ // /\ \/ /       pcg@goof.com      |e|
      -=====/_/_//_/\_,_/ /_/\_\       XX11-RIPE         --+
    The choice of a GNU generation                       |
                                                         |

Re: software raid does not do parallel reads under 2.4?

[Lehmann]

On Fri, Aug 24, 2001 at 08:25:29PM +1000, Neil Brown <neilb@cse.unsw.edu.au> wrote:
> For raid0, the md driver just redirects requests to the right drive.
> It doesn't explicitly serialise or parallelise anything. 2.4 works in
> exactly the same was as 2.2.

then why is 2.2 so much faster?

> With a 2MB chunksize, I would expect a linear read to touch just one
> drive at a time.

why? i read 8mb cvhunks a time (see the dd) so it should read 3x2mb
chunks easily. anyway, i get the same throughput with 64k, 16k etc..
chunksize.

> With a 4K chunk size, I suspect that an linear read would read from
> all the drives in parallel.

maybe, but it's still exactly as fast: 30mb/s, which is slower than one drive
can do.

-- 
      -----==-                                             |
      ----==-- _                                           |
      ---==---(_)__  __ ____  __       Marc Lehmann      +--
      --==---/ / _ \/ // /\ \/ /       pcg@goof.com      |e|
      -=====/_/_//_/\_,_/ /_/\_\       XX11-RIPE         --+
    The choice of a GNU generation                       |
                                                         |