sw raid0 read bottleneck

sw raid0 read bottleneck

[Tomka Gergely]

Hi!

I am currently testing 3ware raid cards. Now i have 15 disks, and on these 
a swraid0. The write speed seems good (700 MBps), but the read performance 
only 350 MBps. Another problem when i try to read with two process, then 
the _sum_ of the read speeds fall back to 200 MBps. So there is a 
bottleneck, or something i need to know, but i dont have ideas. 

The details:

/dev/md0:
        Version : 00.90.03
  Creation Time : Tue Mar 13 16:57:32 2007
     Raid Level : raid0
     Array Size : 7325797440 (6986.43 GiB 7501.62 GB)
   Raid Devices : 15
  Total Devices : 15
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Tue Mar 13 16:57:32 2007
          State : clean
 Active Devices : 15
Working Devices : 15
 Failed Devices : 0
  Spare Devices : 0

     Chunk Size : 64K

# uname -a
Linux ursula 2.6.18-4-686-bigmem #1 SMP Wed Feb 21 17:30:22 UTC 2007 i686 
GNU/Linux

# xfs_info /mnt/
meta-data=/dev/md0               isize=256    agcount=32, agsize=57232784 
blks
         =                       sectsz=512   attr=0
data     =                       bsize=4096   blocks=1831449088, 
imaxpct=25
         =                       sunit=16     swidth=240 blks, unwritten=1
naming   =version 2              bsize=4096  
log      =internal               bsize=4096   blocks=32768, version=1
         =                       sectsz=512   sunit=0 blks
realtime =none                   extsz=983040 blocks=0, rtextents=0

(all software Debian Etch)

four Intel(R) Xeon(TM) CPU 3.00GHz

two 3ware 9590SE-8ML on PCIe

Intel Corporation 5000P Chipset


-- 
Tomka Gergely, gergely@tomka.hu

Re: sw raid0 read bottleneck

[Justin Piszcz]

On Tue, 13 Mar 2007, Tomka Gergely wrote:

> Hi!
>
> I am currently testing 3ware raid cards. Now i have 15 disks, and on these
> a swraid0. The write speed seems good (700 MBps), but the read performance
> only 350 MBps. Another problem when i try to read with two process, then
> the _sum_ of the read speeds fall back to 200 MBps. So there is a
> bottleneck, or something i need to know, but i dont have ideas.
>
> The details:
>
> /dev/md0:
>        Version : 00.90.03
>  Creation Time : Tue Mar 13 16:57:32 2007
>     Raid Level : raid0
>     Array Size : 7325797440 (6986.43 GiB 7501.62 GB)
>   Raid Devices : 15
>  Total Devices : 15
> Preferred Minor : 0
>    Persistence : Superblock is persistent
>
>    Update Time : Tue Mar 13 16:57:32 2007
>          State : clean
> Active Devices : 15
> Working Devices : 15
> Failed Devices : 0
>  Spare Devices : 0
>
>     Chunk Size : 64K
>
> # uname -a
> Linux ursula 2.6.18-4-686-bigmem #1 SMP Wed Feb 21 17:30:22 UTC 2007 i686
> GNU/Linux
>
> # xfs_info /mnt/
> meta-data=/dev/md0               isize=256    agcount=32, agsize=57232784
> blks
>         =                       sectsz=512   attr=0
> data     =                       bsize=4096   blocks=1831449088,
> imaxpct=25
>         =                       sunit=16     swidth=240 blks, unwritten=1
> naming   =version 2              bsize=4096
> log      =internal               bsize=4096   blocks=32768, version=1
>         =                       sectsz=512   sunit=0 blks
> realtime =none                   extsz=983040 blocks=0, rtextents=0
>
> (all software Debian Etch)
>
> four Intel(R) Xeon(TM) CPU 3.00GHz
>
> two 3ware 9590SE-8ML on PCIe
>
> Intel Corporation 5000P Chipset
>
>
> -- 
> Tomka Gergely, gergely@tomka.hu
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Have you tried increasing your readahead values for the md device?

Re: sw raid0 read bottleneck

[Tomka Gergely]

On Tue, 13 Mar 2007, Justin Piszcz wrote:

> Have you tried increasing your readahead values for the md device?

Yes. No real change. According to my humble mental image, readahead not a 
too useful thing, when we read 1-4 thread with sdd. The io subsystem 
already reading with the possible maximum speed, so don't have time to 
read ahead. Correct me, if i wrong. 

-- 
Tomka Gergely, gergely@tomka.hu

Re: sw raid0 read bottleneck

[Tomka Gergely]

On Tue, 13 Mar 2007, Tomka Gergely wrote:

> On Tue, 13 Mar 2007, Justin Piszcz wrote:
> 
> > Have you tried increasing your readahead values for the md device?
> 
> Yes. No real change. According to my humble mental image, readahead not a 
> too useful thing, when we read 1-4 thread with sdd. The io subsystem 
> already reading with the possible maximum speed, so don't have time to 
> read ahead. Correct me, if i wrong. 

I was wrong, readahead can speed things up, to 450 MBps.

-- 
Tomka Gergely, gergely@tomka.hu

Re: sw raid0 read bottleneck

[Justin Piszcz]

Nice.

On Tue, 13 Mar 2007, Tomka Gergely wrote:

> On Tue, 13 Mar 2007, Tomka Gergely wrote:
>
>> On Tue, 13 Mar 2007, Justin Piszcz wrote:
>>
>>> Have you tried increasing your readahead values for the md device?
>>
>> Yes. No real change. According to my humble mental image, readahead not a
>> too useful thing, when we read 1-4 thread with sdd. The io subsystem
>> already reading with the possible maximum speed, so don't have time to
>> read ahead. Correct me, if i wrong.
>
> I was wrong, readahead can speed things up, to 450 MBps.
>
> -- 
> Tomka Gergely, gergely@tomka.hu
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: sw raid0 read bottleneck

[Neil Brown]

On Tuesday March 13, gergely@tomka.hu wrote:
> On Tue, 13 Mar 2007, Tomka Gergely wrote:
> 
> > On Tue, 13 Mar 2007, Justin Piszcz wrote:
> > 
> > > Have you tried increasing your readahead values for the md device?
> > 
> > Yes. No real change. According to my humble mental image, readahead not a 
> > too useful thing, when we read 1-4 thread with sdd. The io subsystem 
> > already reading with the possible maximum speed, so don't have time to 
> > read ahead. Correct me, if i wrong. 
> 
> I was wrong, readahead can speed things up, to 450 MBps.

Can you tell use what read-ahead size you needed?

15 drives and 64K chunks gives 960K per stripe.
The raid0 code should set the read-ahead to twice that: 1920K
which I would have thought would be enough, but apparently not.

Thanks,
NeilBrown

Re: sw raid0 read bottleneck

[Tomka Gergely]

On Wed, 14 Mar 2007, Neil Brown wrote:

> On Tuesday March 13, gergely@tomka.hu wrote:
> > On Tue, 13 Mar 2007, Tomka Gergely wrote:
> > 
> > > On Tue, 13 Mar 2007, Justin Piszcz wrote:
> > > 
> > > > Have you tried increasing your readahead values for the md device?
> > > 
> > > Yes. No real change. According to my humble mental image, readahead not a 
> > > too useful thing, when we read 1-4 thread with sdd. The io subsystem 
> > > already reading with the possible maximum speed, so don't have time to 
> > > read ahead. Correct me, if i wrong. 
> > 
> > I was wrong, readahead can speed things up, to 450 MBps.
> 
> Can you tell use what read-ahead size you needed?
> 
> 15 drives and 64K chunks gives 960K per stripe.
> The raid0 code should set the read-ahead to twice that: 1920K
> which I would have thought would be enough, but apparently not.

blockdev --setra 262144 /dev/md0 gives me 650+ MB/s with 4 threads 
(paralell running sdd). Lower values give lower speeds, greater values not 
giving higher speeds.

-- 
Tomka Gergely, gergely@tomka.hu

Re: sw raid0 read bottleneck

[Mark Hahn]

> 15 drives and 64K chunks gives 960K per stripe.
> The raid0 code should set the read-ahead to twice that: 1920K
> which I would have thought would be enough, but apparently not.

choosing the RA size should depend in some way on speed, shouldn't it?
after all, the goal is to have enough reads queued to avoid a "stall"
while un-read-ahead-ed sectors pass under the head.  for a typical
60 MB/s, 7200 rpm disk, a track is .5 MB.  the heuristic above only reads
ahead .12 MB...

regards, mark hahn.

Re: sw raid0 read bottleneck

[Bill Davidsen]

Mark Hahn wrote:
>> 15 drives and 64K chunks gives 960K per stripe.
>> The raid0 code should set the read-ahead to twice that: 1920K
>> which I would have thought would be enough, but apparently not.
>
> choosing the RA size should depend in some way on speed, shouldn't it?
> after all, the goal is to have enough reads queued to avoid a "stall"
> while un-read-ahead-ed sectors pass under the head.  for a typical
> 60 MB/s, 7200 rpm disk, a track is .5 MB.  the heuristic above only reads
> ahead .12 MB... 

While I'm sure you realize it, some reader won't, so I'll point out that 
a cylinder is all the tracks which can be read without a seek, and 
cylinder size is track size times number of data heads. If you want max 
speed you can read that much off a drive at a time...

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: sw raid0 read bottleneck

[Mark Hahn]

>>> 15 drives and 64K chunks gives 960K per stripe.
>>> The raid0 code should set the read-ahead to twice that: 1920K
>>> which I would have thought would be enough, but apparently not.
>> 
>> choosing the RA size should depend in some way on speed, shouldn't it?
>> after all, the goal is to have enough reads queued to avoid a "stall"
>> while un-read-ahead-ed sectors pass under the head.  for a typical
>> 60 MB/s, 7200 rpm disk, a track is .5 MB.  the heuristic above only reads
>> ahead .12 MB... 
>
> While I'm sure you realize it, some reader won't, so I'll point out that a 
> cylinder is all the tracks which can be read without a seek, and cylinder 
> size is track size times number of data heads. If you want max speed you can 
> read that much off a drive at a time...

hmm.  I certainly agree with your terminology, but afaikt, disks only read
from a head at a time.  the last time I saw a vendor quoting head-switch
times, they were within a factor of two one-track seek times, which explains
why a same-family disk with more heads/surfaces shows only slight speedup.

regards, mark hahn.

Re: sw raid0 read bottleneck

[Bill Davidsen]

Mark Hahn wrote:
>>>> 15 drives and 64K chunks gives 960K per stripe.
>>>> The raid0 code should set the read-ahead to twice that: 1920K
>>>> which I would have thought would be enough, but apparently not.
>>>
>>> choosing the RA size should depend in some way on speed, shouldn't it?
>>> after all, the goal is to have enough reads queued to avoid a "stall"
>>> while un-read-ahead-ed sectors pass under the head.  for a typical
>>> 60 MB/s, 7200 rpm disk, a track is .5 MB.  the heuristic above only 
>>> reads
>>> ahead .12 MB... 
>>
>> While I'm sure you realize it, some reader won't, so I'll point out 
>> that a cylinder is all the tracks which can be read without a seek, 
>> and cylinder size is track size times number of data heads. If you 
>> want max speed you can read that much off a drive at a time...
>
> hmm.  I certainly agree with your terminology, but afaikt, disks only 
> read
> from a head at a time.  the last time I saw a vendor quoting head-switch
> times, they were within a factor of two one-track seek times, which 
> explains
> why a same-family disk with more heads/surfaces shows only slight 
> speedup.
Agreed. The only real gain is avoiding intervening seeks. Makes a case 
for using cylinder size as chunk size? Maybe. Makes a case for being 
sure your readahead is large enough, though, for some sensible value of 
"enough."

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979