Re: Odd (slow) RAID performance

[Bill Davidsen <davidsen@tmr.com>]

Raz Ben-Jehuda(caro) wrote:
> On 12/12/06, Bill Davidsen <davidsen@tmr.com> wrote:
>> Neil Brown wrote:
>> > On Friday December 8, neilb@suse.de wrote:
>> >
>> >> I have measured very slow write throughput for raid5 as well, though
>> >> 2.6.18 does seem to have the same problem.  I'll double check and 
>> do a
>> >> git bisect and see what I can come up with.
>> >>
>> >
>> > Correction... it isn't 2.6.18 that fixes the problem.  It is compiling
>> > without LOCKDEP or PROVE_LOCKING.  I remove those and suddenly a
>> > 3 drive raid5 is faster than a single drive rather than much slower.
>> >
>> > Bill:  Do you have LOCKDEP or PROVE_LOCKING enabled in your .config ??
>>
>> YES and NO respectively. I did try increasing the stripe_cache_size and
>> got better but not anywhere near max performance, perhaps the
>> PROVE_LOCKING is still at fault, although performance of RAID-0 is as
>> expected, so I'm dubious. In any case, by pushing the size from 256 to
>> 1024, 4096, and finally 10240 I was able to raise the speed to 82MB/s,
>> which is right at the edge of what I need. I want to read the doc on
>> stripe_cache_size before going huge, if that's K 10MB is a LOT of cache
>> when 256 works perfectly in RAID-0.
>>
>> I noted that the performance really was bad using 2k write, before
>> increasing the stripe_cache, I will repeat that after doing some other
>> "real work" things.
>>
>> Any additional input appreciated, I would expect the speed to be (Ndisk
>> - 1)*SingleDiskSpeed without a huge buffer, so the fact that it isn't
>> makes me suspect there's unintended serialization or buffering, even
>> when not need (and NOT wanted).
>>
>> Thanks for the feedback, I'm updating the files as I type.
>> http://www.tmr.com/~davidsen/RAID_speed
>> http://www.tmr.com/~davidsen/FC6-config
>>
>> -- 
>> bill davidsen <davidsen@tmr.com>
>>  CTO TMR Associates, Inc
>>  Doing interesting things with small computers since 1979
>
> Bill helllo
> I have been working on raid5 performance write throughout.
> The whole idea is the access pattern.
> One should  buffers with respect to the size of stripe.
> this way you will be able to eiliminate the undesired reads.
> By accessing it correctly I have managed reach a write
> throughout with respect to the number of disks in the raid.
>
>
I'm doing the tests writing 2GB of data to the raw array, in 1MB writes. 
The array is RAID-5 with 256 chunk size. I wouldn't really expect any 
reads, unless I totally misunderstand how all those numbers work 
together. I was really trying to avoid any issues there.However, the 
only other size I have tried was 2K blocks, so I can try other sizes. I 
have a hard time picturing why smaller sizes would be better, but that's 
what testing is for.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979