Re: Performance Characteristics of All Linux RAIDs (mdadm/bonnie++)

Re: Performance Characteristics of All Linux RAIDs (mdadm/bonnie++)

[Justin Piszcz]

On Wed, 28 May 2008, Peter Rabbitson wrote:

> Justin Piszcz wrote:
>> Hardware:
>> 
>> 1. Utilized (6) 400 gigabyte sata hard drives.
>> 2. Everything is on PCI-e (965 chipset & a 2port sata card)
>> 
>> Used the following 'optimizations' for all tests.
>> 
>> # Set read-ahead.
>> echo "Setting read-ahead to 64 MiB for /dev/md3"
>> blockdev --setra 65536 /dev/md3
>
> That's actually 65k x 512byte blocks so 32MiB
Ah whoops, thanks!

> Results are meaningless without a crucial detail - what was the chunk size 
> used during array creation time? Otherwise interesting test :)
Indeed, the chunk size used was 256 KiB for all tests.

Justin.

Re: Performance Characteristics of All Linux RAIDs (mdadm/bonnie++)

[Chris Snook]

Justin Piszcz wrote:
> Hardware:
> 
> 1. Utilized (6) 400 gigabyte sata hard drives.
> 2. Everything is on PCI-e (965 chipset & a 2port sata card)
> 
> Used the following 'optimizations' for all tests.
> 
> # Set read-ahead.
> echo "Setting read-ahead to 64 MiB for /dev/md3"
> blockdev --setra 65536 /dev/md3
> 
> # Set stripe-cache_size for RAID5.
> echo "Setting stripe_cache_size to 16 MiB for /dev/md3"
> echo 16384 > /sys/block/md3/md/stripe_cache_size
> 
> # Disable NCQ on all disks.
> echo "Disabling NCQ on all disks..."
> for i in $DISKS
> do
>   echo "Disabling NCQ on $i"
>   echo 1 > /sys/block/"$i"/device/queue_depth
> done

Given that one of the greatest benefits of NCQ/TCQ is with parity RAID, 
I'd be fascinated to see how enabling NCQ changes your results.  Of 
course, you'd want to use a single SATA controller with a known good NCQ 
implementation, and hard drives known to not do stupid things like 
disable readahead when NCQ is enabled.

-- Chris

Re: Performance Characteristics of All Linux RAIDs (mdadm/bonnie++)

[Justin Piszcz]

On Wed, 28 May 2008, Chris Snook wrote:

> Justin Piszcz wrote:
>> Hardware:
>> 
>> 1. Utilized (6) 400 gigabyte sata hard drives.
>> 2. Everything is on PCI-e (965 chipset & a 2port sata card)
>> 
>> Used the following 'optimizations' for all tests.
>> 
>> # Set read-ahead.
>> echo "Setting read-ahead to 64 MiB for /dev/md3"
>> blockdev --setra 65536 /dev/md3
>> 
>> # Set stripe-cache_size for RAID5.
>> echo "Setting stripe_cache_size to 16 MiB for /dev/md3"
>> echo 16384 > /sys/block/md3/md/stripe_cache_size
>> 
>> # Disable NCQ on all disks.
>> echo "Disabling NCQ on all disks..."
>> for i in $DISKS
>> do
>>   echo "Disabling NCQ on $i"
>>   echo 1 > /sys/block/"$i"/device/queue_depth
>> done
>
> Given that one of the greatest benefits of NCQ/TCQ is with parity RAID, I'd 
> be fascinated to see how enabling NCQ changes your results.  Of course, you'd 
> want to use a single SATA controller with a known good NCQ implementation, 
> and hard drives known to not do stupid things like disable readahead when NCQ 
> is enabled.
Only/usually on multi-threaded jobs/tasks, yes?

Also, I turn off NCQ on all of my hosts that has it enabled by default because
there are many bugs that occur when NCQ is on, they are working on it in the
libata layer but IMO it is not safe at all for running SATA disks w/NCQ as
with it on I have seen drives drop out of the array (with it off, no problems).

Re: Performance Characteristics of All Linux RAIDs (mdadm/bonnie++)

[Chris Snook]

Justin Piszcz wrote:
> 
> 
> On Wed, 28 May 2008, Chris Snook wrote:
> 
>> Justin Piszcz wrote:
>>> Hardware:
>>>
>>> 1. Utilized (6) 400 gigabyte sata hard drives.
>>> 2. Everything is on PCI-e (965 chipset & a 2port sata card)
>>>
>>> Used the following 'optimizations' for all tests.
>>>
>>> # Set read-ahead.
>>> echo "Setting read-ahead to 64 MiB for /dev/md3"
>>> blockdev --setra 65536 /dev/md3
>>>
>>> # Set stripe-cache_size for RAID5.
>>> echo "Setting stripe_cache_size to 16 MiB for /dev/md3"
>>> echo 16384 > /sys/block/md3/md/stripe_cache_size
>>>
>>> # Disable NCQ on all disks.
>>> echo "Disabling NCQ on all disks..."
>>> for i in $DISKS
>>> do
>>>   echo "Disabling NCQ on $i"
>>>   echo 1 > /sys/block/"$i"/device/queue_depth
>>> done
>>
>> Given that one of the greatest benefits of NCQ/TCQ is with parity 
>> RAID, I'd be fascinated to see how enabling NCQ changes your results.  
>> Of course, you'd want to use a single SATA controller with a known 
>> good NCQ implementation, and hard drives known to not do stupid things 
>> like disable readahead when NCQ is enabled.
> Only/usually on multi-threaded jobs/tasks, yes?

Generally, yes, but there's caching and readahead at various layers in 
software that can expose the benefit on certain single-threaded 
workloads as well.

> Also, I turn off NCQ on all of my hosts that has it enabled by default 
> because
> there are many bugs that occur when NCQ is on, they are working on it in 
> the
> libata layer but IMO it is not safe at all for running SATA disks w/NCQ as
> with it on I have seen drives drop out of the array (with it off, no 
> problems).
> 

Are you using SATA drives with RAID-optimized firmware?  Most SATA 
manufacturers have variants of their drives for a few dollars more that 
have firmware that provides bounded latency for error recovery 
operations, for precisely this reason.

-- Chris

Re: Performance Characteristics of All Linux RAIDs (mdadm/bonnie++)

[Justin Piszcz]

On Wed, 28 May 2008, Chris Snook wrote:

> Justin Piszcz wrote:
>> 
>> 
>> On Wed, 28 May 2008, Chris Snook wrote:
>> 
>>> Justin Piszcz wrote:
>>>> Hardware:
>>>> 
>>> Given that one of the greatest benefits of NCQ/TCQ is with parity RAID, 
>>> I'd be fascinated to see how enabling NCQ changes your results.  Of 
>>> course, you'd want to use a single SATA controller with a known good NCQ 
>>> implementation, and hard drives known to not do stupid things like disable 
>>> readahead when NCQ is enabled.
>> Only/usually on multi-threaded jobs/tasks, yes?
>
> Generally, yes, but there's caching and readahead at various layers in 
> software that can expose the benefit on certain single-threaded workloads as 
> well.
>
>> Also, I turn off NCQ on all of my hosts that has it enabled by default 
>> because
>> there are many bugs that occur when NCQ is on, they are working on it in 
>> the
>> libata layer but IMO it is not safe at all for running SATA disks w/NCQ as
>> with it on I have seen drives drop out of the array (with it off, no 
>> problems).
>> 
>
> Are you using SATA drives with RAID-optimized firmware?  Most SATA 
> manufacturers have variants of their drives for a few dollars more that have 
> firmware that provides bounded latency for error recovery operations, for 
> precisely this reason.
I see--however, as I understood it there were bugs utilizing NCQ in libata?

But FYI--
In this test, they were regular SATA drives, not special raid-ones (RE2,etc).

Thanks for the info!

Justin.

Re: Performance Characteristics of All Linux RAIDs (mdadm/bonnie++)

[Kasper Sandberg]

On Wed, 2008-05-28 at 15:27 -0400, Justin Piszcz wrote:
> 
> On Wed, 28 May 2008, Chris Snook wrote:
> 
> > Justin Piszcz wrote:
> >> 
> >> 
> >> On Wed, 28 May 2008, Chris Snook wrote:
> >> 
> >>> Justin Piszcz wrote:
> >>>> Hardware:
> >>>> 
> >>> Given that one of the greatest benefits of NCQ/TCQ is with parity RAID, 
> >>> I'd be fascinated to see how enabling NCQ changes your results.  Of 
> >>> course, you'd want to use a single SATA controller with a known good NCQ 
> >>> implementation, and hard drives known to not do stupid things like disable 
> >>> readahead when NCQ is enabled.
> >> Only/usually on multi-threaded jobs/tasks, yes?
> >
> > Generally, yes, but there's caching and readahead at various layers in 
> > software that can expose the benefit on certain single-threaded workloads as 
> > well.
> >
> >> Also, I turn off NCQ on all of my hosts that has it enabled by default 
> >> because
> >> there are many bugs that occur when NCQ is on, they are working on it in 
> >> the
> >> libata layer but IMO it is not safe at all for running SATA disks w/NCQ as
> >> with it on I have seen drives drop out of the array (with it off, no 
> >> problems).
> >> 
> >
> > Are you using SATA drives with RAID-optimized firmware?  Most SATA 
> > manufacturers have variants of their drives for a few dollars more that have 
> > firmware that provides bounded latency for error recovery operations, for 
> > precisely this reason.
> I see--however, as I understood it there were bugs utilizing NCQ in libata?
You wouldnt happen to have some more information about this? i havent
personally had problems yet, but i havent used it for very long - but
since it comes activated by DEFAULT, i would assume it to be relatively
stable?

> 
> But FYI--
> In this test, they were regular SATA drives, not special raid-ones (RE2,etc).
> 
> Thanks for the info!
> 
> Justin.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Performance Characteristics of All Linux RAIDs (mdadm/bonnie++)

[Justin Piszcz]

On Thu, 29 May 2008, Kasper Sandberg wrote:

> On Wed, 2008-05-28 at 15:27 -0400, Justin Piszcz wrote:
>>
>> On Wed, 28 May 2008, Chris Snook wrote:
>>
>>> Justin Piszcz wrote:
>>>>
>>>>
>>>> On Wed, 28 May 2008, Chris Snook wrote:
>>>>
>>>>> Justin Piszcz wrote:
>>>>>> Hardware:
>>>>>>
>>>>> Given that one of the greatest benefits of NCQ/TCQ is with parity RAID,
>>>>> I'd be fascinated to see how enabling NCQ changes your results.  Of
>>>>> course, you'd want to use a single SATA controller with a known good NCQ
>>>>> implementation, and hard drives known to not do stupid things like disable
>>>>> readahead when NCQ is enabled.
>>>> Only/usually on multi-threaded jobs/tasks, yes?
>>>
>>> Generally, yes, but there's caching and readahead at various layers in
>>> software that can expose the benefit on certain single-threaded workloads as
>>> well.
>>>
>>>> Also, I turn off NCQ on all of my hosts that has it enabled by default
>>>> because
>>>> there are many bugs that occur when NCQ is on, they are working on it in
>>>> the
>>>> libata layer but IMO it is not safe at all for running SATA disks w/NCQ as
>>>> with it on I have seen drives drop out of the array (with it off, no
>>>> problems).
>>>>
>>>
>>> Are you using SATA drives with RAID-optimized firmware?  Most SATA
>>> manufacturers have variants of their drives for a few dollars more that have
>>> firmware that provides bounded latency for error recovery operations, for
>>> precisely this reason.
>> I see--however, as I understood it there were bugs utilizing NCQ in libata?
> You wouldnt happen to have some more information about this? i havent
> personally had problems yet, but i havent used it for very long - but
> since it comes activated by DEFAULT, i would assume it to be relatively
> stable?
Not off-hand, check LKML and my email address from early this year or last 
year and/or the ide-list.

Justin.

Re: Performance Characteristics of All Linux RAIDs (mdadm/bonnie++)

[Chris Snook]

Jens Bäckman wrote:
> On Wed, May 28, 2008 at 10:53 AM, Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
>> Results:
>>
>> http://home.comcast.net/~jpiszcz/raid/20080528/raid-levels.html
>> http://home.comcast.net/~jpiszcz/raid/20080528/raid-levels.txt
> 
> Either the RAID 1 read speed must be wrong, or something is odd in the
> Linux implementation. There's six drives that can be used for reading
> at the same time, as they contain the very same data. 63MB/s
> sequential looks like what you would get from a single drive.

The test is a single thread reading one block at a time, so this is not 
surprising.  If you get this doing multi-megabyte readahead, or with 
several threads, something is very wrong.

-- Chris

Re: Performance Characteristics of All Linux RAIDs (mdadm/bonnie++)

[Bryan Mesich]

On Wed, May 28, 2008 at 06:34:00PM +0200, Jens Bäckman wrote:

> Either the RAID 1 read speed must be wrong, or something is odd in the
> Linux implementation. There's six drives that can be used for reading
> at the same time, as they contain the very same data. 63MB/s
> sequential looks like what you would get from a single drive.

The RAID 1 read speed metrics do not depict multithreaded
processes reading from the array simutaneouly. I would suspect
that the read performance metrics would look better if 2 bonnie
simulations were ran together (for RAID 1 that is).

Bryan 

Re: Performance Characteristics of All Linux RAIDs (mdadm/bonnie++)

[Alan Cox]

> I really don't think that's any part of the issue, the same memory and 
> bridge went 4-5x faster in other read cases. The truth is that the 
> raid-1 performance is really bad, and it's the code causing it AFAIK. If 
> you track the actual io it seems to read one drive at a time, in order, 
> without overlap.

Make sure the readahead is set to be a fair bit over the stripe size if
you are doing bulk data tests for a single file. (Or indeed in the real
world for that specific case ;))

Re: Performance Characteristics of All Linux RAIDs (mdadm/bonnie++)

[Bill Davidsen]

Alan Cox wrote:
>> I really don't think that's any part of the issue, the same memory and 
>> bridge went 4-5x faster in other read cases. The truth is that the 
>> raid-1 performance is really bad, and it's the code causing it AFAIK. If 
>> you track the actual io it seems to read one drive at a time, in order, 
>> without overlap.
>>     
>
> Make sure the readahead is set to be a fair bit over the stripe size if
> you are doing bulk data tests for a single file. (Or indeed in the real
> world for that specific case ;))
>   

IIRC Justin has readahead at 16MB and chunk at 256k. I would think that 
if multiple devices were used at all by the md code, that the chunk 
rather than stripe size would be the issue. In this case the RA seems 
large enough to trigger good behavior, were there are available.

Note: this testing was done with an old(er) kernel, as were all of mine. 
Since my one large raid array has become more mission critical I'm not 
comfortable playing with new kernels. The fate of big, fast, and stable 
machines is to slide into production use. :-(
I suppose that's not a bad way to do it, I now have faith in what I'm 
running.

-- 
Bill Davidsen <davidsen@tmr.com>
  "Woe unto the statesman who makes war without a reason that will still
  be valid when the war is over..." Otto von Bismark 

Re: Performance Characteristics of All Linux RAIDs (mdadm/bonnie++)

[Justin Piszcz]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 978 bytes --]



On Wed, 28 May 2008, Keld Jørn Simonsen wrote:

> I added this in the wiki performance section.
Ok.

> I think it would have been informative if also a test with one drive in
> a non-raid setup was described.
Since the performance of bonnie++ deals with single threads/a raid1 would probably closely match it but yeah I could do that sometime.

>
> Are there any particular findings you want to highlight?
Not in particular, just I could never find this information provided anywhere
that showed all of the raid variation/types in one location that was easy to
read.

>
> Is there some way to estimate random read and writes from this test?
The sequential input and output and re-write speeds are the most important.

>
> Are the XFS file systems completely new when running the tests?
Yes, after the creation of each array, mkfs.xfs -f /dev/md3 was run to ensure
a clean mount and cron was also off throughout all testing.

>
> Best regards
> keld