Re: bitmap-chunk sizing on RAID-10?

[Doug Ledford <dledford@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 3120 bytes --]

On 11/11/2009 06:32 PM, Ben DJ wrote:
> Hi,
> 
> I didn't want to hijack the thread, so a new one here.
> 
> On Wed, Nov 11, 2009 at 2:10 PM, Robin Hill <robin@robinhill.me.uk> wrote:
>> It's true for RAID-10, yes.  You can't physically grow the array, but
>> you can definitely add/remove the bitmap.
> 
> Thanks for clearing that up.  The manpage is a bit unclear to my read.
> 
> I've just been reading threads about proper sizing.  Large
> bitmap-chunk seems good, larger than "1 million bits" -- not, and an
> old bug (resolved?) if bitmap-chunk is smaller than raid10 chunk size.
> 
> I've two arrays --
> 
>    /dev/md0 (RAID-1, across 4x 160MB partitions)
> 
>    /dev/md1 (RAID-10/f2 , across 4x ~1TB partitions, --chunk=256 )
> 
> The first array is so small, that resync takes just a few seconds
> anyway.  Is there any advantage to still installing an internal
> write-intent bitmap on it?

Not in my opinion.  I skip bitmaps on boot arrays and other smallish
arrays like that.

> The second array takes a few hours to resync from scratch, and so the
> bitmap has performance value.  What's the right size for
> --bitmap-chunk for an internal bitmap?  Iiuc, the default that "is
> automatically deteremined to make best use of available space" results
> in 2x-4x (some say 10%) write-performance slowdowns.

It makes for noticeable slowdowns anyway.  How bad is dependent on your
data writing patterns.  Lots of random writes will actually show a
larger slowdown than more sequential writing.  The main thing to
understand is that a bitmap like this is useless if the raid stack
doesn't stop any write going to the disk unless the bit for that write's
sector is set to dirty.  So, when a new write is initiated on a clean
array, the write is help up until the bitmap write to dirty the proper
bits completes, and only then can the normal write proceed.  So, with
lots of random I/O, or even with sequential I/O on very small bitmap
chunk sizes, you end up spending a significant amount of time holding up
writes as you dirty the bits on disk.  Picking a larger bitmap chunk
helps to increase the likelihood that more writes will stream without
having to wait on a new bitmap dirty write.

Given that the only real benefit to the bitmap is reduced resync time in
the event something happens, and given that as you said a 160MB section
of array can resync in a relatively short time, and given that a smaller
bitmap chunk hurts performance *all* the time versus only helping in
rare circumstances, bigger is better in my opinion.

I haven't done specific testing of performance differences with
different size bitmap chunks, but my seat of the pants review puts the
32768 area as a good starting point.  Any given chunk will resync in
just a second or so, but it doesn't cause as much performance slowdown
as the default chunk size.


-- 
Doug Ledford <dledford@redhat.com>
              GPG KeyID: CFBFF194
	      http://people.redhat.com/dledford

Infiniband specific RPMs available at
	      http://people.redhat.com/dledford/Infiniband


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]