bitmap-chunk sizing on RAID-10?

bitmap-chunk sizing on RAID-10?

[Ben DJ]

Hi,

I didn't want to hijack the thread, so a new one here.

On Wed, Nov 11, 2009 at 2:10 PM, Robin Hill <robin@robinhill.me.uk> wrote:
> It's true for RAID-10, yes.  You can't physically grow the array, but
> you can definitely add/remove the bitmap.

Thanks for clearing that up.  The manpage is a bit unclear to my read.

I've just been reading threads about proper sizing.  Large
bitmap-chunk seems good, larger than "1 million bits" -- not, and an
old bug (resolved?) if bitmap-chunk is smaller than raid10 chunk size.

I've two arrays --

   /dev/md0 (RAID-1, across 4x 160MB partitions)

   /dev/md1 (RAID-10/f2 , across 4x ~1TB partitions, --chunk=256 )

The first array is so small, that resync takes just a few seconds
anyway.  Is there any advantage to still installing an internal
write-intent bitmap on it?

The second array takes a few hours to resync from scratch, and so the
bitmap has performance value.  What's the right size for
--bitmap-chunk for an internal bitmap?  Iiuc, the default that "is
automatically deteremined to make best use of available space" results
in 2x-4x (some say 10%) write-performance slowdowns.  I'd like to
avoid that perf hit, but since I did NOT create the array with bitmap,
getting sizing "right" (i.e., not too big for the superblock space) is
important.

Some threads refere to "the math" for different RAID types, but I
haven't figured out what that math is yet.

How do I get to the "right" bitmap-chunk size?

Thanks

BenDJ

Re: bitmap-chunk sizing on RAID-10?

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 3120 bytes --]

On 11/11/2009 06:32 PM, Ben DJ wrote:
> Hi,
> 
> I didn't want to hijack the thread, so a new one here.
> 
> On Wed, Nov 11, 2009 at 2:10 PM, Robin Hill <robin@robinhill.me.uk> wrote:
>> It's true for RAID-10, yes.  You can't physically grow the array, but
>> you can definitely add/remove the bitmap.
> 
> Thanks for clearing that up.  The manpage is a bit unclear to my read.
> 
> I've just been reading threads about proper sizing.  Large
> bitmap-chunk seems good, larger than "1 million bits" -- not, and an
> old bug (resolved?) if bitmap-chunk is smaller than raid10 chunk size.
> 
> I've two arrays --
> 
>    /dev/md0 (RAID-1, across 4x 160MB partitions)
> 
>    /dev/md1 (RAID-10/f2 , across 4x ~1TB partitions, --chunk=256 )
> 
> The first array is so small, that resync takes just a few seconds
> anyway.  Is there any advantage to still installing an internal
> write-intent bitmap on it?

Not in my opinion.  I skip bitmaps on boot arrays and other smallish
arrays like that.

> The second array takes a few hours to resync from scratch, and so the
> bitmap has performance value.  What's the right size for
> --bitmap-chunk for an internal bitmap?  Iiuc, the default that "is
> automatically deteremined to make best use of available space" results
> in 2x-4x (some say 10%) write-performance slowdowns.

It makes for noticeable slowdowns anyway.  How bad is dependent on your
data writing patterns.  Lots of random writes will actually show a
larger slowdown than more sequential writing.  The main thing to
understand is that a bitmap like this is useless if the raid stack
doesn't stop any write going to the disk unless the bit for that write's
sector is set to dirty.  So, when a new write is initiated on a clean
array, the write is help up until the bitmap write to dirty the proper
bits completes, and only then can the normal write proceed.  So, with
lots of random I/O, or even with sequential I/O on very small bitmap
chunk sizes, you end up spending a significant amount of time holding up
writes as you dirty the bits on disk.  Picking a larger bitmap chunk
helps to increase the likelihood that more writes will stream without
having to wait on a new bitmap dirty write.

Given that the only real benefit to the bitmap is reduced resync time in
the event something happens, and given that as you said a 160MB section
of array can resync in a relatively short time, and given that a smaller
bitmap chunk hurts performance *all* the time versus only helping in
rare circumstances, bigger is better in my opinion.

I haven't done specific testing of performance differences with
different size bitmap chunks, but my seat of the pants review puts the
32768 area as a good starting point.  Any given chunk will resync in
just a second or so, but it doesn't cause as much performance slowdown
as the default chunk size.


-- 
Doug Ledford <dledford@redhat.com>
              GPG KeyID: CFBFF194
	      http://people.redhat.com/dledford

Infiniband specific RPMs available at
	      http://people.redhat.com/dledford/Infiniband


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]