mdadm 3.1.x and bitmap chunk

mdadm 3.1.x and bitmap chunk

[Piergiorgio Sartor]

Hi folks,

I just noticed that mdadm 3.1.x sets the default
bitmap chunk to 64MB.

Searching around, I found the reason should be
related with write performances, which seem to
be less compromised with large bitmap chunks.

I've no problem with that, but is it possible
to set the bitmap chunk to the minimum size,
without trying all the possible dimensions?

Thanks a lot in advance,

bye,

-- 

piergiorgio

Re: mdadm 3.1.x and bitmap chunk

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 1799 bytes --]

On 06/09/2010 05:05 PM, Piergiorgio Sartor wrote:
> Hi folks,
> 
> I just noticed that mdadm 3.1.x sets the default
> bitmap chunk to 64MB.
> 
> Searching around, I found the reason should be
> related with write performances, which seem to
> be less compromised with large bitmap chunks.
> 
> I've no problem with that, but is it possible
> to set the bitmap chunk to the minimum size,
> without trying all the possible dimensions?
> 
> Thanks a lot in advance,
> 
> bye,
> 

This depends somewhat on what you mean by "minimum size" to the bitmap
chunk.  The bitmap chunk size can never be smaller than the minimum size
on the device, so 512bytes for most drives, 4k for newer drives.  But
such a chunk size would be horrible for performance as it would make
every single write synchronous on the bitmap update.  However, the
minimum size can also be limited by the available space for the bitmap
when using an internal bitmap.  So, you can only make the size as small
as it will go and the bitmap still fit between the end of the
{array,superblock} and the start of the {superblock,array}.

However, in general, a bitmap that's small enough that you can sync the
dirty section in a second or two should be good enough.  With modern
disks, they can sync 32MB or 64MB in only a second or two, hence the
default size.  Anything less than that is really a case of diminishing
returns.  You are sacrificing write speed on each and every write to the
array in exchange for a *possible* speed up in the event of a crash.
Not a very good trade off IMO.

-- 
Doug Ledford <dledford@redhat.com>
              GPG KeyID: CFBFF194
	      http://people.redhat.com/dledford

Infiniband specific RPMs available at
	      http://people.redhat.com/dledford/Infiniband


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: mdadm 3.1.x and bitmap chunk

[Piergiorgio Sartor]

Hi,

> This depends somewhat on what you mean by "minimum size" to the bitmap
> chunk.  The bitmap chunk size can never be smaller than the minimum size
> on the device, so 512bytes for most drives, 4k for newer drives.  But

as far as I know, there is a limited space for
the bitmap, hence if the chunk is too small,
the bitmap will not fit in that space.

> such a chunk size would be horrible for performance as it would make
> every single write synchronous on the bitmap update.  However, the
> minimum size can also be limited by the available space for the bitmap
> when using an internal bitmap.  So, you can only make the size as small
> as it will go and the bitmap still fit between the end of the
> {array,superblock} and the start of the {superblock,array}.
> 
> However, in general, a bitmap that's small enough that you can sync the
> dirty section in a second or two should be good enough.  With modern
> disks, they can sync 32MB or 64MB in only a second or two, hence the
> default size.  Anything less than that is really a case of diminishing
> returns.  You are sacrificing write speed on each and every write to the
> array in exchange for a *possible* speed up in the event of a crash.
> Not a very good trade off IMO.

Those are my usual USB HDDs in RAID-6 configuration.

Since the setup might be quite fragile, since the
write performances are anyway limited by the USB
and not by the seek time and writinig is anywau slow,
I would like to keep the bitmap resync fast.

That is, if a USB cable fails, I can just replace it
and resync using bitmap, this will be quite fast, if
the bitmap chunk is not too big.

64MB is quite a huge amount of data for USB, so it
will not help to keep the bitmap resync fast.
Which means the chunk should be small, one target
could be the smallest possible, given the available
space in the superblock.

Thanks,

bye,

-- 

piergiorgio

Re: mdadm 3.1.x and bitmap chunk

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 4281 bytes --]

On 08/03/2010 02:36 PM, Piergiorgio Sartor wrote:
> Hi,
> 
>> This depends somewhat on what you mean by "minimum size" to the bitmap
>> chunk.  The bitmap chunk size can never be smaller than the minimum size
>> on the device, so 512bytes for most drives, 4k for newer drives.  But
> 
> as far as I know, there is a limited space for
> the bitmap, hence if the chunk is too small,
> the bitmap will not fit in that space.

For internal bitmaps, this is correct and what I mentioned next.

>> such a chunk size would be horrible for performance as it would make
>> every single write synchronous on the bitmap update.  However, the
>> minimum size can also be limited by the available space for the bitmap
>> when using an internal bitmap.  So, you can only make the size as small
>> as it will go and the bitmap still fit between the end of the
>> {array,superblock} and the start of the {superblock,array}.
>>
>> However, in general, a bitmap that's small enough that you can sync the
>> dirty section in a second or two should be good enough.  With modern
>> disks, they can sync 32MB or 64MB in only a second or two, hence the
>> default size.  Anything less than that is really a case of diminishing
>> returns.  You are sacrificing write speed on each and every write to the
>> array in exchange for a *possible* speed up in the event of a crash.
>> Not a very good trade off IMO.
> 
> Those are my usual USB HDDs in RAID-6 configuration.
> 
> Since the setup might be quite fragile, since the
> write performances are anyway limited by the USB
> and not by the seek time and writinig is anywau slow,

Seeks will *always* limit array speed, even when the link is slow.
Especially on bitmaps where the seek is synchronous to the pending write.

> I would like to keep the bitmap resync fast.
> 
> That is, if a USB cable fails, I can just replace it
> and resync using bitmap, this will be quite fast, if
> the bitmap chunk is not too big.

It will be faster than a full resync regardless.

> 64MB is quite a huge amount of data for USB, so it
> will not help to keep the bitmap resync fast.

It will help, without a doubt.  Any bitmap is better than no bitmap for
resync.  However, any bitmap is a hurt on performance.  How much of a
help versus a hurt just depends on the chunk size.

> Which means the chunk should be small, one target
> could be the smallest possible, given the available
> space in the superblock.

I don't have an easy answer for the smallest chunk size based upon the
space between the superblock and the array.  It depends on the
superblock type you are using and the layout of the superblock on the
device/partition.  You should be able to get the layout info from the
output of mdadm -E on a constituent device (aka, mdadm -E /dev/sde1 or
whatever you use in the array).  For version 1.1 and 1.2 superblocks,
the array will be offset from the beginning of the partition and that
offset will be listed.  Take out 4k for a version 1.1 superblock, or 8k
for a version 1.2 superblock, take out a little more for the bitmap
superblock, and from what's left you can calculate the number of free
sectors, and then each sector is 512 * 8 bits (for a 512byte sector
device).  Then divide the number of available bits into the array size,
then round up to the closest power of two to get the smallest possible
chunk size.  For version 1.0 and 0.90 superblocks, the process is
similar, except you need to know the offset of the superblock from the
beginning of the array, and subtract from that the used dev size, and
what's left is the number of sectors between the end of the data and the
start of the superblock.  Subtract a bit again for the bitmap
superblock, then repeat the math I listed above.

In any case though, I would just try reducing the bitmap chunk to the
old default of 8mb.  At 8mb you are already suffering a 5% or so
performance penalty on write, but 8mb is not too much to resync, even
over USB, so it might be a good compromise in your case.

-- 
Doug Ledford <dledford@redhat.com>
              GPG KeyID: CFBFF194
	      http://people.redhat.com/dledford

Infiniband specific RPMs available at
	      http://people.redhat.com/dledford/Infiniband


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: mdadm 3.1.x and bitmap chunk

[Piergiorgio Sartor]

Hi,

> > Since the setup might be quite fragile, since the
> > write performances are anyway limited by the USB
> > and not by the seek time and writing is anyway slow,
> 
> Seeks will *always* limit array speed, even when the link is slow.
> Especially on bitmaps where the seek is synchronous to the pending write.

not really, in this case.
HDDs have cache and prefetch, being the link slow
these work quite at an optimal point.

Actually, there is a huge difference in case of fast
SATA HDDs (having bitmap or not), but as soon as the
link speed goes down it gets even.
In the end it is a RAID-6, so writes are not easy anyhow...

As a side condition, this USB array is mainly a read
device, it's a storage system.
This means that writes will be almost exclusively
filesystem metadata update (access time), so not
really a big deal.

> > 64MB is quite a huge amount of data for USB, so it
> > will not help to keep the bitmap resync fast.
> 
> It will help, without a doubt.  Any bitmap is better than no bitmap for
> resync.  However, any bitmap is a hurt on performance.  How much of a
> help versus a hurt just depends on the chunk size.

I meant, 64MB will help less than 64KB.

Chunk size is 64KB, that's because it was the default
at the time the arrays were created.

Since the max transfer data chunk for the USB storage
protocol should be 128KB, I wonder if that chunk size
would have been better.
I might try to reshape it.
 
> I don't have an easy answer for the smallest chunk size based upon the
[...]

Thanks for the suggestion, I'll try to see if I can
manage to get the "smaller chunk size".

> In any case though, I would just try reducing the bitmap chunk to the
> old default of 8mb.  At 8mb you are already suffering a 5% or so
> performance penalty on write, but 8mb is not too much to resync, even
> over USB, so it might be a good compromise in your case.

I'll try also this, even if I hope I'll have not
to benchmark... :-)

BTW, a single USB device can transfer up to 30MB/s, with
one HDD limited to 15MB/s (old HDD).

The arrays can achieve around 43MB/s (read, of course),
which seems to be the limit of the USB chipset of the
motherboard (probably todays MBs can do 50~55MB/s).

This 43MB/s seem to be the limit, since this speed is
achieved with 10 HDDs as well as with 4 (a bit less).

Is this data any good to calculate the "optimal" bitmap
chunk size?

Note that with SATA HDDs the read speed I got (4 HDDs in
RAID-5) was almost precisely (n-1)*slowest_disk, i.e.
3*110MB/s.
In case of the USB system it is not (due to the USB
bottleneck) (n-2)*slowest_disk, so the concept of having
a bitmap chunk about around the 1 second transfer is a
bit difficult, for me, to understand.

Thanks again,

bye,

-- 

piergiorgio