Intent Bitmap size and performance

Intent Bitmap size and performance

[Marc MERLIN]

I was reading
http://blog.liw.fi/posts/write-intent-bitmaps/
which warns that intent bitmaps kill performance if they are too small

The mdadm man page does tell me how I can set the intent bitmap size,
but I can't tell what size I have or what is applied by default if you
don't have the size option and create the intent bitmap with --grow

First, I did: mdadm --grow --bitmap=internal /dev/md7

Then, I have:
gargamel:/mnt/btrfs_pool1# mdadm --detail /dev/md7
/dev/md7:
        Version : 1.2
  Creation Time : Thu Mar 25 20:15:00 2010
     Raid Level : raid5
     Array Size : 7814045696 (7452.05 GiB 8001.58 GB)
  Used Dev Size : 1953511424 (1863.01 GiB 2000.40 GB)
   Raid Devices : 5
  Total Devices : 5
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Sat Feb  1 16:08:56 2014
          State : active 
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : gargamel.svh.merlins.org:7  (local to host gargamel.svh.merlins.org)
           UUID : 5884576b:0e402a5d:8629093c:ec020760
         Events : 38030

    Number   Major   Minor   RaidDevice State
       6       8       81        0      active sync   /dev/sdf1
       1       8       49        1      active sync   /dev/sdd1
       2       8       65        2      active sync   /dev/sde1
       3       8        1        3      active sync   /dev/sda1
       7       8       17        4      active sync   /dev/sdb1


How can I tell I got the size for my array size?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Intent Bitmap size and performance

[Marc MERLIN]

On Sat, Feb 01, 2014 at 04:39:15PM -0800, Marc MERLIN wrote:
> How can I tell I got the size for my array size?

Aah, the clue seems to be in the kernel logs:
[669348.274368] md7: bitmap file is out of date (0 < 38029) -- forcing full recovery
[669348.299174] created bitmap (15 pages) for device md7
[669348.316720] md7: bitmap file is out of date, doing full recovery
[669348.380555] md7: bitmap initialized from disk: read 1 pages, set 29809 of 29809 bits

If I got the math right, 30K bits for 8TB is one bit per 266MB.

Given that, I'm going to assume that this is not going to impact system
performance much for most operations.

Is my assumption and conclusion correct?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Intent Bitmap size and performance

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1585 bytes --]

On Sat, 1 Feb 2014 16:56:46 -0800 Marc MERLIN <marc@merlins.org> wrote:

> On Sat, Feb 01, 2014 at 04:39:15PM -0800, Marc MERLIN wrote:
> > How can I tell I got the size for my array size?
> 
> Aah, the clue seems to be in the kernel logs:
> [669348.274368] md7: bitmap file is out of date (0 < 38029) -- forcing full recovery
> [669348.299174] created bitmap (15 pages) for device md7
> [669348.316720] md7: bitmap file is out of date, doing full recovery
> [669348.380555] md7: bitmap initialized from disk: read 1 pages, set 29809 of 29809 bits
> 
> If I got the math right, 30K bits for 8TB is one bit per 266MB.
> 
> Given that, I'm going to assume that this is not going to impact system
> performance much for most operations.
> 
> Is my assumption and conclusion correct?
> 
> Thanks,
> Marc

You can also use "mdadm --examine-bitmap" on one of the component devices to
get more details about the bitmap.

My rule-of-thumb (base on zero hard evidence) is that one bit should
correspond to approximately 1 second of IO.  Your bits correspond to 2 or 3
seconds so that is certainly the right ball park.

As always with RAID, performance is highly dependent on load.
It is quite easy to add and remove bitmaps to/from a live md array so
testing the effect on a particular workload is not that hard.

The default mdadm chooses is a bit complex.  It first chooses an amount of
space to reserve for the bitmap, the it figures what chunk size will allow
the bits to fit in the available space.  Then makes sure that it as least
64Meg.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Intent Bitmap size and performance

[Marc MERLIN]

On Sun, Feb 02, 2014 at 05:28:32PM +1100, NeilBrown wrote:
> On Sat, 1 Feb 2014 16:56:46 -0800 Marc MERLIN <marc@merlins.org> wrote:
> 
> > On Sat, Feb 01, 2014 at 04:39:15PM -0800, Marc MERLIN wrote:
> > > How can I tell I got the size for my array size?
> > 
> > Aah, the clue seems to be in the kernel logs:
> > [669348.274368] md7: bitmap file is out of date (0 < 38029) -- forcing full recovery
> > [669348.299174] created bitmap (15 pages) for device md7
> > [669348.316720] md7: bitmap file is out of date, doing full recovery
> > [669348.380555] md7: bitmap initialized from disk: read 1 pages, set 29809 of 29809 bits
> > 
> > If I got the math right, 30K bits for 8TB is one bit per 266MB.
> > 
> > Given that, I'm going to assume that this is not going to impact system
> > performance much for most operations.
> > 
> > Is my assumption and conclusion correct?
> > 
> > Thanks,
> > Marc
> 
> You can also use "mdadm --examine-bitmap" on one of the component devices to
> get more details about the bitmap.

Thanks, I had managed to miss this in the man page.

Argh, not good then, see:
gargamel:~# mdadm --examine-bitmap /dev/md5
        Filename : /dev/md5
           Magic : 534b554c
mdadm: invalid bitmap magic 0x534b554c, the bitmap file appears to be corrupted
         Version : 16826042
mdadm: unknown bitmap version 16826042, either the bitmap file is corrupted or you need to upgrade your tools
gargamel:~# mdadm --examine-bitmap /dev/md8
        Filename : /dev/md8
           Magic : 534b554c
mdadm: invalid bitmap magic 0x534b554c, the bitmap file appears to be corrupted
         Version : 16826042
mdadm: unknown bitmap version 16826042, either the bitmap file is corrupted or you need to upgrade your tools

md8 I just created a few weeks ago.
md5 is old-ish but I just added the bitmap with --grow.

kernel: 3.12.7
gargamel:~# mdadm --version
mdadm - v3.3 - 3rd September 2013

gargamel:~# apt-get install -t unstable mdadm
Reading package lists... Done
Building dependency tree       
Reading state information... Done
mdadm is already the newest version.

Is debian unstable too old, or do I have another bug?

> My rule-of-thumb (base on zero hard evidence) is that one bit should
> correspond to approximately 1 second of IO.  Your bits correspond to 2 or 3
> seconds so that is certainly the right ball park.
> 
> As always with RAID, performance is highly dependent on load.
> It is quite easy to add and remove bitmaps to/from a live md array so
> testing the effect on a particular workload is not that hard.
> 
> The default mdadm chooses is a bit complex.  It first chooses an amount of
> space to reserve for the bitmap, the it figures what chunk size will allow
> the bits to fit in the available space.  Then makes sure that it as least
> 64Meg.

Thanks for explaining.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Intent Bitmap size and performance

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 3479 bytes --]

On Thu, 6 Feb 2014 11:05:19 -0800 Marc MERLIN <marc@merlins.org> wrote:

> On Sun, Feb 02, 2014 at 05:28:32PM +1100, NeilBrown wrote:
> > On Sat, 1 Feb 2014 16:56:46 -0800 Marc MERLIN <marc@merlins.org> wrote:
> > 
> > > On Sat, Feb 01, 2014 at 04:39:15PM -0800, Marc MERLIN wrote:
> > > > How can I tell I got the size for my array size?
> > > 
> > > Aah, the clue seems to be in the kernel logs:
> > > [669348.274368] md7: bitmap file is out of date (0 < 38029) -- forcing full recovery
> > > [669348.299174] created bitmap (15 pages) for device md7
> > > [669348.316720] md7: bitmap file is out of date, doing full recovery
> > > [669348.380555] md7: bitmap initialized from disk: read 1 pages, set 29809 of 29809 bits
> > > 
> > > If I got the math right, 30K bits for 8TB is one bit per 266MB.
> > > 
> > > Given that, I'm going to assume that this is not going to impact system
> > > performance much for most operations.
> > > 
> > > Is my assumption and conclusion correct?
> > > 
> > > Thanks,
> > > Marc
> > 
> > You can also use "mdadm --examine-bitmap" on one of the component devices to
                                                            ^^^^^^^^^^^^^^^^^
> > get more details about the bitmap.
> 
> Thanks, I had managed to miss this in the man page.
> 
> Argh, not good then, see:
> gargamel:~# mdadm --examine-bitmap /dev/md5

You wanted something like
    mdadm --examine-bitmap /dev/sda1

The bitmap, like other metadata, lives in the component devices, not in the
array.
You certainly aren't the first to suffer this confusion - maybe I should try
to make mdadm catch that failure mode..

NeilBrown


>         Filename : /dev/md5
>            Magic : 534b554c
> mdadm: invalid bitmap magic 0x534b554c, the bitmap file appears to be corrupted
>          Version : 16826042
> mdadm: unknown bitmap version 16826042, either the bitmap file is corrupted or you need to upgrade your tools
> gargamel:~# mdadm --examine-bitmap /dev/md8
>         Filename : /dev/md8
>            Magic : 534b554c
> mdadm: invalid bitmap magic 0x534b554c, the bitmap file appears to be corrupted
>          Version : 16826042
> mdadm: unknown bitmap version 16826042, either the bitmap file is corrupted or you need to upgrade your tools
> 
> md8 I just created a few weeks ago.
> md5 is old-ish but I just added the bitmap with --grow.
> 
> kernel: 3.12.7
> gargamel:~# mdadm --version
> mdadm - v3.3 - 3rd September 2013
> 
> gargamel:~# apt-get install -t unstable mdadm
> Reading package lists... Done
> Building dependency tree       
> Reading state information... Done
> mdadm is already the newest version.
> 
> Is debian unstable too old, or do I have another bug?
> 
> > My rule-of-thumb (base on zero hard evidence) is that one bit should
> > correspond to approximately 1 second of IO.  Your bits correspond to 2 or 3
> > seconds so that is certainly the right ball park.
> > 
> > As always with RAID, performance is highly dependent on load.
> > It is quite easy to add and remove bitmaps to/from a live md array so
> > testing the effect on a particular workload is not that hard.
> > 
> > The default mdadm chooses is a bit complex.  It first chooses an amount of
> > space to reserve for the bitmap, the it figures what chunk size will allow
> > the bits to fit in the available space.  Then makes sure that it as least
> > 64Meg.
> 
> Thanks for explaining.
> 
> Marc


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]