"bitmap" line in /proc/mdstat

"bitmap" line in /proc/mdstat

[Ramon Hofer]

Hi all

When I have added a new RAID5 yesterday I saw that it has an additional 
line (bitmap: 0/30 pages [0KB], 65536KB chunk) in /proc/mdstat for md5 
in contrast to md[1234] not having this line:

# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear]
md5 : active raid5 sdt[4] sds[2] sdr[1] sdi[0]
       11720662272 blocks super 1.2 level 5, 128k chunk, algorithm 2 
[4/4] [UUUU]
       bitmap: 0/30 pages [0KB], 65536KB chunk

md0 : active linear md5[4] md1[0] md4[3] md3[2] md2[1]
       33697299328 blocks super 1.2 0k rounding

md4 : active raid5 sdj[5] sdm[4] sdl[2] sdk[1]
       5860149888 blocks super 1.2 level 5, 128k chunk, algorithm 2 
[4/4] [UUUU]

md3 : active raid5 sdn[0] sdq[4] sdp[2] sdo[1]
       5860540032 blocks super 1.2 level 5, 128k chunk, algorithm 2 
[4/4] [UUUU]

md2 : active raid5 sde[0] sdh[4] sdg[2] sdf[1]
       4395412224 blocks super 1.2 level 5, 128k chunk, algorithm 2 
[4/4] [UUUU]

md1 : active raid5 sda[0] sdd[3] sdc[2] sdb[1]
       5860540032 blocks super 1.2 level 5, 128k chunk, algorithm 2 
[4/4] [UUUU]

unused devices: <none>


What I did yesterday was the following:

# mdadm -C /dev/md5 -c 128 -n4 -l5 /dev/sd[irst]

# mdadm --grow /dev/md0 --add /dev/md5

# xfs_growfs /mnt/media-raid


 From what I was reading [1,2] it is useful for faster rebuilds.
But why is it not present for md[1234] as I used the same command to 
create these RAIDs?
Is it only possible for the last of the linear devices, or is it only 
used for almost empty devices, or was the bitmap introduced in a recent 
update?
I wonder if I should and could add a bitmap for the other devices as well?


Best regards,
Ramon


[1] https://raid.wiki.kernel.org/index.php/Write-intent_bitmap
[2] https://raid.wiki.kernel.org/index.php/Mdstat#bitmap_line

Re: "bitmap" line in /proc/mdstat

[Phil Turmel]

On 10/14/2016 06:06 AM, Ramon Hofer wrote:

> From what I was reading [1,2] it is useful for faster rebuilds.
> But why is it not present for md[1234] as I used the same command to
> create these RAIDs?
> Is it only possible for the last of the linear devices, or is it only
> used for almost empty devices, or was the bitmap introduced in a recent
> update?

Bitmaps have been available for a long time, but including them on a new
array by default is recent.  They are most useful if a device is
unexpectedly disconnected but otherwise OK.  mdadm --re-add will allow
that device to rejoin the array and only the missing data (tracked by
the bitmap) will have to be written to sync it up.

> I wonder if I should and could add a bitmap for the other devices as well?

There is a performance hit, so it should be considered on a case by case
basis.

Phil

Re: "bitmap" line in /proc/mdstat

[Ramon Hofer]

Thanks for your answer, Phil.

On 14/10/16 13:53, Phil Turmel wrote:
> On 10/14/2016 06:06 AM, Ramon Hofer wrote:
>
>>  From what I was reading [1,2] it is useful for faster rebuilds.
>> But why is it not present for md[1234] as I used the same command to
>> create these RAIDs?
>> Is it only possible for the last of the linear devices, or is it only
>> used for almost empty devices, or was the bitmap introduced in a recent
>> update?
>
> Bitmaps have been available for a long time, but including them on a new
> array by default is recent.  They are most useful if a device is
> unexpectedly disconnected but otherwise OK.  mdadm --re-add will allow
> that device to rejoin the array and only the missing data (tracked by
> the bitmap) will have to be written to sync it up.

What could be an unexpected disconnect? If a disc is accidentally 
removed from a slot?


>> I wonder if I should and could add a bitmap for the other devices as well?
>
> There is a performance hit, so it should be considered on a case by case
> basis.

My uneducated guess would be that I don't really need it.

So I should probably remove the bitmap for the newly created device.
Although the performance hit is not really an issue in my case, I guess.
How much disk space is used for a 12 TB RAID5 bitmap?


Best regards,
Ramon

Re: "bitmap" line in /proc/mdstat

[Ramon Hofer]

On 14/10/16 14:32, Ramon Hofer wrote:
> How much disk space is used for a 12 TB RAID5 bitmap?

I have remove the bitmap with

# mdadm -G /dev/md5 -b none

Comparing the array sizes from before and afterwards it appears that the 
internal bitmap does not use any disk space.
Since the internal bitmap is part of the metadata this makes total sense.
Still I would imagine if I assume that a bit per chunck is needed and I 
use 128kB chuncks on the 11720662272 blocks of the 12TB RAID5 this is:
log(11720662272)/log(2) = 34 Bytes

Or am I totally wrong?


Best regards,
Ramon

Re: "bitmap" line in /proc/mdstat

[Andreas Klauer]

On Fri, Oct 14, 2016 at 03:59:20PM +0200, Ramon Hofer wrote:
> Still I would imagine if I assume that a bit per chunck is needed and I 
> use 128kB chuncks on the 11720662272 blocks of the 12TB RAID5 this is:
> log(11720662272)/log(2) = 34 Bytes
> 
> Or am I totally wrong?

It's negligible, a few bytes in the metadata somewhere.

The bitmap has its own size (--bitmap-chunk option). It's fine to set 
this to a large value (128MiB, 1GiB, ...). That might help performance 
and reduce the bytes required for the bitmap itself (not that it matters).

The main issue is performance impact... it's a tradeoff for additional 
bitmap updates each write, or long resync time after a power loss or 
similar event.

If performance is not critical (e.g. it's a NAS, not a database) 
then you can enable it without worries.

Regards
Andreas Klauer

Re: "bitmap" line in /proc/mdstat

[Ramon Hofer]

Thanks for the advice, Andreas.

On 14/10/16 16:10, Andreas Klauer wrote:
> The main issue is performance impact... it's a tradeoff for additional
> bitmap updates each write, or long resync time after a power loss or
> similar event.
>
> If performance is not critical (e.g. it's a NAS, not a database)
> then you can enable it without worries.

Yes, it is some sort of NAS.
I also have a database for ownCloud on it but I am the only user.
And I use an USV for that machine so I am also not really concerned 
about power loss.

Best probably to roll a dice :-)


Regards,
Ramon

RE: "bitmap" line in /proc/mdstat

[Peter Sangas]

Can you quantify the performance decrease and if rebuild time is not as important as performance how does one remove the bitmap?

Thank you,
Peter

-----Original Message-----
From: Phil Turmel [mailto:philip@turmel.org] 
Sent: Friday, October 14, 2016 4:53 AM
To: Ramon Hofer; linux-raid@vger.kernel.org
Subject: Re: "bitmap" line in /proc/mdstat

On 10/14/2016 06:06 AM, Ramon Hofer wrote:

> From what I was reading [1,2] it is useful for faster rebuilds.
> But why is it not present for md[1234] as I used the same command to 
> create these RAIDs?
> Is it only possible for the last of the linear devices, or is it only 
> used for almost empty devices, or was the bitmap introduced in a 
> recent update?

Bitmaps have been available for a long time, but including them on a new array by default is recent.  They are most useful if a device is unexpectedly disconnected but otherwise OK.  mdadm --re-add will allow that device to rejoin the array and only the missing data (tracked by the bitmap) will have to be written to sync it up.

> I wonder if I should and could add a bitmap for the other devices as well?

There is a performance hit, so it should be considered on a case by case basis.

Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: "bitmap" line in /proc/mdstat

[Ramon Hofer]

On 14/10/16 18:08, Peter Sangas wrote:
> Can you quantify the performance decrease and if rebuild time is not as important as performance how does one remove the bitmap?

 From the Linux RAID wiki:
"Some limited measurements I have done suggest up to 10% slowdown, 
though usually less. Possibly some tuning can make it much better." [1]

[1] 
https://raid.wiki.kernel.org/index.php/Write-intent_bitmap#How_Bitmaps_Work_by_Q.26A

Regards,
Ramon

RE: "bitmap" line in /proc/mdstat

[Peter Sangas]

Thanks Ramon.

Do you know why some of your mds were created with bitmaps and others were not?  When I installed Ubuntu 16.04 and used the installer to create a RAID1 some of the mds had bitmaps and some did not.

Thanks,
Pete

-----Original Message-----
From: Ramon Hofer [mailto:ramonhofer@bluewin.ch] 
Sent: Friday, October 14, 2016 9:30 AM
To: Peter Sangas; linux-raid@vger.kernel.org
Subject: Re: "bitmap" line in /proc/mdstat

On 14/10/16 18:08, Peter Sangas wrote:
> Can you quantify the performance decrease and if rebuild time is not as important as performance how does one remove the bitmap?

 From the Linux RAID wiki:
"Some limited measurements I have done suggest up to 10% slowdown, though usually less. Possibly some tuning can make it much better." [1]

[1]
https://raid.wiki.kernel.org/index.php/Write-intent_bitmap#How_Bitmaps_Work_by_Q.26A

Regards,
Ramon
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: "bitmap" line in /proc/mdstat

[Ramon Hofer]

On 14/10/16 19:32, Peter Sangas wrote:
> Do you know why some of your mds were created with bitmaps and others were not?  When I installed Ubuntu 16.04 and used the installer to create a RAID1 some of the mds had bitmaps and some did not.

Yes, I do. Phil Turmel answer my initial question:

On 14/10/16 13:53, Phil Turmel wrote:
 > Bitmaps have been available for a long time, but including them on a
 > new array by default is recent.

http://marc.info/?l=linux-raid&m=147644598622369&w=2


You can remove the bitmap while the RAID is running with

# mdadm -G /dev/md5 -b none

or add it with

# mdadm -G /dev/md5 -b internal


Regards,
Ramon

Re: "bitmap" line in /proc/mdstat

[Andreas Klauer]

On Fri, Oct 14, 2016 at 10:32:47AM -0700, Peter Sangas wrote:
> I installed Ubuntu 16.04 and used the installer to create a RAID1 some of the mds had bitmaps and some did not.

Small ones (/boot, swap) might not get a bitmap by default.
According to the manpage it needs 100G or larger.

|       -b, --bitmap=
|[...]
|              When  creating  an  array  on  devices which are 100G or larger,
|              mdadm automatically adds an internal bitmap as it  will  usually
|              be beneficial.  This can be suppressed with --bitmap=none .



Regards
Andreas Klauer