Bitmap in RAM?

Bitmap in RAM?

[Dark Penguin]

After researching write-intent bitmaps for a while, my understanding is 
that they are used only to speed up "re-adding" drives by avoiding a 
full resync, and to enable --write-mostly --write-behind. However, it 
does introduce some pretty heavy load on whatever device it's on, 
especially if it's an internal bitmap, because the head would have to 
fly all the way to the superblock twice per each write. If it's an 
external bitmap, then the device it's on would be too busy just serving 
it to do anything else.

So if I were to place it on a tmpfs, I could eliminate this problem only 
at the expense of being unable to re-add drives after a reboot, right?.. 
I've read somewhere that bitmaps only work correctly on ext2 or ext3 
filesystems, but that probably means that it's not a good idea to put it 
on a filesystem with delayed allocation like ext4 of zfs, otherwise I 
don't understand why - and so I don't know if there would be any problem 
with it running on tmpfs. Is there?..

By the way, Phil, you are a hero! :) I remember that it was you who 
taught me about the "timeouts and scrubbing" problem a year ago, and you 
always explain things so well! You must have a lot of patience and love 
for all people! :)


-- 
darkpenguin

Re: Bitmap in RAM?

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 722 bytes --]

On Sat, 8 Oct 2016 15:54:26 +0300
Dark Penguin <darkpenguin@yandex.ru> wrote:

> So if I were to place it on a tmpfs, I could eliminate this problem only 
> at the expense of being unable to re-add drives after a reboot, right?.. 

If you don't need that ability, you can just remove bitmap entirely, it's not
mandatory. Run:

  mdadm --grow --bitmap=none /dev/mdX

However I'd say being able to re-add drives is very useful, so first consider
switching to a higher bitmap granularity, 

  mdadm --grow --bitmap=none /dev/mdX
  mdadm --grow --bitmap=internal --bitmap-chunk=131072 /dev/mdX

(or even 262144, 524288) as that will reduce the performance impact of the
bitmap.

-- 
With respect,
Roman

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: Bitmap in RAM?

[Dark Penguin]

If I remove it entirely, I won't be able to re-add drives at all. If I 
move it to a tmpfs, then I can re-add them, but not across reboots - and 
with no downsides, which I want to confirm. So this would be better than 
removing it completely.

I've thought about it and switched the external bitmaps to the 
chunk-size of 65536, which apparently is the default for intermal 
bitmaps. They've become much smaller, which means the default size 
selected for them before was indeed much higher. I'll see if I notice 
any difference the next time I'm moving data around; maybe the load will 
indeed be negligible.


On 08/10/16 19:02, Roman Mamedov wrote:
> On Sat, 8 Oct 2016 15:54:26 +0300
> Dark Penguin <darkpenguin@yandex.ru> wrote:
>
>> So if I were to place it on a tmpfs, I could eliminate this problem only
>> at the expense of being unable to re-add drives after a reboot, right?..
>
> If you don't need that ability, you can just remove bitmap entirely, it's not
> mandatory. Run:
>
>    mdadm --grow --bitmap=none /dev/mdX
>
> However I'd say being able to re-add drives is very useful, so first consider
> switching to a higher bitmap granularity,
>
>    mdadm --grow --bitmap=none /dev/mdX
>    mdadm --grow --bitmap=internal --bitmap-chunk=131072 /dev/mdX
>
> (or even 262144, 524288) as that will reduce the performance impact of the
> bitmap.
>

-- 
darkpenguin

Re: Bitmap in RAM?

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1942 bytes --]

On Sat, Oct 08 2016, Dark Penguin wrote:

> After researching write-intent bitmaps for a while, my understanding is 
> that they are used only to speed up "re-adding" drives by avoiding a 
> full resync, and to enable --write-mostly --write-behind. However, it 

This is not correct.  Speeding up re-adding is certainly one benefit.
The other benefit is speeding up resync after a system crash.


> does introduce some pretty heavy load on whatever device it's on, 
> especially if it's an internal bitmap, because the head would have to 
> fly all the way to the superblock twice per each write. If it's an 
> external bitmap, then the device it's on would be too busy just serving 
> it to do anything else.

It doesn't update the bitmap immediately before and after every write.
Writes are batched, and the bitmap is updated once before each batch of
writes.
There does need to be another update to record that the write has
completed, but that is delayed and usually merged with the update at the
start of the next batch of writes.
So the bitmap is usually updated once per batch of writes.


>
> So if I were to place it on a tmpfs, I could eliminate this problem only 
> at the expense of being unable to re-add drives after a reboot, right?.. 
> I've read somewhere that bitmaps only work correctly on ext2 or ext3 
> filesystems, but that probably means that it's not a good idea to put it 
> on a filesystem with delayed allocation like ext4 of zfs, otherwise I 
> don't understand why - and so I don't know if there would be any problem 
> with it running on tmpfs. Is there?..

You could create a ramdisk, create an ext2 filesystem on that, and put
the bitmap file there.

It probably would make sense to support in-memory bitmaps which never
get written to disk.  It would be fairly easy to do, and would allow
expedited re-add.  I just hasn't been done.
(patches welcome :-)

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]