[linux-lvm] Backup superblock for thin provision?

[linux-lvm] Backup superblock for thin provision?

[Gionatan Danti]

Hi all,
using thin provisioning in production machines (using it mostly for its 
fast snapshot support, rather than for thin provision / storage 
overcommit by itself), I wonder what to do if a critical metadata 
corruption, as the loss of the superblock, should happen.

Filesystems generally have some backup copy of the superblock; should 
the primary one fail, another copy can be used.

So I have the following questions:
- how about thin LVM? Has it a backup superblock somewhere?
- how can the metadata be reliable backupped without shutting down the 
volume?
- more generally, how to deal with metadata backup? Does vgcfgrestore 
works for thin volumes?

Thank you all.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: [linux-lvm] Backup superblock for thin provision?

[Mike Snitzer]

On Fri, May 13 2016 at  6:39am -0400,
Gionatan Danti <g.danti@assyoma.it> wrote:

> Hi all,
> using thin provisioning in production machines (using it mostly for
> its fast snapshot support, rather than for thin provision / storage
> overcommit by itself), I wonder what to do if a critical metadata
> corruption, as the loss of the superblock, should happen.
> 
> Filesystems generally have some backup copy of the superblock;
> should the primary one fail, another copy can be used.
> 
> So I have the following questions:
> - how about thin LVM? Has it a backup superblock somewhere?
> - how can the metadata be reliable backupped without shutting down
> the volume?
> - more generally, how to deal with metadata backup? Does
> vgcfgrestore works for thin volumes?
> 
> Thank you all.

There is more to the thinp metadata than just the metadata superblock.

The DM thin-pool's metadata device was purposely split out from the data
device to allow for additional metadata fault protection using RAID.

I'll defer to the LVM developers for if/how LVM can be used to have
thinp metadata redundancy even if you don't have multiple devices to be
able to use a conventional RAID device.

Re: [linux-lvm] Backup superblock for thin provision?

[Zdenek Kabelac]

On 16.5.2016 20:55, Mike Snitzer wrote:
> On Fri, May 13 2016 at  6:39am -0400,
> Gionatan Danti <g.danti@assyoma.it> wrote:
>
>> Hi all,
>> using thin provisioning in production machines (using it mostly for
>> its fast snapshot support, rather than for thin provision / storage
>> overcommit by itself), I wonder what to do if a critical metadata
>> corruption, as the loss of the superblock, should happen.
>>
>> Filesystems generally have some backup copy of the superblock;
>> should the primary one fail, another copy can be used.
>>
>> So I have the following questions:
>> - how about thin LVM? Has it a backup superblock somewhere?
>> - how can the metadata be reliable backupped without shutting down
>> the volume?
>> - more generally, how to deal with metadata backup? Does
>> vgcfgrestore works for thin volumes?
>>
>> Thank you all.
>
> There is more to the thinp metadata than just the metadata superblock.
>
> The DM thin-pool's metadata device was purposely split out from the data
> device to allow for additional metadata fault protection using RAID.
>
> I'll defer to the LVM developers for if/how LVM can be used to have
> thinp metadata redundancy even if you don't have multiple devices to be
> able to use a conventional RAID device.

There is always the option to take 'metadata' snapshot and just
thin_dump content of metadata to a file (located in some 'safe' place)

However validation of 'restore'  if there are some 'snapshots' is questionable 
as the b-tree describing mapped blocks may change significantly
so 'rescued' content may than reference lots of bad blocks.

If you want to just protect 'superblock' against disk fault - usage
of 'raid' could be an option - but ATM there are some 'related' costs with 
management of 'stacked' device tree.

Regards

Zdenek

Re: [linux-lvm] Backup superblock for thin provision?

[Gionatan Danti]

>
> There is always the option to take 'metadata' snapshot and just
> thin_dump content of metadata to a file (located in some 'safe' place)
>

Something as "dmsetup reserve_metadata_snap", thin_dump, "dmsetup 
release_metadata_snap" ? If so, I'm already using for testing purpose ;)

> However validation of 'restore'  if there are some 'snapshots' is
> questionable as the b-tree describing mapped blocks may change
> significantly
> so 'rescued' content may than reference lots of bad blocks.

Sure, with snapshots the situation is surely more complex, as due to the 
CoW the old metadata can point to stale data, right?
>
> If you want to just protect 'superblock' against disk fault - usage
> of 'raid' could be an option - but ATM there are some 'related' costs
> with management of 'stacked' device tree.
>

Both my tmeta and tdata reside on a RAID6 array provided by a 
BBU-protected RAID card, so I should be safe here.


-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: [linux-lvm] Backup superblock for thin provision?

[Gionatan Danti]

On 16/05/2016 20:55, Mike Snitzer wrote:
>
> There is more to the thinp metadata than just the metadata superblock.
>
> The DM thin-pool's metadata device was purposely split out from the data
> device to allow for additional metadata fault protection using RAID.
>
> I'll defer to the LVM developers for if/how LVM can be used to have
> thinp metadata redundancy even if you don't have multiple devices to be
> able to use a conventional RAID device.
>

Sure - I'm using thinp on a RAID6 volume, with two disks redundancy.

But what about an user error, or a lvm-related bug, which invalidate the 
superblock? In this case, RAID is of no help. On the other side, a 
superblock backup copy can be used to restore the system without major 
data loss.

More in general, how do you deal with metadata backups?
Thanks.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8