* [linux-lvm] how to recover after thin pool metadata did fill up?
@ 2012-10-17 20:21 Andres Toomsalu
2012-10-18 10:30 ` Zdenek Kabelac
2012-10-18 13:35 ` Joe Thornber
0 siblings, 2 replies; 11+ messages in thread
From: Andres Toomsalu @ 2012-10-17 20:21 UTC (permalink / raw)
To: linux-lvm
Hi,
I'm aware that thin provisioning is not yet production ready (no metadata resize) - but is there a way to recover from thin pool failure when pool metadata was filled up?
I did setup 1.95T thin pool and after some usage pool metadata (128MB) was filling up to 99,08% - so all pool thin volumes went into read-only state.
Problem is that I cannot find a way in order to recover from this failure - eg also unable to delete/erase thin volumes and pool - only option seems to be full disk PV re-creation (eg OS re-install).
Is there a way to recover or delete thin pool/volumes - without erasing other (normal) LVs in this Volume Group?
For example dmsetup remove didn't help.
Some diagnostic output:
lvs -a -o+metadata_percent
dm_report_object: report function failed for field data_percent
--- REPEATABLE MSG ---
dm_report_object: report function failed for field data_percent
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert Meta%
pool VolGroupL0 twi-i-tz 1,95t 75,28 99,08
[pool_tdata] VolGroupL0 Twi-aot- 1,95t
[pool_tmeta] VolGroupL0 ewi-aot- 128,00m
root VolGroupL0 -wi-ao-- 10,00g
swap VolGroupL0 -wi-ao-- 16,00g
thin_backup VolGroupL0 Vwi-i-tz 700,00g pool
thin_storage VolGroupL0 Vwi---tz 900,00g pool
thin_storage-snapshot1 VolGroupL0 Vwi-i-tz 700,00g pool thin_storage
thin_storage-snapshot106 VolGroupL0 Vwi-i-tz 900,00g pool thin_storage
thin_storage-snapshot130 VolGroupL0 Vwi-i-tz 900,00g pool thin_storage
thin_storage-snapshot154 VolGroupL0 Vwi-i-tz 900,00g pool thin_storage
thin_storage-snapshot178 VolGroupL0 Vwi-i-tz 900,00g pool thin_storage
thin_storage-snapshot2 VolGroupL0 Vwi-i-tz 700,00g pool thin_storage
thin_storage-snapshot202 VolGroupL0 Vwi-i-tz 900,00g pool thin_storage
dmsetup table
VolGroupL0-thin_storage--snapshot2:
VolGroupL0-thin_storage--snapshot178:
VolGroupL0-swap: 0 33554432 linear 8:2 41945088
VolGroupL0-thin_storage--snapshot1:
VolGroupL0-root: 0 20971520 linear 8:2 20973568
VolGroupL0-thin_storage--snapshot130:
VolGroupL0-pool:
VolGroupL0-thin_backup:
VolGroupL0-thin_storage--snapshot106:
VolGroupL0-thin_storage--snapshot154:
VolGroupL0-pool-tpool: 0 4194304000 thin-pool 253:2 253:3 1024 0 0
VolGroupL0-pool_tdata: 0 2097152000 linear 8:2 75499520
VolGroupL0-pool_tdata: 2097152000 2097152000 linear 8:2 2172913664
VolGroupL0-pool_tmeta: 0 262144 linear 8:2 2172651520
VolGroupL0-thin_storage--snapshot202:
lvremove -f VolGroupL0/pool
Thin pool transaction_id=640, while expected: 643.
Unable to deactivate open VolGroupL0-pool_tdata (253:3)
Unable to deactivate open VolGroupL0-pool_tmeta (253:2)
Failed to deactivate VolGroupL0-pool-tpool
Failed to resume pool.
Failed to update thin pool pool.
--
----------------------------------------------
Andres Toomsalu, andres@active.ee
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [linux-lvm] how to recover after thin pool metadata did fill up?
2012-10-17 20:21 [linux-lvm] how to recover after thin pool metadata did fill up? Andres Toomsalu
@ 2012-10-18 10:30 ` Zdenek Kabelac
2012-10-18 10:42 ` Andres Toomsalu
2012-10-18 13:28 ` Spelic
2012-10-18 13:35 ` Joe Thornber
1 sibling, 2 replies; 11+ messages in thread
From: Zdenek Kabelac @ 2012-10-18 10:30 UTC (permalink / raw)
To: LVM general discussion and development; +Cc: Andres Toomsalu
Dne 17.10.2012 22:21, Andres Toomsalu napsal(a):
> Hi,
>
> I'm aware that thin provisioning is not yet production ready (no metadata resize) - but is there a way to recover from thin pool failure when pool metadata was filled up?
>
> I did setup 1.95T thin pool and after some usage pool metadata (128MB) was filling up to 99,08% - so all pool thin volumes went into read-only state.
> Problem is that I cannot find a way in order to recover from this failure - eg also unable to delete/erase thin volumes and pool - only option seems to be full disk PV re-creation (eg OS re-install).
>
> Is there a way to recover or delete thin pool/volumes - without erasing other (normal) LVs in this Volume Group?
> For example dmsetup remove didn't help.
>
> Some diagnostic output:
>
> lvs -a -o+metadata_percent
> dm_report_object: report function failed for field data_percent
> --- REPEATABLE MSG ---
> dm_report_object: report function failed for field data_percent
> LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert Meta%
> pool VolGroupL0 twi-i-tz 1,95t 75,28 99,08
> [pool_tdata] VolGroupL0 Twi-aot- 1,95t
> [pool_tmeta] VolGroupL0 ewi-aot- 128,00m
> root VolGroupL0 -wi-ao-- 10,00g
> swap VolGroupL0 -wi-ao-- 16,00g
> thin_backup VolGroupL0 Vwi-i-tz 700,00g pool
> thin_storage VolGroupL0 Vwi---tz 900,00g pool
> thin_storage-snapshot1 VolGroupL0 Vwi-i-tz 700,00g pool thin_storage
> thin_storage-snapshot106 VolGroupL0 Vwi-i-tz 900,00g pool thin_storage
> thin_storage-snapshot130 VolGroupL0 Vwi-i-tz 900,00g pool thin_storage
> thin_storage-snapshot154 VolGroupL0 Vwi-i-tz 900,00g pool thin_storage
> thin_storage-snapshot178 VolGroupL0 Vwi-i-tz 900,00g pool thin_storage
> thin_storage-snapshot2 VolGroupL0 Vwi-i-tz 700,00g pool thin_storage
> thin_storage-snapshot202 VolGroupL0 Vwi-i-tz 900,00g pool thin_storage
>
> dmsetup table
> VolGroupL0-thin_storage--snapshot2:
> VolGroupL0-thin_storage--snapshot178:
> VolGroupL0-swap: 0 33554432 linear 8:2 41945088
> VolGroupL0-thin_storage--snapshot1:
> VolGroupL0-root: 0 20971520 linear 8:2 20973568
> VolGroupL0-thin_storage--snapshot130:
> VolGroupL0-pool:
> VolGroupL0-thin_backup:
> VolGroupL0-thin_storage--snapshot106:
> VolGroupL0-thin_storage--snapshot154:
> VolGroupL0-pool-tpool: 0 4194304000 thin-pool 253:2 253:3 1024 0 0
> VolGroupL0-pool_tdata: 0 2097152000 linear 8:2 75499520
> VolGroupL0-pool_tdata: 2097152000 2097152000 linear 8:2 2172913664
> VolGroupL0-pool_tmeta: 0 262144 linear 8:2 2172651520
> VolGroupL0-thin_storage--snapshot202:
>
> lvremove -f VolGroupL0/pool
> Thin pool transaction_id=640, while expected: 643.
> Unable to deactivate open VolGroupL0-pool_tdata (253:3)
> Unable to deactivate open VolGroupL0-pool_tmeta (253:2)
> Failed to deactivate VolGroupL0-pool-tpool
> Failed to resume pool.
> Failed to update thin pool pool.
>
Unfortunately there is no 'easy' advice for now yet - you hit current
Achilles heel of thinp support in lvm2 - we are thinking how to make recovery
usable for user - but it's not easy task since many things are making it very
complex - so it still needs some month of work.
As for now - you would need to download git source and disable certain
security check directly in the source to allow activation of damaged pool.
I guess I could provide you some extra patch that would allow you to active
thin pool in 'read-only' mode (but it's not yet ready for upstream).
Thus you should be able access 'some' data in this mode with a big note -
since devices would be in read-only mode - you cannot even run fsck on such
partition - and since you have transaction_id mismatch - it would need
some analysis to see what actually happened - are you able to provide me
archive files for the history of your recent lvm commands -
it should not allow you to have difference bigger then 1 - so there is
probably some bug (unless you are using some old version of lvm2 with initial
thinp support)
Zdenek
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [linux-lvm] how to recover after thin pool metadata did fill up?
2012-10-18 10:30 ` Zdenek Kabelac
@ 2012-10-18 10:42 ` Andres Toomsalu
2012-10-18 10:55 ` Andres Toomsalu
2012-10-18 11:29 ` Zdenek Kabelac
2012-10-18 13:28 ` Spelic
1 sibling, 2 replies; 11+ messages in thread
From: Andres Toomsalu @ 2012-10-18 10:42 UTC (permalink / raw)
To: LVM general discussion and development
Yes I know that its not currently possible to recover this pool anymore - and we got data copied from read-only mode anyway - but bigger problem is that there is even no known way to reset/recreate pool - as I cant delete/remove the pool - which means that only erasing all PV will help probably. Im just looking for a way to erase faulty pool (with erasing data) and to be able recreate it without full OS reinstall. Is there a way to do it - DM mapper low level commands perhaps?
Cheers,
--
----------------------------------------------
Andres Toomsalu, andres@active.ee
On 18.10.2012, at 13:30, Zdenek Kabelac wrote:
> Dne 17.10.2012 22:21, Andres Toomsalu napsal(a):
>> Hi,
>>
>> I'm aware that thin provisioning is not yet production ready (no metadata resize) - but is there a way to recover from thin pool failure when pool metadata was filled up?
>>
>> I did setup 1.95T thin pool and after some usage pool metadata (128MB) was filling up to 99,08% - so all pool thin volumes went into read-only state.
>> Problem is that I cannot find a way in order to recover from this failure - eg also unable to delete/erase thin volumes and pool - only option seems to be full disk PV re-creation (eg OS re-install).
>>
>> Is there a way to recover or delete thin pool/volumes - without erasing other (normal) LVs in this Volume Group?
>> For example dmsetup remove didn't help.
>>
>> Some diagnostic output:
>>
>> lvs -a -o+metadata_percent
>> dm_report_object: report function failed for field data_percent
>> --- REPEATABLE MSG ---
>> dm_report_object: report function failed for field data_percent
>> LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert Meta%
>> pool VolGroupL0 twi-i-tz 1,95t 75,28 99,08
>> [pool_tdata] VolGroupL0 Twi-aot- 1,95t
>> [pool_tmeta] VolGroupL0 ewi-aot- 128,00m
>> root VolGroupL0 -wi-ao-- 10,00g
>> swap VolGroupL0 -wi-ao-- 16,00g
>> thin_backup VolGroupL0 Vwi-i-tz 700,00g pool
>> thin_storage VolGroupL0 Vwi---tz 900,00g pool
>> thin_storage-snapshot1 VolGroupL0 Vwi-i-tz 700,00g pool thin_storage
>> thin_storage-snapshot106 VolGroupL0 Vwi-i-tz 900,00g pool thin_storage
>> thin_storage-snapshot130 VolGroupL0 Vwi-i-tz 900,00g pool thin_storage
>> thin_storage-snapshot154 VolGroupL0 Vwi-i-tz 900,00g pool thin_storage
>> thin_storage-snapshot178 VolGroupL0 Vwi-i-tz 900,00g pool thin_storage
>> thin_storage-snapshot2 VolGroupL0 Vwi-i-tz 700,00g pool thin_storage
>> thin_storage-snapshot202 VolGroupL0 Vwi-i-tz 900,00g pool thin_storage
>>
>> dmsetup table
>> VolGroupL0-thin_storage--snapshot2:
>> VolGroupL0-thin_storage--snapshot178:
>> VolGroupL0-swap: 0 33554432 linear 8:2 41945088
>> VolGroupL0-thin_storage--snapshot1:
>> VolGroupL0-root: 0 20971520 linear 8:2 20973568
>> VolGroupL0-thin_storage--snapshot130:
>> VolGroupL0-pool:
>> VolGroupL0-thin_backup:
>> VolGroupL0-thin_storage--snapshot106:
>> VolGroupL0-thin_storage--snapshot154:
>> VolGroupL0-pool-tpool: 0 4194304000 thin-pool 253:2 253:3 1024 0 0
>> VolGroupL0-pool_tdata: 0 2097152000 linear 8:2 75499520
>> VolGroupL0-pool_tdata: 2097152000 2097152000 linear 8:2 2172913664
>> VolGroupL0-pool_tmeta: 0 262144 linear 8:2 2172651520
>> VolGroupL0-thin_storage--snapshot202:
>>
>> lvremove -f VolGroupL0/pool
>> Thin pool transaction_id=640, while expected: 643.
>> Unable to deactivate open VolGroupL0-pool_tdata (253:3)
>> Unable to deactivate open VolGroupL0-pool_tmeta (253:2)
>> Failed to deactivate VolGroupL0-pool-tpool
>> Failed to resume pool.
>> Failed to update thin pool pool.
>>
>
>
> Unfortunately there is no 'easy' advice for now yet - you hit current Achilles heel of thinp support in lvm2 - we are thinking how to make recovery usable for user - but it's not easy task since many things are making it very complex - so it still needs some month of work.
>
> As for now - you would need to download git source and disable certain security check directly in the source to allow activation of damaged pool.
>
> I guess I could provide you some extra patch that would allow you to active thin pool in 'read-only' mode (but it's not yet ready for upstream).
>
> Thus you should be able access 'some' data in this mode with a big note - since devices would be in read-only mode - you cannot even run fsck on such partition - and since you have transaction_id mismatch - it would need
> some analysis to see what actually happened - are you able to provide me
> archive files for the history of your recent lvm commands -
> it should not allow you to have difference bigger then 1 - so there is probably some bug (unless you are using some old version of lvm2 with initial thinp support)
>
> Zdenek
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [linux-lvm] how to recover after thin pool metadata did fill up?
2012-10-18 10:42 ` Andres Toomsalu
@ 2012-10-18 10:55 ` Andres Toomsalu
2012-10-18 11:29 ` Zdenek Kabelac
1 sibling, 0 replies; 11+ messages in thread
From: Andres Toomsalu @ 2012-10-18 10:55 UTC (permalink / raw)
To: LVM general discussion and development
[-- Attachment #1: Type: text/plain, Size: 5770 bytes --]
I would happily send you the diagnostic logs - if you could specify what needs to be collected.
--
----------------------------------------------
Andres Toomsalu, andres@active.ee
On 18.10.2012, at 13:42, Andres Toomsalu wrote:
> Yes I know that its not currently possible to recover this pool anymore - and we got data copied from read-only mode anyway - but bigger problem is that there is even no known way to reset/recreate pool - as I cant delete/remove the pool - which means that only erasing all PV will help probably. Im just looking for a way to erase faulty pool (with erasing data) and to be able recreate it without full OS reinstall. Is there a way to do it - DM mapper low level commands perhaps?
>
> Cheers,
> --
> ----------------------------------------------
> Andres Toomsalu, andres@active.ee
>
>
>
>
>
>
> On 18.10.2012, at 13:30, Zdenek Kabelac wrote:
>
>> Dne 17.10.2012 22:21, Andres Toomsalu napsal(a):
>>> Hi,
>>>
>>> I'm aware that thin provisioning is not yet production ready (no metadata resize) - but is there a way to recover from thin pool failure when pool metadata was filled up?
>>>
>>> I did setup 1.95T thin pool and after some usage pool metadata (128MB) was filling up to 99,08% - so all pool thin volumes went into read-only state.
>>> Problem is that I cannot find a way in order to recover from this failure - eg also unable to delete/erase thin volumes and pool - only option seems to be full disk PV re-creation (eg OS re-install).
>>>
>>> Is there a way to recover or delete thin pool/volumes - without erasing other (normal) LVs in this Volume Group?
>>> For example dmsetup remove didn't help.
>>>
>>> Some diagnostic output:
>>>
>>> lvs -a -o+metadata_percent
>>> dm_report_object: report function failed for field data_percent
>>> --- REPEATABLE MSG ---
>>> dm_report_object: report function failed for field data_percent
>>> LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert Meta%
>>> pool VolGroupL0 twi-i-tz 1,95t 75,28 99,08
>>> [pool_tdata] VolGroupL0 Twi-aot- 1,95t
>>> [pool_tmeta] VolGroupL0 ewi-aot- 128,00m
>>> root VolGroupL0 -wi-ao-- 10,00g
>>> swap VolGroupL0 -wi-ao-- 16,00g
>>> thin_backup VolGroupL0 Vwi-i-tz 700,00g pool
>>> thin_storage VolGroupL0 Vwi---tz 900,00g pool
>>> thin_storage-snapshot1 VolGroupL0 Vwi-i-tz 700,00g pool thin_storage
>>> thin_storage-snapshot106 VolGroupL0 Vwi-i-tz 900,00g pool thin_storage
>>> thin_storage-snapshot130 VolGroupL0 Vwi-i-tz 900,00g pool thin_storage
>>> thin_storage-snapshot154 VolGroupL0 Vwi-i-tz 900,00g pool thin_storage
>>> thin_storage-snapshot178 VolGroupL0 Vwi-i-tz 900,00g pool thin_storage
>>> thin_storage-snapshot2 VolGroupL0 Vwi-i-tz 700,00g pool thin_storage
>>> thin_storage-snapshot202 VolGroupL0 Vwi-i-tz 900,00g pool thin_storage
>>>
>>> dmsetup table
>>> VolGroupL0-thin_storage--snapshot2:
>>> VolGroupL0-thin_storage--snapshot178:
>>> VolGroupL0-swap: 0 33554432 linear 8:2 41945088
>>> VolGroupL0-thin_storage--snapshot1:
>>> VolGroupL0-root: 0 20971520 linear 8:2 20973568
>>> VolGroupL0-thin_storage--snapshot130:
>>> VolGroupL0-pool:
>>> VolGroupL0-thin_backup:
>>> VolGroupL0-thin_storage--snapshot106:
>>> VolGroupL0-thin_storage--snapshot154:
>>> VolGroupL0-pool-tpool: 0 4194304000 thin-pool 253:2 253:3 1024 0 0
>>> VolGroupL0-pool_tdata: 0 2097152000 linear 8:2 75499520
>>> VolGroupL0-pool_tdata: 2097152000 2097152000 linear 8:2 2172913664
>>> VolGroupL0-pool_tmeta: 0 262144 linear 8:2 2172651520
>>> VolGroupL0-thin_storage--snapshot202:
>>>
>>> lvremove -f VolGroupL0/pool
>>> Thin pool transaction_id=640, while expected: 643.
>>> Unable to deactivate open VolGroupL0-pool_tdata (253:3)
>>> Unable to deactivate open VolGroupL0-pool_tmeta (253:2)
>>> Failed to deactivate VolGroupL0-pool-tpool
>>> Failed to resume pool.
>>> Failed to update thin pool pool.
>>>
>>
>>
>> Unfortunately there is no 'easy' advice for now yet - you hit current Achilles heel of thinp support in lvm2 - we are thinking how to make recovery usable for user - but it's not easy task since many things are making it very complex - so it still needs some month of work.
>>
>> As for now - you would need to download git source and disable certain security check directly in the source to allow activation of damaged pool.
>>
>> I guess I could provide you some extra patch that would allow you to active thin pool in 'read-only' mode (but it's not yet ready for upstream).
>>
>> Thus you should be able access 'some' data in this mode with a big note - since devices would be in read-only mode - you cannot even run fsck on such partition - and since you have transaction_id mismatch - it would need
>> some analysis to see what actually happened - are you able to provide me
>> archive files for the history of your recent lvm commands -
>> it should not allow you to have difference bigger then 1 - so there is probably some bug (unless you are using some old version of lvm2 with initial thinp support)
>>
>> Zdenek
>>
>> _______________________________________________
>> linux-lvm mailing list
>> linux-lvm@redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-lvm
>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>>
>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
[-- Attachment #2: Type: text/html, Size: 16005 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [linux-lvm] how to recover after thin pool metadata did fill up?
2012-10-18 10:42 ` Andres Toomsalu
2012-10-18 10:55 ` Andres Toomsalu
@ 2012-10-18 11:29 ` Zdenek Kabelac
1 sibling, 0 replies; 11+ messages in thread
From: Zdenek Kabelac @ 2012-10-18 11:29 UTC (permalink / raw)
To: LVM general discussion and development; +Cc: Andres Toomsalu
Dne 18.10.2012 12:42, Andres Toomsalu napsal(a):
> Yes I know that its not currently possible to recover this pool anymore - and we got data copied from read-only mode anyway - but bigger problem is that there is even no known way to reset/recreate pool - as I cant delete/remove the pool - which means that only erasing all PV will help probably. Im just looking for a way to erase faulty pool (with erasing data) and to be able recreate it without full OS reinstall. Is there a way to do it - DM mapper low level commands perhaps?
>
This case is pretty simple this way:
vgcfgbackup -f vg.bak vgname
edit vg.bak and remove all thinp related volumes
vgcfgrestore -f vg.bak vgname
The reason why current upstream code rather avoids modifying metadata by
tools is - to avoid making first worst for recover in the first place.
Please note we already have some work done towards removing mutliple LVs
more efficiently - https://bugzilla.redhat.com/show_bug.cgi?id=817768
Zdenek
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [linux-lvm] how to recover after thin pool metadata did fill up?
2012-10-18 10:30 ` Zdenek Kabelac
2012-10-18 10:42 ` Andres Toomsalu
@ 2012-10-18 13:28 ` Spelic
2012-10-18 13:35 ` Andres Toomsalu
` (2 more replies)
1 sibling, 3 replies; 11+ messages in thread
From: Spelic @ 2012-10-18 13:28 UTC (permalink / raw)
To: LVM general discussion and development; +Cc: Andres Toomsalu, Zdenek Kabelac
On 10/18/12 12:30, Zdenek Kabelac wrote:
> Dne 17.10.2012 22:21, Andres Toomsalu napsal(a):
>> Hi,
>>
>> I'm aware that thin provisioning is not yet production ready (no
>> metadata resize) - but is there a way to recover from thin pool
>> failure when pool metadata was filled up?
>>
>>
>
>
> Unfortunately there is no 'easy' advice for now yet - you hit current
> Achilles heel of thinp support in lvm2 - we are thinking how to make
> recovery usable for user - but it's not easy task since many things
> are making it very complex - so it still needs some month of work.
>
So, supposing one is aware of this problem beforehand, at pool creation
can this problem be worked around by using --poolmetadatasize to make a
metadata volume much larger than the default?
And if yes, do you have any advice on the metadata size we should use?
Thank you
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [linux-lvm] how to recover after thin pool metadata did fill up?
2012-10-18 13:28 ` Spelic
@ 2012-10-18 13:35 ` Andres Toomsalu
2012-10-18 13:43 ` Zdenek Kabelac
2012-10-18 14:01 ` Joe Thornber
2 siblings, 0 replies; 11+ messages in thread
From: Andres Toomsalu @ 2012-10-18 13:35 UTC (permalink / raw)
To: LVM general discussion and development
[-- Attachment #1: Type: text/plain, Size: 1438 bytes --]
It has been commented that about 500MB should be enough by far - but as we could fill 128MB metadata easily with 1TB pool size just rewriting some 600GB during thin snapshot existance - I would go and reserve much more for metadata - so that it wont become an issue for sure. Probably 2-5GB for --poolmetadatasize should guarantee pool lifetime. At least by this time metadata area resizing should be supported I gather :)
--
----------------------------------------------
Andres Toomsalu
On 18.10.2012, at 16:28, Spelic wrote:
> On 10/18/12 12:30, Zdenek Kabelac wrote:
>> Dne 17.10.2012 22:21, Andres Toomsalu napsal(a):
>>> Hi,
>>>
>>> I'm aware that thin provisioning is not yet production ready (no metadata resize) - but is there a way to recover from thin pool failure when pool metadata was filled up?
>>>
>>>
>>
>>
>> Unfortunately there is no 'easy' advice for now yet - you hit current Achilles heel of thinp support in lvm2 - we are thinking how to make recovery usable for user - but it's not easy task since many things are making it very complex - so it still needs some month of work.
>>
>
> So, supposing one is aware of this problem beforehand, at pool creation
> can this problem be worked around by using --poolmetadatasize to make a metadata volume much larger than the default?
>
> And if yes, do you have any advice on the metadata size we should use?
>
> Thank you
>
>
[-- Attachment #2: Type: text/html, Size: 3253 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [linux-lvm] how to recover after thin pool metadata did fill up?
2012-10-18 13:28 ` Spelic
2012-10-18 13:35 ` Andres Toomsalu
@ 2012-10-18 13:43 ` Zdenek Kabelac
2012-10-18 13:47 ` Andres Toomsalu
2012-10-18 14:01 ` Joe Thornber
2 siblings, 1 reply; 11+ messages in thread
From: Zdenek Kabelac @ 2012-10-18 13:43 UTC (permalink / raw)
To: LVM general discussion and development; +Cc: Andres Toomsalu, Spelic
Dne 18.10.2012 15:28, Spelic napsal(a):
> On 10/18/12 12:30, Zdenek Kabelac wrote:
>> Dne 17.10.2012 22:21, Andres Toomsalu napsal(a):
>>> Hi,
>>>
>>> I'm aware that thin provisioning is not yet production ready (no metadata
>>> resize) - but is there a way to recover from thin pool failure when pool
>>> metadata was filled up?
>>>
>>>
>>
>>
>> Unfortunately there is no 'easy' advice for now yet - you hit current
>> Achilles heel of thinp support in lvm2 - we are thinking how to make
>> recovery usable for user - but it's not easy task since many things are
>> making it very complex - so it still needs some month of work.
>>
>
> So, supposing one is aware of this problem beforehand, at pool creation
> can this problem be worked around by using --poolmetadatasize to make a
> metadata volume much larger than the default?
>
> And if yes, do you have any advice on the metadata size we should use?
>
It really depends on the pool size and use-cases.
The max size is ~16GB for metadata.
The reasonable max size is probably within 4GB if you want to stay safe in
the most possible conditions.
With 3.7 kernel and the next release of lvm2 (2.02.99) it's expected full
support for live size extension of metadata device.
Zdenek
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [linux-lvm] how to recover after thin pool metadata did fill up?
2012-10-18 13:43 ` Zdenek Kabelac
@ 2012-10-18 13:47 ` Andres Toomsalu
0 siblings, 0 replies; 11+ messages in thread
From: Andres Toomsalu @ 2012-10-18 13:47 UTC (permalink / raw)
To: LVM general discussion and development
[-- Attachment #1: Type: text/plain, Size: 1514 bytes --]
Is it intended to be backported to RHEL6 kernels as well?
--
----------------------------------------------
Andres Toomsalu
On 18.10.2012, at 16:43, Zdenek Kabelac wrote:
> Dne 18.10.2012 15:28, Spelic napsal(a):
>> On 10/18/12 12:30, Zdenek Kabelac wrote:
>>> Dne 17.10.2012 22:21, Andres Toomsalu napsal(a):
>>>> Hi,
>>>>
>>>> I'm aware that thin provisioning is not yet production ready (no metadata
>>>> resize) - but is there a way to recover from thin pool failure when pool
>>>> metadata was filled up?
>>>>
>>>>
>>>
>>>
>>> Unfortunately there is no 'easy' advice for now yet - you hit current
>>> Achilles heel of thinp support in lvm2 - we are thinking how to make
>>> recovery usable for user - but it's not easy task since many things are
>>> making it very complex - so it still needs some month of work.
>>>
>>
>> So, supposing one is aware of this problem beforehand, at pool creation
>> can this problem be worked around by using --poolmetadatasize to make a
>> metadata volume much larger than the default?
>>
>> And if yes, do you have any advice on the metadata size we should use?
>>
>
> It really depends on the pool size and use-cases.
>
> The max size is ~16GB for metadata.
> The reasonable max size is probably within 4GB if you want to stay safe in the most possible conditions.
>
> With 3.7 kernel and the next release of lvm2 (2.02.99) it's expected full support for live size extension of metadata device.
>
> Zdenek
>
>
[-- Attachment #2: Type: text/html, Size: 4408 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [linux-lvm] how to recover after thin pool metadata did fill up?
2012-10-18 13:28 ` Spelic
2012-10-18 13:35 ` Andres Toomsalu
2012-10-18 13:43 ` Zdenek Kabelac
@ 2012-10-18 14:01 ` Joe Thornber
2 siblings, 0 replies; 11+ messages in thread
From: Joe Thornber @ 2012-10-18 14:01 UTC (permalink / raw)
To: LVM general discussion and development; +Cc: Andres Toomsalu, Zdenek Kabelac
On Thu, Oct 18, 2012 at 03:28:17PM +0200, Spelic wrote:
> So, supposing one is aware of this problem beforehand, at pool creation
> can this problem be worked around by using --poolmetadatasize to
> make a metadata volume much larger than the default?
>
> And if yes, do you have any advice on the metadata size we should use?
There are various factors that effect this.
- block size
- data dev size
- nr snapshots
The rule of thumb I've been giving is:
work out the number of blocks in your data device. ie. data_dev_size / data_block_size.
Then multiply by 64. This gives the metadata size in bytes.
The above calculation should be fine for people who're primarily using
thinp for thin provisioning, and not lots of snapshots. I recommend
these people use a large block size. eg, 4M. I don't think this is
what lvm does by default (at one point it was using a block size of
64k).
Snapshots require extra copies of the metadata for devices. Often the
data is shared, but the btrees for the metadata diverge as cow
exceptions occur. So if you're using snapshots allocate more space.
This is compounded by the fact that it's often better to use small
block sizes for snapshots.
- Joe
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [linux-lvm] how to recover after thin pool metadata did fill up?
2012-10-17 20:21 [linux-lvm] how to recover after thin pool metadata did fill up? Andres Toomsalu
2012-10-18 10:30 ` Zdenek Kabelac
@ 2012-10-18 13:35 ` Joe Thornber
1 sibling, 0 replies; 11+ messages in thread
From: Joe Thornber @ 2012-10-18 13:35 UTC (permalink / raw)
To: LVM general discussion and development
On Wed, Oct 17, 2012 at 11:21:35PM +0300, Andres Toomsalu wrote:
> Hi,
>
> I'm aware that thin provisioning is not yet production ready (no metadata resize) - but is there a way to recover from thin pool failure when pool metadata was filled up?
We don't support online resizing yet. But you can do this offline by following these rough steps:
i) dump your metadata device to a text file using thin_dump. You may
want to eyeball this to see if it's what you expect.
ii) Create a new metadata lv. Better to create a completely new one
than try and extend the old one and risk losing data if you make a
mistake.
iii) Use thin_restore to write the dump to the new metadata volume
iv) Tell lvm to use the new metadata area for your pool. I'm not sure how to do this last step. Kabi can you help please?
- Joe
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2012-10-18 14:01 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-10-17 20:21 [linux-lvm] how to recover after thin pool metadata did fill up? Andres Toomsalu
2012-10-18 10:30 ` Zdenek Kabelac
2012-10-18 10:42 ` Andres Toomsalu
2012-10-18 10:55 ` Andres Toomsalu
2012-10-18 11:29 ` Zdenek Kabelac
2012-10-18 13:28 ` Spelic
2012-10-18 13:35 ` Andres Toomsalu
2012-10-18 13:43 ` Zdenek Kabelac
2012-10-18 13:47 ` Andres Toomsalu
2012-10-18 14:01 ` Joe Thornber
2012-10-18 13:35 ` Joe Thornber
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).