[linux-lvm] Slow boot with large number of thin snapshots

[linux-lvm] Slow boot with large number of thin snapshots

[Andres Toomsalu]

[-- Attachment #1: Type: text/plain, Size: 1216 bytes --]

Hi,

We have experienced a situation where we are left with large number of thin snapshots (around 500) and snapshot removal wont work anymore (we suspect a lock-up situation caused by trying to remove thin snapshots that is still mounted - thou  its not verified yet - just a speculation).  The only path to recover seems to be disabling all thin volumes from fstab and booting into single-user mode - where lvremove -f usually succeeds on these thin snapshots. But LVM init and lvremove take a huge amount of time with large number of snapshots  - initial LVM activation is going on about 6-8 minutes at least and every lvremove takes ca 20-50 sec - depending on snapshot count left. After snapshots are removed thin volumes can be enabled again and fsck-ed and resuming normal operations.

Althou we are not yet sure what is causing the lockup - so that snapshots cant be removed anymore without reboot - its also problematic that with large numbers of snapshots all LVM device operations are dead slow.  I hope this issue can be addressed somehow in the future.

Kind regards,
-- 
----------------------------------------------
Andres Toomsalu, andres@opennodecloud.com
http://www.opennodecloud.com


[-- Attachment #2: Type: text/html, Size: 3397 bytes --]

Re: [linux-lvm] Slow boot with large number of thin snapshots

[Marian Csontos]

On 02/07/2013 01:42 PM, Andres Toomsalu wrote:
> Hi,
>
> We have experienced a situation where we are left with large number
> of thin snapshots (around 500) and snapshot removal wont work anymore
> (we suspect a lock-up situation caused by trying to remove thin
> snapshots that is still mounted - thou  its not verified yet - just a
> speculation).  The only path to recover seems to be disabling all
> thin volumes from fstab and booting into single-user mode - where
> lvremove -f usually succeeds on these thin snapshots. But LVM init
> and lvremove take a huge amount of time with large number of
> snapshots  - initial LVM activation is going on about 6-8 minutes at
> least and every lvremove takes ca 20-50 sec - depending on snapshot
> count left. After snapshots are removed thin volumes can be enabled
> again and fsck-ed and resuming normal operations.

Hello Andres, could you provide the following information:

Which version of lvm?
Is lvmetad used?

     grep lvmetad /etc/lvm/lvm.conf
     pgrep lvmetad

RHEL6.3 version of lvmetad had a serious performance issue with many 
[thin] snapshots, which should be solved in the latest release 
(lvm2-2.02.98)

Could you check the memory usage while the command is running?

Also there were a high memory usage problems fixed in the recent version.

-- Marian

>
> Althou we are not yet sure what is causing the lockup - so that
> snapshots cant be removed anymore without reboot - its also
> problematic that with large numbers of snapshots all LVM device
> operations are dead slow.  I hope this issue can be addressed somehow
> in the future.
>
> Kind regards,
>
>
>
> _______________________________________________ linux-lvm mailing
> list linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO
> at http://tldp.org/HOWTO/LVM-HOWTO/