From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx1.redhat.com (ext-mx14.extmail.prod.ext.phx2.redhat.com [10.5.110.19]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id r17CgJ7Q012360 for ; Thu, 7 Feb 2013 07:42:20 -0500 Received: from mail-lb0-f169.google.com (mail-lb0-f169.google.com [209.85.217.169]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id r17CgGRZ022110 for ; Thu, 7 Feb 2013 07:42:17 -0500 Received: by mail-lb0-f169.google.com with SMTP id m4so2088878lbo.28 for ; Thu, 07 Feb 2013 04:42:15 -0800 (PST) Sender: Andres Toomsalu From: Andres Toomsalu Content-Type: multipart/alternative; boundary="Apple-Mail=_0EA72255-0C19-44E8-A6E1-80CE9A164FE5" Date: Thu, 7 Feb 2013 14:42:12 +0200 Message-Id: Mime-Version: 1.0 (Apple Message framework v1283) Subject: [linux-lvm] Slow boot with large number of thin snapshots Reply-To: LVM general discussion and development List-Id: LVM general discussion and development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , List-Id: To: LVM general discussion and development --Apple-Mail=_0EA72255-0C19-44E8-A6E1-80CE9A164FE5 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi, We have experienced a situation where we are left with large number of = thin snapshots (around 500) and snapshot removal wont work anymore (we = suspect a lock-up situation caused by trying to remove thin snapshots = that is still mounted - thou its not verified yet - just a = speculation). The only path to recover seems to be disabling all thin = volumes from fstab and booting into single-user mode - where lvremove -f = usually succeeds on these thin snapshots. But LVM init and lvremove take = a huge amount of time with large number of snapshots - initial LVM = activation is going on about 6-8 minutes at least and every lvremove = takes ca 20-50 sec - depending on snapshot count left. After snapshots = are removed thin volumes can be enabled again and fsck-ed and resuming = normal operations. Althou we are not yet sure what is causing the lockup - so that = snapshots cant be removed anymore without reboot - its also problematic = that with large numbers of snapshots all LVM device operations are dead = slow. I hope this issue can be addressed somehow in the future. Kind regards, --=20 ---------------------------------------------- Andres Toomsalu, andres@opennodecloud.com http://www.opennodecloud.com --Apple-Mail=_0EA72255-0C19-44E8-A6E1-80CE9A164FE5 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii

andres@opennodecloud.com

= --Apple-Mail=_0EA72255-0C19-44E8-A6E1-80CE9A164FE5-- From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <511A2E14.6060907@redhat.com> Date: Tue, 12 Feb 2013 12:57:08 +0100 From: Marian Csontos MIME-Version: 1.0 References: In-Reply-To: Content-Transfer-Encoding: 7bit Subject: Re: [linux-lvm] Slow boot with large number of thin snapshots Reply-To: LVM general discussion and development List-Id: LVM general discussion and development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , List-Id: Content-Type: text/plain; charset="us-ascii"; format="flowed" To: LVM general discussion and development Cc: Andres Toomsalu On 02/07/2013 01:42 PM, Andres Toomsalu wrote: > Hi, > > We have experienced a situation where we are left with large number > of thin snapshots (around 500) and snapshot removal wont work anymore > (we suspect a lock-up situation caused by trying to remove thin > snapshots that is still mounted - thou its not verified yet - just a > speculation). The only path to recover seems to be disabling all > thin volumes from fstab and booting into single-user mode - where > lvremove -f usually succeeds on these thin snapshots. But LVM init > and lvremove take a huge amount of time with large number of > snapshots - initial LVM activation is going on about 6-8 minutes at > least and every lvremove takes ca 20-50 sec - depending on snapshot > count left. After snapshots are removed thin volumes can be enabled > again and fsck-ed and resuming normal operations. Hello Andres, could you provide the following information: Which version of lvm? Is lvmetad used? grep lvmetad /etc/lvm/lvm.conf pgrep lvmetad RHEL6.3 version of lvmetad had a serious performance issue with many [thin] snapshots, which should be solved in the latest release (lvm2-2.02.98) Could you check the memory usage while the command is running? Also there were a high memory usage problems fixed in the recent version. -- Marian > > Althou we are not yet sure what is causing the lockup - so that > snapshots cant be removed anymore without reboot - its also > problematic that with large numbers of snapshots all LVM device > operations are dead slow. I hope this issue can be addressed somehow > in the future. > > Kind regards, > > > > _______________________________________________ linux-lvm mailing > list linux-lvm@redhat.com > https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO > at http://tldp.org/HOWTO/LVM-HOWTO/