[linux-lvm] LVM snapshot limits

[linux-lvm] LVM snapshot limits

[Shaun Kruger]

[-- Attachment #1.1: Type: text/plain, Size: 874 bytes --]

I'm trying to create 45 LVM snapshots on one of my dev machines.  I have about 
7TB of disk to work with, but I keep having wierd issues.  I can't seem to 
make more than 30 snapshots.  At about 15-20 snapshots they stop being 
created reliably and as I get to 30 snapshots I stop getting snapshots and 
start getting COW table sized logical volumes that are in no way snapshots.  
Some of these snapshots cause kernel panics on RHEL 5 when I try to remove 
them.

If anyone wants to take a look, I have included a script that reproduces the 
issue.  It works on both RHEL 5 and Ubuntu 7.04.  The script only requires a 
few gigabytes of storage, so I don't think the problem has to do with my 
large volume of storage.  

I filed a support ticket with redhat, but I figured I'd float it here to see 
if anyone has any suggestions/experience.

Shaun Kruger

[-- Attachment #1.2: lvbreak --]
[-- Type: application/x-shellscript, Size: 216 bytes --]

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [linux-lvm] LVM snapshot limits

[Milan Broz]

Shaun Kruger wrote:
> I'm trying to create 45 LVM snapshots on one of my dev machines.  I have about 
> 7TB of disk to work with, but I keep having wierd issues.  I can't seem to 
> make more than 30 snapshots.  At about 15-20 snapshots they stop being 
> created reliably and as I get to 30 snapshots I stop getting snapshots and 
> start getting COW table sized logical volumes that are in no way snapshots.  
> Some of these snapshots cause kernel panics on RHEL 5 when I try to remove 
> them.

Hi,
both problems are already known and will be fixed in RHEL 5.1 release.
(See Red Hat Bugzilla bugs 211516 and 211525).
(Upstream kernel is already fixed.)

Anyway, multiple snapshots from the same origin can cause considerable 
performance decrease (there is no shared area for snapshots in current
implementation, so write to origin will cause new writes to every snapshot
copy-on-write area).

Milan
--
mbroz@redhat.com

Re: [linux-lvm] LVM snapshot limits

[Shaun Kruger]

[-- Attachment #1: Type: text/plain, Size: 754 bytes --]

Thanks for the information.  I'll post that to my redhat support ticket and 
see if they can help me with a more immediate fix.

> Anyway, multiple snapshots from the same origin can cause considerable
> performance decrease (there is no shared area for snapshots in current
> implementation, so write to origin will cause new writes to every snapshot
> copy-on-write area).

This copy on write issue had already occoured to us.  We were going to roll 
out with the concept of a persistent or nonpersistent database instance.  The 
nonpersistent ones were going to be removed before refreshing the data on the 
main database source.  There is another script that recreates all the 
developer instances after the update is complete.

Shaun

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [linux-lvm] LVM snapshot limits

[Shaun Kruger]

[-- Attachment #1: Type: text/plain, Size: 1011 bytes --]

> Hi,
> both problems are already known and will be fixed in RHEL 5.1 release.
> (See Red Hat Bugzilla bugs 211516 and 211525).
> (Upstream kernel is already fixed.)

Thanks again for your help on this issue.  Between your response and
redhat support I was able to quickly get a working kernel.

There is however one issue that probably needs attention.  If I create
snapshots until I get an error (about 200 snaps), remove them, and
repeat the creation/removal multiple times it is possible to destroy the
LVM metadata.  When this happens pvscan shows that there is no VG
associated with the physical volume.  Restarting results in full loss of
data.  I was able to reproduce this twice.

While it is not impacting my use of LVM snapshots this is probably a bug
that should be looked into.  It was quite worrying when my volume group
disappeared.

I can provide my scripts and detailed instructions on reproducing the
issue for anyone who is interested in looking into it.

Shaun Kruger

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [linux-lvm] LVM snapshot limits

[Milan Broz]

Shaun Kruger wrote:
> There is however one issue that probably needs attention.  If I create
> snapshots until I get an error (about 200 snaps), remove them, and
> repeat the creation/removal multiple times it is possible to destroy the
> LVM metadata.  When this happens pvscan shows that there is no VG
> associated with the physical volume.  Restarting results in full loss of
> data.  I was able to reproduce this twice.
> 
> While it is not impacting my use of LVM snapshots this is probably a bug
> that should be looked into.  It was quite worrying when my volume group
> disappeared.
> 
> I can provide my scripts and detailed instructions on reproducing the
> issue for anyone who is interested in looking into it.

Please create bug on bugzilla.redhat.com (lvm2 component)
(or report this through Red Hat support)
and attach your scripts there with description of problem, thanks.

Is the problem caused by overfilled metadata area on disk ?
(Will increasing metadata areas by "pvcreate --metadatasize ... " help ?)

Milan
--
mbroz@redhat.com

Re: [linux-lvm] LVM snapshot limits

[Stuart D. Gathman]

On Tue, 30 Oct 2007, Milan Broz wrote:

> > There is however one issue that probably needs attention.  If I create
> > snapshots until I get an error (about 200 snaps), remove them, and
> > repeat the creation/removal multiple times it is possible to destroy the
> > LVM metadata.  When this happens pvscan shows that there is no VG

Cool!  Great torture test!  BwaHaHaHa...

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.