[linux-lvm] 15 snapshot limit?

[linux-lvm] 15 snapshot limit?

[Larry Dickson]

[-- Attachment #1: Type: text/plain, Size: 804 bytes --]

Hi all,

With a 768 GB logical volume in a 2235 GB volume group, and 4 GB of memory,
I try to set up lots of snapshots (of 45 or 50 GB each) and can't get past
15. Sometimes I'll succeed in generating more than 15 COWs in /dev/mapper,
but lvdisplay never shows more than 15 snapshots in the "LV snapshot status
source of" listing under the parent volume. I searched and found 15
snapshots mentioned in a couple places like
wiki.samba.org/index.php/Shadow_Copies_with_Snapshots - but nothing really
authoritative.

Is there really such a limit? Or is it a memory requirement - I seem to
remember so many MB memory per GB of storage, but can't find that either.
Versions are (variants of) lvm2-2.02.38, device-mapper-1.02.26,
kernel-smp-2.6.20 (64 bit).

TIA,

Larry Dickson
Cutting Edge Networked Storage

[-- Attachment #2: Type: text/html, Size: 1231 bytes --]

Re: [linux-lvm] 15 snapshot limit?

[Milan Broz]

Larry Dickson wrote:
> With a 768 GB logical volume in a 2235 GB volume group, and 4 GB of
> memory, I try to set up lots of snapshots (of 45 or 50 GB each) and
> can't get past 15. Sometimes I'll succeed in generating more than 15
> COWs in /dev/mapper, but lvdisplay never shows more than 15 snapshots in
> the "LV snapshot status source of" listing under the parent volume. I
> searched and found 15 snapshots mentioned in a couple places like
> wiki.samba.org/index.php/Shadow_Copies_with_Snapshots
> <http://wiki.samba.org/index.php/Shadow_Copies_with_Snapshots> - but
> nothing really authoritative.
>  
> Is there really such a limit? Or is it a memory requirement - I seem to
> remember so many MB memory per GB of storage, but can't find that
> either. Versions are (variants of) lvm2-2.02.38, device-mapper-1.02.26,
> kernel-smp-2.6.20 (64 bit).

Hi,
no, there is no hard limit for snapshots.

But IIRC patches which solves some problems related with this (using private dm-io
mempool) were commited in 2.6.22 kernel.
But anyway, many snapshot from one origin causes serious performance degradation.
(Current snapsot implementation is not ideal in this situation - every write to origin
is replicated N-times to COW snapshot area - iow: every snapshot is independent and
need copy changed area separately.)

(But technically I tested >50 snapshot from one LV - should be limited only by available
memory in recent kernels).

Milan
--
mbroz@redhat.com

Re: [linux-lvm] 15 snapshot limit?

[Larry Dickson]

[-- Attachment #1: Type: text/plain, Size: 1999 bytes --]

You said:

> (But technically I tested >50 snapshot from one LV - should be limited
> only by available memory in recent kernels)

What is that limit? I suspect that is what I have run up against.

Larry

On 8/29/08, Milan Broz <mbroz@redhat.com> wrote:
>
> Larry Dickson wrote:
> > With a 768 GB logical volume in a 2235 GB volume group, and 4 GB of
> > memory, I try to set up lots of snapshots (of 45 or 50 GB each) and
> > can't get past 15. Sometimes I'll succeed in generating more than 15
> > COWs in /dev/mapper, but lvdisplay never shows more than 15 snapshots in
> > the "LV snapshot status source of" listing under the parent volume. I
> > searched and found 15 snapshots mentioned in a couple places like
> > wiki.samba.org/index.php/Shadow_Copies_with_Snapshots
> > <http://wiki.samba.org/index.php/Shadow_Copies_with_Snapshots> - but
> > nothing really authoritative.
> >
> > Is there really such a limit? Or is it a memory requirement - I seem to
> > remember so many MB memory per GB of storage, but can't find that
> > either. Versions are (variants of) lvm2-2.02.38, device-mapper-1.02.26,
> > kernel-smp-2.6.20 (64 bit).
>
> Hi,
> no, there is no hard limit for snapshots.
>
> But IIRC patches which solves some problems related with this (using
> private dm-io
> mempool) were commited in 2.6.22 kernel.
> But anyway, many snapshot from one origin causes serious performance
> degradation.
> (Current snapsot implementation is not ideal in this situation - every
> write to origin
> is replicated N-times to COW snapshot area - iow: every snapshot is
> independent and
> need copy changed area separately.)
>
> (But technically I tested >50 snapshot from one LV - should be limited only
> by available
> memory in recent kernels).
>
> Milan
> --
> mbroz@redhat.com
>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

[-- Attachment #2: Type: text/html, Size: 2825 bytes --]

Re: [linux-lvm] 15 snapshot limit?

[Larry Dickson]

[-- Attachment #1: Type: text/plain, Size: 2491 bytes --]

Sorry, I did not make that last question clear:

What (if any) is the formula by which I can predict the number of snapshots
that are possible, given a certain size of the parent volume and/or
snapshot and a certain amount of available memory? Milan implied there was
such a limit, and I seem to remember it, but I can't seem to find it spelled
out anywhere.

Larry

On 8/29/08, Larry Dickson <ldickson@cuttedge.com> wrote:

> You said:
>
> > (But technically I tested >50 snapshot from one LV - should be limited
> > only by available memory in recent kernels)
>
> What is that limit? I suspect that is what I have run up against.
>
> Larry
>
>  On 8/29/08, Milan Broz <mbroz@redhat.com> wrote:
>>
>> Larry Dickson wrote:
>> > With a 768 GB logical volume in a 2235 GB volume group, and 4 GB of
>> > memory, I try to set up lots of snapshots (of 45 or 50 GB each) and
>> > can't get past 15. Sometimes I'll succeed in generating more than 15
>> > COWs in /dev/mapper, but lvdisplay never shows more than 15 snapshots in
>> > the "LV snapshot status source of" listing under the parent volume. I
>> > searched and found 15 snapshots mentioned in a couple places like
>> > wiki.samba.org/index.php/Shadow_Copies_with_Snapshots
>> > <http://wiki.samba.org/index.php/Shadow_Copies_with_Snapshots> - but
>> > nothing really authoritative.
>> >
>> > Is there really such a limit? Or is it a memory requirement - I seem to
>> > remember so many MB memory per GB of storage, but can't find that
>> > either. Versions are (variants of) lvm2-2.02.38, device-mapper-1.02.26,
>> > kernel-smp-2.6.20 (64 bit).
>>
>> Hi,
>> no, there is no hard limit for snapshots.
>>
>> But IIRC patches which solves some problems related with this (using
>> private dm-io
>> mempool) were commited in 2.6.22 kernel.
>> But anyway, many snapshot from one origin causes serious performance
>> degradation.
>> (Current snapsot implementation is not ideal in this situation - every
>> write to origin
>> is replicated N-times to COW snapshot area - iow: every snapshot is
>> independent and
>> need copy changed area separately.)
>>
>> (But technically I tested >50 snapshot from one LV - should be limited
>> only by available
>> memory in recent kernels).
>>
>> Milan
>> --
>> mbroz@redhat.com
>>
>>
>> _______________________________________________
>> linux-lvm mailing list
>> linux-lvm@redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-lvm
>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>>
>
>

[-- Attachment #2: Type: text/html, Size: 4203 bytes --]

Re: [linux-lvm] 15 snapshot limit?

[Larry Dickson]

[-- Attachment #1: Type: text/plain, Size: 3771 bytes --]

More details: I found a /var/log/messages note that corresponded exactly to
the following bug:
==========================

http://www.linux-archive.org/debian-kernel/7067-bug-453254-linux-2-6-18-5-xen-686-unable-create-more-than-32-lvm-snapshots-total.html

11-28-2007, 01:34 AM
Peter de Zwart

Default Bug#453254: linux-2.6.18-5-xen-686: Unable to create more than 32
LVM snapshots in total
Package: linux-2.6.18-5-xen-686
Severity: normal


When attempting to create more than 32 LVM snapshots, the following error
message is sent from the Kernel:

device-mapper: table: 254:142: snapshot: Could not create kcopyd client
device-mapper: ioctl: error adding target to table

==========================
(End of reference; no follow-up found.) Except that in my case 15 is the max
snapshot count, not 32. Increasing memory from 4GB to 6GB does not help. I
notice mentions of this "Could not create kcopyd client" bug on the web all
the way back to 2004. Does anybody know any workaround? Our customer is not
happy about a 15-snapshot limit. Versions are (variants of) lvm2-2.02.38,
device-mapper-1.02.26, kernel-smp-2.6.20 (64 bit).

Larry
On 9/2/08, Larry Dickson <ldickson@cuttedge.com> wrote:
>
> Sorry, I did not make that last question clear:
>
> What (if any) is the formula by which I can predict the number of snapshots
> that are possible, given a certain size of the parent volume and/or
> snapshot and a certain amount of available memory? Milan implied there was
> such a limit, and I seem to remember it, but I can't seem to find it spelled
> out anywhere.
>
> Larry
>
> On 8/29/08, Larry Dickson <ldickson@cuttedge.com> wrote:
>
>> You said:
>>
>> > (But technically I tested >50 snapshot from one LV - should be limited
>> > only by available memory in recent kernels)
>>
>> What is that limit? I suspect that is what I have run up against.
>>
>> Larry
>>
>>  On 8/29/08, Milan Broz <mbroz@redhat.com> wrote:
>>>
>>> Larry Dickson wrote:
>>> > With a 768 GB logical volume in a 2235 GB volume group, and 4 GB of
>>> > memory, I try to set up lots of snapshots (of 45 or 50 GB each) and
>>> > can't get past 15. Sometimes I'll succeed in generating more than 15
>>> > COWs in /dev/mapper, but lvdisplay never shows more than 15 snapshots
>>> in
>>> > the "LV snapshot status source of" listing under the parent volume. I
>>> > searched and found 15 snapshots mentioned in a couple places like
>>> > wiki.samba.org/index.php/Shadow_Copies_with_Snapshots
>>> > <http://wiki.samba.org/index.php/Shadow_Copies_with_Snapshots> - but
>>> > nothing really authoritative.
>>> >
>>> > Is there really such a limit? Or is it a memory requirement - I seem to
>>> > remember so many MB memory per GB of storage, but can't find that
>>> > either. Versions are (variants of) lvm2-2.02.38, device-mapper-1.02.26,
>>> > kernel-smp-2.6.20 (64 bit).
>>>
>>> Hi,
>>> no, there is no hard limit for snapshots.
>>>
>>> But IIRC patches which solves some problems related with this (using
>>> private dm-io
>>> mempool) were commited in 2.6.22 kernel.
>>> But anyway, many snapshot from one origin causes serious performance
>>> degradation.
>>> (Current snapsot implementation is not ideal in this situation - every
>>> write to origin
>>> is replicated N-times to COW snapshot area - iow: every snapshot is
>>> independent and
>>> need copy changed area separately.)
>>>
>>> (But technically I tested >50 snapshot from one LV - should be limited
>>> only by available
>>> memory in recent kernels).
>>>
>>> Milan
>>> --
>>> mbroz@redhat.com
>>>
>>>
>>> _______________________________________________
>>> linux-lvm mailing list
>>> linux-lvm@redhat.com
>>> https://www.redhat.com/mailman/listinfo/linux-lvm
>>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>>>
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 6151 bytes --]

Re: [linux-lvm] 15 snapshot limit?

[Milan Broz]

Larry Dickson wrote:
> More details: I found a /var/log/messages note that corresponded exactly

Please try recent kernel (>2.6.22). Or patched distro kernel,
e.g. RHEL/CentOS has patched kernel already since 2007
see  https://bugzilla.redhat.com/show_bug.cgi?id=211516

Also recent upstream kernels have a little bit optimized way how to store
snapshot exceptions in memory, which should help here too.
See https://bugzilla.redhat.com/show_bug.cgi?id=421451

There is no magic formula how many snapshots is possible,
previously it was limited by one common mempool (see bug above),
and for some reason it fails in cca ~32 snapshot - but it is no longer true
(And increasing memory doesn't help with these old unpatched kernels.)

With 6GB of memory you should be able create tens of snapshots at least.
(But it will be just very slow - every snapshot is independent.)

Thanks,
Milan

Re: [linux-lvm] 15 snapshot limit?

[Ehud Karni]

On Tue, 2 Sep 2008 15:19:53 -0700, Larry Dickson wrote:
>
> in my case 15 is the max
> snapshot count, not 32.

I followed this thread from the beginning and I have a simple question.
What is your Maximum Logical Volume number ?
Check it with `vgdisplay' at the "MAX LV" line.

If this is the real problem, you can change it with `vgchange'.

Ehud.


--
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7966-667  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 GnuPG: 98EA398D <http://www.keyserver.net/>    Better Safe Than Sorry

Re: [linux-lvm] 15 snapshot limit?

[Larry Dickson]

[-- Attachment #1: Type: text/plain, Size: 1447 bytes --]

It's MAX LV 0 which I suppose means unlimited (or default?) - though I can't
seem to find it documented anywhere. I'll check out Milan's upgrades. A
"magic formula" sure would be nice, as an unpredictable limit on the number
of snapshots is pretty annoying for a customer product. I wonder if
something is following a "grab half of what's left" strategy; that would
explain a near-independence of memory. There ought to be a way to require
ALL resources to be statically assigned.

Larry

On 9/3/08, Ehud Karni <ehud@unix.mvs.co.il> wrote:
>
> On Tue, 2 Sep 2008 15:19:53 -0700, Larry Dickson wrote:
> >
> > in my case 15 is the max
> > snapshot count, not 32.
>
> I followed this thread from the beginning and I have a simple question.
> What is your Maximum Logical Volume number ?
> Check it with `vgdisplay' at the "MAX LV" line.
>
> If this is the real problem, you can change it with `vgchange'.
>
> Ehud.
>
>
> --
> Ehud Karni           Tel: +972-3-7966-561  /"\
> Mivtach - Simon      Fax: +972-3-7966-667  \ /  ASCII Ribbon Campaign
> Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
> http://www.mvs.co.il  FAX:  1-815-5509341  / \
> GnuPG: 98EA398D <http://www.keyserver.net/>    Better Safe Than Sorry
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

[-- Attachment #2: Type: text/html, Size: 2300 bytes --]