[linux-lvm] Any way to speed up activation of volumes with snapshots?

[linux-lvm] Any way to speed up activation of volumes with snapshots?

[Chris Friesen]

Hi,

I'm running a 3.10 kernel with LVM 2.02.95.

I'm running into a problem where activating snapshots can take quite a long 
time, roughly one minute per 25GB of delta between the snapshot and the origin 
volume.  (See below for my test procedure.)

I realize that my kernel/LVM aren't exactly bleeding edge, and I wondering 
whether more recent versions have done anything to speed up the activation 
process (like maybe making it more lazy-loaded rather than reading in a bunch of 
data up-front).

If anyone is aware of such improvements, I'd appreciate it if you could point me 
at the appropriate changes.


For those that are interested, the test that I did was as follows:

lvcreate -L100G  -n test1 vg
lvcreate -L 100G -s -n test1snapshot /dev/vg/test1
dd if=/dev/zero of=/dev/cinder-volumes/test1snapshot bs=1M count=95000
lvchange -an  /dev/cinder-volumes/test1snapshot
time lvchange -ay  /dev/cinder-volumes/test1snapshot

The final "lvchange" command took about 4m22s.  This works out to 362MB/sec, 
which is almost exactly the speed I got doing a "dd" from the volume to /dev/null.

Thanks,
Chris

Re: [linux-lvm] Any way to speed up activation of volumes with snapshots?

[Chris Friesen]

My apologies, I got the version wrong.  The tests below were actually from the 
3.4 kernel and LVM 2.2.95.   I'm doing tests on the newer versions now.

Chris


On 09/14/2015 12:05 PM, Chris Friesen wrote:
> Hi,
>
> I'm running a 3.10 kernel with LVM 2.02.95.
>
> I'm running into a problem where activating snapshots can take quite a long
> time, roughly one minute per 25GB of delta between the snapshot and the origin
> volume.  (See below for my test procedure.)
>
> I realize that my kernel/LVM aren't exactly bleeding edge, and I wondering
> whether more recent versions have done anything to speed up the activation
> process (like maybe making it more lazy-loaded rather than reading in a bunch of
> data up-front).
>
> If anyone is aware of such improvements, I'd appreciate it if you could point me
> at the appropriate changes.
>
>
> For those that are interested, the test that I did was as follows:
>
> lvcreate -L100G  -n test1 vg
> lvcreate -L 100G -s -n test1snapshot /dev/vg/test1
> dd if=/dev/zero of=/dev/cinder-volumes/test1snapshot bs=1M count=95000
> lvchange -an  /dev/cinder-volumes/test1snapshot
> time lvchange -ay  /dev/cinder-volumes/test1snapshot
>
> The final "lvchange" command took about 4m22s.  This works out to 362MB/sec,
> which is almost exactly the speed I got doing a "dd" from the volume to /dev/null.
>
> Thanks,
> Chris
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Any way to speed up activation of volumes with snapshots?

[Zdenek Kabelac]

Dne 14.9.2015 v 20:05 Chris Friesen napsal(a):
> Hi,
>
> I'm running a 3.10 kernel with LVM 2.02.95.
>
> I'm running into a problem where activating snapshots can take quite a long
> time, roughly one minute per 25GB of delta between the snapshot and the origin
> volume.  (See below for my test procedure.)
>
> I realize that my kernel/LVM aren't exactly bleeding edge, and I wondering
> whether more recent versions have done anything to speed up the activation
> process (like maybe making it more lazy-loaded rather than reading in a bunch
> of data up-front).
>
> If anyone is aware of such improvements, I'd appreciate it if you could point
> me at the appropriate changes.

Hi

There is NO way to accelerate your existing setup, other then placing delta on 
SSD - the format of snapshot was meant to be used 'temporarily' - i.e. until 
you take backup of LV - but not for long-term multi-gigabyte case.

This is major mis-use of this 'snapshot' feature - which repeats over and over...

If you want to use long-living snapshots - you need to use thin-provisioning,
.
> The final "lvchange" command took about 4m22s.  This works out to 362MB/sec,

I've seen users waiting even 1/2 hour - so you are still 'lucky' user ;)



Zdenek

Re: [linux-lvm] Any way to speed up activation of volumes with snapshots?

[Chris Friesen]

On 09/14/2015 12:47 PM, Zdenek Kabelac wrote:
> Dne 14.9.2015 v 20:05 Chris Friesen napsal(a):
>> Hi,
>>
>> I'm running a 3.10 kernel with LVM 2.02.95.
>>
>> I'm running into a problem where activating snapshots can take quite a long
>> time, roughly one minute per 25GB of delta between the snapshot and the origin
>> volume.  (See below for my test procedure.)
>>
>> I realize that my kernel/LVM aren't exactly bleeding edge, and I wondering
>> whether more recent versions have done anything to speed up the activation
>> process (like maybe making it more lazy-loaded rather than reading in a bunch
>> of data up-front).
>>
>> If anyone is aware of such improvements, I'd appreciate it if you could point
>> me at the appropriate changes.
>
> Hi
>
> There is NO way to accelerate your existing setup, other then placing delta on
> SSD - the format of snapshot was meant to be used 'temporarily' - i.e. until you
> take backup of LV - but not for long-term multi-gigabyte case.
>
> This is major mis-use of this 'snapshot' feature - which repeats over and over...
>
> If you want to use long-living snapshots - you need to use thin-provisioning,

Thanks for the quick response.

Presumably we would still have multi-GB of delta between the original volume and 
the snapshot, so what would make the activation of a thin-provisioned snapshot 
faster?  Is the metadata stored differently?

Chris

Re: [linux-lvm] Any way to speed up activation of volumes with snapshots?

[Zdenek Kabelac]

Dne 14.9.2015 v 21:16 Chris Friesen napsal(a):
> On 09/14/2015 12:47 PM, Zdenek Kabelac wrote:
>> Dne 14.9.2015 v 20:05 Chris Friesen napsal(a):
>>> Hi,
>>>
>>> I'm running a 3.10 kernel with LVM 2.02.95.
>>>
>>> I'm running into a problem where activating snapshots can take quite a long
>>> time, roughly one minute per 25GB of delta between the snapshot and the origin
>>> volume.  (See below for my test procedure.)
>>>
>>> I realize that my kernel/LVM aren't exactly bleeding edge, and I wondering
>>> whether more recent versions have done anything to speed up the activation
>>> process (like maybe making it more lazy-loaded rather than reading in a bunch
>>> of data up-front).
>>>
>>> If anyone is aware of such improvements, I'd appreciate it if you could point
>>> me at the appropriate changes.
>>
>> Hi
>>
>> There is NO way to accelerate your existing setup, other then placing delta on
>> SSD - the format of snapshot was meant to be used 'temporarily' - i.e. until
>> you
>> take backup of LV - but not for long-term multi-gigabyte case.
>>
>> This is major mis-use of this 'snapshot' feature - which repeats over and
>> over...
>>
>> If you want to use long-living snapshots - you need to use thin-provisioning,
>
> Thanks for the quick response.
>
> Presumably we would still have multi-GB of delta between the original volume
> and the snapshot, so what would make the activation of a thin-provisioned
> snapshot faster?  Is the metadata stored differently?

Yes - metadata are on separate device(LV) and are using b-trees.
So now you could i.e. chain snap-of-snap-of-snap...
Also snaps do not need to be active with origin and many other goodies.

Drawbacks - all data have to be in same pool and share same space,
so you can't run 'just' 'snapshot' out-of-space - whole pool is out-of-space.
Also min chunk size is 64K (instead of 4K for old-snap)

Anyway - try and play and see what performs better and suit your needs.


Zdenek

Re: [linux-lvm] Any way to speed up activation of volumes with snapshots?

[Zdenek Kabelac]

Dne 14.9.2015 v 21:16 Chris Friesen napsal(a):
> On 09/14/2015 12:47 PM, Zdenek Kabelac wrote:
>> Dne 14.9.2015 v 20:05 Chris Friesen napsal(a):
>>> Hi,
>>>
>>> I'm running a 3.10 kernel with LVM 2.02.95.
>>>
>>> I'm running into a problem where activating snapshots can take quite a long
>>> time, roughly one minute per 25GB of delta between the snapshot and the origin
>>> volume.  (See below for my test procedure.)
>>>
>>> I realize that my kernel/LVM aren't exactly bleeding edge, and I wondering
>>> whether more recent versions have done anything to speed up the activation
>>> process (like maybe making it more lazy-loaded rather than reading in a bunch
>>> of data up-front).
>>>
>>> If anyone is aware of such improvements, I'd appreciate it if you could point
>>> me at the appropriate changes.
>>
>> Hi
>>
>> There is NO way to accelerate your existing setup, other then placing delta on
>> SSD - the format of snapshot was meant to be used 'temporarily' - i.e. until
>> you
>> take backup of LV - but not for long-term multi-gigabyte case.
>>
>> This is major mis-use of this 'snapshot' feature - which repeats over and
>> over...
>>
>> If you want to use long-living snapshots - you need to use thin-provisioning,
>
> Thanks for the quick response.
>
> Presumably we would still have multi-GB of delta between the original volume
> and the snapshot, so what would make the activation of a thin-provisioned
> snapshot faster?  Is the metadata stored differently?

If you don't mind dropping existing snapshot and creating new 'old-snapshot' - 
using larger block size (i.e.  32K  (or even more) instead of 4K) may improve 
speed of disk reading noticeable (at the price of copying larger chunks when 
modifying disk blocks - depends on use-case)....

And if you don't mind replacing your ancient kernel and trying some newer one 
- there have been some 'read' order improvement for this old-snapshot target 
made as well - so you may at least try if there will be any noticeable change
(actually would be interesting just to get report how much faster it boots 
when you only exchange kernel for a boot - you don't have to use it - just 
measure boot speed)

Zdenek

Re: [linux-lvm] Any way to speed up activation of volumes with snapshots?

[Chris Friesen]

On 09/15/2015 02:41 AM, Zdenek Kabelac wrote:

> If you don't mind dropping existing snapshot and creating new 'old-snapshot' -
> using larger block size (i.e.  32K  (or even more) instead of 4K) may improve
> speed of disk reading noticeable (at the price of copying larger chunks when
> modifying disk blocks - depends on use-case)....

Okay, good info to have.

> And if you don't mind replacing your ancient kernel and trying some newer one -
> there have been some 'read' order improvement for this old-snapshot target made
> as well - so you may at least try if there will be any noticeable change
> (actually would be interesting just to get report how much faster it boots when
> you only exchange kernel for a boot - you don't have to use it - just measure
> boot speed)

This is essentially a custom distro, so changing just the kernel is a bit 
complicated.  However, I have a newer version that I want to test with which has 
a newer kernel but the same version of LVM, so that should give some good data.

Chris

Re: [linux-lvm] Any way to speed up activation of volumes with snapshots?

[Chris Friesen]

On 09/14/2015 12:05 PM, Chris Friesen wrote:
> Hi,
>
> I'm running a 3.10 kernel with LVM 2.02.95.
>
> I'm running into a problem where activating snapshots can take quite a long
> time, roughly one minute per 25GB of delta between the snapshot and the origin
> volume.  (See below for my test procedure.)

As a side question, currently we have a script which runs "vgchange -ay <vg>", 
which will activate the VG.

This script appears to activate the LVs within the VG serially rather than in 
parallel.  Does it do any processing to determine in which order to activate the 
LVs?

The reason I ask is that we have an overall timeout on the activate phase, after 
which we'll put any un-activated LVs into an error state.  It would be a shame 
to try to activate a snapshotted LV first, and then timeout without activating 
any of the non-snapshotted LVs.

Chris

Re: [linux-lvm] Any way to speed up activation of volumes with snapshots?

[Zdenek Kabelac]

Dne 16.9.2015 v 01:03 Chris Friesen napsal(a):
> On 09/14/2015 12:05 PM, Chris Friesen wrote:
>> Hi,
>>
>> I'm running a 3.10 kernel with LVM 2.02.95.
>>
>> I'm running into a problem where activating snapshots can take quite a long
>> time, roughly one minute per 25GB of delta between the snapshot and the origin
>> volume.  (See below for my test procedure.)
>
> As a side question, currently we have a script which runs "vgchange -ay <vg>",
> which will activate the VG.
>
> This script appears to activate the LVs within the VG serially rather than in
> parallel.  Does it do any processing to determine in which order to activate
> the LVs?
>
> The reason I ask is that we have an overall timeout on the activate phase,
> after which we'll put any un-activated LVs into an error state.  It would be a
> shame to try to activate a snapshotted LV first, and then timeout without
> activating any of the non-snapshotted LVs.
>


Hi

Sorry, but there is no 'parallel' activation from lvm2 command - as lvm2 is 
not multithreadded app  (and is quite far away from that).

What you can do with recent versions of lvm2 -  you could 'mark' (with 
lvchange --setskipactivation) some LVs to be skipped from normal activation.
Then you could activate such skipped LVs with  'lvchange -ay -K'.

As said lvm2 has not been designed for this case.
(btw during whole activation - your access to VG is actually limited only for 
'read-only')

Zdenek

Re: [linux-lvm] Any way to speed up activation of volumes with snapshots?

[Chris Friesen]

On 09/16/2015 02:27 AM, Zdenek Kabelac wrote:
> Dne 16.9.2015 v 01:03 Chris Friesen napsal(a):

>> As a side question, currently we have a script which runs "vgchange -ay <vg>",
>> which will activate the VG.
>>
>> This script appears to activate the LVs within the VG serially rather than in
>> parallel.  Does it do any processing to determine in which order to activate
>> the LVs?

> What you can do with recent versions of lvm2 -  you could 'mark' (with lvchange
> --setskipactivation) some LVs to be skipped from normal activation.
> Then you could activate such skipped LVs with  'lvchange -ay -K'.
>
> As said lvm2 has not been designed for this case.
> (btw during whole activation - your access to VG is actually limited only for
> 'read-only')

Right...we're planning on testing out thin provisioning but I have to figure out 
a band-aid solution for products already in use.

Just to clarify...what does your "access to VG is actually limited only for 
'read-only'" mean exactly?

Does it mean:

1) I only have read-only access to the LVs within the VG until all LVs in the VG 
are activated?

or

2) I only have readonly access to the VG (so I can't add/remove/modify LVs) but 
I have read/write access to the LVs within the VG.


Thanks,
Chris

Re: [linux-lvm] Any way to speed up activation of volumes with snapshots?

[Chris Friesen]

On 09/16/2015 02:27 AM, Zdenek Kabelac wrote:

> Sorry, but there is no 'parallel' activation from lvm2 command - as lvm2 is not
> multithreadded app  (and is quite far away from that).


Just had a weird thought.  What would happen if I ran the "vgchange -ay" 
command, and then a few seconds later ran a bunch of "lvchange -ay" commands in 
parallel?

Would that maybe let me activate non-snapshotted volumes even if the snapshotted 
volumes took a long time?

Chris

Re: [linux-lvm] Any way to speed up activation of volumes with snapshots?

[Zdenek Kabelac]

Dne 16.9.2015 v 21:19 Chris Friesen napsal(a):
> On 09/16/2015 02:27 AM, Zdenek Kabelac wrote:
>
>> Sorry, but there is no 'parallel' activation from lvm2 command - as lvm2 is not
>> multithreadded app  (and is quite far away from that).
>
>
> Just had a weird thought.  What would happen if I ran the "vgchange -ay"
> command, and then a few seconds later ran a bunch of "lvchange -ay" commands
> in parallel?

Since all of them are 'activation' requests - you just end with success
and everything should be activated.

Much more interesting case is - if you run in parallel activation
and deactivation :) - then the result depends on last executed command.

> Would that maybe let me activate non-snapshotted volumes even if the
> snapshotted volumes took a long time?

Yes - you could activate things in parallel this way - since every command has 
separate udev cookie - so it waits on its set of LVs.

Zdenek

Re: [linux-lvm] Any way to speed up activation of volumes with snapshots?

[Zdenek Kabelac]

Dne 16.9.2015 v 17:30 Chris Friesen napsal(a):
> On 09/16/2015 02:27 AM, Zdenek Kabelac wrote:
>> Dne 16.9.2015 v 01:03 Chris Friesen napsal(a):
>
>>> As a side question, currently we have a script which runs "vgchange -ay <vg>",
>>> which will activate the VG.
>>>
>>> This script appears to activate the LVs within the VG serially rather than in
>>> parallel.  Does it do any processing to determine in which order to activate
>>> the LVs?
>
>> What you can do with recent versions of lvm2 -  you could 'mark' (with lvchange
>> --setskipactivation) some LVs to be skipped from normal activation.
>> Then you could activate such skipped LVs with  'lvchange -ay -K'.
>>
>> As said lvm2 has not been designed for this case.
>> (btw during whole activation - your access to VG is actually limited only for
>> 'read-only')
>
> Right...we're planning on testing out thin provisioning but I have to figure
> out a band-aid solution for products already in use.
>
> Just to clarify...what does your "access to VG is actually limited only for
> 'read-only'" mean exactly?
>
> Does it mean:
>
> 1) I only have read-only access to the LVs within the VG until all LVs in the
> VG are activated?
>
> or
>
> 2) I only have readonly access to the VG (so I can't add/remove/modify LVs)
> but I have read/write access to the LVs within the VG.

2. is correct - you will not be able to modify VG content (as read-lock is 
held for VG - and you need write-lock for updates)

Zdenek