[linux-lvm] Very slow i/o after snapshotting

[linux-lvm] Very slow i/o after snapshotting

[Micky]

On a machine, where weekly snapshots are created for dumping raw data,
the overall disk i/o has drastically degraded to few megabytes per
second. It is quite random and so slow that commands freeze often.

I was using chunk size of 512K and same byte size with dd.

Is it quite common; I read that such freezes require a reboot. Is
there a way to avert?
What am I missing here?

Re: [linux-lvm] Very slow i/o after snapshotting

[Marian Csontos]

On 07/09/2013 03:37 AM, Micky wrote:
> On a machine, where weekly snapshots are created for dumping raw data,
> the overall disk i/o has drastically degraded to few megabytes per
> second. It is quite random and so slow that commands freeze often.
>
> I was using chunk size of 512K and same byte size with dd.
>
> Is it quite common; I read that such freezes require a reboot. Is
> there a way to avert?
> What am I missing here?

What do you mean by weekly snapshots?
If you mean old-style snapshots not thin-ones, many snapshots with 
single origin LV then how many snapshots are you keeping around?

If you have N snapshots up to (N+1)*chunksize is written for every write.

Also if this setup is used on HDDs the seektime may be relevant as well.

Old style snapshots were not meant for long term use and should be used 
while snapshot is copied over to permanent location.

You may be better off using thin-snapshots. Otherwise the solution is 
dumping old snapshots or moving over to tape, whatever, just do not keep 
more of them.

-- Marian

>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

Re: [linux-lvm] Very slow i/o after snapshotting

[Micky]

> What do you mean by weekly snapshots?
> If you mean old-style snapshots not thin-ones, many snapshots with single
> origin LV then how many snapshots are you keeping around?

I meant, running lvcreate ---snapshot, copying the data with dd or pv
and then removing the snapshot.

> If you have N snapshots up to (N+1)*chunksize is written for every write.

It is only one snapshot. A fresh one is created and after use, it's removed.

> Also if this setup is used on HDDs the seektime may be relevant as well.
> Old style snapshots were not meant for long term use and should be used
> while snapshot is copied over to permanent location.

Makes sense. But the snapshots exist no more than just few hours for every LV.

> You may be better off using thin-snapshots. Otherwise the solution is
> dumping old snapshots or moving over to tape, whatever, just do not keep
> more of them.

Sounds like not much of an option since I don't have thin pools.
Haven't had much success with them.

Re: [linux-lvm] Very slow i/o after snapshotting

[Zdenek Kabelac]

Dne 9.7.2013 03:37, Micky napsal(a):
> On a machine, where weekly snapshots are created for dumping raw data,
> the overall disk i/o has drastically degraded to few megabytes per
> second. It is quite random and so slow that commands freeze often.
>
> I was using chunk size of 512K and same byte size with dd.
>
> Is it quite common; I read that such freezes require a reboot. Is
> there a way to avert?
> What am I missing here?

Is this supposed to be a bug report ?

I need crystal ball first ;)

So far I've no idea about disks/tables in use, system/kernel in use, lvm 
version in use....

Zdenek

Re: [linux-lvm] Very slow i/o after snapshotting

[Micky]

> Is this supposed to be a bug report ?

Sorry for bashing the info. I was just answering the questions. LOL!

> So far I've no idea about disks/tables in use, system/kernel in use, lvm
> version in use....

RHEL 6.4 kernel-2.6.32-358.el6 (also tested with self compiled 3.8)
lvm2-2.02.98-9.el6.x86_64
Disk on external mega raid controller with ext4
Running Xen hypervisor but tested without it as well

Re: [linux-lvm] Very slow i/o after snapshotting

[Marian Csontos]

On 07/09/2013 01:51 PM, Micky wrote:
>> Is this supposed to be a bug report ?
>
> Sorry for bashing the info. I was just answering the questions. LOL!
>
>> So far I've no idea about disks/tables in use, system/kernel in use, lvm
>> version in use....
>
> RHEL 6.4 kernel-2.6.32-358.el6 (also tested with self compiled 3.8)
> lvm2-2.02.98-9.el6.x86_64
> Disk on external mega raid controller with ext4

How does it perform without snapshot? I have seen a sluggish RAID array 
when run out of battery.

Is there a reason to use that big chunks?
If the origin is seeing a lot of small dispersed writes using big chunk 
may multiply it manyfold (actually worst case 128-times: every 4k write 
generating 512k read from origin and then written to snapshot.)

-- Marian

> Running Xen hypervisor but tested without it as well
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

Re: [linux-lvm] Very slow i/o after snapshotting

[Micky]

Thanks for the quick response.

It is working fine without the LVM. It is just the snapshot which
makes the things slow. For instance, take a snapshot of an LV that's
running a DomU -- after that point on, the load inside that DomU will
start to increase until the snapshot is removed. Same is with KVM. The
only difference without hypervisor is, it is not terribly slow but
still lags, like, you get 15MB/s after snapshot and ~90-100MB/s
without snapshot being on the same volume!

Even SSH becomes slow, output freezes and you get sudden outbursts
after a few seconds.

As for the reason of copying big chunks out of LVs -- it is simple --
copy on write magic as short term image backup strategy! But I
realized that the magic with LVM comes with a price and that is I/O
latency ;)

Re: [linux-lvm] Very slow i/o after snapshotting

[Zdenek Kabelac]

Dne 9.7.2013 14:43, Micky napsal(a):
> Thanks for the quick response.
>
> It is working fine without the LVM. It is just the snapshot which
> makes the things slow. For instance, take a snapshot of an LV that's
> running a DomU -- after that point on, the load inside that DomU will
> start to increase until the snapshot is removed. Same is with KVM. The
> only difference without hypervisor is, it is not terribly slow but
> still lags, like, you get 15MB/s after snapshot and ~90-100MB/s
> without snapshot being on the same volume!
>
> Even SSH becomes slow, output freezes and you get sudden outbursts
> after a few seconds.
>
> As for the reason of copying big chunks out of LVs -- it is simple --
> copy on write magic as short term image backup strategy! But I
> realized that the magic with LVM comes with a price and that is I/O
> latency ;)
>

Do you write to the snapshot ?

It's known FACT that performance of old snapshot is very far from being ideal 
- it's very simply implementation - for a having consistent system to make a 
backup of the volume - so for backup it doesn't really matter how slow is that 
(it just needs to remain usable)

You have just very simple list of blocks stored in COW devices together with 
list of exceptions.  And modified blocks are simple copied first (so there is 
no reference counting or anything resembling that...)

Using 512kB chunks is actually the worst idea for old snaps (well in fact any 
snaps) especially if they are coded in that way with exception store.

I'd suggest to go with much smaller chunks - i.e. 4-8-16KB - since if you 
update a single 512 sector  -  512KB of data has to be copied!!! so really bad 
idea, unless you overwrite large continuous portion of a device.

And yes - if you have rotational hdd - you need to expect horrible seek times 
as well when reading/writing from snapshot target....

And yes - there are some horrible Segate hdd drives (as I've seen just 
yesterday) were 2 disk reading programs at the same time may degrade 100MB/s 
-> 4MB/s (and there is no  dm involved)

Zdenek

Re: [linux-lvm] Very slow i/o after snapshotting

[Micky]

> Do you write to the snapshot ?

Not so often but there is like 1-5% usage allocation.

> It's known FACT that performance of old snapshot is very far from being
> ideal - it's very simply implementation - for a having consistent system to
> make a backup of the volume - so for backup it doesn't really matter how
> slow is that (it just needs to remain usable)

True. But in case of domains running on a hypervisor, the purpose of doing
a live backup slingshots and dies! I know it's not LVM's fault but
sluggishness is!

> I'd suggest to go with much smaller chunks - i.e. 4-8-16KB - since if you
> update a single 512 sector  -  512KB of data has to be copied!!! so really
> bad idea, unless you overwrite large continuous portion of a device.

I just tried that and got 2-3% improvement.
Here are the gritty details, if someone's interested.
  --- Logical volume ---
  LV Write Access        read/write
  LV snapshot status     active destination for lvma
  LV Status              available
  # open                 1
  LV Size                200.10 GiB
  Current LE             51226
  COW-table size         100.00 GiB
  COW-table LE           25600
  Allocated to snapshot  0.14%
  Snapshot chunk size    4.00 KiB
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     16384
  Block device           253:26

> And yes - if you have rotational hdd - you need to expect horrible seek
> times as well when reading/writing from snapshot target....

Yes, they do. But I reproduced this one with multiple machines (and kernels)!

> And yes - there are some horrible Segate hdd drives (as I've seen just
> yesterday) were 2 disk reading programs at the same time may degrade 100MB/s
> -> 4MB/s (and there is no  dm involved)

Haha, no doubt. Seagates' are the worst ones. IMHO, Hitachi's drives
run cooler and
that's what Nagios tells me!

Re: [linux-lvm] Very slow i/o after snapshotting

[Zdenek Kabelac]

Dne 9.7.2013 16:04, Micky napsal(a):
>> Do you write to the snapshot ?
>
> Not so often but there is like 1-5% usage allocation.
>
>> It's known FACT that performance of old snapshot is very far from being
>> ideal - it's very simply implementation - for a having consistent system to
>> make a backup of the volume - so for backup it doesn't really matter how
>> slow is that (it just needs to remain usable)
>
> True. But in case of domains running on a hypervisor, the purpose of doing
> a live backup slingshots and dies! I know it's not LVM's fault but
> sluggishness is!

Well here we are at lvm list - thus discussing lvm metadata and command line 
issues -  do you see slow command line execution ?

I think you are concerned about the perfomance of dm device - which
is a level below lvm  (kernel level)

Do not take is as some excuse - just we should use correct terms.


>
>> I'd suggest to go with much smaller chunks - i.e. 4-8-16KB - since if you
>> update a single 512 sector  -  512KB of data has to be copied!!! so really
>> bad idea, unless you overwrite large continuous portion of a device.
>
> I just tried that and got 2-3% improvement.
> Here are the gritty details, if someone's interested.
>    --- Logical volume ---
>    LV Write Access        read/write
>    LV snapshot status     active destination for lvma
>    LV Status              available
>    # open                 1
>    LV Size                200.10 GiB
>    Current LE             51226
>    COW-table size         100.00 GiB

Well here is the catch I guess.

While the snapshot might be reasonable enough with sizes like 10GiB,
it's getting much much worse when it scales up.

If you intent to use  100GiB snapshot - please consider thin volumes here.
Use upstream git and report bugs if something doesn't work.
There is not going to be a fix for  old-snaps - the on-disk format it quite 
unscalable.  Thin is the real fix for your problems here.
Also note - you will get horrible start-up times for snapshot of this size...


>> And yes - if you have rotational hdd - you need to expect horrible seek
>> times as well when reading/writing from snapshot target....
>
> Yes, they do. But I reproduced this one with multiple machines (and kernels)!

Once again - there is no hope  old-snaps could become magically faster unless
completely rewritten - and that what's thin provisioning is basically about ;)
We've tried to make everything much faster and smarter.
So do not ask for fixing old snapshots - they are simply unfixable for large
COW sizes - it's been designed for something very different then you try to 
use it for...

>
>> And yes - there are some horrible Segate hdd drives (as I've seen just
>> yesterday) were 2 disk reading programs at the same time may degrade 100MB/s
>> -> 4MB/s (and there is no  dm involved)
>
> Haha, no doubt. Seagates' are the worst ones. IMHO, Hitachi's drives
> run cooler and
> that's what Nagios tells me!

Just simple check is how fast parallel 'dd' you get from  /dev/sda partition - 
if you get approximately halve the speed  of single 'dd' - then you have good 
enough drive (Hitachi is usually pretty good).

Zdenek

Re: [linux-lvm] Very slow i/o after snapshotting

[Micky]

Ahh. I get it. Sorry for using the aging old snap mechanism. Seems no
more luck with it now! I'll have to test the Thin in such an
environment to have my say. But not gonna try it anytime soon. The
power pill I am being referred to has sadly no recovery options ;)
Thanks for the suggestions though!

On Tue, Jul 9, 2013 at 7:18 PM, Zdenek Kabelac <zkabelac@redhat.com> wrote:
> Dne 9.7.2013 16:04, Micky napsal(a):
>
>>> Do you write to the snapshot ?
>>
>>
>> Not so often but there is like 1-5% usage allocation.
>>
>>> It's known FACT that performance of old snapshot is very far from being
>>> ideal - it's very simply implementation - for a having consistent system
>>> to
>>> make a backup of the volume - so for backup it doesn't really matter how
>>> slow is that (it just needs to remain usable)
>>
>>
>> True. But in case of domains running on a hypervisor, the purpose of doing
>> a live backup slingshots and dies! I know it's not LVM's fault but
>> sluggishness is!
>
>
> Well here we are at lvm list - thus discussing lvm metadata and command line
> issues -  do you see slow command line execution ?
>
> I think you are concerned about the perfomance of dm device - which
> is a level below lvm  (kernel level)
>
> Do not take is as some excuse - just we should use correct terms.
>
>
>
>>
>>> I'd suggest to go with much smaller chunks - i.e. 4-8-16KB - since if you
>>> update a single 512 sector  -  512KB of data has to be copied!!! so
>>> really
>>> bad idea, unless you overwrite large continuous portion of a device.
>>
>>
>> I just tried that and got 2-3% improvement.
>> Here are the gritty details, if someone's interested.
>>    --- Logical volume ---
>>    LV Write Access        read/write
>>    LV snapshot status     active destination for lvma
>>    LV Status              available
>>    # open                 1
>>    LV Size                200.10 GiB
>>    Current LE             51226
>>    COW-table size         100.00 GiB
>
>
> Well here is the catch I guess.
>
> While the snapshot might be reasonable enough with sizes like 10GiB,
> it's getting much much worse when it scales up.
>
> If you intent to use  100GiB snapshot - please consider thin volumes here.
> Use upstream git and report bugs if something doesn't work.
> There is not going to be a fix for  old-snaps - the on-disk format it quite
> unscalable.  Thin is the real fix for your problems here.
> Also note - you will get horrible start-up times for snapshot of this
> size...
>
>
>
>>> And yes - if you have rotational hdd - you need to expect horrible seek
>>> times as well when reading/writing from snapshot target....
>>
>>
>> Yes, they do. But I reproduced this one with multiple machines (and
>> kernels)!
>
>
> Once again - there is no hope  old-snaps could become magically faster
> unless
> completely rewritten - and that what's thin provisioning is basically about
> ;)
> We've tried to make everything much faster and smarter.
> So do not ask for fixing old snapshots - they are simply unfixable for large
> COW sizes - it's been designed for something very different then you try to
> use it for...
>
>
>>
>>> And yes - there are some horrible Segate hdd drives (as I've seen just
>>> yesterday) were 2 disk reading programs at the same time may degrade
>>> 100MB/s
>>> -> 4MB/s (and there is no  dm involved)
>>
>>
>> Haha, no doubt. Seagates' are the worst ones. IMHO, Hitachi's drives
>> run cooler and
>> that's what Nagios tells me!
>
>
> Just simple check is how fast parallel 'dd' you get from  /dev/sda partition
> - if you get approximately halve the speed  of single 'dd' - then you have
> good enough drive (Hitachi is usually pretty good).
>
> Zdenek
>

Re: [linux-lvm] Very slow i/o after snapshotting

[Zdenek Kabelac]

Dne 9.7.2013 16:57, Micky napsal(a):
> Ahh. I get it. Sorry for using the aging old snap mechanism. Seems no
> more luck with it now! I'll have to test the Thin in such an
> environment to have my say. But not gonna try it anytime soon. The
> power pill I am being referred to has sadly no recovery options ;)

Well it's getting better every day (literally :)) (using git repo)
The upstream is heavily moving towards some better automated support
for recovery.

You could download tools which are being extensively tested for recovery,
lvm will create pool metadata spare LVs  for automated recovery

So you have some choices:
- very fast snaps - and harder recovery in case of hw fault

- very slow snaps - but you could count on good old 'proven' technology.
   (though you still could have fun with i.e. invalidated snaps)

- write your own dm target to cover your needs.

- using btrfs?

Zdenek




>
> On Tue, Jul 9, 2013 at 7:18 PM, Zdenek Kabelac <zkabelac@redhat.com> wrote:
>> Dne 9.7.2013 16:04, Micky napsal(a):
>>
>>>> Do you write to the snapshot ?
>>>
>>>
>>> Not so often but there is like 1-5% usage allocation.
>>>
>>>> It's known FACT that performance of old snapshot is very far from being
>>>> ideal - it's very simply implementation - for a having consistent system
>>>> to
>>>> make a backup of the volume - so for backup it doesn't really matter how
>>>> slow is that (it just needs to remain usable)
>>>
>>>
>>> True. But in case of domains running on a hypervisor, the purpose of doing
>>> a live backup slingshots and dies! I know it's not LVM's fault but
>>> sluggishness is!
>>
>>
>> Well here we are at lvm list - thus discussing lvm metadata and command line
>> issues -  do you see slow command line execution ?
>>
>> I think you are concerned about the perfomance of dm device - which
>> is a level below lvm  (kernel level)
>>
>> Do not take is as some excuse - just we should use correct terms.
>>
>>
>>
>>>
>>>> I'd suggest to go with much smaller chunks - i.e. 4-8-16KB - since if you
>>>> update a single 512 sector  -  512KB of data has to be copied!!! so
>>>> really
>>>> bad idea, unless you overwrite large continuous portion of a device.
>>>
>>>
>>> I just tried that and got 2-3% improvement.
>>> Here are the gritty details, if someone's interested.
>>>     --- Logical volume ---
>>>     LV Write Access        read/write
>>>     LV snapshot status     active destination for lvma
>>>     LV Status              available
>>>     # open                 1
>>>     LV Size                200.10 GiB
>>>     Current LE             51226
>>>     COW-table size         100.00 GiB
>>
>>
>> Well here is the catch I guess.
>>
>> While the snapshot might be reasonable enough with sizes like 10GiB,
>> it's getting much much worse when it scales up.
>>
>> If you intent to use  100GiB snapshot - please consider thin volumes here.
>> Use upstream git and report bugs if something doesn't work.
>> There is not going to be a fix for  old-snaps - the on-disk format it quite
>> unscalable.  Thin is the real fix for your problems here.
>> Also note - you will get horrible start-up times for snapshot of this
>> size...
>>
>>
>>
>>>> And yes - if you have rotational hdd - you need to expect horrible seek
>>>> times as well when reading/writing from snapshot target....
>>>
>>>
>>> Yes, they do. But I reproduced this one with multiple machines (and
>>> kernels)!
>>
>>
>> Once again - there is no hope  old-snaps could become magically faster
>> unless
>> completely rewritten - and that what's thin provisioning is basically about
>> ;)
>> We've tried to make everything much faster and smarter.
>> So do not ask for fixing old snapshots - they are simply unfixable for large
>> COW sizes - it's been designed for something very different then you try to
>> use it for...
>>
>>
>>>
>>>> And yes - there are some horrible Segate hdd drives (as I've seen just
>>>> yesterday) were 2 disk reading programs at the same time may degrade
>>>> 100MB/s
>>>> -> 4MB/s (and there is no  dm involved)
>>>
>>>
>>> Haha, no doubt. Seagates' are the worst ones. IMHO, Hitachi's drives
>>> run cooler and
>>> that's what Nagios tells me!
>>
>>
>> Just simple check is how fast parallel 'dd' you get from  /dev/sda partition
>> - if you get approximately halve the speed  of single 'dd' - then you have
>> good enough drive (Hitachi is usually pretty good).
>>
>> Zdenek
>>

Re: [linux-lvm] Very slow i/o after snapshotting

[Marian Csontos]

On 07/09/2013 04:57 PM, Micky wrote:
> Ahh. I get it. Sorry for using the aging old snap mechanism. Seems no
> more luck with it now! I'll have to test the Thin in such an
> environment to have my say. But not gonna try it anytime soon. The
> power pill I am being referred to has sadly no recovery options ;)
> Thanks for the suggestions though!

You could still try to allocate the snapshot on another HDD if possible 
to see if HDD seektime may be the issue. Still the performance should 
not be degraded that drastically.

What does the `lsblk -t` say? Could be an alignment issue.

What's `free` saying about the free memory and cache? (dmeventd on 6.4 
is trying to lock a large chunk of address space in RAM (~100M)

-- Marian

>
> On Tue, Jul 9, 2013 at 7:18 PM, Zdenek Kabelac <zkabelac@redhat.com> wrote:
>> Dne 9.7.2013 16:04, Micky napsal(a):
>>
>>>> Do you write to the snapshot ?
>>>
>>>
>>> Not so often but there is like 1-5% usage allocation.
>>>
>>>> It's known FACT that performance of old snapshot is very far from being
>>>> ideal - it's very simply implementation - for a having consistent system
>>>> to
>>>> make a backup of the volume - so for backup it doesn't really matter how
>>>> slow is that (it just needs to remain usable)
>>>
>>>
>>> True. But in case of domains running on a hypervisor, the purpose of doing
>>> a live backup slingshots and dies! I know it's not LVM's fault but
>>> sluggishness is!
>>
>>
>> Well here we are at lvm list - thus discussing lvm metadata and command line
>> issues -  do you see slow command line execution ?
>>
>> I think you are concerned about the perfomance of dm device - which
>> is a level below lvm  (kernel level)
>>
>> Do not take is as some excuse - just we should use correct terms.
>>
>>
>>
>>>
>>>> I'd suggest to go with much smaller chunks - i.e. 4-8-16KB - since if you
>>>> update a single 512 sector  -  512KB of data has to be copied!!! so
>>>> really
>>>> bad idea, unless you overwrite large continuous portion of a device.
>>>
>>>
>>> I just tried that and got 2-3% improvement.
>>> Here are the gritty details, if someone's interested.
>>>     --- Logical volume ---
>>>     LV Write Access        read/write
>>>     LV snapshot status     active destination for lvma
>>>     LV Status              available
>>>     # open                 1
>>>     LV Size                200.10 GiB
>>>     Current LE             51226
>>>     COW-table size         100.00 GiB
>>
>>
>> Well here is the catch I guess.
>>
>> While the snapshot might be reasonable enough with sizes like 10GiB,
>> it's getting much much worse when it scales up.
>>
>> If you intent to use  100GiB snapshot - please consider thin volumes here.
>> Use upstream git and report bugs if something doesn't work.
>> There is not going to be a fix for  old-snaps - the on-disk format it quite
>> unscalable.  Thin is the real fix for your problems here.
>> Also note - you will get horrible start-up times for snapshot of this
>> size...
>>
>>
>>
>>>> And yes - if you have rotational hdd - you need to expect horrible seek
>>>> times as well when reading/writing from snapshot target....
>>>
>>>
>>> Yes, they do. But I reproduced this one with multiple machines (and
>>> kernels)!
>>
>>
>> Once again - there is no hope  old-snaps could become magically faster
>> unless
>> completely rewritten - and that what's thin provisioning is basically about
>> ;)
>> We've tried to make everything much faster and smarter.
>> So do not ask for fixing old snapshots - they are simply unfixable for large
>> COW sizes - it's been designed for something very different then you try to
>> use it for...
>>
>>
>>>
>>>> And yes - there are some horrible Segate hdd drives (as I've seen just
>>>> yesterday) were 2 disk reading programs at the same time may degrade
>>>> 100MB/s
>>>> -> 4MB/s (and there is no  dm involved)
>>>
>>>
>>> Haha, no doubt. Seagates' are the worst ones. IMHO, Hitachi's drives
>>> run cooler and
>>> that's what Nagios tells me!
>>
>>
>> Just simple check is how fast parallel 'dd' you get from  /dev/sda partition
>> - if you get approximately halve the speed  of single 'dd' - then you have
>> good enough drive (Hitachi is usually pretty good).
>>
>> Zdenek
>>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

Re: [linux-lvm] Very slow i/o after snapshotting

[Micky]

> Well it's getting better every day (literally :)) (using git repo)
> The upstream is heavily moving towards some better automated support
> for recovery.
>
> You could download tools which are being extensively tested for recovery,
> lvm will create pool metadata spare LVs  for automated recovery
>
> So you have some choices:
> - very fast snaps - and harder recovery in case of hw fault
>
> - very slow snaps - but you could count on good old 'proven' technology.
>   (though you still could have fun with i.e. invalidated snaps)
>
> - write your own dm target to cover your needs.
>
> - using btrfs?
>


Quite fascinating. Definitely a new dimension and project seems
heading in right direction, there's no doubt. I already have had fun
(recently autoextend died on me after few extensions).

It might change the topic but I am just gonna put it out. THE Future
looks promising but that's what they said about KVM too (wink wink)!

Re: [linux-lvm] Very slow i/o after snapshotting

[Micky]

> You could still try to allocate the snapshot on another HDD if possible to
> see if HDD seektime may be the issue. Still the performance should not be
> degraded that drastically.

How does that work?
lvcreate -n name -L size VG pv ?


> What does the `lsblk -t` say? Could be an alignment issue.

0 through 31

> What's `free` saying about the free memory and cache? (dmeventd on 6.4 is
> trying to lock a large chunk of address space in RAM (~100M)

Cached mem looks good.
Dmeventd. Right. It is. Isn't it spawed everytime an LV is created?
root      6813  0.0  1.4 197056 11044 ?        S<s  May26   2:44 /sbin/dmeventd

Re: [linux-lvm] Very slow i/o after snapshotting

[Micky]

I meant alignment for all dm entries 0 through 31 is zero!


>> What does the `lsblk -t` say? Could be an alignment issue.
>
> 0 through 31
>
>> What's `free` saying about the free memory and cache? (dmeventd on 6.4 is
>> trying to lock a large chunk of address space in RAM (~100M)
>
> Cached mem looks good.
> Dmeventd. Right. It is. Isn't it spawed everytime an LV is created?
> root      6813  0.0  1.4 197056 11044 ?        S<s  May26   2:44 /sbin/dmeventd

Re: [linux-lvm] Very slow i/o after snapshotting

[matthew patton]

I don't know the internals of a VMware snapshot (aka journal disk) but what's the impact to writing to an LVM snapshot if the original is never written to again? Think of the original LV as a "golden master" and I create 10 snapshots of it and each DomU is presented with a snapshot device as it's /dev/hda. Does the indirection even work?

I would have thought that the metadata in the snapshot device would have a bitmap (corresponding to specified "page" size) wherein the bitmap completely covered the original device and whenever a write got sent to the base block device (and the original copied to the snap) the flag would get updated such that reads to the snapshot device computes the "page" address of the desired block, checks the flag and if set, reads from the master device, otherwise find it in the snapshot (b-tree search? or hash+linked-list?)

Re: [linux-lvm] Very slow i/o after snapshotting

[Zdenek Kabelac]

Dne 9.7.2013 19:59, matthew patton napsal(a):
> I don't know the internals of a VMware snapshot (aka journal disk) but what's the impact to writing to an LVM snapshot if the original is never written to again? Think of the original LV as a "golden master" and I create 10 snapshots of it and each DomU is presented with a snapshot device as it's /dev/hda. Does the indirection even work?
>
> I would have thought that the metadata in the snapshot device would have a bitmap (corresponding to specified "page" size) wherein the bitmap completely covered the original device and whenever a write got sent to the base block device (and the original copied to the snap) the flag would get updated such that reads to the snapshot device computes the "page" address of the desired block, checks the flag and if set, reads from the master device, otherwise find it in the snapshot (b-tree search? or hash+linked-list?)
>



As said above:

'Smart and cool' code is thin provisioning
'Old and very simple' is old snapshot code.

And you may try to guess which one of those two is  matching and even 
significantly surpassing your ideas ;)

Zdenel

Re: [linux-lvm] Very slow i/o after snapshotting

[Zdenek Kabelac]

Dne 9.7.2013 17:39, Micky napsal(a):
> I meant alignment for all dm entries 0 through 31 is zero!
>
>
>>> What does the `lsblk -t` say? Could be an alignment issue.
>>
>> 0 through 31
>>
>>> What's `free` saying about the free memory and cache? (dmeventd on 6.4 is
>>> trying to lock a large chunk of address space in RAM (~100M)
>>
>> Cached mem looks good.
>> Dmeventd. Right. It is. Isn't it spawed everytime an LV is created?
>> root      6813  0.0  1.4 197056 11044 ?        S<s  May26   2:44 /sbin/dmeventd
>

There is only one dmeventd running - and lvm is spawning it only when it's not
available - and in fact spawning is not the right term if you use systemd 
enabled system (like Fedora)

Also so far you still have not show actually any 'real' numbers even when you 
run plain good old  'dd' command.

So what is the performance of 'dd'  reading 10G >/dev/null
or raw device,  dm origin, dm snapshot  (with  iflag=direct)

What is performance of write ?

What is the performance when 2 of them are running in parallel.

Also it's probably more easier to resolve this through #irc.

Zdenek

Re: [linux-lvm] Very slow i/o after snapshotting

[Micky]

Read/write speed is just fine (~100-200MB/s) as I described in my
first few emails without LVM snapshots.


On Tue, Jul 9, 2013 at 11:47 PM, Zdenek Kabelac <zkabelac@redhat.com> wrote:
> Dne 9.7.2013 17:39, Micky napsal(a):
>
>> I meant alignment for all dm entries 0 through 31 is zero!
>>
>>
>>>> What does the `lsblk -t` say? Could be an alignment issue.
>>>
>>>
>>> 0 through 31
>>>
>>>> What's `free` saying about the free memory and cache? (dmeventd on 6.4
>>>> is
>>>> trying to lock a large chunk of address space in RAM (~100M)
>>>
>>>
>>> Cached mem looks good.
>>> Dmeventd. Right. It is. Isn't it spawed everytime an LV is created?
>>> root      6813  0.0  1.4 197056 11044 ?        S<s  May26   2:44
>>> /sbin/dmeventd
>>
>>
>
> There is only one dmeventd running - and lvm is spawning it only when it's
> not
> available - and in fact spawning is not the right term if you use systemd
> enabled system (like Fedora)
>
> Also so far you still have not show actually any 'real' numbers even when
> you run plain good old  'dd' command.
>
> So what is the performance of 'dd'  reading 10G >/dev/null
> or raw device,  dm origin, dm snapshot  (with  iflag=direct)
>
> What is performance of write ?
>
> What is the performance when 2 of them are running in parallel.
>
> Also it's probably more easier to resolve this through #irc.
>
> Zdenek
>
>

Re: [linux-lvm] Very slow i/o after snapshotting

[Micky]

Fellows,
While we were getting out of options, I have finally managed to fix
the read/write speed from LVM snapshots. Turns out, it was default
scheduler CFQ's fault or not-fault.
Switching to NOOP gave nearly a 40% performance boost (getting 30MB/s
write speed instead of 10-15MB/s when dumping or copying raw data from
snapshot).

And if I use Deadline scheduler, it quite randomly jumps to 50MB/s and
load goes down and stays constant (3.0 while dd'ing a fresh snapshot)!

Considering the normal disk speeds of 100MB/s of these standard
off-the-shelf HDDs, what I am getting now out of LVM snapshots is not
bad at all.

Leaving the details here so it can help others who may coming looking
for answers!

On Wed, Jul 10, 2013 at 4:15 AM, Micky <mickylmartin@gmail.com> wrote:
> Read/write speed is just fine (~100-200MB/s) as I described in my
> first few emails without LVM snapshots.
>
>
> On Tue, Jul 9, 2013 at 11:47 PM, Zdenek Kabelac <zkabelac@redhat.com> wrote:
>> Dne 9.7.2013 17:39, Micky napsal(a):
>>
>>> I meant alignment for all dm entries 0 through 31 is zero!
>>>
>>>
>>>>> What does the `lsblk -t` say? Could be an alignment issue.
>>>>
>>>>
>>>> 0 through 31
>>>>
>>>>> What's `free` saying about the free memory and cache? (dmeventd on 6.4
>>>>> is
>>>>> trying to lock a large chunk of address space in RAM (~100M)
>>>>
>>>>
>>>> Cached mem looks good.
>>>> Dmeventd. Right. It is. Isn't it spawed everytime an LV is created?
>>>> root      6813  0.0  1.4 197056 11044 ?        S<s  May26   2:44
>>>> /sbin/dmeventd
>>>
>>>
>>
>> There is only one dmeventd running - and lvm is spawning it only when it's
>> not
>> available - and in fact spawning is not the right term if you use systemd
>> enabled system (like Fedora)
>>
>> Also so far you still have not show actually any 'real' numbers even when
>> you run plain good old  'dd' command.
>>
>> So what is the performance of 'dd'  reading 10G >/dev/null
>> or raw device,  dm origin, dm snapshot  (with  iflag=direct)
>>
>> What is performance of write ?
>>
>> What is the performance when 2 of them are running in parallel.
>>
>> Also it's probably more easier to resolve this through #irc.
>>
>> Zdenek
>>
>>