[linux-lvm] write performance with active snapshot

[linux-lvm] write performance with active snapshot

[Peter Daum]

Hi,

for an application I am just working on it looks like lvm snapshots would
be just what I need as far as functionality is concerned. Unfortunately,
I am experiencing such a massive degradation in performance, that the
result is almost useless.

I'm working on a fairly fast machine (Quadcore, 8GB RAM) with a big
hardware RAID array and lvm2 (Debian Lenny; Linux 2.6.26-1-amd64;
LVM version:2.02.39 (2008-06-27)
Library version: 1.02.27 (2008-06-25)
Driver version:  4.13.0)

Sequentially writing to a file (ext3) on a logical volume, I get a
sustained performance of ~ 250 MB/sec. When I create a snapshot
volume, the write throughput drops to 7-8 MB/secs (on the original
volume; writing to the snapshot I see a significant degradation,
but not nearly, as bad; read performance is o.k.).Is this "normal"
or is there anything I can do to about it?

I looked in this list and searched the WWW but couldn't find any
concrete information on the performance impact of snapshots
(except http://www.nikhef.nl/~dennisvd/lvmcrap.html).
It seems like write performance should probably be less then 1/3
of the original throughput, because every write to the source
volume causes 3 I/O operations plus some overhead for meta data.
More difficult to estimate would be the time lost by additional
head movements. Still, a throughput degradation by a factor of 30
seems pretty extreme.

Any ideas?

Regards,
                          Peter Daum

Re: [linux-lvm] write performance with active snapshot

[Larry Dickson]

[-- Attachment #1: Type: text/plain, Size: 2073 bytes --]

My guess is that you are getting the typical seek overhead. Have you tried
making a volume group out of two separate RAID arrays (driving different
spindles), and using lvdisplay --maps to make sure the parent volume is on
one array, the snapshot(s) on the other?

Larry Dickson
Cutting Edge Networked Storage

On 11/9/08, Peter Daum <gator_ml@yahoo.de> wrote:
>
> Hi,
>
> for an application I am just working on it looks like lvm snapshots would
> be just what I need as far as functionality is concerned. Unfortunately,
> I am experiencing such a massive degradation in performance, that the
> result is almost useless.
>
> I'm working on a fairly fast machine (Quadcore, 8GB RAM) with a big
> hardware RAID array and lvm2 (Debian Lenny; Linux 2.6.26-1-amd64;
> LVM version:2.02.39 (2008-06-27)
> Library version: 1.02.27 (2008-06-25)
> Driver version:  4.13.0)
>
> Sequentially writing to a file (ext3) on a logical volume, I get a
> sustained performance of ~ 250 MB/sec. When I create a snapshot
> volume, the write throughput drops to 7-8 MB/secs (on the original
> volume; writing to the snapshot I see a significant degradation,
> but not nearly, as bad; read performance is o.k.).Is this "normal"
> or is there anything I can do to about it?
>
> I looked in this list and searched the WWW but couldn't find any
> concrete information on the performance impact of snapshots
> (except http://www.nikhef.nl/~dennisvd/lvmcrap.html).
> It seems like write performance should probably be less then 1/3
> of the original throughput, because every write to the source
> volume causes 3 I/O operations plus some overhead for meta data.
> More difficult to estimate would be the time lost by additional
> head movements. Still, a throughput degradation by a factor of 30
> seems pretty extreme.
>
> Any ideas?
>
> Regards,
>                         Peter Daum
>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

[-- Attachment #2: Type: text/html, Size: 3017 bytes --]

Re: [linux-lvm] write performance with active snapshot

[Stuart D. Gathman]

On Mon, 10 Nov 2008, Larry Dickson wrote:

> My guess is that you are getting the typical seek overhead. Have you tried
> making a volume group out of two separate RAID arrays (driving different
> spindles), and using lvdisplay --maps to make sure the parent volume is on
> one array, the snapshot(s) on the other?

At the expense of memory, LVM could copy larger pieces to snapshot on
write.  This would improve sequential write performance with snapshots
on the same spindle.  Is there a tuning knob for that?  Does read_ahead
affect COW?

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

[linux-lvm] Re: write performance with active snapshot

[Peter Daum]

Hi,

Larry Dickson wrote:
> My guess is that you are getting the typical seek overhead. Have you 
> tried making a volume group out of two separate RAID arrays (driving 
> different spindles), and using lvdisplay --maps to make sure the parent 
> volume is on one array, the snapshot(s) on the other?

That was my suspicion, too (although I could not imagine such an extreme
impact). Just for testing I added a single disk to the same volume group
and put the snapshot onto that disk - amazingly it made hardly any
difference (Actually, I'm almost glad about that, because the combination
of a 12-disk-array with a single disk would be under almost all other
aspects foolish).

One thing that does improve the performance a little (actually by 100%,
which in this case meens still pretty lousy 16 MB/sec) is to increase
the chunk size to 512kb. (I don't know yet, how this might affect
performance when dealing with many small files) ...

Regards,
                     Peter


> On 11/9/08, *Peter Daum* <gator_ml@yahoo.de <mailto:gator_ml@yahoo.de>> 
> wrote:
> 
>     Hi,
> 
>     for an application I am just working on it looks like lvm snapshots
>     would
>     be just what I need as far as functionality is concerned. Unfortunately,
>     I am experiencing such a massive degradation in performance, that the
>     result is almost useless.
> 
>     I'm working on a fairly fast machine (Quadcore, 8GB RAM) with a big
>     hardware RAID array and lvm2 (Debian Lenny; Linux 2.6.26-1-amd64;
>     LVM version:2.02.39 (2008-06-27)
>     Library version: 1.02.27 (2008-06-25)
>     Driver version:  4.13.0)
> 
>     Sequentially writing to a file (ext3) on a logical volume, I get a
>     sustained performance of ~ 250 MB/sec. When I create a snapshot
>     volume, the write throughput drops to 7-8 MB/secs (on the original
>     volume; writing to the snapshot I see a significant degradation,
>     but not nearly, as bad; read performance is o.k.).Is this "normal"
>     or is there anything I can do to about it?
> 
>     I looked in this list and searched the WWW but couldn't find any
>     concrete information on the performance impact of snapshots
>     (except http://www.nikhef.nl/~dennisvd/lvmcrap.html).
>     It seems like write performance should probably be less then 1/3
>     of the original throughput, because every write to the source
>     volume causes 3 I/O operations plus some overhead for meta data.
>     More difficult to estimate would be the time lost by additional
>     head movements. Still, a throughput degradation by a factor of 30
>     seems pretty extreme.
> 
>     Any ideas?
> 
>     Regards,
>                             Peter Daum

Re: [linux-lvm] Re: write performance with active snapshot

[thomas62186218]

Hi Peter,

I am following your thread on this topic...have any solutions emerged? 
I as well have seen miserably performance when snapshots are active.

Thank you in advance for your feedback.

Regards,
Thomas


-----Original Message-----
From: Peter Daum <gator_ml@yahoo.de>
To: linux-lvm@redhat.com
Sent: Tue, 11 Nov 2008 8:04 am
Subject: [linux-lvm] Re: write performance with active snapshot









Hi, 
 

Larry Dickson wrote: 

> My guess is that you are getting the typical seek overhead. Have you
> tried making a volume group out of two separate RAID arrays (driving
> different spindles), and using lvdisplay --maps to make sure the 
parent
> volume is on one array, the snapshot(s) on the other? 
 

That was my suspicion, too (although I could not imagine such an 
extreme 

impact). Just for testing I added a single disk to the same volume 
group 

and put the snapshot onto that disk - amazingly it made hardly any 

difference (Actually, I'm almost glad about that, because the 
combination 

of a 12-disk-array with a single disk would be under almost all other 

aspects foolish). 
 

One thing that does improve the performance a little (actually by 100%, 

which in this case meens still pretty lousy 16 MB/sec) is to increase 

the chunk size to 512kb. (I don't know yet, how this might
 affect 

performance when dealing with many small files) ... 
 

Regards, 

                    Peter 
 


> On 11/9/08, *Peter Daum* <gator_ml@yahoo.de 
<mailto:gator_ml@yahoo.de>>
> wrote: 

>
>     Hi, 

>
>     for an application I am just working on it looks like lvm 
snapshots 

>     would 

>     be just what I need as far as functionality is concerned. 
Unfortunately, 

>     I am experiencing such a massive degradation in performance, that 
the 

>     result is almost useless. 

>
>     I'm working on a fairly fast machine (Quadcore, 8GB RAM) with a 
big 

>     hardware RAID array and lvm2 (Debian Lenny; Linux 2.6.26-1-amd64; 

>     LVM version:2.02.39 (2008-06-27) 

>     Library version: 1.02.27 (2008-06-25) 

>     Driver version:  4.13.0) 

>
>     Sequentially writing to a file (ext3) on a logical volume, I get 
a 

>     sustained performance of ~ 250 MB/sec. When I create a snapshot 

>     volume, the write throughput drops to 7-8 MB/secs (on the 
original 

>     volume; writing to the snapshot I see a significant degradation, 

>     but not nearly, as bad; read performance is o.k.).Is this 
"normal" 

>     or is there a
nything I can do to about it? 

>
>     I looked in this list and searched the WWW but couldn't find any 

>     concrete information on the performance impact of snapshots 

>     (except http://www.nikhef.nl/~dennisvd/lvmcrap.html). 

>     It seems like write performance should probably be less then 1/3 

>     of the original throughput, because every write to the source 

>     volume causes 3 I/O operations plus some overhead for meta data. 

>     More difficult to estimate would be the time lost by additional 

>     head movements. Still, a throughput degradation by a factor of 30 

>     seems pretty extreme. 

>
>     Any ideas? 

>
>     Regards, 

>                             Peter Daum 
 

_______________________________________________ 

linux-lvm mailing list 

linux-lvm@redhat.com 

https://www.redhat.com/mailman/listinfo/linux-lvm 

read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ 





Hi

[linux-lvm] Re: write performance with active snapshot

[Peter Daum]

Hi Thomas,

thomas62186218@aol.com wrote:
> I am following your thread on this topic...have any solutions emerged? I 
> as well have seen miserably performance when snapshots are active.

I am sorry, at least I still don't know any solution
(except avoiding snapshots wherever performance matters )-:

Regards,
                   Peter

> -----Original Message-----
> From: Peter Daum <gator_ml@yahoo.de>
> To: linux-lvm@redhat.com
> Sent: Tue, 11 Nov 2008 8:04 am
> Subject: [linux-lvm] Re: write performance with active snapshot
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Hi, 
>  
> 
> Larry Dickson wrote: 
> 
>> My guess is that you are getting the typical seek overhead. Have you
>> tried making a volume group out of two separate RAID arrays (driving
>> different spindles), and using lvdisplay --maps to make sure the 
> parent
>> volume is on one array, the snapshot(s) on the other? 
>  
> 
> That was my suspicion, too (although I could not imagine such an extreme 
> 
> impact). Just for testing I added a single disk to the same volume group 
> 
> and put the snapshot onto that disk - amazingly it made hardly any 
> 
> difference (Actually, I'm almost glad about that, because the combination 
> 
> of a 12-disk-array with a single disk would be under almost all other 
> 
> aspects foolish). 
>  
> 
> One thing that does improve the performance a little (actually by 100%, 
> 
> which in this case meens still pretty lousy 16 MB/sec) is to increase 
> 
> the chunk size to 512kb. (I don't know yet, how this might
> affect 
> 
> performance when dealing with many small files) ... 
>  
> 
> Regards, 
> 
>                     Peter 
>  
> 
> 
>> On 11/9/08, *Peter Daum* <gator_ml@yahoo.de 
> <mailto:gator_ml@yahoo.de>>
>> wrote: 
> 
>>
>>     Hi, 
> 
>>
>>     for an application I am just working on it looks like lvm 
> snapshots 
> 
>>     would 
> 
>>     be just what I need as far as functionality is concerned. 
> Unfortunately, 
> 
>>     I am experiencing such a massive degradation in performance, that 
> the 
> 
>>     result is almost useless. 
> 
>>
>>     I'm working on a fairly fast machine (Quadcore, 8GB RAM) with a 
> big 
> 
>>     hardware RAID array and lvm2 (Debian Lenny; Linux 2.6.26-1-amd64; 
> 
>>     LVM version:2.02.39 (2008-06-27) 
> 
>>     Library version: 1.02.27 (2008-06-25) 
> 
>>     Driver version:  4.13.0) 
> 
>>
>>     Sequentially writing to a file (ext3) on a logical volume, I get 
> a 
> 
>>     sustained performance of ~ 250 MB/sec. When I create a snapshot 
> 
>>     volume, the write throughput drops to 7-8 MB/secs (on the 
> original 
> 
>>     volume; writing to the snapshot I see a significant degradation, 
> 
>>     but not nearly, as bad; read performance is o.k.).Is this 
> "normal" 
> 
>>     or is there a
> nything I can do to about it? 
> 
>>
>>     I looked in this list and searched the WWW but couldn't find any 
> 
>>     concrete information on the performance impact of snapshots 
> 
>>     (except http://www.nikhef.nl/~dennisvd/lvmcrap.html). 
> 
>>     It seems like write performance should probably be less then 1/3 
> 
>>     of the original throughput, because every write to the source 
> 
>>     volume causes 3 I/O operations plus some overhead for meta data. 
> 
>>     More difficult to estimate would be the time lost by additional 
> 
>>     head movements. Still, a throughput degradation by a factor of 30 
> 
>>     seems pretty extreme. 
> 
>>
>>     Any ideas? 
> 
>>
>>     Regards, 
> 
>>                             Peter Daum 
>  
> 
> _______________________________________________ 
> 
> linux-lvm mailing list 
> 
> linux-lvm@redhat.com 
> 
> https://www.redhat.com/mailman/listinfo/linux-lvm 
> 
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ 
> 
> 
> 
> 
> 
> Hi
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>