[linux-lvm] very slow fw performance when snapshots active

[linux-lvm] very slow fw performance when snapshots active

[Falko Zurell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi List,

I have an actual CentOS Linux with LVM on hardware RAID1+0.

fs layout looks like this:

[root@C20-DB1-00 ~]# lvs
  LV     VG   Attr   LSize   Origin Snap%  Move Log Copy%
  LVOpt  VG0  -wi-ao 300.00G
  LVRoot VG0  -wi-ao  43.94G
  LVSwap VG0  -wi-ao  23.44G

Then I create a snapshot vor VG0/LVOpt with 40 GB size.

Then I edit a 2 GB text file on /opt (XFS filesystem) with vi (just
hitting "enter" and save this minor change). This will take minutes to
finish the vi during saving.
I see the kcopyd in the process list which fills up the snapshot.

When the snapshot is off then saving the same file with the same minor
change will take only seconds.

I understand that snapshots are copy-on-write. But I can't explain why
this takes minutes for a 2 GB file which has changed only in a few bytes.

Shouldn't only the changed blocks be copied to the snapshot space? Even
if only whole PEs are copied then it shouldn't take only seconds to copy
to the snapshot.

- From the actual point of view activating snapshots on this database
machine absolutly kills performance.

Did I something wrong on that?
- --

Falko Zurell
Skype: zero_data
/i-d media AG
Ohlauer Stra�e 43
D-10999 Berlin

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHFxvWsdVNK8p2VTARArgkAJ9KK9hpQBH6gxhzMQiYdZMH87Q0GwCgwOSy
w9tcXzNveVWCnTfB4EO5bKU=
=u+DI
-----END PGP SIGNATURE-----

Re: [linux-lvm] very slow fw performance when snapshots active

[Gabriel Barazer]

Hi,

On 10/18/2007 10:39:50 AM +0200, Falko Zurell <falko.zurell@idmedia.com> 
wrote:
> 
> Then I edit a 2 GB text file on /opt (XFS filesystem) with vi (just
> hitting "enter" and save this minor change). This will take minutes to
> finish the vi during saving.

This problem is not lvm-related: when opening a file with vim, it reads 
the entire file into memory (even if the file is 2GB), marks the changes 
into a .swp file. When saving, vim rewrites the entire file in place, 
effectively changing 2GB of copy on write blocks. This happens every 
time, with or without snapshot, the latter being worse because of the 
cow overhead.

for reading-only very large files, "less" or "more" read files on-demand 
which is a far more effecient way.

for writing, I don't know any method allowing to safely write a small 
change into a large file, except using awk or perl parsing (which reads 
the entire streamed file anyway)

Gabriel

AW: Re: [linux-lvm] very slow fw performance when snapshots active

[Zurell, Falko]

[-- Attachment #1: Type: text/plain, Size: 2330 bytes --]

Ok, i understand the point with the swap file of vi. But why is this with snapshots activated that much slower than without? 
Why can the file be saved in seconds without cow instead of about 2 minutes with cow. Needing twice the time for saving would be reasonable.

Editing with vi was just a performance test ;-)

Thanks and kind regards


Falko Zurell

Head of Application Management
falko.zurell@idmedia.com
Mobil +49 (0)160 - 3 62 52 77

/i-d media AG
Ohlauer Straße 43
D-10999 Berlin

Tel +49 (0)30 - 2 59 47 - 357
Fax +49 (0)30 - 2 59 47 - 471
www.idmedia.com

_______________________________________________

Board: Regine Haschka-Helmer (CEO), Franz Klose (CFO)
Chairman of the Supervisory Board: Christian A. Hufnagl
Headquarters: Berlin, Place of Jurisdiction: Berlin HRB 76342
UST-Id. DE812698340
_______________________________________________


----- Originalnachricht -----
Von: linux-lvm-bounces@redhat.com <linux-lvm-bounces@redhat.com>
An: LVM general discussion and development <linux-lvm@redhat.com>
Gesendet: Thu Oct 18 15:52:36 2007
Betreff: Re: [linux-lvm] very slow fw performance when snapshots active

Hi,

On 10/18/2007 10:39:50 AM +0200, Falko Zurell <falko.zurell@idmedia.com> 
wrote:
> 
> Then I edit a 2 GB text file on /opt (XFS filesystem) with vi (just
> hitting "enter" and save this minor change). This will take minutes to
> finish the vi during saving.

This problem is not lvm-related: when opening a file with vim, it reads 
the entire file into memory (even if the file is 2GB), marks the changes 
into a .swp file. When saving, vim rewrites the entire file in place, 
effectively changing 2GB of copy on write blocks. This happens every 
time, with or without snapshot, the latter being worse because of the 
cow overhead.

for reading-only very large files, "less" or "more" read files on-demand 
which is a far more effecient way.

for writing, I don't know any method allowing to safely write a small 
change into a large file, except using awk or perl parsing (which reads 
the entire streamed file anyway)

Gabriel

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[-- Attachment #2: Type: text/html, Size: 3162 bytes --]

Re: AW: Re: [linux-lvm] very slow fw performance when snapshots active

[Gabriel Barazer]

On 10/18/2007 4:01:40 PM +0200, "Zurell, Falko" 
<Falko.Zurell@idmedia.com> wrote:
> Ok, i understand the point with the swap file of vi. But why is this 
> with snapshots activated that much slower than without?
> Why can the file be saved in seconds without cow instead of about 2 
> minutes with cow. Needing twice the time for saving would be reasonable.

I think this is because the re-writing operation on a standard block 
device is done using a comparison between the data to write, and the 
data already written (which is nearly all the time read from the OS file 
cache in your test case, instead of the block device itself, because the 
file written was entirely read just before) whereas the LVM snapshot 
target redirect any write to the COW space without this comparison step. 
Maybe there is a possible optimization of the snapshot dm target which 
could compare block before deciding to write them on the COW space. 
Obvisouly, such optimization process add a slight overhead an can 
possibly have a negative impact on the performance.

Gabriel

Re: [linux-lvm] very slow fw performance when snapshots active

[malahal]

Reducing chunksize may help a bit but you may end up having
performance problems when you read snapshot data later.

Thanks, Malahal.

Gabriel Barazer [gabriel@oxeva.fr] wrote:
> On 10/18/2007 4:01:40 PM +0200, "Zurell, Falko" 
> <Falko.Zurell@idmedia.com> wrote:
> >Ok, i understand the point with the swap file of vi. But why is this 
> >with snapshots activated that much slower than without?
> >Why can the file be saved in seconds without cow instead of about 2 
> >minutes with cow. Needing twice the time for saving would be reasonable.
> 
> I think this is because the re-writing operation on a standard block 
> device is done using a comparison between the data to write, and the 
> data already written (which is nearly all the time read from the OS file 
> cache in your test case, instead of the block device itself, because the 
> file written was entirely read just before) whereas the LVM snapshot 
> target redirect any write to the COW space without this comparison step. 
> Maybe there is a possible optimization of the snapshot dm target which 
> could compare block before deciding to write them on the COW space. 
> Obvisouly, such optimization process add a slight overhead an can 
> possibly have a negative impact on the performance.
> 
> Gabriel
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/