[linux-lvm] syncing with snapshots takes forever?

[linux-lvm] syncing with snapshots takes forever?

[Sean Oh]

I am having difficult with LVM snapshots and syncing.
I have tested LVM 1.0.6 and also LVM2. Both generate the same problem.

My testing environment is as follows:

VG vg01 Size 100GB
LV lv01 Size 50GB
LV lv01_snapshot1 size 10GB
LV lv01_snapshot2 size 10GB
LV lv01_snapshot3 size 10GB

as you can see above, there 4 LVs, one LV and 3 snapshots for the LV
and also the system has SCSI raid controller(Mylex) and 1024GB of RAM, 
though these does not seem to matter, since the problem happens to other 
systems, also.
The FS does not matter here, since Reiserfs and XFS both produce the same 
syncing problem.

For LVM 1.0.6, I have applied VFS lock patch and lvm-1.0.6 patch to the 
kernel 2.4.20. ---> same problem. Kernel 2.4.20 without any patches gives 
the same problem,too.

For LVM2 ---> I removed all the LVs and VG that made with LVM1 and remake 
them with LVM2. No diifference at all. BTW, LVM2 genereates a stragne error 
message to console 'invalidate: busy buffer'

After copying a file with 1GB size to lv01 and typing 'sync' takes more 30 
minitus or sometime forever. During these time period, access to lv01 is 
neary blocked, producing very slow performance(0 ~ 400B/s). If I remove the 
snapshots, sync returns normally. So it is the issue with LVM making the 
changes in LV to 3 snapshots while locking the LV.

Is there any way to make the snapshots syncing faster? Is that what LVM2 
promise? Is there any alternative to LVM for snapshoting?

Any help and comments are apperciated.


_________________________________________________________________
MSN 8 with e-mail virus protection service: 2 months FREE* 
http://join.msn.com/?page=features/virus

Re: [linux-lvm] syncing with snapshots takes forever?

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 613 bytes --]

On Fri, 2003-01-24 22:07:31 +0900, Sean Oh <benew666@hotmail.com>
wrote in message <F59HdlQgdYXSVvR0Wss000091d0@hotmail.com>:

> and also the system has SCSI raid controller(Mylex) and 1024GB of RAM, 
                                                          ^^^^^^

Wow... I'd really like to get in touch with that box:-)

SCNR, JBG

-- 
   Jan-Benedict Glaw       jbglaw@lug-owl.de    . +49-172-7608481
   "Eine Freie Meinung in  einem Freien Kopf    | Gegen Zensur
    fuer einen Freien Staat voll Freier Bürger" | im Internet!
   Shell Script APT-Proxy: http://lug-owl.de/~jbglaw/software/ap2/

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [linux-lvm] syncing with snapshots takes forever?

[Sean Oh]

Sorry about the typo. The size of RAM was 1GB

----- Original Message ----- 
From: "Jan-Benedict Glaw" <jbglaw@lug-owl.de>
To: <linux-lvm@sistina.com>
Sent: Friday, January 24, 2003 10:32 PM
Subject: Re: [linux-lvm] syncing with snapshots takes forever?

Re: [linux-lvm] syncing with snapshots takes forever?

[Andreas Dilger]

On Jan 24, 2003  22:07 +0900, Sean Oh wrote:
> I am having difficult with LVM snapshots and syncing.
> I have tested LVM 1.0.6 and also LVM2. Both generate the same problem.
> 
> My testing environment is as follows:
> 
> VG vg01 Size 100GB
> LV lv01 Size 50GB
> LV lv01_snapshot1 size 10GB
> LV lv01_snapshot2 size 10GB
> LV lv01_snapshot3 size 10GB
> 
> as you can see above, there 4 LVs, one LV and 3 snapshots for the LV
> and also the system has SCSI raid controller(Mylex) and 1024GB of RAM, 
> though these does not seem to matter, since the problem happens to other 
> systems, also.
> The FS does not matter here, since Reiserfs and XFS both produce the same 
> syncing problem.
> 
> After copying a file with 1GB size to lv01 and typing 'sync' takes more 30 
> minitus or sometime forever. During these time period, access to lv01 is 
> neary blocked, producing very slow performance(0 ~ 400B/s). If I remove the 
> snapshots, sync returns normally. So it is the issue with LVM making the 
> changes in LV to 3 snapshots while locking the LV.

The problem is that when you type "sync" you are actually writing (most of)
your file to disk 4 times - once for the initial LV (which was mostly still
in RAM I'm guessing), and once for each snapshot.  Since you now have 4x
the write load and it is not streaming, but random I/O it causes your disks
to thrash badly.  Add to that the fact that you are writing to a RAID which
needs to seek _every_ disk for each write, and you are in really bad shape.

I'm guessing that if you put each snapshot on its own physical disk (maybe
only keeping the original LV on the RAID device) you will see a lot of
improvement.  Also, it would be interesting to know what the real performance
of even the single 1GB write is (i.e. write + sync) because I'm guessing
most of the data is still in memory.  At, say, 50MB/s to disk you are still
talking 3 or 4 minutes to write out 1GB, so even an ideal LV snapshot case
would take 9-15 minutes to write out 4GB.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: [linux-lvm] syncing with snapshots takes forever?

[Joe Thornber]

On Fri, Jan 24, 2003 at 11:15:15AM -0700, Andreas Dilger wrote:
> The problem is that when you type "sync" you are actually writing (most of)
> your file to disk 4 times - once for the initial LV (which was mostly still
> in RAM I'm guessing), and once for each snapshot.  Since you now have 4x
> the write load and it is not streaming, but random I/O it causes your disks
> to thrash badly.  Add to that the fact that you are writing to a RAID which
> needs to seek _every_ disk for each write, and you are in really bad shape.

30 minutes still seems _far_ too long for LVM2, although totally
reasonable for LVM1.  At the risk of sounding patronising; when you
tried LVM2 are you certain you were not running LVM1 by mistake ?
(eg. was the LVM1 kernel driver present? did your volumes appear in
/dev/mapper/ ?).

- Joe

Re: [linux-lvm] syncing with snapshots takes forever?

[Sean Oh]

----- Original Message -----
From: "Joe Thornber" <joe@fib011235813.fsnet.co.uk>
To: <linux-lvm@sistina.com>
Sent: Saturday, January 25, 2003 3:26 AM
Subject: Re: [linux-lvm] syncing with snapshots takes forever?

>
> 30 minutes still seems _far_ too long for LVM2, although totally
> reasonable for LVM1.  At the risk of sounding patronising; when you
> tried LVM2 are you certain you were not running LVM1 by mistake ?
> (eg. was the LVM1 kernel driver present? did your volumes appear in
> /dev/mapper/ ?).
>

LVM1 was removed from the system cleanly. And LVM1 kernel driver was
removed,too.
Yes, /dev/mapper/? is there and /dev/vg01/?? is linked to /dev/mapper/??.

Thanks.

Re: [linux-lvm] syncing with snapshots takes forever?

[Sean Oh]

----- Original Message -----
From: "Andreas Dilger" <adilger@clusterfs.com>
To: <linux-lvm@sistina.com>
Sent: Saturday, January 25, 2003 3:15 AM
Subject: Re: [linux-lvm] syncing with snapshots takes forever?

> The problem is that when you type "sync" you are actually writing (most
of)
> your file to disk 4 times - once for the initial LV (which was mostly
still
> in RAM I'm guessing), and once for each snapshot.

I understand that it is writing from memory to disk 4 times.(Yes, the data
is kept in memery until kernel dumps to disk, since the application I am
using does _async_ operation)

> Since you now have 4x
> the write load and it is not streaming, but random I/O it causes your
disks
> to thrash badly.  Add to that the fact that you are writing to a RAID
which
> needs to seek _every_ disk for each write, and you are in really bad
shape.


I do not think the RAID is one of the significant factor for the slowdown
here, since systems without RAID have the pretty much same problem.

> I'm guessing that if you put each snapshot on its own physical disk (maybe
> only keeping the original LV on the RAID device) you will see a lot of
> improvement.

I did.. see below

> Also, it would be interesting to know what the real performance
> of even the single 1GB write is (i.e. write + sync) because I'm guessing
> most of the data is still in memory.

Well, I have tested on the different system than my orignal posting. This
system has 4GB of RAM  and FC external RAID5 system, composing 4 RAID5
volumes with 5 FC disks each.

By doing 'sync ; time dd if=/dev/zero of=1G bs=1024k count=1000 ; time sync'
.

The dd write + sync speed on LV without snapshots was about110MB/s.
The dd write + sync speed on LV with 1 snapshots was about 5MB/s,too
The dd write + sync speed on LV with 2 snapshots was about 3.5MB/s,too
The dd write + sync speed on LV with 3 snapshots was about 3.2MB/s.

After putting the snapshot PVs on the different RAID volume:
The dd write + sync speed on LV with 1 snapshots was about 12MB/s,too
The dd write + sync speed on LV with 2 snapshots was about 9MB/s,too
The dd write + sync speed on LV with 2 snapshots was about 7MB/s,too

When I test with NFS(async mode) and SAMBA,they go fast until let's say a
few hundreds of MB is filled up in the memory, needed to be flushed and then
the trasfer suffers, nearly _STOPPED_ responding, producing 0 to a few
hundred _bytes_ for the transfer speed.

 > At, say, 50MB/s to disk you are still
> talking 3 or 4 minutes to write out 1GB, so even an ideal LV snapshot case
> would take 9-15 minutes to write out 4GB.


> Cheers, Andreas

Well, even with 1 snapshot, the throughput goes down to 1/10.

For my purpose, I would like to have NFS and SAMBA to transfer data
consistently with async mode. It seems to me that some logic in lvm snapshot
or kernel is causing those apps to stoped for a few tens of minutes when
memory is full and synced.
Do you know any patches to kernel or lvm that does not block(?) the I/O
operations while syncing or do something smarter for handling above issues?