[linux-lvm] Server hangs when writing to LVM volume due to LVM Snapshots

[linux-lvm] Server hangs when writing to LVM volume due to LVM Snapshots

[Veselin Kantsev]

Hello,
I'm using LVM snapshots on the following configuration:

Debian lenny (2.6.24-1-amd64)

LVM version:     2.02.33 (2008-01-31)
Library version: 1.02.24 (2007-12-20)
Driver version:  4.12.0

3ware 9000 Storage Controller device driver for Linux v2.26.02.010

I have a Logical volume running off a RAID5 for all my data.
I recently started making daily LVM snapshots of this Logical volume.

The problem I'm experiencing now is that the server freezes if I try
write a big chunk of data to the LVM volume.

Writing is OK on the second RAID (which holds the Debian OS)

Is there some known problem with writing lots of data to LVs while using
LVM snapshots?


Thank you.

-- 
Regards,
Veselin

Re: [linux-lvm] Server hangs when writing to LVM volume due to LVM Snapshots

[Tomasz Chmielewski]

Veselin Kantsev schrieb:
> Hello,
> I'm using LVM snapshots on the following configuration:
> 
> Debian lenny (2.6.24-1-amd64)
> 
> LVM version:     2.02.33 (2008-01-31)
> Library version: 1.02.24 (2007-12-20)
> Driver version:  4.12.0
> 
> 3ware 9000 Storage Controller device driver for Linux v2.26.02.010
> 
> I have a Logical volume running off a RAID5 for all my data.
> I recently started making daily LVM snapshots of this Logical volume.
> 
> The problem I'm experiencing now is that the server freezes if I try
> write a big chunk of data to the LVM volume.
> 
> Writing is OK on the second RAID (which holds the Debian OS)
> 
> Is there some known problem with writing lots of data to LVs while using
> LVM snapshots?

What is your RAM usage as you write to the snapshot?

Snapshots take some memory, and if you have lots of snapshots or they're 
just big, you can run out of memory.

Is it a possibility for you?


-- 
Tomasz Chmielewski
http://wpkg.org

Re: [linux-lvm] Server hangs when writing to LVM volume due to LVM Snapshots

[Veselin Kantsev]

Hello Tomasz,
thank you much for the prompt reply.

The server has 6119216k of RAM in total.
And most of it is shown as used all the time (To my knowledge thats just how
linux utilizes ram). 

But from memory, the last time the server froze during copying (goes
back to normal once I kill the copy process) I didn't notice any extreme
RAM usage or swapping.


As I'm rotating LVM snapshots weekly, there are 7 snapshots on the system
at all times, and 1 of them is active(the last one taken).

The script that I use for creating the snapshots, checks the current day
of the week removes last weeks snapshot and creates new one.

So:
On monday, check if snapshot.monday exists, lvremove it and lvcreate new
snapshot.monday.
On tuesday, remove snapshot.tuesday if exists, then create new
snapshot.tuesday.
And so on.


Do you see a flaw in this process that might be causing the issues?

Regards,
Veselin


On Mon, Jun 16, 2008 at 03:57:46PM +0200, Tomasz Chmielewski wrote:
> Veselin Kantsev schrieb:
>> Hello,
>> I'm using LVM snapshots on the following configuration:
>>
>> Debian lenny (2.6.24-1-amd64)
>>
>> LVM version:     2.02.33 (2008-01-31)
>> Library version: 1.02.24 (2007-12-20)
>> Driver version:  4.12.0
>>
>> 3ware 9000 Storage Controller device driver for Linux v2.26.02.010
>>
>> I have a Logical volume running off a RAID5 for all my data.
>> I recently started making daily LVM snapshots of this Logical volume.
>>
>> The problem I'm experiencing now is that the server freezes if I try
>> write a big chunk of data to the LVM volume.
>>
>> Writing is OK on the second RAID (which holds the Debian OS)
>>
>> Is there some known problem with writing lots of data to LVs while using
>> LVM snapshots?
>
> What is your RAM usage as you write to the snapshot?
>
> Snapshots take some memory, and if you have lots of snapshots or they're  
> just big, you can run out of memory.
>
> Is it a possibility for you?
>
>
> -- 
> Tomasz Chmielewski
> http://wpkg.org
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Server hangs when writing to LVM volume due to LVM Snapshots

[Tomasz Chmielewski]

Veselin Kantsev schrieb:
> Hello Tomasz,
> thank you much for the prompt reply.
> 
> The server has 6119216k of RAM in total.

That's quite a bit..
You would need terabytes of snapshots to fill it.
So perhaps it hangs for a different reason.

Capacitors? Hardware?


> And most of it is shown as used all the time (To my knowledge thats just how
> linux utilizes ram). 

Please show the output of "free" command. The "buffers/cache" line is 
your current RAM usage, excluding buffers (more or less).


> But from memory, the last time the server froze during copying (goes
> back to normal once I kill the copy process) I didn't notice any extreme
> RAM usage or swapping.

Kernel data in memory used for snapshots will not be swapped out.


> As I'm rotating LVM snapshots weekly, there are 7 snapshots on the system
> at all times, and 1 of them is active(the last one taken).

(...)

> Do you see a flaw in this process that might be causing the issues?

Unless you have terabytes of data in snapshots, it shouldn't be a problem.

Does it freeze for good? Machine stops responding, stops logging, etc.?


-- 
Tomasz Chmielewski
http://wpkg.org

Re: [linux-lvm] Server hangs when writing to LVM volume due to LVM Snapshots

[Veselin Kantsev]

Hi Tomasz,

Here is some more hardware info related to HDDs and RAID card:

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache AVrfy
------------------------------------------------------------------------------
u0    RAID-1    OK             -       -       -       232.82    ON    OFF
u1    RAID-5    OK             -       -       64K     2095.44   ON    OFF

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
p0     OK               u0     232.88 GB   488397168     6RY461CT
p1     OK               u0     232.88 GB   488397168     6RY44PSH
p2     OK               u1     698.63 GB   1465149168    WD-WCAPT0527412
p3     OK               u1     698.63 GB   1465149168    WD-WCAPT0537910
p4     OK               u1     698.63 GB   1465149168    WD-WCAPT0527026
p5     OK               u1     698.63 GB   1465149168    WD-WCAPT0527156
p6     NOT-PRESENT      -      -           -             -
p7     NOT-PRESENT      -      -           -             -

Name  OnlineState  BBUReady  Status    Volt     Temp     Hours LastCapTest
---------------------------------------------------------------------------
bbu   On           Yes       OK        OK       OK       0     xx-xxx-xxxx



Output from "free":

             total       used       free     shared    buffers cached
Mem:       6119216    5964204     155012          0     393880 2291796
-/+ buffers/cache:    3278528    2840688
Swap:      3903784       1940    3901844



The snapshots are 180GB each.

The current writing process freezes and working with the LV becomes
impossible both locally and through the network.

I've also noticed that any users browsing the LV via samba or
netatalk (the LV is shared on the network) report that the network share
slows down to the point where they cannot browse the folders any more.

Otherwise the system remains operational as it is installed on a
sepparate RAID1. 

When the problem occurs, killing the process thats writing to the LV, 
helps.



By the way, there is a second server with different specs but using LVM
snapshots in the same fashion and I've managed to reproduce the same
problem.

Regards,
Veselin

On Mon, Jun 16, 2008 at 04:51:36PM +0200, Tomasz Chmielewski wrote:
> Veselin Kantsev schrieb:
>> Hello Tomasz,
>> thank you much for the prompt reply.
>>
>> The server has 6119216k of RAM in total.
>
> That's quite a bit..
> You would need terabytes of snapshots to fill it.
> So perhaps it hangs for a different reason.
>
> Capacitors? Hardware?
>
>
>> And most of it is shown as used all the time (To my knowledge thats just how
>> linux utilizes ram). 
>
> Please show the output of "free" command. The "buffers/cache" line is  
> your current RAM usage, excluding buffers (more or less).
>
>
>> But from memory, the last time the server froze during copying (goes
>> back to normal once I kill the copy process) I didn't notice any extreme
>> RAM usage or swapping.
>
> Kernel data in memory used for snapshots will not be swapped out.
>
>
>> As I'm rotating LVM snapshots weekly, there are 7 snapshots on the system
>> at all times, and 1 of them is active(the last one taken).
>
> (...)
>
>> Do you see a flaw in this process that might be causing the issues?
>
> Unless you have terabytes of data in snapshots, it shouldn't be a problem.
>
> Does it freeze for good? Machine stops responding, stops logging, etc.?
>
>
> -- 
> Tomasz Chmielewski
> http://wpkg.org
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Server hangs when writing to LVM volume due to LVM Snapshots

[Tomasz Chmielewski]

Veselin Kantsev schrieb:

> The snapshots are 180GB each.

So that makes 7 snapshots, each is 180GB? What percentage is filled?


> The current writing process freezes and working with the LV becomes
> impossible both locally and through the network.

LVM-2 doesn't scale very well.

If you have I/O intensive workloads, I'd discourage using (multiple) 
snapshots.

A single write to the origin mean additional writes to each and every 
snapshot of that origin.


-- 
Tomasz Chmielewski
http://wpkg.org

Re: [linux-lvm] Server hangs when writing to LVM volume due to LVM Snapshots

[Dan Kegel]

On Mon, Jun 16, 2008 at 12:18 PM, Tomasz Chmielewski <mangoo@wpkg.org> wrote:
> LVM-2 doesn't scale very well.
>
> If you have I/O intensive workloads, I'd discourage using (multiple)
> snapshots.
>
> A single write to the origin mean additional writes to each and every
> snapshot of that origin.

Right.  This is one of the things zumastor.org snapshots do better;
all snapshots for a volume share a common snapshot store,
so performance doesn't fall as you add more snapshots.
You might want to have a look at it... join the zumastor
mailing list or drop in on our irc if you have questions.
- Dan

p.s.
> Tomasz Chmielewski
> http://wpkg.org

wpkg.org looks very cool.  I haven't used it much yet,
but I hope to use it as a kind of automated real world test of
wine sometime.

Re: [linux-lvm] Server hangs when writing to LVM volume due to LVM Snapshots

[Tomasz Chmielewski]

Dan Kegel schrieb:
> On Mon, Jun 16, 2008 at 12:18 PM, Tomasz Chmielewski <mangoo@wpkg.org> wrote:
>> LVM-2 doesn't scale very well.
>>
>> If you have I/O intensive workloads, I'd discourage using (multiple)
>> snapshots.
>>
>> A single write to the origin mean additional writes to each and every
>> snapshot of that origin.
> 
> Right.  This is one of the things zumastor.org snapshots do better;
> all snapshots for a volume share a common snapshot store,
> so performance doesn't fall as you add more snapshots.
> You might want to have a look at it... join the zumastor
> mailing list or drop in on our irc if you have questions.

Speaking of Zumastor - is it equivalent (or just a part of something 
bigger) of still discussed, future LVM-3?

There was supposed to be a presentation titled "LVM3 Advanced Volume 
Manager for Linux" by Daniel Phillips on LinuxTag 2008 in May[1] - do 
you know if it's available anywhere (yet)?


>> Tomasz Chmielewski
>> http://wpkg.org
> 
> wpkg.org looks very cool.  I haven't used it much yet,
> but I hope to use it as a kind of automated real world test of
> wine sometime.

Thanks.
For application installation it should work just fine (as it is what it 
does), but for some more thorough GUI testing...? Anyway, we're getting 
off-topic here ;)

[1] 
http://www.linuxtag.org/2008/de/conf/events/vp-mittwoch/vortragsdetails.html?talkid=51

-- 
Tomasz Chmielewski
http://wpkg.org

Re: [linux-lvm] Server hangs when writing to LVM volume due to LVM Snapshots

[Dan Kegel]

On Mon, Jun 16, 2008 at 1:02 PM, Tomasz Chmielewski <mangoo@wpkg.org> wrote:
> Speaking of Zumastor - is it equivalent (or just a part of something bigger)
> of still discussed, future LVM-3?

I think it's more like a preview of a small part of it
(e.g. improved COW scheme for snapshots).

> There was supposed to be a presentation titled "LVM3 Advanced Volume Manager
> for Linux" by Daniel Phillips on LinuxTag 2008 in May[1] - do you know if
> it's available anywhere (yet)?

He says he'll put the slides online in the next couple days.

See also the talk by the LVM2 maintainer there,
http://www.linuxtag.org/2008/de/conf/events/vp-events/vortragsdetails.html?talkid=75
(No, I don't know if his slides are online.)
I think he and Daniel are in violent agreement about
what LVM needs :-)
- Dan

Re: [linux-lvm] Server hangs when writing to LVM volume due to LVM Snapshots

[Bryn M. Reeves]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dan Kegel wrote:
> See also the talk by the LVM2 maintainer there,
> http://www.linuxtag.org/2008/de/conf/events/vp-events/vortragsdetails.html?talkid=75
> (No, I don't know if his slides are online.)

http://people.redhat.com/agk/talks/

The LinuxTag one isn't there yet.

Cheers,
Bryn.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFIV4AY6YSQoMYUY94RArp0AJ9bd0UiQWLzO6s01HpIfkV677AfdwCfdKmY
wqgOPPUjvonbxkWAF1gkzLc=
=SeVF
-----END PGP SIGNATURE-----

Re: [linux-lvm] Server hangs when writing to LVM volume due to LVM Snapshots

[Veselin Kantsev]

Thank you Dan,
I'll certainly have a look at the zumastor project.

Regards,
Veselin
On Mon, Jun 16, 2008 at 12:38:23PM -0700, Dan Kegel wrote:
> On Mon, Jun 16, 2008 at 12:18 PM, Tomasz Chmielewski <mangoo@wpkg.org> wrote:
> > LVM-2 doesn't scale very well.
> >
> > If you have I/O intensive workloads, I'd discourage using (multiple)
> > snapshots.
> >
> > A single write to the origin mean additional writes to each and every
> > snapshot of that origin.
> 
> Right.  This is one of the things zumastor.org snapshots do better;
> all snapshots for a volume share a common snapshot store,
> so performance doesn't fall as you add more snapshots.
> You might want to have a look at it... join the zumastor
> mailing list or drop in on our irc if you have questions.
> - Dan
> 
> p.s.
> > Tomasz Chmielewski
> > http://wpkg.org
> 
> wpkg.org looks very cool.  I haven't used it much yet,
> but I hope to use it as a kind of automated real world test of
> wine sometime.
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Server hangs when writing to LVM volume due to LVM Snapshots

[Veselin Kantsev]

Hi Thomas,
I think I've misunderstood the concept of snapshots.

So If I have 7 snapshots of the same volume (created at different times)
and a file is changed, would the change affect all 7 snapshots or 
just the latest one created?

If the former, then I imagine changing a 1GB+ file, in my
situation will result in an unpleasant experience with the LVM snapshots.


Regards,
Veselin

On Mon, Jun 16, 2008 at 09:18:32PM +0200, Tomasz Chmielewski wrote:
> Veselin Kantsev schrieb:
>
>> The snapshots are 180GB each.
>
> So that makes 7 snapshots, each is 180GB? What percentage is filled?
>
>
>> The current writing process freezes and working with the LV becomes
>> impossible both locally and through the network.
>
> LVM-2 doesn't scale very well.
>
> If you have I/O intensive workloads, I'd discourage using (multiple)  
> snapshots.
>
> A single write to the origin mean additional writes to each and every  
> snapshot of that origin.
>
>
> -- 
> Tomasz Chmielewski
> http://wpkg.org
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Server hangs when writing to LVM volume due to LVM Snapshots

[Tomasz Chmielewski]

Veselin Kantsev schrieb:
> Hi Thomas,
> I think I've misunderstood the concept of snapshots.
> 
> So If I have 7 snapshots of the same volume (created at different times)
> and a file is changed, would the change affect all 7 snapshots or 
> just the latest one created?
> 
> If the former, then I imagine changing a 1GB+ file, in my
> situation will result in an unpleasant experience with the LVM snapshots.

Yes, writing 1 GB will in effect end up in writing 8 GB if you have 7 
snapshots.


-- 
Tomasz Chmielewski
http://wpkg.org

Re: [linux-lvm] Server hangs when writing to LVM volume due to LVM Snapshots

[Dan Kegel]

On Mon, Jun 16, 2008 at 2:18 PM, Veselin Kantsev
<veselin@campbell-lange.net> wrote:
> I think I've misunderstood the concept of snapshots.

No, only how lvm works under the hood.

> So If I have 7 snapshots of the same volume (created at different times)
> and a file is changed, would the change affect all 7 snapshots or
> just the latest one created?

If the file had not changed since the beginning of time,
then when you change it, each block needs to be
written to each snapshot because of the way COW
is implemented in LVM snapshots.

> If the former, then I imagine changing a 1GB+ file, in my
> situation will result in an unpleasant experience with the LVM snapshots.

Correct.  That's why people have been asking for that common
snapshot store for a long time.  Zumastor is more or less a
preview of this long-sought lvm feature.
- Dan