[linux-lvm] Automatic snapshot system?

[linux-lvm] Automatic snapshot system?

[Jason L Tibbitts III]

Has anyone worked up a system that automatically snapshots volumes at
regular intervals and then expires the snapshots as necessary?  If
not, I may take a stab at it.

 - J<

Re: [linux-lvm] Automatic snapshot system?

[Petro]

On Thu, Nov 29, 2001 at 01:08:33AM -0600, Jason L Tibbitts III wrote:
> Has anyone worked up a system that automatically snapshots volumes at
> regular intervals and then expires the snapshots as necessary?  If
> not, I may take a stab at it.

    We have a home-brewed perl script called from cron that does a snap,
    rsync's out the data, then releases the snapshot. 

    Is that what you're talking about? 

-- 
Share and Enjoy. 

Re: [linux-lvm] Automatic snapshot system?

[Jason L Tibbitts III]

>>>>> "p" == petro  <petro@auctionwatch.com> writes:

p> We have a home-brewed perl script called from cron that does a
p> snap, rsync's out the data, then releases the snapshot.

Interesting, but not quite what I'm trying to do.  Essentially I'd
like to use snapshots as a sort of quick online backup to guard
against users nuking their files.  Of course we do backups nightly,
but restoring from a snapshot is much quicker than messing with tapes.

So, depending on how snapshots actually work and the performance
penalty they incur and various other bits, I was thinking of doing a
daily or twice-daily snapshot, keeping them around for a few days,
then deleting them.  But that entails having around 100 snapshots
sitting around, and I don't know if there are performance problems
with doing that.

 - J<

Re: [linux-lvm] Automatic snapshot system?

[Petro]

On Thu, Nov 29, 2001 at 02:42:58PM -0600, Jason L Tibbitts III wrote:
> >>>>> "p" == petro  <petro@auctionwatch.com> writes:
> 
> p> We have a home-brewed perl script called from cron that does a
> p> snap, rsync's out the data, then releases the snapshot.
> Interesting, but not quite what I'm trying to do.  Essentially I'd
> like to use snapshots as a sort of quick online backup to guard
> against users nuking their files.  Of course we do backups nightly,
> but restoring from a snapshot is much quicker than messing with tapes.

    Well, I won't get into the debate on whether it's better to protect
    the users or educate them (ie easy to access backups make users
    sloppy kind of argument), but were I tasked with such a thing, I'd
    do it thus: 

    I'm assuming a situation were you have "working hours", but if
    that's not the case, then only the times have to be modified: 

    Establish a cron-job to run at <work-start-time>, <work-start+4> and
    <work-end>. This cron job calls a script that expires the old snap
    and creates a new one. If you want (for whatever reason) to keep
    multiple old snapshots around (and it can be done I believe) you
    would just make the script smarter and have it expire <current-<x>>
    snapshots. 

> So, depending on how snapshots actually work and the performance
> penalty they incur and various other bits, I was thinking of doing a
> daily or twice-daily snapshot, keeping them around for a few days,
> then deleting them.  But that entails having around 100 snapshots
> sitting around, and I don't know if there are performance problems
> with doing that.

    I'd imagine there is, since the snap doesn't "store" all the data,
    but rather it stores the unchanged version of changed data, meaning
    that as long as the snap is active, it's getting writes. You may be
    able to deactivate a snap and keep it around, but I don't know how
    that works. 

-- 
Share and Enjoy. 

Re: [linux-lvm] Automatic snapshot system?

[Jason L Tibbitts III]

>>>>> "p" == petro  <petro@auctionwatch.com> writes:

p> Well, I won't get into the debate on whether it's better to
p> protect the users or educate them (ie easy to access backups make
p> users sloppy kind of argument),

Uh, well, I've been an administrator for 13 years and a computer user
for 24, and I still accidentally overwrite a file or delete a
directory on occasion.  And when this happens, I don't particularly
want to have to mess with the tape drive.  The the LVM system already
includes a method for making snapshots; why not make use of it?
Educating users doesn't really come into it.

p> Establish a cron-job to run at <work-start-time>, <work-start+4>
p> and <work-end>. This cron job calls a script that expires the old
p> snap and creates a new one.

Well, I understand how to do it.  I was simply asking if anyone had
already done it so I can save myself some work.  (It's actually mildly
complicated.  You have to parse various /proc files to get the LVM
state and figure out which snapshots you have active and whether any
of them have filled and thus become unusable.  You also have to update
/etc/fstab so a reboot doesn't lose your snapshots.)

 - J<

Re: [linux-lvm] Automatic snapshot system?

[Joe Thornber]

On Thu, Nov 29, 2001 at 02:42:58PM -0600, Jason L Tibbitts III wrote:
> >>>>> "p" == petro  <petro@auctionwatch.com> writes:
> 
> p> We have a home-brewed perl script called from cron that does a
> p> snap, rsync's out the data, then releases the snapshot.
> 
> Interesting, but not quite what I'm trying to do.  Essentially I'd
> like to use snapshots as a sort of quick online backup to guard
> against users nuking their files.  Of course we do backups nightly,
> but restoring from a snapshot is much quicker than messing with tapes.
> 
> So, depending on how snapshots actually work and the performance
> penalty they incur and various other bits, I was thinking of doing a
> daily or twice-daily snapshot, keeping them around for a few days,
> then deleting them.  But that entails having around 100 snapshots
> sitting around, and I don't know if there are performance problems
> with doing that.

It's a nice idea, but with the current implementation there's no way
I'd want 100 snapshots of the same origin hanging around.

- Joe

Re: [linux-lvm] Automatic snapshot system?

[Jason L Tibbitts III]

>>>>> "JT" == Joe Thornber <thornber@btconnect.com> writes:

JT> It's a nice idea, but with the current implementation there's no
JT> way I'd want 100 snapshots of the same origin hanging around.

They wouldn't all be of the same LV, fortunately; I split my VG up
into lots of manageable chunks.  (Large users get a volume to
themselves, small users get grouped into various user volumes.)  But
still, I assume that you have some details of performance with the
current snapshot implementation.  Is it that bad?

 - J<

Re: [linux-lvm] Automatic snapshot system?

[Heinz J . Mauelshagen]

On Fri, Nov 30, 2001 at 09:12:53AM -0600, Jason L Tibbitts III wrote:
> >>>>> "JT" == Joe Thornber <thornber@btconnect.com> writes:
> 
> JT> It's a nice idea, but with the current implementation there's no
> JT> way I'd want 100 snapshots of the same origin hanging around.
> 
> They wouldn't all be of the same LV, fortunately; I split my VG up
> into lots of manageable chunks.  (Large users get a volume to
> themselves, small users get grouped into various user volumes.)  But
> still, I assume that you have some details of performance with the
> current snapshot implementation.  Is it that bad?

Well, it does

- copy on write (COW) synchronously *before* et allows the write to
  the original LV to proceed

- do multiple reads in case you`ve got multiple snapshots

- does the COWs for multiple snapshots sequentially

This leads to an unecessary read per additional snapshot on the original
LV and therefore to something around n-times delay in case you've got
n snapshots.

We will have an asynchrounous implementation which avoids multiple
reads with LVM2.

> 
>  - J<
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html

-- 

Regards,
Heinz    -- The LVM Guy --

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-