[linux-lvm] Keep snapshots active for 24 hours?

[linux-lvm] Keep snapshots active for 24 hours?

[Dave Alden]

Hi,
  I'm in the process of evaluating LVM (with XFS) for our ~500G RAID NFS
server.  I'm using hardware RAID, so I'm only using LVM is for snapshots.
My original plan was to take a snapshot around midnight everyday, run the
backup, then remove the snapshot (just as the FAQ says to :-).  However,
I'd like to offer my users the ability to have access to the previous days
snapshot (so they can easily restore an accidentally deleted file).  I don't
need all 500G (we're currently using only ~60G), so I was thinking of either
50G or 100G for the snapshot (although I only expect ~5G/day to be changed).
I was wondering what kind of performance hit would I take if I kept the
snapshot for 24 hours?
...thnx,
...dave alden

Re: [linux-lvm] Keep snapshots active for 24 hours?

[Andreas Dilger]

On Dec 14, 2001  07:35 -0500, Dave Alden wrote:
>   I'm in the process of evaluating LVM (with XFS) for our ~500G RAID NFS
> server.  I'm using hardware RAID, so I'm only using LVM is for snapshots.
> My original plan was to take a snapshot around midnight everyday, run the
> backup, then remove the snapshot (just as the FAQ says to :-).  However,
> I'd like to offer my users the ability to have access to the previous days
> snapshot (so they can easily restore an accidentally deleted file).  I don't
> need all 500G (we're currently using only ~60G), so I was thinking of either
> 50G or 100G for the snapshot (although I only expect ~5G/day to be changed).
> I was wondering what kind of performance hit would I take if I kept the
> snapshot for 24 hours?

Well, it will basically mean [read-]write-write for all data going to the
disk.  Generally, this will not matter a huge amount unless you are doing
tons of I/O.  You could start with a large snapshot size, and then get an
idea of how much data really changes each day.  You could then actually
keep multiple snapshots if you had the space.  If you were really clever,
you could write a script which checked the free space in the snapshots
every minute, and if they had less than X PEs free, you extend the snapshot
LV to have more free space.  If you run out of free PEs, you delete the
oldest snapshot.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

RE: [linux-lvm] Keep snapshots active for 24 hours?

[Steve Wray]

I'd been wondering if snapshotting could be used for;

- providing 'virtual journalling' any filesystem that happens to
be on a logical volume.

- take a snapshot, install something or try something out
that might break something, then restore *directly* from
the snapshot if anything goes wrong; without having to actually
back it up to media and restore it.


> -----Original Message-----
> From: linux-lvm-admin@sistina.com [mailto:linux-lvm-admin@sistina.com]On
> Behalf Of Andreas Dilger
> Sent: Saturday, 15 December 2001 07:38
> To: Dave Alden
> Cc: linux-lvm@sistina.com
> Subject: Re: [linux-lvm] Keep snapshots active for 24 hours?
>
>
> On Dec 14, 2001  07:35 -0500, Dave Alden wrote:
> >   I'm in the process of evaluating LVM (with XFS) for our ~500G RAID NFS
> > server.  I'm using hardware RAID, so I'm only using LVM is for
> snapshots.
> > My original plan was to take a snapshot around midnight
> everyday, run the
> > backup, then remove the snapshot (just as the FAQ says to :-).  However,
> > I'd like to offer my users the ability to have access to the
> previous days
> > snapshot (so they can easily restore an accidentally deleted
> file).  I don't
> > need all 500G (we're currently using only ~60G), so I was
> thinking of either
> > 50G or 100G for the snapshot (although I only expect ~5G/day to
> be changed).
> > I was wondering what kind of performance hit would I take if I kept the
> > snapshot for 24 hours?
>
> Well, it will basically mean [read-]write-write for all data going to the
> disk.  Generally, this will not matter a huge amount unless you are doing
> tons of I/O.  You could start with a large snapshot size, and then get an
> idea of how much data really changes each day.  You could then actually
> keep multiple snapshots if you had the space.  If you were really clever,
> you could write a script which checked the free space in the snapshots
> every minute, and if they had less than X PEs free, you extend
> the snapshot
> LV to have more free space.  If you run out of free PEs, you delete the
> oldest snapshot.
>
> Cheers, Andreas
> --
> Andreas Dilger
> http://sourceforge.net/projects/ext2resize/
> http://www-mddsp.enel.ucalgary.ca/People/adilger/
>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
>

Re: [linux-lvm] Keep snapshots active for 24 hours?

[Andreas Dilger]

On Dec 15, 2001  10:52 +1300, Steve Wray wrote:
> I'd been wondering if snapshotting could be used for;
> 
> - providing 'virtual journalling' any filesystem that happens to
> be on a logical volume.

It could, but you would have to make a _lot_ of snapshots for it to
be worthwhile (i.e. one every 10-30 seconds or so).  If you want this
for "non-journaled" filesystems, you can use the ext3 journaling code
to add this to most any block based filesystem.  It isn't trivial,
but at least possible.

> - take a snapshot, install something or try something out
> that might break something, then restore *directly* from
> the snapshot if anything goes wrong; without having to actually
> back it up to media and restore it.

Yes, this would be clever.  I'd rather that RPM/DEB package tools just
become smarter (like AIX LPP) where it allows you arbitrary amounts of
back-out from updated packages (as long as you have enough disk space).
After you have "applied" a package, you can test it out, and either
commit it or revert to the old version.  The rpm and dpkg tools could
just do this by checksumming any existing files from the previous install
and only backing up those that had changed between versions.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

RE: [linux-lvm] Keep snapshots active for 24 hours?

[Steve Wray]

> From: linux-lvm-admin@sistina.com [mailto:linux-lvm-admin@sistina.com]On
> Behalf Of Andreas Dilger
> 
> On Dec 15, 2001  10:52 +1300, Steve Wray wrote:
> > I'd been wondering if snapshotting could be used for;
> > 
> > - providing 'virtual journalling' any filesystem that happens to
> > be on a logical volume.
> 
> It could, but you would have to make a _lot_ of snapshots for it to
> be worthwhile (i.e. one every 10-30 seconds or so).  If you want this
> for "non-journaled" filesystems, you can use the ext3 journaling code
> to add this to most any block based filesystem.  It isn't trivial,
> but at least possible.

Fair enough for snapshotting as it is today,
but maybe the underlying technology could be adapted?

 
> > - take a snapshot, install something or try something out
> > that might break something, then restore *directly* from
> > the snapshot if anything goes wrong; without having to actually
> > back it up to media and restore it.
> 
> Yes, this would be clever.  I'd rather that RPM/DEB package tools just
> become smarter (like AIX LPP) where it allows you arbitrary amounts of
> back-out from updated packages (as long as you have enough disk space).
> After you have "applied" a package, you can test it out, and either
> commit it or revert to the old version.  The rpm and dpkg tools could
> just do this by checksumming any existing files from the previous install
> and only backing up those that had changed between versions.
> 
> Cheers, Andreas
> --
> Andreas Dilger
> http://sourceforge.net/projects/ext2resize/
> http://www-mddsp.enel.ucalgary.ca/People/adilger/
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
> 

[linux-lvm] catch-22

[Erick Calder]

a drive part of a volume crashed on me... I've looked through the man pages
and HOWTO but find no happiness... help!

the volume used to be called LVM (/dev/LVM) and had /dev/hda3 and
/dev/hdc... hdc crashed and I've replaced it with an identical drive.  I did
a "pvcreate /dev/hdc" after installation but cannot seem to add it to the
volume:

# vgextend LVM /dev/hdc
vgextend -- volume group "LVM" doesn't exist

and if I try to create it:

# vgcreate LVM /dev/hdc
vgcreate -- volume group directory or file already exists
vgcreate -- please choose a different name

I look in:

# ls /dev/LVM
total 104
dr-xr-xr-x    2 root     root         4096 Oct 26 12:54 ./
drwxr-xr-x   14 root     root        98304 Dec 13 17:46 ../
crw-r-----    1 root     disk     109,   0 Oct 26 12:54 group
brw-rw----    1 root     disk      58,   0 Oct 26 12:54 mp3z

# ls /proc/lvm/VGs
total 0
dr-xr-xr-x    2 root     root            0 Dec 14 17:52 ./
dr-xr-xr-x    3 root     root            0 Dec 14 17:52 ../

what is the correct procedure to follow? shall I just blow away the /dev/LVM
directory and do a "pvcreate -ff /dev/hda3"?

1k thx - e

Re: [linux-lvm] catch-22

[Wolfgang Weisselberg]

Hi, Erick!

Erick Calder (e@arix.com) wrote 43 lines:

> a drive part of a volume crashed on me... I've looked through the man pages
> and HOWTO but find no happiness... help!

> # vgextend LVM /dev/hdc
> vgextend -- volume group "LVM" doesn't exist

> # vgcreate LVM /dev/hdc
> vgcreate -- volume group directory or file already exists
> vgcreate -- please choose a different name

> what is the correct procedure to follow? shall I just blow away the /dev/LVM
> directory and do a "pvcreate -ff /dev/hda3"?

I am no expert.  I don't even try to guarantee that you'll
not blow up your data (and the whole planet, too).  But AFAICT
/dev/LVM is created as needed with vgscan.  You did run vgscan
after the HW-change?

You did a vgcfgrestore on the new/replaced /dev/hdc PV?

-Wolfgang

PS: I'd still advise a current, working backup. :-)

Re: [linux-lvm] Keep snapshots active for 24 hours?

[Wolfgang Weisselberg]

Hi, Steve!

Steve Wray (steve.wray@the.net.nz) wrote 70 lines:

> - take a snapshot, install something or try something out
> that might break something, then restore *directly* from
> the snapshot if anything goes wrong; without having to actually
> back it up to media and restore it.

Even better would be writable snapshots for that: 
- make a snapshot of (everything where the install writes to
- install the program
- test it on the snapshot
- junk the snapshot if the install is bad

and, of course, you might want to migrate changes from the
snapshot to the original FS (or vice versa).  That however
is a second feature.  Until then you'd have to do the install
all over again -- once we actually get writable shnapshots.

-Wolfgang

Re: [linux-lvm] Keep snapshots active for 24 hours?

[Kirby C. Bohling]

Wolfgang,

	While the concept of a writeable LVM is cool, I believe LVM operates a 
low enough level that the migrations you speak of is impossible at the 
LVM level (in the kernel).  There could be filesystem level utilities, 
but the LVM is really nothing more then a blank harddisk partition. 
Harddisks shouldn't know a thing about files, directories, or 
filesystems in general.  It would be cool to have an filesystem diff, 
but on a live filesystem that could be quite tricky.  Essentially, that 
would be a differential backup utility.

	A writeable LVM would solve the problem of taking a snapshot of a 
journalling filesystem.  Somebody recently wrote in with a problem with 
ReiserFS that when he mounted the snapshot it kept giving him an error 
that the journal couldn't be rolled forward as it was mounted read only. 
  I am not sure if ReiserFS works okay with this or not, but the error 
is because there is uncommitted changes in the journal, no commits 
allowed because its read-only.

	It would be nice to essentially get a chance to install software in a 
controlled manner and once your sure it is working then roll it forward. 
  About the only issue I can see with that is having something that you 
install on /usr (which you took a snapshot of), that has to write to 
/etc which might take some trickery to get to be written to a 
snapshotted version of /etc.  The only thing I can think of off hand is 
to snapshot / and all the other filesystems, then mount the snap shot of 
/ on /myChrootJail then chroot to that and mount all of the various 
snapshots under that system.  All of this would work, but when you got 
done with the install you would have to do it all on the live device 
system.  The only usefulness I can imagine for that is if you are a 
scripting god and script it all up in the play environment and then do 
the real deal using the scripts you developed or as a dry practice run 
of an install.  Would be handy in a very controlled environment where 
repeatability is of the utmost importance.

	Okay so for the highly far fetched, it would be fun to tinker with this 
where you chroot jailed everybody in that environment that ssh'ed in, so 
some foolish hacker could break into the box, screw up a snapshot and 
then you just refresh the snapshot.  It would be fun to use in a 
honeypot situation.

		Kirby


Wolfgang Weisselberg wrote:

> Hi, Steve!
> 
> Steve Wray (steve.wray@the.net.nz) wrote 70 lines:
> 
> 
>>- take a snapshot, install something or try something out
>>that might break something, then restore *directly* from
>>the snapshot if anything goes wrong; without having to actually
>>back it up to media and restore it.
>>
> 
> Even better would be writable snapshots for that: 
> - make a snapshot of (everything where the install writes to
> - install the program
> - test it on the snapshot
> - junk the snapshot if the install is bad
> 
> and, of course, you might want to migrate changes from the
> snapshot to the original FS (or vice versa).  That however
> is a second feature.  Until then you'd have to do the install
> all over again -- once we actually get writable shnapshots.
> 
> -Wolfgang
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
> 


-- 
Real Programmers view electronic multimedia files with a hex editor.

Re: [linux-lvm] Keep snapshots active for 24 hours?

[Petro]

On Sun, Dec 16, 2001 at 04:07:31PM -0600, Kirby C. Bohling wrote:
> 	It would be nice to essentially get a chance to install software in 
> 	a controlled manner and once your sure it is working then roll it forward. 

    There is. They are called "testing machines". 

    I interviewed at a firm that traded on a financial market in Chicago
    many years ago, and on every traders desk there were 3 machines. Two
    production machines running identical software more or less in
    parallel, and a third "testing" machine that the did the same stuff,
    only with new versions of the software. 

    Software spent 6 months in that "testing" environment before being
    rolled live. 

    While I think LVM is a really neat hammer, not everthing is a nail. 

-- 
Share and Enjoy. 

Re: [linux-lvm] Keep snapshots active for 24 hours?

[Wolfgang Weisselberg]

Hi, Kirby!

Kirby C. Bohling (kbohling@birddog.com) wrote 88 lines:

> 	While the concept of a writeable LVM is cool, I believe LVM operates a 
> low enough level that the migrations you speak of is impossible at the 
> LVM level (in the kernel).

Moving the *complete* snapshot onto the LVM should be easy:
lock the FS (or be very clever), replay the changes in the
snapshot and migrate them onto the parent (i.e. look up the
page mapping and write the used pages onto their respective
pages in the parent).

Moving just selective parts has to be done as a FS-level
tool, though, you are right there.

> It would be cool to have an filesystem diff, 
> but on a live filesystem that could be quite tricky.  Essentially, that 
> would be a differential backup utility.

Well, that's what a snapshot is, isn't it?  Just that it
operates on (usually) 4M pages...

> 	A writeable LVM would solve the problem of taking a snapshot of a 
> journalling filesystem.

Actually there is a patch to do that even while the journalled
FS is mounted.  You also have to allow that change by enabling
it in the LVM-kernel-sourcecode by changing a #define.

Cleanly umounted partitions should be no problem, they have
no log that needs a replay.

Anyway, there are more useful things you can do with
writable snapshots; just think of an error-resistant system.
Everything happens on the big-enough snapshots, but you can
reset the system in a snap; just drop all the snapshots and
recreate them.  This is far faster than copying or extracting
a .tar.gz, and the 'base system' is easier to upgrade, too.

-Wolfgang

RE: [linux-lvm] Keep snapshots active for 24 hours?

[Steve Wray]

> From: linux-lvm-admin@sistina.com [mailto:linux-lvm-admin@sistina.com]On
> Behalf Of Wolfgang Weisselberg
[big snip]
> Anyway, there are more useful things you can do with
> writable snapshots; just think of an error-resistant system.
> Everything happens on the big-enough snapshots, but you can
> reset the system in a snap; just drop all the snapshots and
> recreate them.  This is far faster than copying or extracting
> a .tar.gz, and the 'base system' is easier to upgrade, too.

Yes! And when can we have it, please?
8)

Re: [linux-lvm] Keep snapshots active for 24 hours?

[Wolfgang Weisselberg]

Hi, Steve!

Steve Wray (steve.wray@the.net.nz) wrote 20 lines:

> > From: linux-lvm-admin@sistina.com [mailto:linux-lvm-admin@sistina.com]On
> > Behalf Of Wolfgang Weisselberg

> > Anyway, there are more useful things you can do with
> > writable snapshots; just think of an error-resistant system.
> > Everything happens on the big-enough snapshots, but you can
[...]

> Yes! And when can we have it, please?
> 8)

BTDTGTTS.
But that was without snapshots and not as perfect as I'd
liked to have it.

-Wolfgang

Re: [linux-lvm] Keep snapshots active for 24 hours?

[Anselm Kruis]

Hi Wolfgang,

On Sun, 16 Dec 2001, Wolfgang Weisselberg wrote:

> Hi, Steve!
> 
> Steve Wray (steve.wray@the.net.nz) wrote 70 lines:
> 
> > - take a snapshot, install something or try something out
> > that might break something, then restore *directly* from
> > the snapshot if anything goes wrong; without having to actually
> > back it up to media and restore it.
> 
> Even better would be writable snapshots for that: 
> - make a snapshot of (everything where the install writes to
> - install the program
> - test it on the snapshot
> - junk the snapshot if the install is bad
> 
> and, of course, you might want to migrate changes from the
> snapshot to the original FS (or vice versa).  That however
> is a second feature.  Until then you'd have to do the install
> all over again -- once we actually get writable shnapshots.

I posted a patch for writeable snapshots about a month ago. Look for the
subject "[PATCH] writeable snapshots". The patch sould apply clean against
current LVM.

-Anselm

> 
> -Wolfgang
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
> 

---
 Anselm Kruis                                 Tel. +49 (0)89-356386-74
 science + computing ag		              FAX  +49 (0)89-356386-37
 Ingolstädter Str. 22             mailto: A.Kruis@science-computing.de
 D-80807 München                 WWW: http://www.science-computing.de/

**********************************************************************
***     CeBIT 2002                                                 ***
***     Besuchen Sie uns auf der CeBIT vom 13.-20.03.2002          ***
***     auf dem Messegelände in Hannover, Halle 11                 ***
**********************************************************************

Re: [linux-lvm] Keep snapshots active for 24 hours?

[Terje Kvernes]

* Dave Alden

> I'm in the process of evaluating LVM (with XFS) for our ~500G RAID
> NFS server. I'm using hardware RAID, so I'm only using LVM is for
> snapshots. My original plan was to take a snapshot around midnight
> everyday, run the backup, then remove the snapshot (just as the FAQ
> says to :-). However, I'd like to offer my users the ability to have
> access to the previous days snapshot (so they can easily restore an
> accidentally deleted file). I don't need all 500G (we're currently
> using only ~60G), so I was thinking of either 50G or 100G for the
> snapshot (although I only expect ~5G/day to be changed). I was
> wondering what kind of performance hit would I take if I kept the
> snapshot for 24 hours?

* Andreas Dilger

> Well, it will basically mean [read-]write-write for all data going
> to the disk. Generally, this will not matter a huge amount unless
> you are doing tons of I/O. You could start with a large snapshot
> size, and then get an idea of how much data really changes each day.

  this is something I'm hoping to get done -- eventually. the idea of
  having at least a snapshot taken an hour ago will be very nice for
  our dearest tape-robot-backup-thingy. not to mention the network. :)

> You could then actually keep multiple snapshots if you had the
> space. 

  hm. this will add one extra write for every currently open snapshot,
  right? 

> If you were really clever, you could write a script which checked
> the free space in the snapshots every minute, and if they had less
> than X PEs free, you extend the snapshot LV to have more free space.
> If you run out of free PEs, you delete the oldest snapshot.

  heh. I was thinking of trying to get a snapshot taken every three
  hours or so, and storing it until the next one came about. but of
  course, you're right, I should check how much free space I have, and
  keep a backlog until I _need_ the space. 

  come to think of it, it doesn't take that complicated a shell-script
  to get this working. hm... :)

-- 
Terje - who finally gets to be with his girlfriend again, after 4
        months and 10 days... *yay*!

Re: [linux-lvm] Keep snapshots active for 24 hours?

[Andreas Dilger]

On Dec 15, 2001  02:00 +0100, Terje Kvernes wrote:
> * Andreas Dilger
> > If you were really clever, you could write a script which checked
> > the free space in the snapshots every minute, and if they had less
> > than X PEs free, you extend the snapshot LV to have more free space.
> > If you run out of free PEs, you delete the oldest snapshot.
> 
>   heh. I was thinking of trying to get a snapshot taken every three
>   hours or so, and storing it until the next one came about. but of
>   course, you're right, I should check how much free space I have, and
>   keep a backlog until I _need_ the space. 
> 
>   come to think of it, it doesn't take that complicated a shell-script
>   to get this working. hm... :)

In fact, the LVM snapshots have a facility where you can have it wake a
process when the snapshot is more than X percent full.  You probably
couldn't do it with a shell script (it is an ioctl), but you could
either write a simple executable (e.g. lvmsnapwait 95 /dev/vgtest/lvtest),
or do it with Perl (there is a Perl ioctl module, whose name I have
forgotten).

This avoids polling for lots of snapshots of lots of LVs, and also prevents
you from over-committing PEs to snapshots.

Note that there is a system-wide hard limit of 255 LVs (ABS_MAX_LV).  The
problem is that LVM only has a single block major number, with 256 minor
numers, so you might have to do some kernel/tool hackery to exceed that.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/