[linux-lvm] Using LVM snapshots for hourly backups

[linux-lvm] Using LVM snapshots for hourly backups

[Anthony Wright]

I'm trying to work out if there's a way to use LVM snapshots to create 
hourly backups of my data. I realise that this doesn't protect me 
against disk/machine failure, but it would really handy in cases where I 
want easy access to historical files, e.g where I accidentally 
change/delete a file or I want to compare historical files/directories 
to the current versions to see what's been changed.

I thought a really simple and ingeneous approach would be to use LVM 
snapshots so that each hour I make a snapshot, and remove the snapshot 
from 24 hours ago, thus having a rolling hourly backup. However when I 
tried this I ended up with 24 snapshots of the original volume which 
means a single change to the original volume causes each snapshot to do 
a Copy on Write of the changed block, so if I change one block on the 
original volume the snapshots create 24 copies of the original volume 
which isn't very good for disk performance or disk usage!!

What I'd really want is to daisy-chain the snapshots so that 
snapshot-3hours is a snapshot of snapshot-2hours, which is a snapshot of 
snapshot-1hour, which is a snapshot of the original volume, thus if a 
block in the original volume changes only snapshot-1hour is changed and 
once a snapshot is older than an hour it cannot change any more.

I notice that you can't make snapshots of snapshots, so I suspect what 
I'm looking for isn't possible, but I thought I check and if it isn't 
possible suggest it as a really useful feature for the future. Your 
thoughts/comments would be welcome.

Thanks,

Tony Wright.

Re: [linux-lvm] Using LVM snapshots for hourly backups

[Jonathan E Brassow]

[-- Attachment #1: Type: text/plain, Size: 2183 bytes --]

I think everyone feels the same way about the current snapshot 
implementation, and there is growing momentum for change.  See 
"[dm-devel] [RFC] DM Snapshot scalability - Common snap store approach" 
(Oct 9th on dm-devel).

Certainly, any changes will take a while though.

  brassow

On Oct 10, 2006, at 3:00 PM, Anthony Wright wrote:

> I'm trying to work out if there's a way to use LVM snapshots to create 
> hourly backups of my data. I realise that this doesn't protect me 
> against disk/machine failure, but it would really handy in cases where 
> I want easy access to historical files, e.g where I accidentally 
> change/delete a file or I want to compare historical files/directories 
> to the current versions to see what's been changed.
>
> I thought a really simple and ingeneous approach would be to use LVM 
> snapshots so that each hour I make a snapshot, and remove the snapshot 
> from 24 hours ago, thus having a rolling hourly backup. However when I 
> tried this I ended up with 24 snapshots of the original volume which 
> means a single change to the original volume causes each snapshot to 
> do a Copy on Write of the changed block, so if I change one block on 
> the original volume the snapshots create 24 copies of the original 
> volume which isn't very good for disk performance or disk usage!!
>
> What I'd really want is to daisy-chain the snapshots so that 
> snapshot-3hours is a snapshot of snapshot-2hours, which is a snapshot 
> of snapshot-1hour, which is a snapshot of the original volume, thus if 
> a block in the original volume changes only snapshot-1hour is changed 
> and once a snapshot is older than an hour it cannot change any more.
>
> I notice that you can't make snapshots of snapshots, so I suspect what 
> I'm looking for isn't possible, but I thought I check and if it isn't 
> possible suggest it as a really useful feature for the future. Your 
> thoughts/comments would be welcome.
>
> Thanks,
>
> Tony Wright.
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

[-- Attachment #2: Type: text/enriched, Size: 2140 bytes --]

I think everyone feels the same way about the current snapshot
implementation, and there is growing momentum for change.  See
"<bold>[dm-devel] [RFC] DM Snapshot scalability - Common snap store
approach</bold>" (Oct 9th on dm-devel).


Certainly, any changes will take a while though.


 brassow


On Oct 10, 2006, at 3:00 PM, Anthony Wright wrote:


<excerpt>I'm trying to work out if there's a way to use LVM snapshots
to create hourly backups of my data. I realise that this doesn't
protect me against disk/machine failure, but it would really handy in
cases where I want easy access to historical files, e.g where I
accidentally change/delete a file or I want to compare historical
files/directories to the current versions to see what's been changed.


I thought a really simple and ingeneous approach would be to use LVM
snapshots so that each hour I make a snapshot, and remove the snapshot
from 24 hours ago, thus having a rolling hourly backup. However when I
tried this I ended up with 24 snapshots of the original volume which
means a single change to the original volume causes each snapshot to
do a Copy on Write of the changed block, so if I change one block on
the original volume the snapshots create 24 copies of the original
volume which isn't very good for disk performance or disk usage!!


What I'd really want is to daisy-chain the snapshots so that
snapshot-3hours is a snapshot of snapshot-2hours, which is a snapshot
of snapshot-1hour, which is a snapshot of the original volume, thus if
a block in the original volume changes only snapshot-1hour is changed
and once a snapshot is older than an hour it cannot change any more.


I notice that you can't make snapshots of snapshots, so I suspect what
I'm looking for isn't possible, but I thought I check and if it isn't
possible suggest it as a really useful feature for the future. Your
thoughts/comments would be welcome.


Thanks,


Tony Wright.


_______________________________________________

linux-lvm mailing list

linux-lvm@redhat.com

https://www.redhat.com/mailman/listinfo/linux-lvm

read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/


</excerpt>

Re: [linux-lvm] Using LVM snapshots for hourly backups

[Nick]

Sounds like it's not available in LVM but I achieve simmilar results 
with rsync. Take a look at this for some pointers.

http://www.mikerubel.org/computers/rsync_snapshots/

Nick

Jonathan E Brassow wrote:
> I think everyone feels the same way about the current snapshot 
> implementation, and there is growing momentum for change. See 
> "*[dm-devel] [RFC] DM Snapshot scalability - Common snap store 
> approach*" (Oct 9th on dm-devel).
>
> Certainly, any changes will take a while though.
>
> brassow
>

RE: [linux-lvm] Using LVM snapshots for hourly backups

[Roger Lucas]

Hi Nick,

RSYNC is useful for this, but it works at the file level.  If you had, for example, a 1GB e-mail file (e.g. Outlook PST) or a large
database file, and it was getting small changes every day (e.g. receiving a dozen 10KB e-mails => 120KB of changes), then RSYNC
would quickly eat up your disk space as each backup taken with RSYNC a complete new copy of the whole 1GB file.

Cascaded LVM snapshots, on the other hand, would allow the just the changes within the files to be kept, dramatically reducing the
disk usage.

I'm sure that you know this already, but it was worth explicitly stating the difference between RSYCN and LVM snapshots in case
someone reading the list wasn't as sure.

BR,

Roger 

> -----Original Message-----
> From: linux-lvm-bounces@redhat.com [mailto:linux-lvm-bounces@redhat.com] On Behalf Of Nick
> Sent: 13 October 2006 11:14
> To: LVM general discussion and development
> Subject: Re: [linux-lvm] Using LVM snapshots for hourly backups
> 
> Sounds like it's not available in LVM but I achieve simmilar results
> with rsync. Take a look at this for some pointers.
> 
> http://www.mikerubel.org/computers/rsync_snapshots/
> 
> Nick
> 
> Jonathan E Brassow wrote:
> > I think everyone feels the same way about the current snapshot
> > implementation, and there is growing momentum for change. See
> > "*[dm-devel] [RFC] DM Snapshot scalability - Common snap store
> > approach*" (Oct 9th on dm-devel).
> >
> > Certainly, any changes will take a while though.
> >
> > brassow
> >
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Using LVM snapshots for hourly backups

[Greg Freemyer]

On 10/13/06, Roger Lucas <roger@planbit.co.uk> wrote:
> Hi Nick,
>
> RSYNC is useful for this, but it works at the file level.  If you had, for example, a 1GB e-mail file (e.g. Outlook PST) or a large
> database file, and it was getting small changes every day (e.g. receiving a dozen 10KB e-mails => 120KB of changes), then RSYNC
> would quickly eat up your disk space as each backup taken with RSYNC a complete new copy of the whole 1GB file.
>
> Cascaded LVM snapshots, on the other hand, would allow the just the changes within the files to be kept, dramatically reducing the
> disk usage.
>
> I'm sure that you know this already, but it was worth explicitly stating the difference between RSYCN and LVM snapshots in case
> someone reading the list wasn't as sure.
>
> BR,
>
> Roger

For now, you may want to checkout rdiff-backup.  It uses rsync like
functionality to find deltas in your files and then it only backs up
the deltas.

IIRC, it actually keeps a current copy of your file, plus a series of
deltas that let you get to older versions.

Greg
-- 
Greg Freemyer
The Norcross Group
Forensics for the 21st Century