[linux-lvm] Block-Level Backup

[linux-lvm] Block-Level Backup

[Rocky Lee]

>What exactly shall this be good for?
>I immediately thought about ISO files: CD-ROM images. You can do the
>very same to any block device:
>dd if=/dev/hda1 of=/tmp/disk.bak
>Of course, the drive mounted ot /tmp has to be larger than /dev/hda1, in
>this case... and it's filesystem should probably support files > 2GB.
>Compression might help, too.
>Not sure if that's whah you're looking for...
>Markus

Thank you Markus!

I forget "dd", it really is a good way to do a simple block-level backup, :)

but it'll be great if there's a tool,
that supoorts block-level increment backup....

Rocky

Re: [linux-lvm] Block-Level Backup

[Frank Van Damme]

On Thursday 07 August 2003 11:14, Rocky Lee wrote:

> Thank you Markus!
>
> I forget "dd", it really is a good way to do a simple block-level backup,
> :)
>
> but it'll be great if there's a tool,
> that supoorts block-level increment backup....
>
> Rocky

Maybe partimage is useful for you too?

-- 
Frank Van Damme    http://www.openstandaarden.be 
~~~~~~~~~~~~~~~    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Je pense, donc je suis breveté."

Re: [linux-lvm] Block-Level Backup

[Jose Luis Domingo Lopez]

On Thursday, 07 August 2003, at 17:14:30 +0800,
Rocky Lee wrote:

> but it'll be great if there's a tool,
> that supoorts block-level increment backup....
> 
Well, seems you are looking for something like "rsync", but at block
level, not filesystem level. If you could convince "rsync" to consider a
block device as if it where a (huge) file directory, maybe it could work.

However "rsync" insists on treating special (block device) files as
such, an refuses to open them like they where files from which to read.

Regards,

-- 
Jose Luis Domingo Lopez
Linux Registered User #189436     Debian Linux Sid (Linux 2.6.0-test2-mm2)

Re: [linux-lvm] Block-Level Backup

[Joe Thornber]

On Thu, Aug 07, 2003 at 05:14:30PM +0800, Rocky Lee wrote:
> >What exactly shall this be good for?
> >I immediately thought about ISO files: CD-ROM images. You can do the
> >very same to any block device:
> >dd if=/dev/hda1 of=/tmp/disk.bak
> >Of course, the drive mounted ot /tmp has to be larger than /dev/hda1, in
> >this case... and it's filesystem should probably support files > 2GB.
> >Compression might help, too.
> >Not sure if that's whah you're looking for...
> >Markus
> 
> Thank you Markus!
> 
> I forget "dd", it really is a good way to do a simple block-level backup, :)
> 
> but it'll be great if there's a tool,
> that supoorts block-level increment backup....

Icremental backup would be good, from the kernel side all you'd need
is a dm target that records which blocks have changed - anyone
interested in coding this ?

- Joe

Re: [linux-lvm] Block-Level Backup

[Christophe Saout]

Am Do, 2003-08-07 um 11.36 schrieb Joe Thornber:

> > but it'll be great if there's a tool,
> > that supoorts block-level increment backup....
> 
> Icremental backup would be good, from the kernel side all you'd need
> is a dm target that records which blocks have changed - anyone
> interested in coding this ?

I'm still here. ;-)

You already wrote something like this. I was thinking about it. Perhaps
we could do some brainstorming how to exactly do this, not from the
programming side, I could find that out myself, but from the "output"
side.

Which options, what data format, what gets logged and where.

--
Christophe Saout <christophe@saout.de>
Please avoid sending me Word or PowerPoint attachments.
See http://www.fsf.org/philosophy/no-word-attachments.html

Re: [linux-lvm] Block-Level Backup

[Herbert Pötzl]

On Thu, Aug 07, 2003 at 02:47:46PM +0200, Christophe Saout wrote:
> Am Do, 2003-08-07 um 11.36 schrieb Joe Thornber:
> 
> > > but it'll be great if there's a tool,
> > > that supoorts block-level increment backup....
> > 
> > Icremental backup would be good, from the kernel side all you'd need
> > is a dm target that records which blocks have changed - anyone
> > interested in coding this ?
> 
> I'm still here. ;-)
> 
> You already wrote something like this. I was thinking about it. Perhaps
> we could do some brainstorming how to exactly do this, not from the
> programming side, I could find that out myself, but from the "output"
> side.
> 
> Which options, what data format, what gets logged and where.

IMHO dm isn't required and not very useful, if for
example you want to do something like incremental
backups, you'll probably end up doing towers of
hanoi with changing intervals.

anyway, as pointed out in another response, there is
no good way, at least none I know of, to find the
blocks changed since the last backup ...

 - one could make checksums over blocks
   store them, and compare them later ...
 - there could be a blockdevice change journal
   extension, where block accesses are stored
   with time/date (100GB storage with 4k blocks
   would still require ~100MB change journal)

on the other hand, maybe the 'original' backup
could be used to find the differences, if it
can be accessed somehow (this requires at least
twice the size of your block device ...

this is not an issue for something like block
rsync, because you can easily compare source and
destination (live system and backup) either via
checksums or byte per byte ...

IMHO, compression and encryption are useful features
too, but YMMV

best,
Herbert

Re: [linux-lvm] Block-Level Backup

[Bill Rugolsky Jr.]

On Thu, Aug 07, 2003 at 03:44:51PM +0200, Herbert Pötzl wrote:
> anyway, as pointed out in another response, there is
> no good way, at least none I know of, to find the
> blocks changed since the last backup ...
 
The DM snapshot target already keeps track of which blocks have
been written.  For incremental backup, we really only need
to know the block numbers.  Why not a new block-meta-data-only
snapshot target that just records block numbers?  That
should have *much* lower overhead and consume insignificant
disk space.

For archival storage, I've lately become enamored of hashed-indexing
as used in Plan 9's Venti (for a single server target) or the various
distributed-hash systems (Chord, etc.) for distributed, replicated
storage.

Regards,

	Bill Rugolsky

Re: [linux-lvm] Block-Level Backup

[Joe Thornber]

On Thu, Aug 07, 2003 at 11:11:27AM -0400, Bill Rugolsky Jr. wrote:
> On Thu, Aug 07, 2003 at 03:44:51PM +0200, Herbert P?tzl wrote:
> > anyway, as pointed out in another response, there is
> > no good way, at least none I know of, to find the
> > blocks changed since the last backup ...
>  
> The DM snapshot target already keeps track of which blocks have
> been written.  For incremental backup, we really only need
> to know the block numbers.  Why not a new block-meta-data-only
> snapshot target that just records block numbers?  That
> should have *much* lower overhead and consume insignificant
> disk space.

This scheme only works if you don't use the device while you are
taking an incremental backup.  I think the real solution will be based
around snapshots.

- Joe

Re: [linux-lvm] Block-Level Backup

[Bill Rugolsky Jr.]

On Thu, Aug 07, 2003 at 04:54:21PM +0100, Joe Thornber wrote:
> This scheme only works if you don't use the device while you are
> taking an incremental backup.  I think the real solution will be based
> around snapshots.

Right.  But I think the point is this: one won't mind keeping multiple
(tower of hanoi) long-lived (week or more) block change indices, because
the performance impact should be very low.

When it it time to do an actual backup, we simultaneously create a
new block change index, take a regular snapshot, and freeze (or copy)
the block-change index of interest.  Then we just back up the blocks
from the snapshot volume indicated in the frozen block-change index and
then delete the frozen block-change index and the snapshot volume.

For a typical filesystem, the snapshot volume persists for seconds to
minutes while the data is copied, not the tens of minutes to hours often
required to traverse the whole device with a tool like dump(8).

Regards,

	Bill Rugolsky

Re: [linux-lvm] Block-Level Backup

[Dieter Stueken]

Bill Rugolsky Jr. wrote:
> On Thu, Aug 07, 2003 at 04:54:21PM +0100, Joe Thornber wrote:
> 
>>This scheme only works if you don't use the device while you are
>>taking an incremental backup.  I think the real solution will be based
>>around snapshots.
> 
> 
> Right.  But I think the point is this: one won't mind keeping multiple
> (tower of hanoi) long-lived (week or more) block change indices, because
> the performance impact should be very low.
 >
> When it it time to do an actual backup, we simultaneously create a
> new block change index, take a regular snapshot, and freeze (or copy)
> the block-change index of interest.  Then we just back up the blocks
> from the snapshot volume indicated in the frozen block-change index and
> then delete the frozen block-change index and the snapshot volume.

For me this looks very similar to using rsync, but simultaneously
saving all files that are changed. Instead you optimize towards saving
the changed blocks only. Fine! but have you thought about how to restore
any file again if it was changed multiple times? It sounds a little bit
like my favorite unbeatable compression algorithm: it just counts all
1 and all 0 bits and writes just two numbers to tape :-)

But seriously: there is some other interesting possibility. After you
made the snapshot, don't try do save it elsewhere. Instead just keep it!
I use a similar system since a year now. It holds a full mirror of
all my data, thus its some kind of full backup. Each night I
synchronize all modified files, but keep the previous state, too.
Its like a  snapshot. As all unchanged data is shared between the
snapshots, the whole thing grows quite moderately compared to its
total size (200Gb).

Thus I have a snapshot of all my data for each day. I save them daily
for about a week. Then I thin them out, keeping the Sundays only.
After a few weeks I keep one snapshot per month etc. This is equivalent
to dealing with a bundle of tapes, but much much easier.

Also remember, I have the full filesystem on direct access for each
snapshot. No untaring, no merging of files etc. just "cp -a". Disk
devices become cheaper and larger each day. If my disk capacity is
exhausted, I put the disk offline and by a new one (about once a year).
And you have to compare the costs of the HD to tapes. There is no
big different any more and you save an expensive tape drive, too.

OK, I implemented all this by a completely different approach using
hardlinks. Thus shared data is kept file based. But the idea could
be implemented more effective based on shared data blocks. You need
some big block device, for each snapshot an array of block numbers.
Also an copy-on-write algorithm and probably a reference count for
each block (or some garbage algorithm after deleting a snapshot).
This is very similar to LVM snapshots, but not exactly the same...

Dieter.
-- 
Dieter St�ken, con terra GmbH, M�nster
     stueken@conterra.de
     http://www.conterra.de/
     (0)251-7474-501

Re: [linux-lvm] Block-Level Backup

[Wolfgang Weisselberg]

Dieter Stueken wrote 67 lines:

> But seriously: there is some other interesting possibility. After you
> made the snapshot, don't try do save it elsewhere. Instead just keep it!
> I use a similar system since a year now. It holds a full mirror of
> all my data, thus its some kind of full backup. Each night I
> synchronize all modified files, but keep the previous state, too.
> Its like a  snapshot. As all unchanged data is shared between the
> snapshots, the whole thing grows quite moderately compared to its
> total size (200Gb).

> Thus I have a snapshot of all my data for each day. I save them daily
> for about a week. Then I thin them out, keeping the Sundays only.
> After a few weeks I keep one snapshot per month etc. This is equivalent
> to dealing with a bundle of tapes, but much much easier.

Sounds interesting.  How does defend against:
- hard disk head crashes 
- a lightning striking your PC, roasting your HDs (and any tape
  drive, and all the rest)
- software errors (e.g. in LVM, in your scripts)
- operator errors (removing the wrong snapshot(s))
- cracker attacks
- corrupted main file system (even fdisk giving up), due to
  some software/hardware problem (e.g. cable came half off)

With a backup which migrates/copies the data to tapes, I can at
least:
- replace broken HDs/computers
- know the data was at least readable at the time of the
  backup, and any later accidents are recoverable
- it's much harder to nuke 2+ tapes/backup sets by operator
  error.

-Wolfgang

Re: [linux-lvm] Block-Level Backup

[Mark H. Wood]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, 7 Aug 2003, Rocky Lee wrote:
[snip]
> but it'll be great if there's a tool,
> that supoorts block-level increment backup....

"Incremental backup" usually means "just the stuff that changed since last
time."  Filesystems typically have a last-write date in the metadata for
each file.  Where will you find the last-write date for each block of a
block device?  Or what do you mean by "incremental backup" of a block
device?

What problem are you trying to solve?

- -- 
Mark H. Wood, Lead System Programmer   mwood@IUPUI.Edu
MS Windows *is* user-friendly, but only for certain values of "user".
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)
Comment: pgpenvelope 2.10.2 - http://pgpenvelope.sourceforge.net/

iD8DBQE/MlO2s/NR4JuTKG8RAptlAJsEqXT0BUeIlhlbPXn40RB8UNBGsQCgiJek
iWddJfaYyccPHEmpBnR5FPw=
=x5lK
-----END PGP SIGNATURE-----

[linux-lvm] Block-Level Backup

[Rocky Lee]

Hi all,

How to do block-level Backup with LVM?
"tar" seems like only support file-level backup.
any backup tool recommand?

Have a nice Day.
Rocky

Re: [linux-lvm] Block-Level Backup

[Markus Schiltknecht]

On Thu, 2003-08-07 at 10:02, Rocky Lee wrote:
> Hi all,
> 
> How to do block-level Backup with LVM?
> "tar" seems like only support file-level backup.
> any backup tool recommand?

What exactly shall this be good for?

I immediately thought about ISO files: CD-ROM images. You can do the
very same to any block device:

dd if=/dev/hda1 of=/tmp/disk.bak

Of course, the drive mounted ot /tmp has to be larger than /dev/hda1, in
this case... and it's filesystem should probably support files > 2GB.
Compression might help, too.

Not sure if that's whah you're looking for...

Markus