Re: bdar: efficiently backup allocated bytes in file systems

[Andreas Dilger <adilger@Sun.COM>]

On Mar 17, 2008  18:13 -0700, Zach Brown wrote:
> So, I had a fun time throwing together a utility last weekend.  I
> thought I'd share it sooner rather than later.
> 
> I found myself wanting to backup a copy of an ancient ~75g ext3 file
> system.  I got frustrated by of our utilities which don't saturate
> storage.  I wanted dd line rates but I also only wanted to copy
> referenced data.
> 
> So I threw something together which does that.  I made it work roughly
> like tar so that people have some idea what to expect.  So you can do
> something like:
> 
>  $ bdar -cf - /dev/sda3 | gzip -c > /tmp/sda3-backup.bdar.gz
> ...
>  $ zcat /tmp/sda3-backup.bdar.gz | bdar -xf - /dev/sda3
> 
> and it will do exactly what you would guess it would do after reading
> those command lines.
> 
> The bdar file format is just a header and then a series of regions of
> bytes described by their length and offset.  To create a bdar file from
> a file system bdar needs to know enough to figure out what extents are
> referenced.  Restoring a bdar is generic, though, it just stamps bytes
> into the target file.

So the question is whether the ".bdar" file is specific to the filesystem
being backed up, and if it only allows backing up the whole filesystem?
Does it create a dense output file or a sparse one?  Does it store the
data as chunks of blocks in a full-device map or on a per file basis?

If you can't restore a .bdar backup file to a smaller device than the
source device that makes it less useful than most of the other tools.

> I only taught it the most basic knowledge of ext[234].  Just enough to
> show that generating the bdar is ~4x faster than tar and ~2x faster than
> dump :).  There's still some available disk bandwidth to consume with
> read-ahead, but it's pretty close.  (single spindle, ~5g of kernel
> trees, beefy cpus.)

The question is whether the 2x speed improvement is worth the lack of
portability compared to even dump?

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.