incremental full file backups to smaller mediums possible?

incremental full file backups to smaller mediums possible?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 2702 bytes --]

Hey.

I wondered whether this is possible in btrfs (or could be
implemented),... it's in a way similar to send/receive, but AFAIU not
fully solvable with that.

What I want to do is making incremental backups of a (btrfs) filesystem
to smaller mediums (that is for example: from a big RAID filesystem to
many BluRays or similar things).


Right now I make this as follows:
Every time backups should be made, I create a ext4 image which would
just fit inside the UDF fs of a BluRay (the ext4 image in turn is
dm-crypt encrypted) and copy as many *complete*[0] files from the source
filesystem to that image as possible. That image file is then burned to
BluRay. I also write an SHA512 sum of the whole image file to the BluRay
so that one can check whether it can be still read correctly.
Then the process continues with the remaining files.


The main idea behind writing only complete files to the separate parts
of the backup is, that I don't want to have a backup set, where I may
get into troubles if just one of the disks fails.


Obviously, when I would do regularly backups that way, then I'd write
each time a big amount of data which hasn't changed at all, which is not
just a costly issue, but also quite time consuming.
So ideally, I'd want to do just an incremental backup of all the files
that have been added/modified + the information of what has been
deleted/moved/changed it's properties.

All that already smells quite a lot after making a snapshot at the last
backup, and then doing send/receive for the next one.


Now the problem, though, is, that I don't want to loose the property of
working in terms of complete files (and I guess send/recieve does not),
i.e. when a big file has just one block changed, I still would like to
have the complete file on the next incremental dump (and not just
information about the "diff").
The only exception is when file metadata changes (i.e. pathname, date,
permissions, xattrs, etc.). If *just* one of these changes I wouldn't
want to backup the full raw data of the file again.

Another issue is, that send gives me one big file, while I would need
however n files of max size m, where m is e.g. what fits in a BluRay's
UDF... or perhaps on a 3,5" HD Floppy ;)
Because of the "complete files" constraint I cannot just split the file
from btrfs send,... because the split would need to happen so that no
files are split up.


Any ideas whether and how this could be done / assisted by btrfs?


Thanks,
Chris.

[0] Obviously there is a problem, when a single file would be larger
than a BluRay, but so far this hasn't happened to me. If so, I'd of
course need to split the file.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5313 bytes --]

Re: incremental full file backups to smaller mediums possible?

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2717 bytes --]

On Thu, Apr 09, 2015 at 06:14:33PM +0200, Christoph Anton Mitterer wrote:
> Hey.
> 
> I wondered whether this is possible in btrfs (or could be
> implemented),... it's in a way similar to send/receive, but AFAIU not
> fully solvable with that.
> 
> What I want to do is making incremental backups of a (btrfs) filesystem
> to smaller mediums (that is for example: from a big RAID filesystem to
> many BluRays or similar things).
> 
> 
> Right now I make this as follows:
> Every time backups should be made, I create a ext4 image which would
> just fit inside the UDF fs of a BluRay (the ext4 image in turn is
> dm-crypt encrypted) and copy as many *complete*[0] files from the source
> filesystem to that image as possible. That image file is then burned to
> BluRay. I also write an SHA512 sum of the whole image file to the BluRay
> so that one can check whether it can be still read correctly.
> Then the process continues with the remaining files.
> 
> 
> The main idea behind writing only complete files to the separate parts
> of the backup is, that I don't want to have a backup set, where I may
> get into troubles if just one of the disks fails.
> 
> 
> Obviously, when I would do regularly backups that way, then I'd write
> each time a big amount of data which hasn't changed at all, which is not
> just a costly issue, but also quite time consuming.
> So ideally, I'd want to do just an incremental backup of all the files
> that have been added/modified + the information of what has been
> deleted/moved/changed it's properties.
> 
> All that already smells quite a lot after making a snapshot at the last
> backup, and then doing send/receive for the next one.
> 
> 
> Now the problem, though, is, that I don't want to loose the property of
> working in terms of complete files (and I guess send/recieve does not),
> i.e. when a big file has just one block changed, I still would like to
> have the complete file on the next incremental dump (and not just
> information about the "diff").
> The only exception is when file metadata changes (i.e. pathname, date,
> permissions, xattrs, etc.). If *just* one of these changes I wouldn't
> want to backup the full raw data of the file again.

> Any ideas whether and how this could be done / assisted by btrfs?

   btrfs sub find-new might be more helpful to you here. That will
give you the list of changed files; then just feed that list to your
existing bin-packing algorithm for working out what goes on which
disks, and you're done.

   Hugo.

-- 
Hugo Mills             | Dullest spy film ever: The Eastbourne Ultimatum
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |                                       The Thick of It

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: incremental full file backups to smaller mediums possible?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 658 bytes --]

On Thu, 2015-04-09 at 16:33 +0000, Hugo Mills wrote: 
>    btrfs sub find-new might be more helpful to you here. That will
> give you the list of changed files; then just feed that list to your
> existing bin-packing algorithm for working out what goes on which
> disks, and you're done.

hmm that sounds interesting, but
a) it lacks much documentation (how does that with the generations work?
when is there a new generation? what, if the generation overflows?)
b) it seems to only show me new files, but not files that have been
moved, or removed, or when properties have changed.
c) it seems to work only for one subvolume?

Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5313 bytes --]

Re: incremental full file backups to smaller mediums possible?

[Martin Steigerwald]

[-- Attachment #1: Type: text/plain, Size: 1320 bytes --]

Am Samstag, 18. April 2015, 01:03:23 schrieb Christoph Anton Mitterer:
> On Thu, 2015-04-09 at 16:33 +0000, Hugo Mills wrote:
> >    btrfs sub find-new might be more helpful to you here. That will
> > 
> > give you the list of changed files; then just feed that list to your
> > existing bin-packing algorithm for working out what goes on which
> > disks, and you're done.
> 
> hmm that sounds interesting, but
> a) it lacks much documentation (how does that with the generations work?
> when is there a new generation? what, if the generation overflows?)

I have no idea about when there is a new generation. I think it depends on 
the commit intervall.

You can see the current generation with:

merkaba:~> btrfs subvol show / | grep Generation
        Generation (Gen):       468143

> b) it seems to only show me new files, but not files that have been moved,
> or removed, or when properties have changed.

It should show renamed or moved files. But yeah, if it does, at least not 
in a way that shows that a rename or move operation took place.

> c) it seems to work only for one subvolume?

Its per subvolume, I think. Thats why it is under "btrfs subvolume"

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 181 bytes --]