btrfs and backups

btrfs and backups

[James Courtier-Dutton]

Hi,

I have a local btrfs file system with various sub-volumes that have
had snapshots done on them.

Is there some tool like rsync that I could copy all the data and
snapshots to a backup system, but still only use the same amount of
space as the source filesystem.
I see a problem being getting a consistent and steady state during the rsync.
I was thinking that I might be able to do this with LVM snapshots, but
that would require something along these lines:
1) pause the btrfs filesystem into a consistent state that can be
mounted cleanly
2) Do LVM snapshot on it.
3) un-pause btrfs filesystem.

I can then do a block level backup of the LVM snapshot and it should
be mountable on the backup server.
So, the snapshot is not a snapshot of the current filesystem, it is a
snapshot of all the snapshots and all the sub-volumes at a particular
time, that is in a stable state to be backed up.

I don't know if 1 is supported?
I suppose I am hoping for 1,2,3 to already be supported by some
special btrfs command.


Any ideas?

Kind Regards

James

Re: btrfs and backups

[Felix Blanke]

On 3/26/12 10:30 AM, James Courtier-Dutton wrote:
> Hi,
>
> I have a local btrfs file system with various sub-volumes that have
> had snapshots done on them.
>
> Is there some tool like rsync that I could copy all the data and
> snapshots to a backup system, but still only use the same amount of
> space as the source filesystem.
> I see a problem being getting a consistent and steady state during the rsync.
> I was thinking that I might be able to do this with LVM snapshots, but
> that would require something along these lines:
> 1) pause the btrfs filesystem into a consistent state that can be
> mounted cleanly
> 2) Do LVM snapshot on it.
> 3) un-pause btrfs filesystem.
>
> I can then do a block level backup of the LVM snapshot and it should
> be mountable on the backup server.
> So, the snapshot is not a snapshot of the current filesystem, it is a
> snapshot of all the snapshots and all the sub-volumes at a particular
> time, that is in a stable state to be backed up.
>
> I don't know if 1 is supported?
> I suppose I am hoping for 1,2,3 to already be supported by some
> special btrfs command.
>
>
> Any ideas?


I'm not sure if I understand your problem right, but I would suggest:

1) Snapshot the subvolume on the source
2) rsync the snapshot to the destination
3) Snapshot the destination

Kind regards,
Felix

>
> Kind Regards
>
> James
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs and backups

[Fajar A. Nugraha]

On Mon, Mar 26, 2012 at 3:56 PM, Felix Blanke <felixblanke@gmail.com> wrote:
> On 3/26/12 10:30 AM, James Courtier-Dutton wrote:
>> Is there some tool like rsync that I could copy all the data and
>> snapshots to a backup system, but still only use the same amount of
>> space as the source filesystem.


> I'm not sure if I understand your problem right, but I would suggest:
>
> 1) Snapshot the subvolume on the source
> 2) rsync the snapshot to the destination
> 3) Snapshot the destination

James did say "only use the same amount of space as the source
filesystem." Your approach would increase the usage when one or more
subvolume shares the same space (e.g. when one subvolume starts as
snapshot).

AFAIK the (planned) way to do this is using "btrfs send | receive",
which is not available yet.

-- 
Fajar

Re: btrfs and backups

[Duncan]

Fajar A. Nugraha posted on Mon, 26 Mar 2012 16:01:54 +0700 as excerpted:

> On Mon, Mar 26, 2012 at 3:56 PM, Felix Blanke <felixblanke@gmail.com>
> wrote:
>> On 3/26/12 10:30 AM, James Courtier-Dutton wrote:
>>> Is there some tool like rsync that I could copy all the data and
>>> snapshots to a backup system, but still only use the same amount of
>>> space as the source filesystem.
> 
> 
>> I'm not sure if I understand your problem right, but I would suggest:
>>
>> 1) Snapshot the subvolume on the source 2) rsync the snapshot to the
>> destination 3) Snapshot the destination
> 
> James did say "only use the same amount of space as the source
> filesystem." Your approach would increase the usage when one or more
> subvolume shares the same space (e.g. when one subvolume starts as
> snapshot).
> 
> AFAIK the (planned) way to do this is using "btrfs send | receive",
> which is not available yet.

What about rsyncing the snapshots, one at a time, snapshotting on the 
destination after each one?  I think that's what Felix' idea was.

On the source filesystem, if each of the snapshots is mounted in turn and 
rsynced across, then rsync should only catch the differences between the 
previous and currently rsyncing snapshots.

On the destination, after the first rsync, a snapshot could be taken and 
mounted, so the second rsync is cumulative.  Then a second snapshot can 
be taken, then it mounted, for the next rsync.  Given COW, I'd think 
that'd work.

That's in contrast to an attempted rsync of the root filesystem, which 
would appear to rsync as if each snapshot was a separate directory tree, 
which would indeed kill the data sharing between them, thus taking up N 
times the space of one snapshot on the destination.

But if each snapshot is mounted in turn on both sides, destination of 
course trailing source by one snapshot, in theory at least, it should 
work, tho it depends on the rsync implementation being COW-friendly and 
I'm not positive it is but expect that it should be.

Here, I'd probably do it manually the first few snapshot generations, 
checking usage on the destination to see that it was working as intended 
as I went and ensuring I had the process down, then script parts of it, 
automating the parts, before ultimately combining the scripts into a full 
automation that, depending on the intent, could ideally be run from a cron 
job or the like.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: btrfs and backups

[Alexander Block]

On Mon, Mar 26, 2012 at 4:26 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> Fajar A. Nugraha posted on Mon, 26 Mar 2012 16:01:54 +0700 as excerpt=
ed:
>
>> On Mon, Mar 26, 2012 at 3:56 PM, Felix Blanke <felixblanke@gmail.com=
>
>> wrote:
>>> On 3/26/12 10:30 AM, James Courtier-Dutton wrote:
>>>> Is there some tool like rsync that I could copy all the data and
>>>> snapshots to a backup system, but still only use the same amount o=
f
>>>> space as the source filesystem.
>>
>>
>>> I'm not sure if I understand your problem right, but I would sugges=
t:
>>>
>>> 1) Snapshot the subvolume on the source 2) rsync the snapshot to th=
e
>>> destination 3) Snapshot the destination
>>
>> James did say "only use the same amount of space as the source
>> filesystem." Your approach would increase the usage when one or more
>> subvolume shares the same space (e.g. when one subvolume starts as
>> snapshot).
>>
>> AFAIK the (planned) way to do this is using "btrfs send | receive",
>> which is not available yet.
>
> What about rsyncing the snapshots, one at a time, snapshotting on the
> destination after each one? =A0I think that's what Felix' idea was.
>
> On the source filesystem, if each of the snapshots is mounted in turn=
 and
> rsynced across, then rsync should only catch the differences between =
the
> previous and currently rsyncing snapshots.
>
> On the destination, after the first rsync, a snapshot could be taken =
and
> mounted, so the second rsync is cumulative. =A0Then a second snapshot=
 can
> be taken, then it mounted, for the next rsync. =A0Given COW, I'd thin=
k
> that'd work.
>
> That's in contrast to an attempted rsync of the root filesystem, whic=
h
> would appear to rsync as if each snapshot was a separate directory tr=
ee,
> which would indeed kill the data sharing between them, thus taking up=
 N
> times the space of one snapshot on the destination.
>
> But if each snapshot is mounted in turn on both sides, destination of
> course trailing source by one snapshot, in theory at least, it should
> work, tho it depends on the rsync implementation being COW-friendly a=
nd
> I'm not positive it is but expect that it should be.
>
> Here, I'd probably do it manually the first few snapshot generations,
> checking usage on the destination to see that it was working as inten=
ded
> as I went and ensuring I had the process down, then script parts of i=
t,
> automating the parts, before ultimately combining the scripts into a =
full
> automation that, depending on the intent, could ideally be run from a=
 cron
> job or the like.
>
> --
> Duncan - List replies preferred. =A0 No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master." =A0Richard Stallman
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs=
" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at =A0http://vger.kernel.org/majordomo-info.html

I used this style of incremental backups for some time and it works wel=
l
if you ignore the low performance when backing up very large files
(e.g. VM images).
I'm not 100% sure about this, but I think it's important to use the --i=
nplace
option of rsync if you want to preserve COW. If not used with --inplace=
,
rsync will create a new file every and after syncing, replace the orifi=
nal. This
will prevent COW at the moment.
Maybe this won't be required in the future when something like auto
deduplication is implemented, but I currently don't know about the plan=
s
for this feature.
however,
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html