Are there plans for full btrfs filesystem backups?

Are there plans for full btrfs filesystem backups?

[Marc MERLIN]

I use send/receive to backup some subvolumes, but there is no good way
to back up all of my subvolumes in one swoop.
Backing up with tar/cp/rsync is of course no good since it loses all the
btrfs attributes including COW deduped blocks.

For one, my recent btrfs filesystem crash reminded me that docker
creates a boatload of subvolumes which my backups didn't cacch, so now
it's all gone.

Are there any plans for a full filesystem send/receive, not just a
subvolume send/receive?

Note that this is the same problem when you are migtrating to new drives
or bcache, and you can't play tricks with adding a virtual drive of 8TB
just to mirror to it and remove the original mirror later.

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Are there plans for full btrfs filesystem backups?

[Duncan]

Marc MERLIN posted on Sun, 10 May 2015 09:49:44 -0700 as excerpted:

> Are there any plans for a full filesystem send/receive, not just a
> subvolume send/receive?

Recent discussion suggests yes, tho it's a relatively new plan.

More specifically, the recent discussion seemed to settle around the need 
for a send/receive --recursive option, which would recurse into nested 
subvolumes.  IIRC, the systemd folks requested that for their container 
management, among other things.  And if I remember the thread correctly, 
I /believe/ one of the devs said they agreed and were taking on the 
project.

That would of course allow recursing from ID5/the-root-subvolume, thus 
filling your request, but it would be more flexible than full filesystem 
send/receive, as it could handle arbitrary subvolume subtrees, too.

Previous to that, I don't believe it was on the roadmap.  It reads like a 
bunch of people, now including you, independently becoming aware that it 
was a problem at roughly the same time.


Meanwhile, there's another potential alternative as well, for the whole 
filesystem non-incremental case.  If you're backing up everything, you 
can of course dd/netcat the individual component devices making literal 
byte for byte copies, tho you'd want to do it to a remote machine, to 
avoid the kernel detecting a local copy as part of the same filesystem, 
due to UUID duplication.  But incrementals, parallel to send/receive with 
parent, would seem to be impractical.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Are there plans for full btrfs filesystem backups?

[Chris Murphy]

On Sun, May 10, 2015 at 9:55 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> Marc MERLIN posted on Sun, 10 May 2015 09:49:44 -0700 as excerpted:
>
>> Are there any plans for a full filesystem send/receive, not just a
>> subvolume send/receive?
>
> Recent discussion suggests yes, tho it's a relatively new plan.
>
> More specifically, the recent discussion seemed to settle around the need
> for a send/receive --recursive option, which would recurse into nested
> subvolumes.  IIRC, the systemd folks requested that for their container
> management, among other things.

That was for snapshot creation and destruction, not send receive.

To me, an exact Btrfs clone suggests only a Btrfs destination at the
moment, because there's no superset functionality elsewhere for
everything in Btrfs to exist. One way to get to an "exact clone" would
be Btrfs seed device to effectively freeze the original, and then
"send" the entire seed somehow rather than just a subvolume.

A btrfs send file on non-Btrfs destinations, is a maybe, the problem
there is no checksums exist in the send file and maybe some other
things (?) but presumably that could be extended so that the file
could be a complete representation of the file system warts and all -
which itself may not be desirable so it's like,  how exactly do we
really want these things to be? Hmm?



-- 
Chris Murphy

Re: Are there plans for full btrfs filesystem backups?

[Duncan]

Chris Murphy posted on Sun, 10 May 2015 22:57:24 -0600 as excerpted:

> On Sun, May 10, 2015 at 9:55 PM, Duncan <1i5t5.duncan@cox.net> wrote:
>> Marc MERLIN posted on Sun, 10 May 2015 09:49:44 -0700 as excerpted:
>>
>>> Are there any plans for a full filesystem send/receive, not just a
>>> subvolume send/receive?
>>
>> Recent discussion suggests yes, tho it's a relatively new plan.
>>
>> More specifically, the recent discussion seemed to settle around the
>> need for a send/receive --recursive option, which would recurse into
>> nested subvolumes.  IIRC, the systemd folks requested that for their
>> container management, among other things.
> 
> That was for snapshot creation and destruction, not send receive.

I believe that was it, yes.

But of course recursive snapshot creation/deletion would be a practical 
prerequisite to recursive send/receive, since in that case read-only 
snapshots would need to be created in ordered to be sent, and similarly 
created on the receive side to recreate the nested snapshot tree on the 
send side.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Are there plans for full btrfs filesystem backups?

[csirac2]

Full-filesystem backups also beg the question of full-filesystem snapshots; this level of atomicity may be desirable for such a thing.

Perhaps this off topic (and I am sorry to bring it up again :D), but normal snazzer [1] usage operates on all subvolumes on all mounted btrfs filesystems, although I should revisit docker assumptions (I normally exclude them). snazzer-receive man page is at [2]. Still working on coping with manually mounted subvols [3].

[1] https://github.com/csirac2/snazzer
[2] https://github.com/csirac2/snazzer/blob/master/doc/snazzer-receive.md
[3] https://github.com/csirac2/snazzer/issues/2



Sent from my android device.

-----Original Message-----
From: Paul Harvey <csirac2@gmail.com>
To: Marc MERLIN <marc@merlins.org>
Cc: linux-btrfs <linux-btrfs@vger.kernel.org>
Sent: Mon, 11 May 2015 8:13
Subject: Re: Are there plans for full btrfs filesystem backups?

Full-filesystem backups also beg the question of full-filesystem snapshots;
this level of atomicity may be desirable for such a thing.

Perhaps this off topic (and I am sorry to bring it up again :D), but normal
snazzer [1] usage operates on all subvolumes on all mounted btrfs
filesystems, although I should revisit docker assumptions (I normally
exclude them). snazzer-receive man page is at [2]. Still working on coping
with manually mounted subvols [3].

[1] https://github.com/csirac2/snazzer
[2] https://github.com/csirac2/snazzer/blob/master/doc/snazzer-receive.md
[3] https://github.com/csirac2/snazzer/issues/2
On 11 May 2015 02:56, "Marc MERLIN" <marc@merlins.org> wrote:

> I use send/receive to backup some subvolumes, but there is no good way
> to back up all of my subvolumes in one swoop.
> Backing up with tar/cp/rsync is of course no good since it loses all the
> btrfs attributes including COW deduped blocks.
>
> For one, my recent btrfs filesystem crash reminded me that docker
> creates a boatload of subvolumes which my backups didn't cacch, so now
> it's all gone.
>
> Are there any plans for a full filesystem send/receive, not just a
> subvolume send/receive?
>
> Note that this is the same problem when you are migtrating to new drives
> or bcache, and you can't play tricks with adding a virtual drive of 8TB
> just to mirror to it and remove the original mirror later.
>
> Thanks,
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" -
> A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet
> cooking
> Home page: http://marc.merlins.org/                         | PGP
> 1024R/763BE901
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>