Copying related snapshots to another server with btrfs send/receive?

Copying related snapshots to another server with btrfs send/receive?

[Marc MERLIN]

Another question I just came up with.

If I have historical snapshots like so:
backup
backup.sav1
backup.sav2
backup.sav3

If I want to copy them up to another server, can btrfs send/receive
let me copy all of the to another btrfs pool while keeping the
duplicated block relationship between all of them?
Note that the backup.sav dirs will never change, so I won't need
incremental backups on those, just a one time send.
I believe this is supposed to work, correct?

The only part I'm not clear about is am I supposed to copy them all at
once in the same send command, or one by one?

If they had to be copied together and if I create a new snapshot of
backup: backup.sav4

If I use btrfs send to that same destination, is btrfs send/receive indeed 
able to keep the shared block relationship?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Copying related snapshots to another server with btrfs send/receive?

[Brendan Hide]

On 2014/05/04 05:12 AM, Marc MERLIN wrote:
> Another question I just came up with.
>
> If I have historical snapshots like so:
> backup
> backup.sav1
> backup.sav2
> backup.sav3
>
> If I want to copy them up to another server, can btrfs send/receive
> let me copy all of the to another btrfs pool while keeping the
> duplicated block relationship between all of them?
> Note that the backup.sav dirs will never change, so I won't need
> incremental backups on those, just a one time send.
> I believe this is supposed to work, correct?
>
> The only part I'm not clear about is am I supposed to copy them all at
> once in the same send command, or one by one?
>
> If they had to be copied together and if I create a new snapshot of
> backup: backup.sav4
>
> If I use btrfs send to that same destination, is btrfs send/receive indeed
> able to keep the shared block relationship?
>
> Thanks,
> Marc
I'm not sure if they can be sent in one go. :-/

Sending one-at-a-time, the shared-data relationship will be kept by 
using the -p (parent) parameter. Send will only send the differences and 
receive will create a new snapshot, adjusting for those differences, 
even when the receive is run on a remote server.

$ btrfs send backup | btrfs receive $path/
$ btrfs send -p backup backup.sav1 | btrfs receive $path/
$ btrfs send -p backup.sav1 backup.sav2 | btrfs receive $path/
$ btrfs send -p backup.sav2 backup.sav3 | btrfs receive $path/
$ btrfs send -p backup.sav3 backup.sav4 | btrfs receive $path/

-- 
__________
Brendan Hide
http://swiftspirit.co.za/
http://www.webafrica.co.za/?AFF1E97

Re: Copying related snapshots to another server with btrfs send/receive?

[Marc MERLIN]

On Sun, May 04, 2014 at 09:16:02AM +0200, Brendan Hide wrote:
> Sending one-at-a-time, the shared-data relationship will be kept by
> using the -p (parent) parameter. Send will only send the differences
> and receive will create a new snapshot, adjusting for those
> differences, even when the receive is run on a remote server.
> 
> $ btrfs send backup | btrfs receive $path/
> $ btrfs send -p backup backup.sav1 | btrfs receive $path/
> $ btrfs send -p backup.sav1 backup.sav2 | btrfs receive $path/
> $ btrfs send -p backup.sav2 backup.sav3 | btrfs receive $path/
> $ btrfs send -p backup.sav3 backup.sav4 | btrfs receive $path/

So this is exactly the same than what I do incremental backups with
brrfs send, but -p only works if the snapshot is read only, does it not?
I do use that for my incremental syncs and don't mind read only
snapshots there, but if I have read/write snapshots that are there for
other reasons than btrfs send incrementals, can I still send them that
way with -p?
(I thought that wouldn't work)

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Copying related snapshots to another server with btrfs send/receive?

[Brendan Hide]

On 2014/05/04 09:28 AM, Marc MERLIN wrote:
> On Sun, May 04, 2014 at 09:16:02AM +0200, Brendan Hide wrote:
>> Sending one-at-a-time, the shared-data relationship will be kept by
>> using the -p (parent) parameter. Send will only send the differences
>> and receive will create a new snapshot, adjusting for those
>> differences, even when the receive is run on a remote server.
>>
>> $ btrfs send backup | btrfs receive $path/
>> $ btrfs send -p backup backup.sav1 | btrfs receive $path/
>> $ btrfs send -p backup.sav1 backup.sav2 | btrfs receive $path/
>> $ btrfs send -p backup.sav2 backup.sav3 | btrfs receive $path/
>> $ btrfs send -p backup.sav3 backup.sav4 | btrfs receive $path/
> So this is exactly the same than what I do incremental backups with
> brrfs send, but -p only works if the snapshot is read only, does it not?
> I do use that for my incremental syncs and don't mind read only
> snapshots there, but if I have read/write snapshots that are there for
> other reasons than btrfs send incrementals, can I still send them that
> way with -p?
> (I thought that wouldn't work)
>
> Thanks,
> Marc
Yes, -p (parent) and -c (clone source) are the only ways I'm aware of to 
push subvolumes across while ensuring data-sharing relationship remains 
intact. This will end up being much the same as doing incremental backups:
 From the man page section on -c:
     "You must not specify clone sources unless you guarantee that these 
snapshots are exactly in the same state on both sides, the sender and 
the receiver. It is allowed to omit the '-p <parent>' option when '-c 
<clone-src>' options are given, in which case 'btrfs send' will 
determine a suitable parent among the clone sources itself."

-p does require that the sources be read-only. I suspect -c does as 
well. This means that it won't be so simple as you want your sources to 
be read-write. Probably the only way then would be to make read-only 
snapshots whenever you want to sync these over while also ensuring that 
you keep at least one read-only snapshot intact - again, much like 
incremental backups.

-- 
__________
Brendan Hide
http://swiftspirit.co.za/
http://www.webafrica.co.za/?AFF1E97

Re: Copying related snapshots to another server with btrfs send/receive?

[Marc MERLIN]

On Sun, May 04, 2014 at 09:54:38AM +0200, Brendan Hide wrote:
> Yes, -p (parent) and -c (clone source) are the only ways I'm aware
> of to push subvolumes across while ensuring data-sharing
> relationship remains intact. This will end up being much the same as
> doing incremental backups:
> From the man page section on -c:
>     "You must not specify clone sources unless you guarantee that
> these snapshots are exactly in the same state on both sides, the
> sender and the receiver. It is allowed to omit the '-p <parent>'
> option when '-c <clone-src>' options are given, in which case 'btrfs
> send' will determine a suitable parent among the clone sources
> itself."

Right. I had read that, but it was not super clear to me how it can be
useful, especially if it's supposed to find the source clone by itself.
>From what you said and what I read, I think the source might be allowed
to be read write, otherwise it would be simpler for btrfs send to know
that the source has not changed.

I think I'll have to do more testing with this when I get some time.

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Copying related snapshots to another server with btrfs send/receive?

[Duncan]

Brendan Hide posted on Sun, 04 May 2014 09:54:38 +0200 as excerpted:

>  From the man page section on -c:
>      "You must not specify clone sources unless you guarantee that these
> snapshots are exactly in the same state on both sides, the sender and
> the receiver. It is allowed to omit the '-p <parent>' option when '-c
> <clone-src>' options are given, in which case 'btrfs send' will
> determine a suitable parent among the clone sources itself."
> 
> -p does require that the sources be read-only. I suspect -c does as
> well. This means that it won't be so simple as you want your sources to
> be read-write. Probably the only way then would be to make read-only
> snapshots whenever you want to sync these over while also ensuring that
> you keep at least one read-only snapshot intact - again, much like
> incremental backups.

I don't claim in any way to be a send/receive expert as I don't use it 
for my use-case at all.  However...

It's worth noting in the context of that manpage quote, that really the 
only practical way to guarantee that the snapshots are exactly the same 
on both sides is to have them read-only the entire time.  Because the 
moment you make them writable on either side all bets are off as to 
whether something has been written, thereby killing the exact-same-state 
guarantee. =:^(

*However*: snapshotting a read-only snapshot and making the new one 
writable is easy enough[1].  Just keep the originals read-only so they 
can be used as parents/clones, and make a second, writable snapshot of 
the first, to do your writable stuff in.

---
[1]  Snapshotting a snapshot: I'm getting a metaphorical flashing light 
saying I need to go check the wiki FAQ that deals with this again before 
I post, but unfortunately I can't check out why ATM as I just upgraded 
firefox and cairo and am currently getting a blank window where the 
firefox content should be, that will hopefully be gone and the content 
displayed after I reboot and get rid of the still loaded old libs, so 
unfortunately I can't check that flashing light ATM and am writing 
blind.  Hopefully that flashing light warning isn't for something /too/ 
major that I'm overlooking!  =:^(

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Copying related snapshots to another server with btrfs send/receive?

[Marc MERLIN]

On Mon, May 05, 2014 at 03:24:45AM +0000, Duncan wrote:
> *However*: snapshotting a read-only snapshot and making the new one 
> writable is easy enough[1].  Just keep the originals read-only so they 
> can be used as parents/clones, and make a second, writable snapshot of 
> the first, to do your writable stuff in.
> 
> ---
> [1]  Snapshotting a snapshot: I'm getting a metaphorical flashing light 

I already snapshot ro snapshots as rw snapshots and that works fine.
It's actually rely on this in my script:
http://marc.merlins.org/perso/btrfs/post_2014-03-22_Btrfs-Tips_-Doing-Fast-Incremental-Backups-With-Btrfs-Send-and-Receive.html
(skip to the bottom)
# We make a read-write snapshot in case you want to use it for a chroot
# and some testing with a writeable filesystem or want to boot from a
# last good known snapshot.
btrfs subvolume snapshot "$src_newsnap" "$src_newsnaprw"
$ssh btrfs subvolume snapshot "$dest_pool/$src_newsnap" "$dest_pool/$src_newsnaprw"

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901