Send/receive snapshot from/between backup

Send/receive snapshot from/between backup

[René Bühlmann]

Hi all,

I have a system running on btrfs which is backed up to two (one local
USB and one through SSH) external drives (also using btrfs) using
send/receive (with the help of btrbk). For each backup a snapshot is
created and transferred to the external drives. After some time, the
snapshot on the origin is removed but kept on the two backups.

Now my problem is, that the SSH-drive has not received any snapshots for
some time such that there is no snapshot any more which is on the origin
AND the SSH-drive. I would like to prevent transferring a full snapshot
over the network and am now trying to work around it.

Here the current situation (S? for a Snapshot):

Origin: S2 S3

USB: S1 S2

SSH: S1

Transferring S3 to USB is no problem as S2 is on both btrfs drives. But
how can I transfer S3 to SSH?

I tried to transfer...

1. S3 from Origin to SSH -> does not work as there is no common snapshot.

2. S2 from USB to SSH -> did not work.

3. S1 from USB to Origin (such that there is a common snapshot with SSH)
-> did not work.

Is it correct that 1. would work if a common snapshot is present on
Origin and SSH?

Is it expected that 2. and 3. do not work?

Is there some other way to achieve it?

Thanks for the help

René

Re: Send/receive snapshot from/between backup

[Hans van Kranenburg]

On 11/02/2016 03:23 PM, René Bühlmann wrote:
> 
> I have a system running on btrfs which is backed up to two (one local
> USB and one through SSH) external drives (also using btrfs) using
> send/receive (with the help of btrbk). For each backup a snapshot is
> created and transferred to the external drives. After some time, the
> snapshot on the origin is removed but kept on the two backups.
> 
> Now my problem is, that the SSH-drive has not received any snapshots for
> some time such that there is no snapshot any more which is on the origin
> AND the SSH-drive. I would like to prevent transferring a full snapshot
> over the network and am now trying to work around it.
> 
> Here the current situation (S? for a Snapshot):
> 
> Origin: S2 S3
> 
> USB: S1 S2
> 
> SSH: S1
> 
> Transferring S3 to USB is no problem as S2 is on both btrfs drives. But
> how can I transfer S3 to SSH?

Since there is no snapshot left that is present on both drives, you
cannot use incremental send.

btrbk does not remember what snapshots were last transferred or were
present on the missing disk, since the tool does not do any extra meta
administation on top of just having the snapshots on btrfs level. So
it's figuring out the relationships between each of them (combining it
with info from the remotes) again every time it runs.

> I tried to transfer...
> 
> 1. S3 from Origin to SSH -> does not work as there is no common snapshot.
> 
> 2. S2 from USB to SSH -> did not work.
> 
> 3. S1 from USB to Origin (such that there is a common snapshot with SSH)
> -> did not work.
> 
> Is it correct that 1. would work if a common snapshot is present on
> Origin and SSH?
> 
> Is it expected that 2. and 3. do not work?
> 
> Is there some other way to achieve it?

At home, I do a similar thing, I periodically send/receive changes of a
filesystem to two external disks, also using btrbk. Since I don't want
to have both backup disks and the originating filesystem all online and
in the same geographical location at the same moment, I have the same
problem.

What I did is just setting expiry the snapshots on the origin manually,
and keep the meta-administration in my head. I don't do it that often
anyway.

-- 
Hans van Kranenburg

Re: Send/receive snapshot from/between backup

[René Bühlmann]

On 11/02/2016 03:49 PM, Hans van Kranenburg wrote:
> On 11/02/2016 03:23 PM, René Bühlmann wrote:
>> I have a system running on btrfs which is backed up to two (one local
>> USB and one through SSH) external drives (also using btrfs) using
>> send/receive (with the help of btrbk). For each backup a snapshot is
>> created and transferred to the external drives. After some time, the
>> snapshot on the origin is removed but kept on the two backups.
>>
>> Now my problem is, that the SSH-drive has not received any snapshots for
>> some time such that there is no snapshot any more which is on the origin
>> AND the SSH-drive. I would like to prevent transferring a full snapshot
>> over the network and am now trying to work around it.
>>
>> Here the current situation (S? for a Snapshot):
>>
>> Origin: S2 S3
>>
>> USB: S1 S2
>>
>> SSH: S1
>>
>> Transferring S3 to USB is no problem as S2 is on both btrfs drives. But
>> how can I transfer S3 to SSH?
> Since there is no snapshot left that is present on both drives, you
> cannot use incremental send.
>
> btrbk does not remember what snapshots were last transferred or were
> present on the missing disk, since the tool does not do any extra meta
> administation on top of just having the snapshots on btrfs level. So
> it's figuring out the relationships between each of them (combining it
> with info from the remotes) again every time it runs.
Yes this explains why 1. did not work. But why can't I transfer S1 from
USB back to Origin as incremental to S2?
And why can't I transfer S2 from USB to SSH? Is S1 not recognized as the
same snapshot there?
>
>> I tried to transfer...
>>
>> 1. S3 from Origin to SSH -> does not work as there is no common snapshot.
>>
>> 2. S2 from USB to SSH -> did not work.
>>
>> 3. S1 from USB to Origin (such that there is a common snapshot with SSH)
>> -> did not work.
>>
>> Is it correct that 1. would work if a common snapshot is present on
>> Origin and SSH?
>>
>> Is it expected that 2. and 3. do not work?
>>
>> Is there some other way to achieve it?
> At home, I do a similar thing, I periodically send/receive changes of a
> filesystem to two external disks, also using btrbk. Since I don't want
> to have both backup disks and the originating filesystem all online and
> in the same geographical location at the same moment, I have the same
> problem.
>
> What I did is just setting expiry the snapshots on the origin manually,
> and keep the meta-administration in my head. I don't do it that often
> anyway.
>

Re: Send/receive snapshot from/between backup (also restore example)

[Hans van Kranenburg]

On 11/02/2016 04:14 PM, René Bühlmann wrote:
> On 11/02/2016 03:49 PM, Hans van Kranenburg wrote:
>> On 11/02/2016 03:23 PM, René Bühlmann wrote:
>>> I have a system running on btrfs which is backed up to two (one local
>>> USB and one through SSH) external drives (also using btrfs) using
>>> send/receive (with the help of btrbk). For each backup a snapshot is
>>> created and transferred to the external drives. After some time, the
>>> snapshot on the origin is removed but kept on the two backups.
>>>
>>> Now my problem is, that the SSH-drive has not received any snapshots for
>>> some time such that there is no snapshot any more which is on the origin
>>> AND the SSH-drive. I would like to prevent transferring a full snapshot
>>> over the network and am now trying to work around it.
>>>
>>> Here the current situation (S? for a Snapshot):
>>>
>>> Origin: S2 S3
>>>
>>> USB: S1 S2
>>>
>>> SSH: S1
>>>
>>> Transferring S3 to USB is no problem as S2 is on both btrfs drives. But
>>> how can I transfer S3 to SSH?
>> Since there is no snapshot left that is present on both drives, you
>> cannot use incremental send.
>>
>> btrbk does not remember what snapshots were last transferred or were
>> present on the missing disk, since the tool does not do any extra meta
>> administation on top of just having the snapshots on btrfs level. So
>> it's figuring out the relationships between each of them (combining it
>> with info from the remotes) again every time it runs.
> Yes this explains why 1. did not work. But why can't I transfer S1 from
> USB back to Origin as incremental to S2?

You can, but what you end up with is not the same as the original S1 any
more. It will have a new uuid, which is not the received_uuid of the S1
that is still on SSH, so they won't be 'common'.

Allow me to demonstrate... A slightly different use case, but based on
this you can debug your own scenario again in the same way:

The following is how to snapshot to a remote, how to roll back to a
snapshot that is only on the remote (send it back), and then continue
sending incrementals again to the remote again.

Let's create some snapshots, and send them:

/mnt/zooi 12-# mkdir send
/mnt/zooi 12-# mkdir receive
/mnt/zooi 12-# mount /dev/zooi/send send/
/mnt/zooi 12-# mount /dev/zooi/receive receive/
/mnt/zooi 12-# cd send/

/mnt/zooi/send 12-# btrfs sub create A
Create subvolume './A'

/mnt/zooi/send 12-# dd if=/dev/zero of=A/1 bs=1 count=0 seek=1G; shred
-n 1 A/1
/mnt/zooi/send 12-# btrfs sub snap -r A A1
Create a readonly snapshot of 'A' in './A1'
/mnt/zooi/send 12-# btrfs send A1 | btrfs receive ../receive/
At subvol A1
At subvol A1

/mnt/zooi/send 12-# dd if=/dev/zero of=A/2 bs=1 count=0 seek=1G; shred
-n 1 A/2
/mnt/zooi/send 12-# btrfs sub snap -r A A2
Create a readonly snapshot of 'A' in './A2'
/mnt/zooi/send 12-# btrfs send -p A1 A2 | btrfs receive ../receive/
At subvol A2
At snapshot A2

/mnt/zooi/send 12-# dd if=/dev/zero of=A/3 bs=1 count=0 seek=1G; shred
-n 1 A/3
/mnt/zooi/send 12-# btrfs sub snap -r A A3
Create a readonly snapshot of 'A' in './A3'
/mnt/zooi/send 12-# btrfs send -p A2 A3 | btrfs receive ../receive/
At subvol A3
At snapshot A3

Now, the list of subvolumes at the send side looks like (linebreaks added):

/mnt/zooi/send 12-# btrfs sub list -u -q -R .
ID 271
 gen 40
 top level 5
 parent_uuid -
 received_uuid -
 uuid f08112b1-223d-cb46-b54b-2d90c8339617
 path A
ID 274
 gen 39
 top level 5
 parent_uuid f08112b1-223d-cb46-b54b-2d90c8339617
 received_uuid -
 uuid 862f3808-ccdf-cb40-8cf1-c31b8cbea635
 path A1
ID 276
 gen 40
 top level 5
 parent_uuid f08112b1-223d-cb46-b54b-2d90c8339617
 received_uuid -
 uuid ab4bd82b-39f9-a24c-9ead-02023e392e9f
 path A2
ID 278
 gen 41
 top level 5
 parent_uuid f08112b1-223d-cb46-b54b-2d90c8339617
 received_uuid -
 uuid d6d29fd8-d0db-cf49-81f5-8a7993134363
 path A3

You can see that all of the snapshots have f08112b1 as parent.

At the receiving side:

/mnt/zooi/receive 12-# btrfs sub list -u -q -R .

ID 263
 gen 31
 top level 5
 parent_uuid -
 received_uuid 862f3808-ccdf-cb40-8cf1-c31b8cbea635
 uuid 41a4495c-89bd-f949-8bfe-601e197eb96d
 path A1
ID 265
 gen 34
 top level 5
 parent_uuid 41a4495c-89bd-f949-8bfe-601e197eb96d
 received_uuid ab4bd82b-39f9-a24c-9ead-02023e392e9f
 uuid 332fd67a-cfd4-7e45-9114-a910fc8d5b95
 path A2
ID 267
 gen 36
 top level 5
 parent_uuid 332fd67a-cfd4-7e45-9114-a910fc8d5b95
 received_uuid d6d29fd8-d0db-cf49-81f5-8a7993134363
 uuid bb3a14fd-668c-6c4d-be1b-fc11a4cda6a0
 path A3

Here we see that if -p is used for send, the receiving side snapshots
the subvolume which has the sender-side uuid of the one referenced with
-p in its received_uuid field, and based on that clone (which on the
receiving side now has the previous received one as parent_uuid), it
adds the diff and it ends up as new read-only snapshot.

The original parent_uuid two steps back, is nowhere to be found any more.

Now let's remove some snapshots from the origin...

/mnt/zooi/send 12-# btrfs sub del A1 A2
Delete subvolume (no-commit): '/mnt/zooi/send/A1'
Delete subvolume (no-commit): '/mnt/zooi/send/A2'

I keep one snapshot in common, otherwise it's end of the game.

/mnt/zooi 12-# tree
.
├── receive
│   ├── A1
│   │   └── 1
│   ├── A2
│   │   ├── 1
│   │   └── 2
│   └── A3
│       ├── 1
│       ├── 2
│       └── 3
└── send
    ├── A
    │   ├── 1
    │   ├── 2
    │   └── 3
    └── A3
        ├── 1
        ├── 2
        └── 3

I can send back A1 or A2 now, using A3 as reference:

/mnt/zooi/receive 12-# btrfs send -v -p A3 A1 | btrfs receive -v ../send/
At subvol A1
BTRFS_IOC_SEND returned 0
joining genl thread
At snapshot A1
receiving snapshot A1
 uuid=862f3808-ccdf-cb40-8cf1-c31b8cbea635, ctransid=30
 parent_uuid=d6d29fd8-d0db-cf49-81f5-8a7993134363, parent_ctransid=38
BTRFS_IOC_SET_RECEIVED_SUBVOL
 uuid=862f3808-ccdf-cb40-8cf1-c31b8cbea635, stransid=30

What this does, is at the send side (which is now receiving back)
snapshotting the d6d29fd8 (which is A3) into A1 again, and then applying
the differences on top, setting received uuid to the uuid of the sent
A3. No actual data was sent.

And now:

/mnt/zooi/send -# btrfs sub list -u -q -R .

ID 271
 gen 41
 top level 5
 parent_uuid -
 received_uuid -
 uuid f08112b1-223d-cb46-b54b-2d90c8339617
 path A
ID 278
 gen 42
 top level 5
 parent_uuid f08112b1-223d-cb46-b54b-2d90c8339617
 received_uuid -
 uuid d6d29fd8-d0db-cf49-81f5-8a7993134363
 path A3
ID 279
 gen 43
 top level 5
 parent_uuid d6d29fd8-d0db-cf49-81f5-8a7993134363
 received_uuid 862f3808-ccdf-cb40-8cf1-c31b8cbea635
 uuid c787af91-874c-534f-ae8a-f3d78c18c971
 path A1

The parent_uuid of A1 is now A3, and not A!

But, we want to stop using A, because something terrible happened
between A1 and A2, and continue working based on A1, before the disaster.

/mnt/zooi/send 12-# mv A A-borken
/mnt/zooi/send 12-# btrfs sub snap A1 B
Create a snapshot of 'A1' in './B'
/mnt/zooi/send 12-# dd if=/dev/zero of=B/4 bs=1 count=0 seek=1G; shred
-n 1 B/4
/mnt/zooi/send 12-# btrfs sub snap -r B B4
Create a readonly snapshot of 'B' in './B4'

/mnt/zooi/send 12-# btrfs sub list -u -q -R .
[...]
ID 280
 gen 47
 top level 5
 parent_uuid c787af91-874c-534f-ae8a-f3d78c18c971
 received_uuid -
 uuid b1616848-a3e8-2149-9a96-9042d23c94c3
 path B
ID 282
 gen 47
 top level 5
 parent_uuid b1616848-a3e8-2149-9a96-9042d23c94c3
 received_uuid -
 uuid a5432e89-cf87-7443-9abf-8927d9c52ca0
 path B4

Fun thing is that you can actually continue the chain of incremental
sends for B4 on top of A1... which is quite nice, because it allows to
do a restore from the remote to an older snapshot, and then continue
with incrementals only again.

/mnt/zooi/send 12-# btrfs send -v -p A1 B4 | btrfs receive -v ../receive/
At subvol B4
At snapshot B4
receiving snapshot B4
 uuid=a5432e89-cf87-7443-9abf-8927d9c52ca0, ctransid=46
 parent_uuid=862f3808-ccdf-cb40-8cf1-c31b8cbea635, parent_ctransid=43
BTRFS_IOC_SEND returned 0
joining genl thread
BTRFS_IOC_SET_RECEIVED_SUBVOL
 uuid=a5432e89-cf87-7443-9abf-8927d9c52ca0, stransid=46

(No GiBs of data transferred at all!)

> And why can't I transfer S2 from USB to SSH? Is S1 not recognized as the
> same snapshot there?

Because S1 on USB and S1 on SSH are not the same thing. :| You can
compare their uuid fields, like I did above and see.

Btrfs send/receive is not as powerful as git with clone and merge. :] I
would certainly like it to be, that would be absolutely fabulous, but
it's not, right now.

>>> I tried to transfer...

Origin: S2 S3
USB: S1 S2
SSH: S1

>>> 1. S3 from Origin to SSH -> does not work as there is no common snapshot.

True. Not incremental

>>> 2. S2 from USB to SSH -> did not work.

What does 'did not work' mean?
You can send it, but not incremental, see above.

>>> 3. S1 from USB to Origin (such that there is a common snapshot with SSH)
>>> -> did not work.

When you send S1 back from USB to Origin, based on common S2, it will be
something new, having S2 as parent_uuid, and a new uuid of its own.

>>>
>>> Is it correct that 1. would work if a common snapshot is present on
>>> Origin and SSH?
>>>
>>> Is it expected that 2. and 3. do not work?
>>>
>>> Is there some other way to achieve it?
>> At home, I do a similar thing, I periodically send/receive changes of a
>> filesystem to two external disks, also using btrbk. Since I don't want
>> to have both backup disks and the originating filesystem all online and
>> in the same geographical location at the same moment, I have the same
>> problem.
>>
>> What I did is just setting expiry the snapshots on the origin manually,
>> and keep the meta-administration in my head. I don't do it that often
>> anyway.
>>

Btrfs send/receive is not as powerful as git with clone and merge. :] I
would certainly like it to be, that would be absolutely fabulous, but
it's not, right now.

-- 
Hans van Kranenburg

Re: Send/receive snapshot from/between backup

[Piotr Pawłow]

On 02.11.2016 15:23, René Bühlmann wrote:
> Origin: S2 S3
>
> USB: S1 S2
>
> SSH: S1
>
> Transferring S3 to USB is no problem as S2 is on both btrfs drives. But
> how can I transfer S3 to SSH?

If I understand correctly how send / receive works, for the incremental
receive to work there must be a subvolume on the destination which has
"received uuid" equal to the uuid of parent choosen for the incremental
send.

> I tried to transfer...
>
> 1. S3 from Origin to SSH -> does not work as there is no common snapshot.
>
> 2. S2 from USB to SSH -> did not work.

The "received uuid" of S1 on SSH is the uuid S1 had on Origin. The uuid
of S1 on USB is different, so when choosen as parent for the incremental
send it doesn't match.

> 3. S1 from USB to Origin (such that there is a common snapshot with SSH)
> -> did not work.

There are no previously received subvolumes on Origin at all, so it
isn't going to work.

> Is it correct that 1. would work if a common snapshot is present on
> Origin and SSH?

If there was a snapshot received from Origin that still exists on
Origin, then yes, you could use it as a clone source for incremental send.

> Is it expected that 2. and 3. do not work?
>
> Is there some other way to achieve it?

I doubt you can do it without some "hacking" to fool btrfs receive.

You would need a tool that can issue BTRFS_IOC_SET_RECEIVED_SUBVOL ioctl
to change the received uuid. Then you could:

1. Change received uuid of S1 on SSH to match S1 uuid on USB.
2. Send incremental S1-S2 from USB to SSH.
3. Change received uuid of S2 on SSH to match S2 on Origin.
4. Send incremental S2-S3 from Origin to SSH.

Regards

Re: Send/receive snapshot from/between backup

[René Bühlmann]

On 11/02/2016 05:13 PM, Piotr Pawłow wrote:
> On 02.11.2016 15:23, René Bühlmann wrote:
>> Origin: S2 S3
>>
>> USB: S1 S2
>>
>> SSH: S1
>>
>> Transferring S3 to USB is no problem as S2 is on both btrfs drives. But
>> how can I transfer S3 to SSH?
> If I understand correctly how send / receive works, for the incremental
> receive to work there must be a subvolume on the destination which has
> "received uuid" equal to the uuid of parent choosen for the incremental
> send.
>
>> I tried to transfer...
>>
>> 1. S3 from Origin to SSH -> does not work as there is no common snapshot.
>>
>> 2. S2 from USB to SSH -> did not work.
> The "received uuid" of S1 on SSH is the uuid S1 had on Origin. The uuid
> of S1 on USB is different, so when choosen as parent for the incremental
> send it doesn't match.
>
>> 3. S1 from USB to Origin (such that there is a common snapshot with SSH)
>> -> did not work.
> There are no previously received subvolumes on Origin at all, so it
> isn't going to work.
>
>> Is it correct that 1. would work if a common snapshot is present on
>> Origin and SSH?
> If there was a snapshot received from Origin that still exists on
> Origin, then yes, you could use it as a clone source for incremental send.
>
>> Is it expected that 2. and 3. do not work?
>>
>> Is there some other way to achieve it?
> I doubt you can do it without some "hacking" to fool btrfs receive.
>
> You would need a tool that can issue BTRFS_IOC_SET_RECEIVED_SUBVOL ioctl
> to change the received uuid. Then you could:
>
> 1. Change received uuid of S1 on SSH to match S1 uuid on USB.
> 2. Send incremental S1-S2 from USB to SSH.
> 3. Change received uuid of S2 on SSH to match S2 on Origin.
> 4. Send incremental S2-S3 from Origin to SSH.
>
> Regards
>
Thanks for all the input,

I did successfully try this approach, could change the "received uuid"
and then transfer a snapshot from a different source. So far so good.

But:

Due to a lot of errors during btrfs check on SSH, I decided to recreate
the BTRFS filesystem on SSH still with the goal to not transfer all the
data over the network.

These were the steps:

1. Create a new btrfs (calling it SSH')

2. Full transfer S1 from SSH to SSH'

3. Incremental transfer S2 from USB to SSH' (S1 as parent)

4. Incremental transfer S3 from Origin to SSH' (S2 as parent)

5. Btrfs check SSH'

6. Used rsync (with checksum-diff) to verify that S3 on Origin and SSH'
contain the same files.


Step 2 did work and beside of a single checksum error on SSH the
transfer completed without errors.

Step 3 and 4 did work as well and surprisingly, I did not even had to
update the "received uuid". They cant be full transfers as that would
have taken months with my bandwidth. How can this be?

Step 5 did not return any errors

Step 6 did find a single file differing which is due to the checksum
error on step 2.


So, everything seems to be fine now, I just do not understand why this
did work without any updating of the UUID.
Do you have an explanation for that?

In any case, thanks for your help.
René

Re: Send/receive snapshot from/between backup

[Piotr Pawłow]

On 16.11.2016 21:32, René Bühlmann wrote:
>
>> 1. Change received uuid of S1 on SSH to match S1 uuid on USB.
>> [...]
> 2. Full transfer S1 from SSH to SSH'
>
> 3. Incremental transfer S2 from USB to SSH' (S1 as parent)
>
> 4. Incremental transfer S3 from Origin to SSH' (S2 as parent)
>
> [..]
> Step 3 and 4 did work as well and surprisingly, I did not even had to
> update the "received uuid". They cant be full transfers as that would
> have taken months with my bandwidth. How can this be?

Hans van Kranenburg wrote in another post that you can send back an
incremental snapshot, so I guess the algorithm is more complicated.

Since S1 on SSH was faked to be received from USB, and S1 on SSH' is
received from S1 on SSH, and SSH is the same host as SSH' so both
subvolumes are visible by btrfs receive tool, maybe it can follow the
path and see that S1 on SSH' came from USB? I don't know.

I wonder why does it even matter for send / receive where a subvolume
comes from. Why not base it only on the content? Do I have a subvolume
that has the same content as the parent used for incremental send? - if
yes, then use it for receive.

If btrfs used a Merkle tree and had strong checksums, then a subvolume
checksum could be used as the content identifier (does it use a Merkle
tree? - Wikipedia article about the tree lists btrfs, but I can't find
any source).

Otherwise there could be a "fake" content identifier. A random "content
uuid" generated when setting subvolume read-only, and removed when
setting read/write (read only snapshot would preserve it, and rw
snapshot would remove it). Send / receive would preserve "content uuid",
and compare parent's "content uuid" for incremental send. Wouldn't it be
easier and more flexible than the current "received uuid" scheme? Am I
missing something?

Regards