back&forth send/receiving?

back&forth send/receiving?

[Christoph Anton Mitterer]

Hey.

I've had asked[0] this a while ago already and back then the conclusion
seems to have been that it wasn't (yet) possible but could in principle
be done.

The idea was that one main usage of btrfs' send/receive is probably
making incremental backups.

So one might have a setup, where a master is send/received to multiple
copies of that like in:

master-+-> copy1
       |-> copy2
       \-> copy3

with those all being different storage mediums.

At some point, master's HDD may e.g. break or at least reach an age
where one wants to replace it with a new one.
So either one would make e.g. copy1 the new master, or buy new HDD for
that, while keeping all the other copyN.

Problem is now, how to best continue with the above schema?

Either one just uses dd and copy over the old master to the new one
(keeping the very same btrfs).
This of course doesn't work if the old master really broke. And even if
it didn't it may be impractical (consider e.g. a 20 TB HDD which is
only half filled - why copying 10TB of empty space).


What one would IMO want is, that send/receive also works like that:
a) copy1/new-master-+-> copy2
                    |-> copy3
                    \-> copy4 (new)

b) once:
   copy1 ---> new-master

   following that: 
   new-master-+-> copy1
              |-> copy2
              \-> copy3


Did anything change with respect to v2 send/receive? Or would it be
possible to get such feature sooner or later? :-)


Thanks,
Chris.



[0] https://lore.kernel.org/linux-btrfs/157ed91bb66820d1fef89eb05d00e65c25607938.camel@scientia.net/

Re: back&forth send/receiving?

[Andrei Borzenkov]

On 03.02.2023 19:42, Christoph Anton Mitterer wrote:
> Hey.
> 
> I've had asked[0] this a while ago already and back then the conclusion
> seems to have been that it wasn't (yet) possible but could in principle
> be done.
> 
> The idea was that one main usage of btrfs' send/receive is probably
> making incremental backups.
> 
> So one might have a setup, where a master is send/received to multiple
> copies of that like in:
> 
> master-+-> copy1
>         |-> copy2
>         \-> copy3
> 
> with those all being different storage mediums.
> 

Then you do not have incremental replication and your question is moot. 
Incremental btrfs send/recieve is possible only between the same pair of 
filesystems.

> At some point, master's HDD may e.g. break or at least reach an age
> where one wants to replace it with a new one.
> So either one would make e.g. copy1 the new master, or buy new HDD for
> that, while keeping all the other copyN.
> 
> Problem is now, how to best continue with the above schema?
> 
> Either one just uses dd and copy over the old master to the new one
> (keeping the very same btrfs).
> This of course doesn't work if the old master really broke. And even if
> it didn't it may be impractical (consider e.g. a 20 TB HDD which is
> only half filled - why copying 10TB of empty space).
> 
> 
> What one would IMO want is, that send/receive also works like that:
> a) copy1/new-master-+-> copy2
>                      |-> copy3
>                      \-> copy4 (new)
> 
> b) once:
>     copy1 ---> new-master
> 
>     following that:
>     new-master-+-> copy1
>                |-> copy2
>                \-> copy3
> 
> 
> Did anything change with respect to v2 send/receive? Or would it be
> possible to get such feature sooner or later? :-)
> 
> 

You seem to misunderstand how btrfs send/receive works. There is no 
inherent relationship between copy1, copy2 etc nor between master and 
copy1, copy2, ... As mentioned as each copy1, copy2, ... is on separate 
medium, each one is the complete copy of master and you can make new 
complete copies as you need.

Re: back&forth send/receiving?

[Christoph Anton Mitterer]

On Sat, 2023-02-04 at 09:10 +0300, Andrei Borzenkov wrote:
> > So one might have a setup, where a master is send/received to
> > multiple
> > copies of that like in:
> > 
> > master-+-> copy1
> >         |-> copy2
> >         \-> copy3
> > 
> > with those all being different storage mediums.
> > 
> 
> Then you do not have incremental replication and your question is
> moot. 
> Incremental btrfs send/recieve is possible only between the same pair
> of 
> filesystems.

I don't quite get why this shouldn't be incremental backups.

Whenever the files on master have changed and a new snapshot is created
on it, and the copies shall be synced with the current state of master,
the new snapshot from that is send|receive(d) to the copies, using
previous snapshots on them as -p parent.



> You seem to misunderstand how btrfs send/receive works. There is no 
> inherent relationship between copy1, copy2 etc nor between master and
> copy1, copy2, ... As mentioned as each copy1, copy2, ... is on
> separate 
> medium, each one is the complete copy of master and you can make new 
> complete copies as you need.

I rather believe that there was a misunderstanding of my setup
respectively what I do :-)


Cheers,
Chris.

Re: back&forth send/receiving?

[Andrei Borzenkov]

On 06.02.2023 06:07, Christoph Anton Mitterer wrote:
> On Sat, 2023-02-04 at 09:10 +0300, Andrei Borzenkov wrote:
>>> So one might have a setup, where a master is send/received to
>>> multiple
>>> copies of that like in:
>>>
>>> master-+-> copy1
>>>          |-> copy2
>>>          \-> copy3
>>>
>>> with those all being different storage mediums.
>>>
>>
>> Then you do not have incremental replication and your question is
>> moot.
>> Incremental btrfs send/recieve is possible only between the same pair
>> of
>> filesystems.
> 
> I don't quite get why this shouldn't be incremental backups.
> 
> Whenever the files on master have changed and a new snapshot is created
> on it, and the copies shall be synced with the current state of master,
> the new snapshot from that is send|receive(d) to the copies, using
> previous snapshots on them as -p parent.
> 

You stated that copies are on different media which means there is no 
previous snapshot to apply changes to. If I misunderstood you, then you 
need to explain what "all being on different mediums" means.

> 
> 
>> You seem to misunderstand how btrfs send/receive works. There is no
>> inherent relationship between copy1, copy2 etc nor between master and
>> copy1, copy2, ... As mentioned as each copy1, copy2, ... is on
>> separate
>> medium, each one is the complete copy of master and you can make new
>> complete copies as you need.
> 
> I rather believe that there was a misunderstanding of my setup
> respectively what I do :-)
> 
> 
> Cheers,
> Chris.

Re: back&forth send/receiving?

[Christoph Anton Mitterer]

On Mon, 2023-02-06 at 07:16 +0300, Andrei Borzenkov wrote:
> You stated that copies are on different media which means there is no
> previous snapshot to apply changes to. If I misunderstood you, then
> you 
> need to explain what "all being on different mediums" means.

There are 4 HDDs:
master
copy1
copy2
copy3

- Files are added/removed/modified on master.
- Once new backups shall be made:
  - a new ro-snapshot from the actual data-subvolume on master
    is made, e.g. with the name of the current date
  - that snapshot is then send|receive(d) to each of the copyN HDDs
    - for the first time, this will obviously be non-incremental
    - subsequently, the most recent common parent subvol on copyN and
      master will be used with -p
    - eventually, older subvols will be removed on all the HDDs;
      on master of course only, if the respective subvol is no longer
      needed for updating any of the copyN HDDs

Figured that's a pretty plain and common backup cycle using btrfs, so I
didn't explain it any further in my original post.


At a certain point in time, the scenario I've described in the original
post occurs and e.g. master breaks or shall simply be replaced with a
new HDD.
Thus, a new master is needed, either the new HDD, or maybe one of copyN
(just as described in the OP).

But at that point, at least that was my understanding from the
discussion some months ago, I cannot just e.g.:
- send|receive from copy1 to new-master, continue there (on new-master)
  to add/remove/edit data, make a new snapshot and then *incrementally*
  send|receive back to the copyN HDDs.
or alternatively:
- send|receive incrementally from copy1 (which would then be the "new"
  master) to copy2 and copy3.

Being able to do so, would be the request of this post ;-)


I mean the data should all be there, isn't it?


Cheers,
Chris.

Re: back&forth send/receiving?

[Andrei Borzenkov]

On Mon, Feb 6, 2023 at 7:31 AM Christoph Anton Mitterer
<calestyo@scientia.org> wrote:
>
> On Mon, 2023-02-06 at 07:16 +0300, Andrei Borzenkov wrote:
> > You stated that copies are on different media which means there is no
> > previous snapshot to apply changes to. If I misunderstood you, then
> > you
> > need to explain what "all being on different mediums" means.
>
> There are 4 HDDs:
> master
> copy1
> copy2
> copy3
>
> - Files are added/removed/modified on master.
> - Once new backups shall be made:
>   - a new ro-snapshot from the actual data-subvolume on master
>     is made, e.g. with the name of the current date
>   - that snapshot is then send|receive(d) to each of the copyN HDDs
>     - for the first time, this will obviously be non-incremental
>     - subsequently, the most recent common parent subvol on copyN and
>       master will be used with -p
>     - eventually, older subvols will be removed on all the HDDs;
>       on master of course only, if the respective subvol is no longer
>       needed for updating any of the copyN HDDs
>

OK, that's not exactly "all on different mediums", but rather several
(independent) chains of incremental snapshots, with each chain on a
different medium.

> Figured that's a pretty plain and common backup cycle using btrfs, so I
> didn't explain it any further in my original post.
>
>
> At a certain point in time, the scenario I've described in the original
> post occurs and e.g. master breaks or shall simply be replaced with a
> new HDD.
> Thus, a new master is needed, either the new HDD, or maybe one of copyN
> (just as described in the OP).
>
> But at that point, at least that was my understanding from the
> discussion some months ago, I cannot just e.g.:
> - send|receive from copy1 to new-master, continue there (on new-master)
>   to add/remove/edit data, make a new snapshot and then *incrementally*
>   send|receive back to the copyN HDDs.

You can always incrementally send, there is no problem. To actually
receive *and apply* incremental stream you need base snapshot (or
identical copy of base snapshot) to apply the changes to. So if you
make copy1 your new master, there is no way to continue incremental
send/receive to copy2, copy3, ... because copy2, copy3, ... do not
have snapshots present on copy1.

Same if you would copy/clone copy1 to another HDD.

> or alternatively:
> - send|receive incrementally from copy1 (which would then be the "new"
>   master) to copy2 and copy3.
>

See above.

> Being able to do so, would be the request of this post ;-)
>
>
> I mean the data should all be there, isn't it?
>

You need data at the exact point in time. According to your
description each of copy1, copy2, ... captures different points in
time.

To implement what you want one would need cross-filesystem support for
computing changes (compute difference between snapA on copy1 and snapB
on copy2 and recreate snapB on copy1 to get a common base snapshot).

Re: back&forth send/receiving?

[Christoph Anton Mitterer]

Hey Andrei.



On Mon, 2023-02-06 at 09:46 +0300, Andrei Borzenkov wrote:
> OK, that's not exactly "all on different mediums", but rather several
> (independent) chains of incremental snapshots, with each chain on a
> different medium.

Yes.



> You can always incrementally send, there is no problem. To actually
> receive *and apply* incremental stream you need base snapshot (or
> identical copy of base snapshot) to apply the changes to.

Back then in my first thread[0] about this, I had the impression that
it is not ought to work:
Hugo Mills said so in [1].
You also commented and I'm not sure whether I understood it correctly
back then,... but the following paragraph of yours I'd understand that
it's not intended to work, right?

> So if you
> make copy1 your new master, there is no way to continue incremental
> send/receive to copy2, copy3, ... because copy2, copy3, ... do not
> have snapshots present on copy1.
> 
> Same if you would copy/clone copy1 to another HDD.

Below you'll find a simulation of what I'd do in my use case.
First starting off with one master and a few copies (each of which
receive the line of incremental snapshots) and then I decommission
master and introduce a new-master (filling it with data from copy1) and
then continue with the incremental snapshots from that new-master to
the copies.

And... it does seem to work (already).
I'm just still not sure whether it's really safe or just a
"coincidence" that it does work:

--------------------------------------------------------------------

// make a few mountpoints and empty image-files (with btrfs on them)
// that will be used to simulate different HDDs
// new-master is not used in the beginning
// <dir># being a root-user prompt

~# mkdir -p mnt/{master,new-master,copy1,copy2}
~# truncate -s 1G {master,new-master,copy1,copy2}.img
~# for n in master new-master copy1 copy2; do mkfs.btrfs -L $n $n.img; done
...
~# mount master.img mnt/master/
~# mount copy1.img mnt/copy1/
~# mount copy2.img mnt/copy2/


// simulate adding some data

~# cd mnt/master/
~/mnt/master# btrfs subvolume create data
Create subvolume './data'
~/mnt/master# echo REV1 > data/small-file


// the first ever round of snapshots (on master) and backups of that
// snapshot from master to the copyN HDDs
// send|receive is obviously non-incremental
//
// I run receive with /usr/bin/time -v to see how much was actually written:
// "File system outputs" is in 512B blocks

~/mnt/master# btrfs subvolume snapshot -r data/ snap-1
Create a readonly snapshot of 'data/' in './snap-1'
~/mnt/master# btrfs send snap-1 | /usr/bin/time -v btrfs receive ~/mnt/copy1/
At subvol snap-1
At subvol snap-1
	Command being timed: "btrfs receive /root/mnt/copy1/"
	User time (seconds): 0.00
	System time (seconds): 0.00
	Percent of CPU this job got: 6%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.09
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 3524
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 186
	Voluntary context switches: 9
	Involuntary context switches: 11
	Swaps: 0
	File system inputs: 0
	File system outputs: 424
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0
~/mnt/master# btrfs send snap-1 | /usr/bin/time -v btrfs receive ~/mnt/copy2/
At subvol snap-1
At subvol snap-1
	Command being timed: "btrfs receive /root/mnt/copy2/"
	User time (seconds): 0.00
	System time (seconds): 0.00
	Percent of CPU this job got: 1%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.40
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 3488
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 184
	Voluntary context switches: 7
	Involuntary context switches: 10
	Swaps: 0
	File system inputs: 0
	File system outputs: 424
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

// at this point, snap-1 from master should be on copy1 and copy2


// simulate some more changes of files on the master using a "big" file
// this time to see whether incremental send|receive works

~/mnt/master# dd if=/dev/urandom of=data/big-file bs=1M count=512
512+0 records in
512+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 2,74663 s, 195 MB/s
~/mnt/master# echo REV2 > data/small-file


// the second round of snapshots (on master) and backups of that
// snapshot from master to the copyN HDDs
// this time with INCREMENTAL send|receive

~/mnt/master# btrfs subvolume snapshot -r data/ snap-2
Create a readonly snapshot of 'data/' in './snap-2'
~/mnt/master# btrfs send -p snap-1 snap-2 | /usr/bin/time -v btrfs receive ~/mnt/copy1/
At subvol snap-2
At snapshot snap-2
	Command being timed: "btrfs receive /root/mnt/copy1/"
	User time (seconds): 0.12
	System time (seconds): 0.82
	Percent of CPU this job got: 29%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:03.27
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 3600
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 201
	Voluntary context switches: 156
	Involuntary context switches: 7802
	Swaps: 0
	File system inputs: 0
	File system outputs: 1049640
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0
~/mnt/master# btrfs send -p snap-1 snap-2 | /usr/bin/time -v btrfs receive ~/mnt/copy2/
At subvol snap-2
At snapshot snap-2
	Command being timed: "btrfs receive /root/mnt/copy2/"
	User time (seconds): 0.06
	System time (seconds): 0.76
	Percent of CPU this job got: 35%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.32
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 3576
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 203
	Voluntary context switches: 52
	Involuntary context switches: 11726
	Swaps: 0
	File system inputs: 0
	File system outputs: 1049512
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

// at this point, both copyN HDDs should have snap-2
// in bot cases File system outputs: was around 1049640
// which is ~500MB... obviously, since the big-file was new


// just for convenience, some UUIDs from the (old) master

~/mnt/master# cd 
~# btrfs subvolume list -paguqR mnt/master/
ID 256 gen 8 parent 5 top level 5 parent_uuid -                                    received_uuid -                                    uuid 5bc13acd-01f8-774f-913b-2833d0b99ed2 path data
ID 257 gen 7 parent 5 top level 5 parent_uuid 5bc13acd-01f8-774f-913b-2833d0b99ed2 received_uuid -                                    uuid 6fb6191a-413e-4d46-aa65-0fb30dd8a606 path snap-1
ID 265 gen 8 parent 5 top level 5 parent_uuid 5bc13acd-01f8-774f-913b-2833d0b99ed2 received_uuid -                                    uuid f38aaffc-226f-d748-949e-5c9677440e14 path snap-2


// at this point, master HDD breaks or is simply scheduled to be
// replaced by a new one (here I really use a new one, and DON'T make
// e.g. copy1 the new master)

~# umount mnt/master 
~# rm -f master.img ; rmdir mnt/master/
~# mount new-master.img mnt/new-master/


// next, I populate the new master with the latest data from copy1 HDD
// obviously, non-incrementally

~# cd mnt/copy1/
~/mnt/copy1# btrfs send snap-2 | /usr/bin/time -v btrfs receive ~/mnt/new-master/
At subvol snap-2
At subvol snap-2
	Command being timed: "btrfs receive /root/mnt/new-master/"
	User time (seconds): 0.09
	System time (seconds): 0.55
	Percent of CPU this job got: 22%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.85
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 3452
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 197
	Voluntary context switches: 63
	Involuntary context switches: 8513
	Swaps: 0
	File system inputs: 0
	File system outputs: 1049608
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

// again, ~500MB transferred, just as expected


// now, I create a new working area of the received snapshot on the new master
// this is simply a RW snapshot of that, and will be the place where I add new files
// modify or delete existing ones

~/mnt/copy1# cd ../new-master/
~/mnt/new-master# btrfs subvolume snapshot snap-2 data
Create a snapshot of 'snap-2' in './data'


// just a poor check whether the files are still in the most recent state

~/mnt/new-master# cat data/small-file 
REV2
~/mnt/new-master# diff data/big-file ../copy1/snap-2/big-file  ; echo $?
0


// simulate another round of modifying files on the (now new) master

~/mnt/new-master# dd if=/dev/urandom of=data/medium-file bs=1M count=5
5+0 records in
5+0 records out
5242880 bytes (5,2 MB, 5,0 MiB) copied, 0,0339809 s, 154 MB/s
~/mnt/new-master# echo REV3 > data/small-file


// the third round of snapshots (now on/from the new master) and
// backups of that snapshot from (new) master to the copyN HDDs
// important point: again INCREMENTAL send|receive

~/mnt/new-master# btrfs subvolume snapshot -r data/ snap-3
Create a readonly snapshot of 'data/' in './snap-3'
~/mnt/new-master# btrfs send -p snap-2 snap-3 | /usr/bin/time -v btrfs receive ~/mnt/copy1/
At subvol snap-3
At snapshot snap-3
	Command being timed: "btrfs receive /root/mnt/copy1/"
	User time (seconds): 0.00
	System time (seconds): 0.01
	Percent of CPU this job got: 5%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.33
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 3524
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 200
	Voluntary context switches: 27
	Involuntary context switches: 184
	Swaps: 0
	File system inputs: 0
	File system outputs: 10696
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0
~/mnt/new-master# btrfs send -p snap-2 snap-3 | /usr/bin/time -v btrfs receive ~/mnt/copy2/
At subvol snap-3
At snapshot snap-3
	Command being timed: "btrfs receive /root/mnt/copy2/"
	User time (seconds): 0.00
	System time (seconds): 0.01
	Percent of CPU this job got: 6%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.39
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 3564
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 198
	Voluntary context switches: 19
	Involuntary context switches: 192
	Swaps: 0
	File system inputs: 0
	File system outputs: 10824
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

// at this point, the copyN HDDs should have snap-3 from the new master
// created incrementally, using their snap-2 (which they've still received
// from the OLD master... and which was also received on new master from copy1)
// File system outputs is in both cases ~ 10824, which is a bit more than 5 MB,...
// just the newly added medium-file in run 3... especially no 500 MB from
// the big-file, so seems, that incremental send|receive did in fact work


// "verifying" this:

~/mnt/new-master# cd ..
~/mnt# md5sum new-master/data/small-file copy1/snap-3/small-file copy2/snap-3/small-file
78a0b89f84b68d690067b06522b39452  new-master/data/small-file
78a0b89f84b68d690067b06522b39452  copy1/snap-3/small-file
78a0b89f84b68d690067b06522b39452  copy2/snap-3/small-file
~/mnt# md5sum new-master/data/medium-file copy1/snap-3/medium-file copy2/snap-3/medium-file 
06352daf5656d20a870405d82d1e6d79  new-master/data/medium-file
06352daf5656d20a870405d82d1e6d79  copy1/snap-3/medium-file
06352daf5656d20a870405d82d1e6d79  copy2/snap-3/medium-file
~/mnt# md5sum new-master/data/big-file copy1/snap-3/big-file copy2/snap-3/big-file 
4b0fe1b47cd2be7dcdf7337a7b0782ae  new-master/data/big-file
4b0fe1b47cd2be7dcdf7337a7b0782ae  copy1/snap-3/big-file
4b0fe1b47cd2be7dcdf7337a7b0782ae  copy2/snap-3/big-file



// looking at the UUIDs

~/mnt# btrfs subvolume list -paguqR  new-master/
ID 256 gen  9 parent 5 top level 5 parent_uuid -                                    received_uuid f38aaffc-226f-d748-949e-5c9677440e14 uuid 4a098904-a9a0-214b-8009-88a70f0bbe77 path snap-2
ID 265 gen 11 parent 5 top level 5 parent_uuid 4a098904-a9a0-214b-8009-88a70f0bbe77 received_uuid -                                    uuid 22d82a1b-f764-5243-8f43-3f8f492b718e path data
ID 266 gen 10 parent 5 top level 5 parent_uuid 22d82a1b-f764-5243-8f43-3f8f492b718e received_uuid -                                    uuid 76db9591-2d8c-094a-9f78-c976bd4d3264 path snap-3
~/mnt# btrfs subvolume list -paguqR  copy1/
ID 256 gen  9 parent 5 top level 5 parent_uuid -                                    received_uuid 6fb6191a-413e-4d46-aa65-0fb30dd8a606 uuid fbc5489d-3738-694c-ba5e-ba36987f832b path snap-1
ID 257 gen 13 parent 5 top level 5 parent_uuid fbc5489d-3738-694c-ba5e-ba36987f832b received_uuid f38aaffc-226f-d748-949e-5c9677440e14 uuid 720bda11-1b92-fc45-8bb0-2a35a4c86f9d path snap-2
ID 266 gen 16 parent 5 top level 5 parent_uuid 720bda11-1b92-fc45-8bb0-2a35a4c86f9d received_uuid 76db9591-2d8c-094a-9f78-c976bd4d3264 uuid a77f24e1-f357-2148-9ca5-1240c4056cf3 path snap-3
~/mnt# btrfs subvolume list -paguqR  copy2/
ID 256 gen 10 parent 5 top level 5 parent_uuid -                                    received_uuid 6fb6191a-413e-4d46-aa65-0fb30dd8a606 uuid 5e73c53e-98b5-b44c-9d39-812c67f43fad path snap-1
ID 257 gen 14 parent 5 top level 5 parent_uuid 5e73c53e-98b5-b44c-9d39-812c67f43fad received_uuid f38aaffc-226f-d748-949e-5c9677440e14 uuid 33d07ed9-e216-954e-9c0c-b8f115546522 path snap-2
ID 264 gen 16 parent 5 top level 5 parent_uuid 33d07ed9-e216-954e-9c0c-b8f115546522 received_uuid 76db9591-2d8c-094a-9f78-c976bd4d3264 uuid 6bbf4bfd-5f9e-a347-89e4-6f58d6bfd035 path snap-3

from above:
~# btrfs subvolume list -paguqR mnt/master/
ID 256 gen 8 parent 5 top level 5 parent_uuid -                                    received_uuid -                                    uuid 5bc13acd-01f8-774f-913b-2833d0b99ed2 path data
ID 257 gen 7 parent 5 top level 5 parent_uuid 5bc13acd-01f8-774f-913b-2833d0b99ed2 received_uuid -                                    uuid 6fb6191a-413e-4d46-aa65-0fb30dd8a606 path snap-1
ID 265 gen 8 parent 5 top level 5 parent_uuid 5bc13acd-01f8-774f-913b-2833d0b99ed2 received_uuid -                                    uuid f38aaffc-226f-d748-949e-5c9677440e14 path snap-2


// AFAICS, the UUIDs are all correct (as far as they should have been in a chain)
--------------------------------------------------------------------


So from what I can tell... it actually does already work.
I'm just not really sure whether this is expected and can be considered
to be production ready... or whether it's just coincidence, that I
could do this "back & forth" send|receive.



> You need data at the exact point in time. According to your
> description each of copy1, copy2, ... captures different points in
> time.

They may or may not be.
Sometimes, I just send the same snapshot (e.g. from some specific date)
to all copyN ... sometimes the most recent one of all of them is
different.

But I always keep on (old) master the snapshot that are most recent on
each of copyN, so that I can continue from there, when I do the next
round of snapshot.

Of course, for the case when I switch from a old to a new master, I
would need the exactly same snapshot on all of copyN, so that I can
incrementally send|receive from the NEW master to the copyN.

It wouldn't need to be the most recent snapshot, AFAIU.

If e.g. I had the following snapshots:
copy1:
2023-01-01
2023-02-02

copy2:
2023-01-01
2023-01-15

and then want to switch from old to new master:
I'd use first copy1/2023-01-01 (NOT the most recent) to populate new-
master. And then copy1/2023-02-02 to bring new-master up2date.

Now in order to send a future new-master/2023-03-20 to copy1, I could
use 2023-02-02 as parent.
But not so for copy2.

However, I'd expect/hope, that for copy2, I could use the new-
master/2023-01-01 to still incrementally send|receive from new-master
to copy2.
It might of course require a bit more data than if I'd were able to use
2023-02-02, but still better than sending several TB from scratch.




Thanks,
Chris.


[0] https://lore.kernel.org/linux-btrfs/157ed91bb66820d1fef89eb05d00e65c25607938.camel@scientia.net/
[1]
https://lore.kernel.org/linux-btrfs/20210129192058.GN4090@savella.carfax.org.uk/

Re: back&forth send/receiving?

[Andrei Borzenkov]

On 17.02.2023 20:12, Christoph Anton Mitterer wrote:
> ~/mnt/master# btrfs send snap-1 | /usr/bin/time -v btrfs receive ~/mnt/copy1/
> ~/mnt/master# btrfs send snap-1 | /usr/bin/time -v btrfs receive ~/mnt/copy2/
> ~/mnt/master# btrfs subvolume snapshot -r data/ snap-2
> Create a readonly snapshot of 'data/' in './snap-2'
> ~/mnt/master# btrfs send -p snap-1 snap-2 | /usr/bin/time -v btrfs receive ~/mnt/copy1/
> ~/mnt/master# btrfs send -p snap-1 snap-2 | /usr/bin/time -v btrfs receive ~/mnt/copy2/

So copy1 and copy2 are identical. This is not what you said earlier (at 
least, it is not how what you said earlier sounded).

Of course you can use either of them and continue incremental chain on 
another.
...
> 
> They may or may not be.
> Sometimes, I just send the same snapshot (e.g. from some specific date)
> to all copyN ... sometimes the most recent one of all of them is
> different.
> 

So are all copyN identical or not?

> But I always keep on (old) master the snapshot that are most recent on
> each of copyN, so that I can continue from there, when I do the next
> round of snapshot.
> 

It does not matter what you keep on old master because old master is 
gone. What matter is what you keep on each copyN.

Re: back&forth send/receiving?

[Christoph Anton Mitterer]

On Fri, 2023-02-17 at 20:25 +0300, Andrei Borzenkov wrote:
> So copy1 and copy2 are identical. This is not what you said earlier
> (at 
> least, it is not how what you said earlier sounded).

Well that are based on the same master, but their most recent snapshot
from that *may* be a different one.

E.g.:

master could have:
data  (with the following snapshots of that)
data_2022-10-01
data_2022-11-01
data_2022-12-01
data_2023-01-01
data_2023-02-01

copy1 could have:
data_2022-10-01
data_2022-12-01
data_2023-01-01

copy2 could have

data_2022-10-01
data_2022-12-01
data_2023-02-01



> So are all copyN identical or not?

Well, except for perhaps the most recent snapshot.

The reason is in my case simply that I keep the copyN HDDs never at the
same place.... so there may be some time where on is not up to date.
And sometimes I may even skip a snapshot when I up date one of the
copyN HDDs, instead taking an already newer one.


> > But I always keep on (old) master the snapshot that are most recent
> > on
> > each of copyN, so that I can continue from there, when I do the
> > next
> > round of snapshot.
> > 
> 
> It does not matter what you keep on old master because old master is 
> gone. What matter is what you keep on each copyN.

That was meant for the time hen old master is still there any I make
the regular snapshots.
Of course I'll do the same on new master, too. Simply to be able to
incrementally send|receive to the copyN.


Cheers,
Chri.

Re: back&forth send/receiving?

[Andrei Borzenkov]

On 18.02.2023 03:29, Christoph Anton Mitterer wrote:
> On Fri, 2023-02-17 at 20:25 +0300, Andrei Borzenkov wrote:
>> So copy1 and copy2 are identical. This is not what you said earlier
>> (at
>> least, it is not how what you said earlier sounded).
> 
> Well that are based on the same master, but their most recent snapshot
> from that *may* be a different one.
> 

We are going in circles. To continue incremental chain on another copy 
after switching to a new master you need common base snapshot on the new 
master and the other copy. If such base snapshot exists, you can 
continue - starting with this base snapshot. If such base snapshot does 
not exist - you cannot.