Using send/receive to keep two rootfs-partitions in sync fails with "ERROR: snapshot: cannot find parent subvolume"

Using send/receive to keep two rootfs-partitions in sync fails with "ERROR: snapshot: cannot find parent subvolume"

[Clemens Eisserer]

Hi,

I would like to use send/receive to keep two root-filesystems in sync,
as I've been using it for years now for backups where it really does
wonders (thanks a lot!).

Both disks contain a read-only snapshot which is kept in-sync between
the filesystems (int and ext are the mountpoints of the two disks,
original_disk is just used for initial data):
   btrfs send original_disk/root-ro | btrfs receive int/ #send
snapshot of the original disk to the first of the two filesystens
(disk "int")
   btrfs send int/root-ro | btrfs receive ext/ #now replicate the same
to disk "ext"
so both disks start with a snapshot "root-ro" with equal content.

in case I would like to work with one of the two disks, I create a rw
snapshot based no root-ro:
  btrfs sub snap ext/root-ro ext/root-rw

  touch ext/root-wr/create-a-new-file # perform some modifications

once modifications in root-rw are done, the following steps are
performed to sync the filesystems:
  btrfs sub snap -r ext/root-rw ext/root-ro-new #create a root-ro-new
read-only snapshot based on the rw-snapshot with modfications (so it
can be used with btrfs send)
  btrfs send -p ext/root-ro extern/root-ro-new | btrfs receive int/
#send root-ro-new to "int" filesystem
  btrfs sub del ext/root-ro # delete the original root-ro snapshot, as
it is no longer needed for differential btrfs send
  mv ext/root-ro-new ext/root-ro #rename root-ro-new to root-ro, as
this is the current state of the other (int) filesystem
  btrfs sub del int/root-ro # delete root-ro in "int" too, as it is no
longer needed for differential btrfs receive
  mv int/root-ro-new int/root-ro #rename root-ro-new to root-ro
  btrfs sub snap int/root-ro int/root-rw # create a working copy of
root-ro which is writeable

this works great - i can add/modify files in one root-rw folder, call
the synchronization script and everything is found on the other
filesystem.
When exchanging int and ext in the script above it actually works in
both directions, so this is exactly what I was hoping to achieve.
Even when executing the script multiple times int->ext, ext->int,
int->ext ... with modifications in between, everything works as
expected.
Awesome :)

However, when actually using the file-systems as rootfs, this seem to
break when performing the following steps:
- create rw snapshot of root-ro on disk "ext": btrfs sub snap
ext/root-ro ext/root-rw
- boot the system with rootfs=ext-uuid and rootflags=subvol=/root-rw
(etc/fstab was adapted accordingly)
- use the system, modify file system etc and shutdown again
- start separate system to synchronize disks (not based on int or ext
rootfs) and call sync script ext->int (shown above)

it now suddenly fails at btrfs receive with:
btrfs send -p ext/root-ro ext/root-ro-new | btrfs receive intern/
ERROR: snapshot: cannot find parent subvolume
4ed11491-7563-fb49-99e7-86cb47cfb510

which I, to be honest, don't understand.
Exactly the same sequence of commands worked multiple times when the
root-rw snapshot was not booted from but modified directly on the sync
system, even with umounts between modification & send/receive.
Does it make a difference for btrfs if it is used as rootfs vs normal
writeable mount?
Or does it just work in the non-boot case because of some side-effects?

Thanks & best regards, Clemens

Re: Using send/receive to keep two rootfs-partitions in sync fails with "ERROR: snapshot: cannot find parent subvolume"

[Andrei Borzenkov]

On 07.01.2024 10:06, Clemens Eisserer wrote:
> Hi,
> 
> I would like to use send/receive to keep two root-filesystems in sync,
> as I've been using it for years now for backups where it really does
> wonders (thanks a lot!).
> 
> Both disks contain a read-only snapshot which is kept in-sync between
> the filesystems (int and ext are the mountpoints of the two disks,
> original_disk is just used for initial data):
>     btrfs send original_disk/root-ro | btrfs receive int/ #send
> snapshot of the original disk to the first of the two filesystens
> (disk "int")
>     btrfs send int/root-ro | btrfs receive ext/ #now replicate the same
> to disk "ext"
> so both disks start with a snapshot "root-ro" with equal content.
> 
> in case I would like to work with one of the two disks, I create a rw
> snapshot based no root-ro:
>    btrfs sub snap ext/root-ro ext/root-rw
> 
>    touch ext/root-wr/create-a-new-file # perform some modifications
> 

There was no "root-wr" before.

> once modifications in root-rw are done, the following steps are
> performed to sync the filesystems:
>    btrfs sub snap -r ext/root-rw ext/root-ro-new #create a root-ro-new
> read-only snapshot based on the rw-snapshot with modfications (so it
> can be used with btrfs send)
>    btrfs send -p ext/root-ro extern/root-ro-new | btrfs receive int/
> #send root-ro-new to "int" filesystem

There was no "extern" before.

Never describe computer commands. Copy and paste them in full with 
complete output.

>    btrfs sub del ext/root-ro # delete the original root-ro snapshot, as
> it is no longer needed for differential btrfs send
>    mv ext/root-ro-new ext/root-ro #rename root-ro-new to root-ro, as
> this is the current state of the other (int) filesystem
>    btrfs sub del int/root-ro # delete root-ro in "int" too, as it is no
> longer needed for differential btrfs receive
>    mv int/root-ro-new int/root-ro #rename root-ro-new to root-ro
>    btrfs sub snap int/root-ro int/root-rw # create a working copy of
> root-ro which is writeable
> 
> this works great - i can add/modify files in one root-rw folder, call
> the synchronization script and everything is found on the other
> filesystem.
> When exchanging int and ext in the script above it actually works in
> both directions, so this is exactly what I was hoping to achieve.
> Even when executing the script multiple times int->ext, ext->int,
> int->ext ... with modifications in between, everything works as
> expected.
> Awesome :)
> 
> However, when actually using the file-systems as rootfs, this seem to
> break when performing the following steps:
> - create rw snapshot of root-ro on disk "ext": btrfs sub snap
> ext/root-ro ext/root-rw
> - boot the system with rootfs=ext-uuid and rootflags=subvol=/root-rw
> (etc/fstab was adapted accordingly)
> - use the system, modify file system etc and shutdown again
> - start separate system to synchronize disks (not based on int or ext
> rootfs) and call sync script ext->int (shown above)
> 

It is absolutely unclear what it means. As mentioned, provide full 
transcript of your actions as well as the output of

btrfs subvolume list -pqu /mount-point

for all filesystems involved in these commands.

> it now suddenly fails at btrfs receive with:
> btrfs send -p ext/root-ro ext/root-ro-new | btrfs receive intern/
> ERROR: snapshot: cannot find parent subvolume
> 4ed11491-7563-fb49-99e7-86cb47cfb510
> 
> which I, to be honest, don't understand.
> Exactly the same sequence of commands worked multiple times when the
> root-rw snapshot was not booted from but modified directly on the sync
> system, even with umounts between modification & send/receive.
> Does it make a difference for btrfs if it is used as rootfs vs normal
> writeable mount?
> Or does it just work in the non-boot case because of some side-effects?
> 
> Thanks & best regards, Clemens
> 

Re: Using send/receive to keep two rootfs-partitions in sync fails with "ERROR: snapshot: cannot find parent subvolume"

[Clemens Eisserer]

Hello Andrei,

Thanks for taking a look.

The exact commands executed where:

mkdir intern
mkdir extern

mount /dev/mapper/ext extern
mout /dev/mapper/int intern

ls extern/
# output: root-ro

ls intern/
# output: -> empty

btrfs send extern/root-ro | btrfs receive intern/ #initial ext -> int
btrfs sub snap intern/root-ro intern/root-rw # rw snapshot to modify int
touch intern/root-rw/newfile #actual modification

umount intern
umount extern

sh sync_int_to_ext.sh #source of script at bottom

ls extern/root-ro/newfile
# output: extern/root-ro/newfile -> sync int->ext was successful

# now modify ext fs
mount /dev/mapper/ext extern/
touch extern/root-rw/anothernewfile
umount extern

sh sync_ext_to_int.sh

mount /dev/mapper/int intern/

ls intern/root-ro/anothernewfile
# output intern/root-ro/anothernewfile -> sync ext->int was successful

umount intern

sh sync_int_to_ext.sh # just to make sure

# boot into extern/root-rw with rootflags=subvol/root-rw
# both newfile and anothernewfile are visible in the root-fs

# reboot into other OS used for syncing disks

sh sync_ext_to_int.sh #to mirror modifications made in ext during it
was rootfs back to intern, worked
sh sync_int_to_ext.sh # just to make sure

# boot into extern/root-rw with rootflags=subvol/root-rw
# perform a few modifications

# reboot into other OS used for syncing disks

sh sync_ext_to_int.sh
# command btrfs send -p extern/root-ro extern/root-ro-new | btrfs
receive intern/ fails with:
ERROR: clone: cannot find source subvol 29fca96e-ca6a-3d4b-b7c9-566f1240d978


btrfs sub list -pqu intern/
ID 330 gen 426 parent 5 top level 5 parent_uuid
29fca96e-ca6a-3d4b-b7c9-566f1240d978 uuid
6409bfb7-1af0-7e4b-8a0f-d5a44e34a15c path root-ro
ID 331 gen 426 parent 5 top level 5 parent_uuid
6409bfb7-1af0-7e4b-8a0f-d5a44e34a15c uuid
258c1fe5-b14e-654b-8ad5-5591268c9095 path root-rw

btrfs sub list -pqu extern/
ID 291 gen 418 parent 5 top level 5 parent_uuid
1c738933-0bb2-2547-ad5f-326bfc6263b3 uuid
939c1ed9-9589-6c4b-ace7-2fcdeb970303 path root-rw
ID 292 gen 418 parent 5 top level 5 parent_uuid
939c1ed9-9589-6c4b-ace7-2fcdeb970303 uuid
155f4370-32f0-5f4b-b288-2e8d7302d279 path root-ro

Best regards, Clemens



script sync_ext_to_int.sh:

unalias cp
mount -o compress=zstd:6 /dev/mapper/ext extern/
mount -o compress=zstd:6 /dev/mapper/int intern/

btrfs sub snap -r extern/root-rw extern/root-ro-new
btrfs send -p extern/root-ro extern/root-ro-new | btrfs receive intern/

btrfs sub del extern/root-ro
mv extern/root-ro-new extern/root-ro

btrfs sub del intern/root-ro
mv intern/root-ro-new intern/root-ro
btrfs sub del intern/root-rw
btrfs sub snap intern/root-ro intern/root-rw

#cp cfgint/fstab intern/root-rw/etc/
#cp cfgint/crypttab intern/root-rw/etc/
#cp cfgint/grub intern/root-rw/etc/default/

umount extern;
umount intern;



script sync_int_to_ext.sh:

unalias cp
mount -o compress=zstd:6 /dev/mapper/ext extern/
mount -o compress=zstd:6 /dev/mapper/int intern/

btrfs sub snap -r intern/root-rw intern/root-ro-new
btrfs send -p intern/root-ro intern/root-ro-new | btrfs receive extern/

btrfs sub del intern/root-ro
mv intern/root-ro-new intern/root-ro

btrfs sub del extern/root-ro
mv extern/root-ro-new extern/root-ro
btrfs sub del extern/root-rw
btrfs sub snap extern/root-ro extern/root-rw

cp cfgext/fstab extern/root-rw/etc/
cp cfgext/crypttab extern/root-rw/etc/
cp cfgext/grub extern/root-rw/etc/default/

umount extern;
umount intern;

Am So., 7. Jan. 2024 um 08:19 Uhr schrieb Andrei Borzenkov
<arvidjaar@gmail.com>:
>
> On 07.01.2024 10:06, Clemens Eisserer wrote:
> > Hi,
> >
> > I would like to use send/receive to keep two root-filesystems in sync,
> > as I've been using it for years now for backups where it really does
> > wonders (thanks a lot!).
> >
> > Both disks contain a read-only snapshot which is kept in-sync between
> > the filesystems (int and ext are the mountpoints of the two disks,
> > original_disk is just used for initial data):
> >     btrfs send original_disk/root-ro | btrfs receive int/ #send
> > snapshot of the original disk to the first of the two filesystens
> > (disk "int")
> >     btrfs send int/root-ro | btrfs receive ext/ #now replicate the same
> > to disk "ext"
> > so both disks start with a snapshot "root-ro" with equal content.
> >
> > in case I would like to work with one of the two disks, I create a rw
> > snapshot based no root-ro:
> >    btrfs sub snap ext/root-ro ext/root-rw
> >
> >    touch ext/root-wr/create-a-new-file # perform some modifications
> >
>
> There was no "root-wr" before.
>
> > once modifications in root-rw are done, the following steps are
> > performed to sync the filesystems:
> >    btrfs sub snap -r ext/root-rw ext/root-ro-new #create a root-ro-new
> > read-only snapshot based on the rw-snapshot with modfications (so it
> > can be used with btrfs send)
> >    btrfs send -p ext/root-ro extern/root-ro-new | btrfs receive int/
> > #send root-ro-new to "int" filesystem
>
> There was no "extern" before.
>
> Never describe computer commands. Copy and paste them in full with
> complete output.
>
> >    btrfs sub del ext/root-ro # delete the original root-ro snapshot, as
> > it is no longer needed for differential btrfs send
> >    mv ext/root-ro-new ext/root-ro #rename root-ro-new to root-ro, as
> > this is the current state of the other (int) filesystem
> >    btrfs sub del int/root-ro # delete root-ro in "int" too, as it is no
> > longer needed for differential btrfs receive
> >    mv int/root-ro-new int/root-ro #rename root-ro-new to root-ro
> >    btrfs sub snap int/root-ro int/root-rw # create a working copy of
> > root-ro which is writeable
> >
> > this works great - i can add/modify files in one root-rw folder, call
> > the synchronization script and everything is found on the other
> > filesystem.
> > When exchanging int and ext in the script above it actually works in
> > both directions, so this is exactly what I was hoping to achieve.
> > Even when executing the script multiple times int->ext, ext->int,
> > int->ext ... with modifications in between, everything works as
> > expected.
> > Awesome :)
> >
> > However, when actually using the file-systems as rootfs, this seem to
> > break when performing the following steps:
> > - create rw snapshot of root-ro on disk "ext": btrfs sub snap
> > ext/root-ro ext/root-rw
> > - boot the system with rootfs=ext-uuid and rootflags=subvol=/root-rw
> > (etc/fstab was adapted accordingly)
> > - use the system, modify file system etc and shutdown again
> > - start separate system to synchronize disks (not based on int or ext
> > rootfs) and call sync script ext->int (shown above)
> >
>
> It is absolutely unclear what it means. As mentioned, provide full
> transcript of your actions as well as the output of
>
> btrfs subvolume list -pqu /mount-point
>
> for all filesystems involved in these commands.
>
> > it now suddenly fails at btrfs receive with:
> > btrfs send -p ext/root-ro ext/root-ro-new | btrfs receive intern/
> > ERROR: snapshot: cannot find parent subvolume
> > 4ed11491-7563-fb49-99e7-86cb47cfb510
> >
> > which I, to be honest, don't understand.
> > Exactly the same sequence of commands worked multiple times when the
> > root-rw snapshot was not booted from but modified directly on the sync
> > system, even with umounts between modification & send/receive.
> > Does it make a difference for btrfs if it is used as rootfs vs normal
> > writeable mount?
> > Or does it just work in the non-boot case because of some side-effects?
> >
> > Thanks & best regards, Clemens
> >
>

Re: Using send/receive to keep two rootfs-partitions in sync fails with "ERROR: snapshot: cannot find parent subvolume"

[Andrei Borzenkov]

On 07.01.2024 11:10, Clemens Eisserer wrote:
> Hello Andrei,
> 
> Thanks for taking a look.
> 
> The exact commands executed where:
> 
> mkdir intern
> mkdir extern
> 
> mount /dev/mapper/ext extern
> mout /dev/mapper/int intern
> 
> ls extern/
> # output: root-ro
> 
> ls intern/
> # output: -> empty
> 
> btrfs send extern/root-ro | btrfs receive intern/ #initial ext -> int
> btrfs sub snap intern/root-ro intern/root-rw # rw snapshot to modify int
> touch intern/root-rw/newfile #actual modification
> 
> umount intern
> umount extern
> 
> sh sync_int_to_ext.sh #source of script at bottom
> 
> ls extern/root-ro/newfile
> # output: extern/root-ro/newfile -> sync int->ext was successful
> 
> # now modify ext fs
> mount /dev/mapper/ext extern/
> touch extern/root-rw/anothernewfile
> umount extern
> 
> sh sync_ext_to_int.sh
> 
> mount /dev/mapper/int intern/
> 
> ls intern/root-ro/anothernewfile
> # output intern/root-ro/anothernewfile -> sync ext->int was successful
> 
> umount intern
> 
> sh sync_int_to_ext.sh # just to make sure
> 
> # boot into extern/root-rw with rootflags=subvol/root-rw
> # both newfile and anothernewfile are visible in the root-fs
> 
> # reboot into other OS used for syncing disks
> 
> sh sync_ext_to_int.sh #to mirror modifications made in ext during it
> was rootfs back to intern, worked
> sh sync_int_to_ext.sh # just to make sure
> 
> # boot into extern/root-rw with rootflags=subvol/root-rw
> # perform a few modifications
> 
> # reboot into other OS used for syncing disks
> 
> sh sync_ext_to_int.sh
> # command btrfs send -p extern/root-ro extern/root-ro-new | btrfs
> receive intern/ fails with:
> ERROR: clone: cannot find source subvol 29fca96e-ca6a-3d4b-b7c9-566f1240d978
> 
> 
> btrfs sub list -pqu intern/
> ID 330 gen 426 parent 5 top level 5 parent_uuid
> 29fca96e-ca6a-3d4b-b7c9-566f1240d978 uuid
> 6409bfb7-1af0-7e4b-8a0f-d5a44e34a15c path root-ro
> ID 331 gen 426 parent 5 top level 5 parent_uuid
> 6409bfb7-1af0-7e4b-8a0f-d5a44e34a15c uuid
> 258c1fe5-b14e-654b-8ad5-5591268c9095 path root-rw
> 
> btrfs sub list -pqu extern/
> ID 291 gen 418 parent 5 top level 5 parent_uuid
> 1c738933-0bb2-2547-ad5f-326bfc6263b3 uuid
> 939c1ed9-9589-6c4b-ace7-2fcdeb970303 path root-rw
> ID 292 gen 418 parent 5 top level 5 parent_uuid
> 939c1ed9-9589-6c4b-ace7-2fcdeb970303 uuid
> 155f4370-32f0-5f4b-b288-2e8d7302d279 path root-ro
> 
> Best regards, Clemens
> 
> 
> 
> script sync_ext_to_int.sh:
> 
> unalias cp
> mount -o compress=zstd:6 /dev/mapper/ext extern/
> mount -o compress=zstd:6 /dev/mapper/int intern/
> 
> btrfs sub snap -r extern/root-rw extern/root-ro-new
> btrfs send -p extern/root-ro extern/root-ro-new | btrfs receive intern/
> 
> btrfs sub del extern/root-ro
> mv extern/root-ro-new extern/root-ro
> 
> btrfs sub del intern/root-ro
> mv intern/root-ro-new intern/root-ro
> btrfs sub del intern/root-rw
> btrfs sub snap intern/root-ro intern/root-rw
> 
> #cp cfgint/fstab intern/root-rw/etc/
> #cp cfgint/crypttab intern/root-rw/etc/
> #cp cfgint/grub intern/root-rw/etc/default/
> 
> umount extern;
> umount intern;
> 
> 
> 
> script sync_int_to_ext.sh:
> 
> unalias cp
> mount -o compress=zstd:6 /dev/mapper/ext extern/
> mount -o compress=zstd:6 /dev/mapper/int intern/
> 
> btrfs sub snap -r intern/root-rw intern/root-ro-new
> btrfs send -p intern/root-ro intern/root-ro-new | btrfs receive extern/
> 
> btrfs sub del intern/root-ro
> mv intern/root-ro-new intern/root-ro
> 
> btrfs sub del extern/root-ro
> mv extern/root-ro-new extern/root-ro
> btrfs sub del extern/root-rw
> btrfs sub snap extern/root-ro extern/root-rw
> 
> cp cfgext/fstab extern/root-rw/etc/
> cp cfgext/crypttab extern/root-rw/etc/
> cp cfgext/grub extern/root-rw/etc/default/
> 
> umount extern;
> umount intern;
> 
> Am So., 7. Jan. 2024 um 08:19 Uhr schrieb Andrei Borzenkov
> <arvidjaar@gmail.com>:
>>
>> On 07.01.2024 10:06, Clemens Eisserer wrote:
>>> Hi,
>>>
>>> I would like to use send/receive to keep two root-filesystems in sync,
>>> as I've been using it for years now for backups where it really does
>>> wonders (thanks a lot!).
>>>
>>> Both disks contain a read-only snapshot which is kept in-sync between
>>> the filesystems (int and ext are the mountpoints of the two disks,
>>> original_disk is just used for initial data):
>>>      btrfs send original_disk/root-ro | btrfs receive int/ #send
>>> snapshot of the original disk to the first of the two filesystens
>>> (disk "int")
>>>      btrfs send int/root-ro | btrfs receive ext/ #now replicate the same
>>> to disk "ext"
>>> so both disks start with a snapshot "root-ro" with equal content.
>>>
>>> in case I would like to work with one of the two disks, I create a rw
>>> snapshot based no root-ro:
>>>     btrfs sub snap ext/root-ro ext/root-rw
>>>
>>>     touch ext/root-wr/create-a-new-file # perform some modifications
>>>
>>
>> There was no "root-wr" before.
>>
>>> once modifications in root-rw are done, the following steps are
>>> performed to sync the filesystems:
>>>     btrfs sub snap -r ext/root-rw ext/root-ro-new #create a root-ro-new
>>> read-only snapshot based on the rw-snapshot with modfications (so it
>>> can be used with btrfs send)
>>>     btrfs send -p ext/root-ro extern/root-ro-new | btrfs receive int/
>>> #send root-ro-new to "int" filesystem
>>
>> There was no "extern" before.
>>
>> Never describe computer commands. Copy and paste them in full with
>> complete output.
>>
>>>     btrfs sub del ext/root-ro # delete the original root-ro snapshot, as
>>> it is no longer needed for differential btrfs send
>>>     mv ext/root-ro-new ext/root-ro #rename root-ro-new to root-ro, as
>>> this is the current state of the other (int) filesystem
>>>     btrfs sub del int/root-ro # delete root-ro in "int" too, as it is no
>>> longer needed for differential btrfs receive
>>>     mv int/root-ro-new int/root-ro #rename root-ro-new to root-ro
>>>     btrfs sub snap int/root-ro int/root-rw # create a working copy of
>>> root-ro which is writeable
>>>
>>> this works great - i can add/modify files in one root-rw folder, call
>>> the synchronization script and everything is found on the other
>>> filesystem.
>>> When exchanging int and ext in the script above it actually works in
>>> both directions, so this is exactly what I was hoping to achieve.
>>> Even when executing the script multiple times int->ext, ext->int,
>>> int->ext ... with modifications in between, everything works as
>>> expected.
>>> Awesome :)
>>>
>>> However, when actually using the file-systems as rootfs, this seem to
>>> break when performing the following steps:
>>> - create rw snapshot of root-ro on disk "ext": btrfs sub snap
>>> ext/root-ro ext/root-rw
>>> - boot the system with rootfs=ext-uuid and rootflags=subvol=/root-rw
>>> (etc/fstab was adapted accordingly)
>>> - use the system, modify file system etc and shutdown again
>>> - start separate system to synchronize disks (not based on int or ext
>>> rootfs) and call sync script ext->int (shown above)
>>>
>>
>> It is absolutely unclear what it means. As mentioned, provide full
>> transcript of your actions as well as the output of
>>
>> btrfs subvolume list -pqu /mount-point
>>

Apologies, could you post

btrfs subvolume list -pqRu ...

to get received UUID.

>> for all filesystems involved in these commands.
>>
>>> it now suddenly fails at btrfs receive with:
>>> btrfs send -p ext/root-ro ext/root-ro-new | btrfs receive intern/
>>> ERROR: snapshot: cannot find parent subvolume
>>> 4ed11491-7563-fb49-99e7-86cb47cfb510
>>>
>>> which I, to be honest, don't understand.
>>> Exactly the same sequence of commands worked multiple times when the
>>> root-rw snapshot was not booted from but modified directly on the sync
>>> system, even with umounts between modification & send/receive.
>>> Does it make a difference for btrfs if it is used as rootfs vs normal
>>> writeable mount?
>>> Or does it just work in the non-boot case because of some side-effects?
>>>
>>> Thanks & best regards, Clemens
>>>
>>
> 

Re: Using send/receive to keep two rootfs-partitions in sync fails with "ERROR: snapshot: cannot find parent subvolume"

[Clemens Eisserer]

Hi Andrei,

Sure, the output is:

root@fedora:/home/ce# btrfs sub list -pqRu extern/
ID 291 gen 418 parent 5 top level 5 parent_uuid
1c738933-0bb2-2547-ad5f-326bfc6263b3 received_uuid -  uuid
939c1ed9-9589-6c4b-ace7-2fcdeb970303 path root-rw
ID 292 gen 418 parent 5 top level 5 parent_uuid
939c1ed9-9589-6c4b-ace7-2fcdeb970303 received_uuid - uuid
155f4370-32f0-5f4b-b288-2e8d7302d279 path root-ro

btrfs sub list -pqRu intern/
ID 330 gen 426 parent 5 top level 5 parent_uuid
29fca96e-ca6a-3d4b-b7c9-566f1240d978 received_uuid - uuid
6409bfb7-1af0-7e4b-8a0f-d5a44e34a15c path root-ro
ID 331 gen 426 parent 5 top level 5 parent_uuid
6409bfb7-1af0-7e4b-8a0f-d5a44e34a15c received_uuid -  uuid
258c1fe5-b14e-654b-8ad5-5591268c9095 path root-rw

Thanks, Clemens

Re: Using send/receive to keep two rootfs-partitions in sync fails with "ERROR: snapshot: cannot find parent subvolume"

[Clemens Eisserer]

Hi again,

I've reproduced it now a few times and still can't draw any
conclusions from the observed behaviour.
I can modify the rw-snapshots manually or even chroot into them,
install packages via dnf etc and everything works as expected, the
sync scripts run through without any issues a dozen times - so my
guess is the scripts actually do what they are supposed to.

However once I've used a rw-snapshot once or twice(?) as "real" rootfs
(not chroot) everything falls apart later when executing the scripts
with "ERROR: snapshot: cannot find parent subvolume".

Could it be that fedora (the distribution contained in the subvolumes
) as part of its boot process somehow modifies the subvolume or
performs operations within the subvolume which disturb a offline
send/receive performed later? As far as I know I am not using software
which explicitly uses btrfs-fetures like snapper - the only thing I
could think of is podman (running with unchanged default config).

Thanks & best regards, Clemens

Re: Using send/receive to keep two rootfs-partitions in sync fails with "ERROR: snapshot: cannot find parent subvolume"

[Andrei Borzenkov]

On 07.01.2024 11:10, Clemens Eisserer wrote:
> Hello Andrei,
> 
> Thanks for taking a look.
> 
> The exact commands executed where:
> 
> mkdir intern
> mkdir extern
> 
> mount /dev/mapper/ext extern
> mout /dev/mapper/int intern
> 
> ls extern/
> # output: root-ro
> 
> ls intern/
> # output: -> empty
> 
> btrfs send extern/root-ro | btrfs receive intern/ #initial ext -> int
> btrfs sub snap intern/root-ro intern/root-rw # rw snapshot to modify int
> touch intern/root-rw/newfile #actual modification
> 
> umount intern
> umount extern
> 
> sh sync_int_to_ext.sh #source of script at bottom
> 
> ls extern/root-ro/newfile

That cannot show anything because previous script unmounts "extern".

> # output: extern/root-ro/newfile -> sync int->ext was successful
>  > # now modify ext fs
> mount /dev/mapper/ext extern/
> touch extern/root-rw/anothernewfile
> umount extern
> 
> sh sync_ext_to_int.sh
> 
> mount /dev/mapper/int intern/
> 
> ls intern/root-ro/anothernewfile
> # output intern/root-ro/anothernewfile -> sync ext->int was successful
> 
> umount intern
> 
> sh sync_int_to_ext.sh # just to make sure
> 
> # boot into extern/root-rw with rootflags=subvol/root-rw
> # both newfile and anothernewfile are visible in the root-fs
> 
> # reboot into other OS used for syncing disks
> 
> sh sync_ext_to_int.sh #to mirror modifications made in ext during it
> was rootfs back to intern, worked
> sh sync_int_to_ext.sh # just to make sure
> 
> # boot into extern/root-rw with rootflags=subvol/root-rw
> # perform a few modifications
> 
> # reboot into other OS used for syncing disks
> 
> sh sync_ext_to_int.sh
> # command btrfs send -p extern/root-ro extern/root-ro-new | btrfs
> receive intern/ fails with:
> ERROR: clone: cannot find source subvol 29fca96e-ca6a-3d4b-b7c9-566f1240d978
> 
> 
> btrfs sub list -pqu intern/
> ID 330 gen 426 parent 5 top level 5 parent_uuid
> 29fca96e-ca6a-3d4b-b7c9-566f1240d978 uuid
> 6409bfb7-1af0-7e4b-8a0f-d5a44e34a15c path root-ro
> ID 331 gen 426 parent 5 top level 5 parent_uuid
> 6409bfb7-1af0-7e4b-8a0f-d5a44e34a15c uuid
> 258c1fe5-b14e-654b-8ad5-5591268c9095 path root-rw
> 
> btrfs sub list -pqu extern/
> ID 291 gen 418 parent 5 top level 5 parent_uuid
> 1c738933-0bb2-2547-ad5f-326bfc6263b3 uuid
> 939c1ed9-9589-6c4b-ace7-2fcdeb970303 path root-rw
> ID 292 gen 418 parent 5 top level 5 parent_uuid
> 939c1ed9-9589-6c4b-ace7-2fcdeb970303 uuid
> 155f4370-32f0-5f4b-b288-2e8d7302d279 path root-ro
> 


The error is correct. There is no subvolume with UUID 
29fca96e-ca6a-3d4b-b7c9-566f1240d978 and according to the output in your 
other mail there are no received UUIDs which breaks send/receive chain.

Unfortunately the output you sent was *after* your script already 
destroyed the original state of both filesystems - it recreated 
subvolumes without running successful send/receive first. So we still do 
not know the state when error happened.

Modify your scripts to output "btrfs sub list -pqRu ..." for both 
filesystems at the very beginning, before doing anything, and post full 
output.

> Best regards, Clemens
> 
> 
> 
> script sync_ext_to_int.sh:
> 
> unalias cp
> mount -o compress=zstd:6 /dev/mapper/ext extern/
> mount -o compress=zstd:6 /dev/mapper/int intern/
> 
> btrfs sub snap -r extern/root-rw extern/root-ro-new
> btrfs send -p extern/root-ro extern/root-ro-new | btrfs receive intern/
> 
> btrfs sub del extern/root-ro
> mv extern/root-ro-new extern/root-ro
> 
> btrfs sub del intern/root-ro
> mv intern/root-ro-new intern/root-ro
> btrfs sub del intern/root-rw
> btrfs sub snap intern/root-ro intern/root-rw
> 
> #cp cfgint/fstab intern/root-rw/etc/
> #cp cfgint/crypttab intern/root-rw/etc/
> #cp cfgint/grub intern/root-rw/etc/default/
> 
> umount extern;
> umount intern;
> 
> 
> 
> script sync_int_to_ext.sh:
> 
> unalias cp
> mount -o compress=zstd:6 /dev/mapper/ext extern/
> mount -o compress=zstd:6 /dev/mapper/int intern/
> 
> btrfs sub snap -r intern/root-rw intern/root-ro-new
> btrfs send -p intern/root-ro intern/root-ro-new | btrfs receive extern/
> 
> btrfs sub del intern/root-ro
> mv intern/root-ro-new intern/root-ro
> 
> btrfs sub del extern/root-ro
> mv extern/root-ro-new extern/root-ro
> btrfs sub del extern/root-rw
> btrfs sub snap extern/root-ro extern/root-rw
> 
> cp cfgext/fstab extern/root-rw/etc/
> cp cfgext/crypttab extern/root-rw/etc/
> cp cfgext/grub extern/root-rw/etc/default/
> 
> umount extern;
> umount intern;
> 
> Am So., 7. Jan. 2024 um 08:19 Uhr schrieb Andrei Borzenkov
> <arvidjaar@gmail.com>:
>>
>> On 07.01.2024 10:06, Clemens Eisserer wrote:
>>> Hi,
>>>
>>> I would like to use send/receive to keep two root-filesystems in sync,
>>> as I've been using it for years now for backups where it really does
>>> wonders (thanks a lot!).
>>>
>>> Both disks contain a read-only snapshot which is kept in-sync between
>>> the filesystems (int and ext are the mountpoints of the two disks,
>>> original_disk is just used for initial data):
>>>      btrfs send original_disk/root-ro | btrfs receive int/ #send
>>> snapshot of the original disk to the first of the two filesystens
>>> (disk "int")
>>>      btrfs send int/root-ro | btrfs receive ext/ #now replicate the same
>>> to disk "ext"
>>> so both disks start with a snapshot "root-ro" with equal content.
>>>
>>> in case I would like to work with one of the two disks, I create a rw
>>> snapshot based no root-ro:
>>>     btrfs sub snap ext/root-ro ext/root-rw
>>>
>>>     touch ext/root-wr/create-a-new-file # perform some modifications
>>>
>>
>> There was no "root-wr" before.
>>
>>> once modifications in root-rw are done, the following steps are
>>> performed to sync the filesystems:
>>>     btrfs sub snap -r ext/root-rw ext/root-ro-new #create a root-ro-new
>>> read-only snapshot based on the rw-snapshot with modfications (so it
>>> can be used with btrfs send)
>>>     btrfs send -p ext/root-ro extern/root-ro-new | btrfs receive int/
>>> #send root-ro-new to "int" filesystem
>>
>> There was no "extern" before.
>>
>> Never describe computer commands. Copy and paste them in full with
>> complete output.
>>
>>>     btrfs sub del ext/root-ro # delete the original root-ro snapshot, as
>>> it is no longer needed for differential btrfs send
>>>     mv ext/root-ro-new ext/root-ro #rename root-ro-new to root-ro, as
>>> this is the current state of the other (int) filesystem
>>>     btrfs sub del int/root-ro # delete root-ro in "int" too, as it is no
>>> longer needed for differential btrfs receive
>>>     mv int/root-ro-new int/root-ro #rename root-ro-new to root-ro
>>>     btrfs sub snap int/root-ro int/root-rw # create a working copy of
>>> root-ro which is writeable
>>>
>>> this works great - i can add/modify files in one root-rw folder, call
>>> the synchronization script and everything is found on the other
>>> filesystem.
>>> When exchanging int and ext in the script above it actually works in
>>> both directions, so this is exactly what I was hoping to achieve.
>>> Even when executing the script multiple times int->ext, ext->int,
>>> int->ext ... with modifications in between, everything works as
>>> expected.
>>> Awesome :)
>>>
>>> However, when actually using the file-systems as rootfs, this seem to
>>> break when performing the following steps:
>>> - create rw snapshot of root-ro on disk "ext": btrfs sub snap
>>> ext/root-ro ext/root-rw
>>> - boot the system with rootfs=ext-uuid and rootflags=subvol=/root-rw
>>> (etc/fstab was adapted accordingly)
>>> - use the system, modify file system etc and shutdown again
>>> - start separate system to synchronize disks (not based on int or ext
>>> rootfs) and call sync script ext->int (shown above)
>>>
>>
>> It is absolutely unclear what it means. As mentioned, provide full
>> transcript of your actions as well as the output of
>>
>> btrfs subvolume list -pqu /mount-point
>>
>> for all filesystems involved in these commands.
>>
>>> it now suddenly fails at btrfs receive with:
>>> btrfs send -p ext/root-ro ext/root-ro-new | btrfs receive intern/
>>> ERROR: snapshot: cannot find parent subvolume
>>> 4ed11491-7563-fb49-99e7-86cb47cfb510
>>>
>>> which I, to be honest, don't understand.
>>> Exactly the same sequence of commands worked multiple times when the
>>> root-rw snapshot was not booted from but modified directly on the sync
>>> system, even with umounts between modification & send/receive.
>>> Does it make a difference for btrfs if it is used as rootfs vs normal
>>> writeable mount?
>>> Or does it just work in the non-boot case because of some side-effects?
>>>
>>> Thanks & best regards, Clemens
>>>
>>
> 

Re: Using send/receive to keep two rootfs-partitions in sync fails with "ERROR: snapshot: cannot find parent subvolume"

[Clemens Eisserer]

Hi Andrei,

> The error is correct. There is no subvolume with UUID
> 29fca96e-ca6a-3d4b-b7c9-566f1240d978 and according to the output in your
> other mail there are no received UUIDs which breaks send/receive chain.
>
> Unfortunately the output you sent was *after* your script already
> destroyed the original state of both filesystems - it recreated
> subvolumes without running successful send/receive first. So we still do
> not know the state when error happened.

I am really sorry, I didn't think of this.

I've followed your suggestions and tried to reproduce it once again.
This time I booted from extern/root-rw two times and performed
ext->int->ext in between, but didn't manually change files like in the
previous runs.
After the second boot send/receive failed as expected. I've omitted
manual mounts/unmounts this time from the command-list.


I would be really glad if you could - once again - have a look at the results:

btrfs send source_disk/root-ro | btrfs receive extern/ #initial source->ext
btrfs send extern/root-ro | btrfs receive intern/ #initial ext -> int
btrfs sub snap extern/root-ro extern/root-rw # rw snapshot to modify
ext, will be used as subvol when booting
btrfs sub snap intern/root-ro intern/root-rw # rw snapshot to modify
int, won't be used

# boot fedora from extern (+ install firefox via dnf) + shutdown again

btrfs subvolume list -pqRu intern/
ID 386 gen 576 parent 5 top level 5 parent_uuid -
              received_uuid 8d1ee193-2522-014b-b436-032f0a4fc461 uuid
a88bb4ed-65c8-db4f-b209-9ca07eaf3d21 path root-ro
ID 387 gen 576 parent 5 top level 5 parent_uuid
a88bb4ed-65c8-db4f-b209-9ca07eaf3d21 received_uuid -
                 uuid f4d1c9cc-3e4b-4248-9b02-626bed3ad238 path
root-rw

btrfs subvolume list -pqRu extern/
ID 345 gen 588 parent 5 top level 5 parent_uuid -
              received_uuid 8d1ee193-2522-014b-b436-032f0a4fc461 uuid
21cabdfd-02f5-ab4b-943a-6ebf88326dae path root-ro
ID 346 gen 611 parent 5 top level 5 parent_uuid
21cabdfd-02f5-ab4b-943a-6ebf88326dae received_uuid -
                 uuid 0c9d9843-32f3-d341-b657-a33de46c9d0e path
root-rw

sh sync_ext_to_int.sh

btrfs subvolume list -pqRu intern/
ID 388 gen 583 parent 5 top level 5 parent_uuid
a88bb4ed-65c8-db4f-b209-9ca07eaf3d21 received_uuid
b326e1fe-7295-3948-a401-b6de850e213c uuid
f055e3e8-d048-744c-abb4-c3efd87c4875 path root-ro
ID 389 gen 583 parent 5 top level 5 parent_uuid
f055e3e8-d048-744c-abb4-c3efd87c4875 received_uuid -
                 uuid da51f30f-f10a-b04c-b7a7-5e718118a49c path
root-rw

btrfs subvolume list -pqRu extern/
ID 346 gen 615 parent 5 top level 5 parent_uuid
21cabdfd-02f5-ab4b-943a-6ebf88326dae received_uuid -
                 uuid 0c9d9843-32f3-d341-b657-a33de46c9d0e path
root-rw
ID 347 gen 615 parent 5 top level 5 parent_uuid
0c9d9843-32f3-d341-b657-a33de46c9d0e received_uuid -
                 uuid b326e1fe-7295-3948-a401-b6de850e213c path
root-ro

sh sync_int_to_ext.sh (intern was not changed, just to keep ext->int,
int->ext in order)

btrfs subvolume list -pqRu intern/
ID 389 gen 588 parent 5 top level 5 parent_uuid
f055e3e8-d048-744c-abb4-c3efd87c4875 received_uuid -
                 uuid da51f30f-f10a-b04c-b7a7-5e718118a49c path
root-rw
ID 390 gen 588 parent 5 top level 5 parent_uuid
da51f30f-f10a-b04c-b7a7-5e718118a49c received_uuid -
                 uuid 8a1f263b-581d-1847-a0da-0fc17e1ca2c4 path
root-ro

btrfs subvolume list -pqRu extern/
ID 348 gen 623 parent 5 top level 5 parent_uuid
b326e1fe-7295-3948-a401-b6de850e213c received_uuid
8a1f263b-581d-1847-a0da-0fc17e1ca2c4 uuid
c481acb9-b077-7841-a4b6-a0273f7aa12f path root-ro
ID 349 gen 623 parent 5 top level 5 parent_uuid
c481acb9-b077-7841-a4b6-a0273f7aa12f received_uuid -
                 uuid cf80f17f-3d5d-7744-83f4-096b7688288c path
root-rw

# boot fedora from extern (remove google-chrome and chromium via dnf)
+ shutdown again

btrfs subvolume list -pqRu extern/
ID 348 gen 623 parent 5 top level 5 parent_uuid
b326e1fe-7295-3948-a401-b6de850e213c received_uuid
8a1f263b-581d-1847-a0da-0fc17e1ca2c4 uuid
c481acb9-b077-7841-a4b6-a0273f7aa12f path root-ro
ID 349 gen 633 parent 5 top level 5 parent_uuid
c481acb9-b077-7841-a4b6-a0273f7aa12f received_uuid -
                 uuid cf80f17f-3d5d-7744-83f4-096b7688288c path
root-rw

sh sync_ext_to_int.sh

manually executed line-by-line in expecting it would fail it failed at:
btrfs send -p extern/root-ro extern/root-ro-new | btrfs receive intern/
At subvol extern/root-ro-new
At snapshot root-ro-new
ERROR: clone: cannot find source subvol 8a1f263b-581d-1847-a0da-0fc17e1ca2c4

interestingly, despite send-receive failed/aborted with the message
above, intern/root-ro-new was created and contains contents but left
in writable state:

btrfs sub show intern/root-ro-new/
root-ro-new
        Name:                   root-ro-new
        UUID:                   d32788b9-5572-5743-bf27-46b57c8b92c1
        Parent UUID:            8a1f263b-581d-1847-a0da-0fc17e1ca2c4
        Received UUID:          -
        Creation time:          2024-01-08 20:14:59 +0100
        Subvolume ID:           391
        Generation:             596
        Gen at creation:        594
        Parent ID:              5
        Top level ID:           5
        Flags:                  -
        Send transid:           0
        Send time:              2024-01-08 20:14:59 +0100
        Receive transid:        0
        Receive time:           -
        Snapshot(s):
        Quota group:            n/a


btrfs subvolume list -pqRu intern/
ID 389 gen 588 parent 5 top level 5 parent_uuid
f055e3e8-d048-744c-abb4-c3efd87c4875 received_uuid -
                 uuid da51f30f-f10a-b04c-b7a7-5e718118a49c path
root-rw
ID 390 gen 594 parent 5 top level 5 parent_uuid
da51f30f-f10a-b04c-b7a7-5e718118a49c received_uuid -
                 uuid 8a1f263b-581d-1847-a0da-0fc17e1ca2c4 path
root-ro
ID 391 gen 595 parent 5 top level 5 parent_uuid
8a1f263b-581d-1847-a0da-0fc17e1ca2c4 received_uuid -
                 uuid d32788b9-5572-5743-bf27-46b57c8b92c1 path
root-ro-new


btrfs subvolume list -pqRu extern/
ID 348 gen 623 parent 5 top level 5 parent_uuid
b326e1fe-7295-3948-a401-b6de850e213c received_uuid
8a1f263b-581d-1847-a0da-0fc17e1ca2c4 uuid
c481acb9-b077-7841-a4b6-a0273f7aa12f path root-ro
ID 349 gen 636 parent 5 top level 5 parent_uuid
c481acb9-b077-7841-a4b6-a0273f7aa12f received_uuid -
                 uuid cf80f17f-3d5d-7744-83f4-096b7688288c path
root-rw
ID 350 gen 636 parent 5 top level 5 parent_uuid
cf80f17f-3d5d-7744-83f4-096b7688288c received_uuid -
                 uuid d73b4afe-0837-8046-b798-21102cb82d4d path
root-ro-new

Looking at the last output I have to admit I am confused.
The error mentioned missing parent with
8a1f263b-581d-1847-a0da-0fc17e1ca2c4 for intern/root-ro-new seems to
be there. Isn't this, as expected intern/root-ro with
8a1f263b-581d-1847-a0da-0fc17e1ca2c4.

Thanks for all your patience and help!

Best regards, Clemens

Re: Using send/receive to keep two rootfs-partitions in sync fails with "ERROR: snapshot: cannot find parent subvolume"

[Andrei Borzenkov]

On 08.01.2024 22:36, Clemens Eisserer wrote:
> Hi Andrei,
> 
>> The error is correct. There is no subvolume with UUID
>> 29fca96e-ca6a-3d4b-b7c9-566f1240d978 and according to the output in your
>> other mail there are no received UUIDs which breaks send/receive chain.
>>
>> Unfortunately the output you sent was *after* your script already
>> destroyed the original state of both filesystems - it recreated
>> subvolumes without running successful send/receive first. So we still do
>> not know the state when error happened.
> 
> I am really sorry, I didn't think of this.
> 
> I've followed your suggestions and tried to reproduce it once again.
> This time I booted from extern/root-rw two times and performed
> ext->int->ext in between, but didn't manually change files like in the
> previous runs.
> After the second boot send/receive failed as expected. I've omitted
> manual mounts/unmounts this time from the command-list.
> 
> 
> I would be really glad if you could - once again - have a look at the results:
> 
> btrfs send source_disk/root-ro | btrfs receive extern/ #initial source->ext
> btrfs send extern/root-ro | btrfs receive intern/ #initial ext -> int
> btrfs sub snap extern/root-ro extern/root-rw # rw snapshot to modify
> ext, will be used as subvol when booting
> btrfs sub snap intern/root-ro intern/root-rw # rw snapshot to modify
> int, won't be used
> 
> # boot fedora from extern (+ install firefox via dnf) + shutdown again
> 
> btrfs subvolume list -pqRu intern/
> ID 386 gen 576 parent 5 top level 5 parent_uuid -
>                received_uuid 8d1ee193-2522-014b-b436-032f0a4fc461 uuid
> a88bb4ed-65c8-db4f-b209-9ca07eaf3d21 path root-ro
> ID 387 gen 576 parent 5 top level 5 parent_uuid
> a88bb4ed-65c8-db4f-b209-9ca07eaf3d21 received_uuid -
>                   uuid f4d1c9cc-3e4b-4248-9b02-626bed3ad238 path
> root-rw
> 
> btrfs subvolume list -pqRu extern/
> ID 345 gen 588 parent 5 top level 5 parent_uuid -
>                received_uuid 8d1ee193-2522-014b-b436-032f0a4fc461 uuid
> 21cabdfd-02f5-ab4b-943a-6ebf88326dae path root-ro
> ID 346 gen 611 parent 5 top level 5 parent_uuid
> 21cabdfd-02f5-ab4b-943a-6ebf88326dae received_uuid -
>                   uuid 0c9d9843-32f3-d341-b657-a33de46c9d0e path
> root-rw
> 
> sh sync_ext_to_int.sh
> 
> btrfs subvolume list -pqRu intern/
> ID 388 gen 583 parent 5 top level 5 parent_uuid
> a88bb4ed-65c8-db4f-b209-9ca07eaf3d21 received_uuid
> b326e1fe-7295-3948-a401-b6de850e213c uuid
> f055e3e8-d048-744c-abb4-c3efd87c4875 path root-ro
> ID 389 gen 583 parent 5 top level 5 parent_uuid
> f055e3e8-d048-744c-abb4-c3efd87c4875 received_uuid -
>                   uuid da51f30f-f10a-b04c-b7a7-5e718118a49c path
> root-rw
> 
> btrfs subvolume list -pqRu extern/
> ID 346 gen 615 parent 5 top level 5 parent_uuid
> 21cabdfd-02f5-ab4b-943a-6ebf88326dae received_uuid -
>                   uuid 0c9d9843-32f3-d341-b657-a33de46c9d0e path
> root-rw
> ID 347 gen 615 parent 5 top level 5 parent_uuid
> 0c9d9843-32f3-d341-b657-a33de46c9d0e received_uuid -
>                   uuid b326e1fe-7295-3948-a401-b6de850e213c path
> root-ro
> 
> sh sync_int_to_ext.sh (intern was not changed, just to keep ext->int,
> int->ext in order)
> 

This should have failed already and the real question is why it did not. 
The parent subvolume is identified by rceieved_uuid which is missing.

> btrfs subvolume list -pqRu intern/
> ID 389 gen 588 parent 5 top level 5 parent_uuid
> f055e3e8-d048-744c-abb4-c3efd87c4875 received_uuid -
>                   uuid da51f30f-f10a-b04c-b7a7-5e718118a49c path
> root-rw
> ID 390 gen 588 parent 5 top level 5 parent_uuid
> da51f30f-f10a-b04c-b7a7-5e718118a49c received_uuid -
>                   uuid 8a1f263b-581d-1847-a0da-0fc17e1ca2c4 path
> root-ro
> 
> btrfs subvolume list -pqRu extern/
> ID 348 gen 623 parent 5 top level 5 parent_uuid
> b326e1fe-7295-3948-a401-b6de850e213c received_uuid
> 8a1f263b-581d-1847-a0da-0fc17e1ca2c4 uuid
> c481acb9-b077-7841-a4b6-a0273f7aa12f path root-ro
> ID 349 gen 623 parent 5 top level 5 parent_uuid
> c481acb9-b077-7841-a4b6-a0273f7aa12f received_uuid -
>                   uuid cf80f17f-3d5d-7744-83f4-096b7688288c path
> root-rw
> 
> # boot fedora from extern (remove google-chrome and chromium via dnf)
> + shutdown again
> 
> btrfs subvolume list -pqRu extern/
> ID 348 gen 623 parent 5 top level 5 parent_uuid
> b326e1fe-7295-3948-a401-b6de850e213c received_uuid
> 8a1f263b-581d-1847-a0da-0fc17e1ca2c4 uuid
> c481acb9-b077-7841-a4b6-a0273f7aa12f path root-ro
> ID 349 gen 633 parent 5 top level 5 parent_uuid
> c481acb9-b077-7841-a4b6-a0273f7aa12f received_uuid -
>                   uuid cf80f17f-3d5d-7744-83f4-096b7688288c path
> root-rw
> 
> sh sync_ext_to_int.sh
> 
> manually executed line-by-line in expecting it would fail it failed at:
> btrfs send -p extern/root-ro extern/root-ro-new | btrfs receive intern/
> At subvol extern/root-ro-new
> At snapshot root-ro-new
> ERROR: clone: cannot find source subvol 8a1f263b-581d-1847-a0da-0fc17e1ca2c4
> 
> interestingly, despite send-receive failed/aborted with the message
> above, intern/root-ro-new was created and contains contents but left
> in writable state:
> 

Correct. Where "btrfs receive" fails is cloning extents from the parent 
subvolume. The difference is that rebooting performs much larger 
modifications of the root-rw subvolume which triggers "clone" operation 
during send/receive. Your test runs basically do nothing. Post output of

btrfs send -p extern/root-ro extern/root-ro-new | btrfs receive --dump

I do not know off-hand how to trigger "clone" operation.

It is definitely "btrfs receive" bug. Either it should fail from the 
very beginning due to missing received_uuid or it should consistently 
fall back to subvolume uuid in all cases. But starting with subvolume 
that cannot be later used to apply incremental changes is certainly the 
wrong thing to do.

> btrfs sub show intern/root-ro-new/
> root-ro-new
>          Name:                   root-ro-new
>          UUID:                   d32788b9-5572-5743-bf27-46b57c8b92c1
>          Parent UUID:            8a1f263b-581d-1847-a0da-0fc17e1ca2c4
>          Received UUID:          -
>          Creation time:          2024-01-08 20:14:59 +0100
>          Subvolume ID:           391
>          Generation:             596
>          Gen at creation:        594
>          Parent ID:              5
>          Top level ID:           5
>          Flags:                  -
>          Send transid:           0
>          Send time:              2024-01-08 20:14:59 +0100
>          Receive transid:        0
>          Receive time:           -
>          Snapshot(s):
>          Quota group:            n/a
> 
> 
> btrfs subvolume list -pqRu intern/
> ID 389 gen 588 parent 5 top level 5 parent_uuid
> f055e3e8-d048-744c-abb4-c3efd87c4875 received_uuid -
>                   uuid da51f30f-f10a-b04c-b7a7-5e718118a49c path
> root-rw
> ID 390 gen 594 parent 5 top level 5 parent_uuid
> da51f30f-f10a-b04c-b7a7-5e718118a49c received_uuid -
>                   uuid 8a1f263b-581d-1847-a0da-0fc17e1ca2c4 path
> root-ro
> ID 391 gen 595 parent 5 top level 5 parent_uuid
> 8a1f263b-581d-1847-a0da-0fc17e1ca2c4 received_uuid -
>                   uuid d32788b9-5572-5743-bf27-46b57c8b92c1 path
> root-ro-new
> 
> 
> btrfs subvolume list -pqRu extern/
> ID 348 gen 623 parent 5 top level 5 parent_uuid
> b326e1fe-7295-3948-a401-b6de850e213c received_uuid
> 8a1f263b-581d-1847-a0da-0fc17e1ca2c4 uuid
> c481acb9-b077-7841-a4b6-a0273f7aa12f path root-ro
> ID 349 gen 636 parent 5 top level 5 parent_uuid
> c481acb9-b077-7841-a4b6-a0273f7aa12f received_uuid -
>                   uuid cf80f17f-3d5d-7744-83f4-096b7688288c path
> root-rw
> ID 350 gen 636 parent 5 top level 5 parent_uuid
> cf80f17f-3d5d-7744-83f4-096b7688288c received_uuid -
>                   uuid d73b4afe-0837-8046-b798-21102cb82d4d path
> root-ro-new
> 
> Looking at the last output I have to admit I am confused.
> The error mentioned missing parent with
> 8a1f263b-581d-1847-a0da-0fc17e1ca2c4 for intern/root-ro-new seems to
> be there. Isn't this, as expected intern/root-ro with
> 8a1f263b-581d-1847-a0da-0fc17e1ca2c4.
> 
> Thanks for all your patience and help!
> 
> Best regards, Clemens

Re: Using send/receive to keep two rootfs-partitions in sync fails with "ERROR: snapshot: cannot find parent subvolume"

[Clemens Eisserer]

Hi Andrei,

> Correct. Where "btrfs receive" fails is cloning extents from the parent
> subvolume. The difference is that rebooting performs much larger
> modifications of the root-rw subvolume which triggers "clone" operation
> during send/receive. Your test runs basically do nothing. Post output of
>
> btrfs send -p extern/root-ro extern/root-ro-new | btrfs receive --dump
>
> I do not know off-hand how to trigger "clone" operation.

I've uploaded the receive dump to:
https://drive.google.com/file/d/1aluawM35xFxm7Au4dWtUzBcXbM3hyK-P/view?usp=sharing
The dump contains 4 clone operations, seemingly caused by an
in-process database used by some kde component:

At subvol extern/root-ro-new
clone
./root-ro-new/root/.local/share/kactivitymanagerd/resources/working-backup/database-wal
offset=4096 len=61440
from=./root-ro-new/root/.local/share/kactivitymanager
d/resources/test-backup/database-wal clone_offset=4096
clone
./root-ro-new/root/.local/share/kactivitymanagerd/resources/working-backup/database-wal
offset=65536 len=147456
from=./root-ro-new/root/.local/share/kactivitymanag
erd/resources/test-backup/database-wal clone_offset=65536
clone
./root-ro-new/root/.local/share/kactivitymanagerd/resources/working-backup/database-shm
offset=4096 len=12288
from=./root-ro-new/root/.local/share/kactivitymanager
d/resources/test-backup/database-shm clone_offset=4096
clone
./root-ro-new/root/.local/share/kactivitymanagerd/resources/working-backup/database-shm
offset=16384 len=8192
from=./root-ro-new/root/.local/share/kactivitymanager
d/resources/test-backup/database-shm clone_offset=16384

This would also explain why it worked for the first boot (fresh
profile, db-files are created) and fails the second time (db files are
modified leading to clone).

> It is definitely "btrfs receive" bug. Either it should fail from the
> very beginning due to missing received_uuid or it should consistently
> fall back to subvolume uuid in all cases.

Do you have any idea why the received_uuid is missing in the first place?

Should I file a bug-report against btrfs receive?

Thanks a lot for taking the time for such an in-depth analysis!

- Clemens

Re: Using send/receive to keep two rootfs-partitions in sync fails with "ERROR: snapshot: cannot find parent subvolume"

[Andrei Borzenkov]

On 12.01.2024 11:44, Clemens Eisserer wrote:
> Hi Andrei,
> 
>> Correct. Where "btrfs receive" fails is cloning extents from the parent
>> subvolume. The difference is that rebooting performs much larger
>> modifications of the root-rw subvolume which triggers "clone" operation
>> during send/receive. Your test runs basically do nothing. Post output of
>>
>> btrfs send -p extern/root-ro extern/root-ro-new | btrfs receive --dump
>>
>> I do not know off-hand how to trigger "clone" operation.
> 
> I've uploaded the receive dump to:
> https://drive.google.com/file/d/1aluawM35xFxm7Au4dWtUzBcXbM3hyK-P/view?usp=sharing
> The dump contains 4 clone operations, seemingly caused by an
> in-process database used by some kde component:
> 
> At subvol extern/root-ro-new
> clone
> ./root-ro-new/root/.local/share/kactivitymanagerd/resources/working-backup/database-wal
> offset=4096 len=61440
> from=./root-ro-new/root/.local/share/kactivitymanager
> d/resources/test-backup/database-wal clone_offset=4096
> clone
> ./root-ro-new/root/.local/share/kactivitymanagerd/resources/working-backup/database-wal
> offset=65536 len=147456
> from=./root-ro-new/root/.local/share/kactivitymanag
> erd/resources/test-backup/database-wal clone_offset=65536
> clone
> ./root-ro-new/root/.local/share/kactivitymanagerd/resources/working-backup/database-shm
> offset=4096 len=12288
> from=./root-ro-new/root/.local/share/kactivitymanager
> d/resources/test-backup/database-shm clone_offset=4096
> clone
> ./root-ro-new/root/.local/share/kactivitymanagerd/resources/working-backup/database-shm
> offset=16384 len=8192
> from=./root-ro-new/root/.local/share/kactivitymanager
> d/resources/test-backup/database-shm clone_offset=16384
> 

Good. So this confirms the root cause. To be absolutely sure you could 
also enable verbose logging in "btrfs receive" to get the exact point 
where it stops.

> This would also explain why it worked for the first boot (fresh
> profile, db-files are created) and fails the second time (db files are
> modified leading to clone).
> 

Correct. New files result in "btrfs send" instructions to create and 
fill in data, not to clone existing data.

>> It is definitely "btrfs receive" bug. Either it should fail from the
>> very beginning due to missing received_uuid or it should consistently
>> fall back to subvolume uuid in all cases.
> 
> Do you have any idea why the received_uuid is missing in the first place?
> 

Where should it come from? Received_uuid is set by "btrfs receive" on 
the target of replication and is always equal to UUID of the source 
subvolume data. But in your workflow when you sync in reverse direction 
the former source subvolumes are not the result of replication and so do 
not have received_uuid.

When you perform int/root-ro-new -> ext/root-ro-new, ext/root-ro-new 
gets received_uuid equal to int/root-ro-new UUID. But int/root-ro-new 
does not have any received_uuid because it was just created locally on int.

As Hugo mentioned, the current design of btrfs replication with 
send/receive is fundamentally unidirectional.

To allow bidirectional replication in the current design you must always 
have the same data with the same received_uuid on both sides. So 
replication will become something like

1. Start with initial replica having the same received_uuid (look 
carefully at your btrfs subvolume list output)

int/root-ro <-> ext/root-ro (*both* received_uuid == UUID1)

2. Do incremental update with initial replica as parent

btrfs send -p int/root-ro int/root-ro-new

int/root-ro-new (*empty* received_uuid) -> ext/root-ro-new 
(received_uuid == UUID2 == int/root-ro-new UUID)

3. Delete int/root-ro-new and perform reverse incremental sync

btrfs sub del int/root-ro-new
btrfs send -p ext/root-ro ext/root-ro-new

which results in

int/root-ro-new <-> ext/root-ro-new (*both* now have received_uuid == UUID2)

now you are on the square one and can again resync in every direction 
but must follow the same procedure every time.

It does mean extra time and bandwidth. Of course you can take the 
sledgehammer and simply manually set int/root-ro-new received_uuid to be 
equal its own UUID ... but I am not sure if this would not violate some 
of the current or future btrfs checks (there is no legitimate workflow 
where received_uuid would be equal to UUID).

> Should I file a bug-report against btrfs receive?
> 

I guess it never hurts, but I am not sure whether btrfs is using any 
formal bug tracker.

Thinking more about it - the fallback to subvolume UUID if received_uuid 
is missing is wrong. We have no way to verify that subvolume data was 
not really changed. It was one of the most common problem with 
send/receive - people flipped read-only -> read-write, made some 
changes, then flipped back read-write -> read-only and now source and 
target replicas were no more in sync and incremental stream created on 
one side could not be applied to another side.

Recent btrfs progs will clear received_uuid when subvolume is changed to 
read-write which offers some protection against such mishandling, but if 
we will fall back to plain UUID anyway, any bets are off.

Re: Using send/receive to keep two rootfs-partitions in sync fails with "ERROR: snapshot: cannot find parent subvolume"

[Hugo Mills]

On Sun, Jan 07, 2024 at 08:06:36AM +0100, Clemens Eisserer wrote:
> Hi,
> 
> I would like to use send/receive to keep two root-filesystems in sync,
> as I've been using it for years now for backups where it really does
> wonders (thanks a lot!).

   I'm afraid that btrfs incremental send/receive is *only* designed
for one way backups, as you've been using it up to now. It will not
work for bidirectional syncing, and no amount of gymnastics will make
it work. If you want to do bidirectional sync, you'll need to use some
other tool like rsync.

   Hugo.

-- 
Hugo Mills             | Anyone using a computer to generate random numbers
hugo@... carfax.org.uk | is, of course, in a state of sin.
http://carfax.org.uk/  |
PGP: E2AB1DE4          |                                       Jon von Neumann

Re: Using send/receive to keep two rootfs-partitions in sync fails with "ERROR: snapshot: cannot find parent subvolume"

[Clemens Eisserer]

Hi Hugo,

> I'm afraid that btrfs incremental send/receive is *only* designed
> for one way backups, as you've been using it up to now.
But it seems to work perfectly fine as long as I don't actually use
the subvolume as rootfs.
I still wonder what is so special using a subvolume as rootfs...

> It will not
> work for bidirectional syncing, and no amount of gymnastics will make
> it work. If you want to do bidirectional sync, you'll need to use some
> other tool like rsync.
But why?
What is the technical difference between on-way backup-style transfer
and using send/receive for bidirectional synchronization - isn't it
just applying the diff between two snapshots in both use-cases?

Best regards, Clemens