Re: Using send/receive to keep two rootfs-partitions in sync fails with "ERROR: snapshot: cannot find parent subvolume"

[Andrei Borzenkov <arvidjaar@gmail.com>]

On 12.01.2024 11:44, Clemens Eisserer wrote:
> Hi Andrei,
> 
>> Correct. Where "btrfs receive" fails is cloning extents from the parent
>> subvolume. The difference is that rebooting performs much larger
>> modifications of the root-rw subvolume which triggers "clone" operation
>> during send/receive. Your test runs basically do nothing. Post output of
>>
>> btrfs send -p extern/root-ro extern/root-ro-new | btrfs receive --dump
>>
>> I do not know off-hand how to trigger "clone" operation.
> 
> I've uploaded the receive dump to:
> https://drive.google.com/file/d/1aluawM35xFxm7Au4dWtUzBcXbM3hyK-P/view?usp=sharing
> The dump contains 4 clone operations, seemingly caused by an
> in-process database used by some kde component:
> 
> At subvol extern/root-ro-new
> clone
> ./root-ro-new/root/.local/share/kactivitymanagerd/resources/working-backup/database-wal
> offset=4096 len=61440
> from=./root-ro-new/root/.local/share/kactivitymanager
> d/resources/test-backup/database-wal clone_offset=4096
> clone
> ./root-ro-new/root/.local/share/kactivitymanagerd/resources/working-backup/database-wal
> offset=65536 len=147456
> from=./root-ro-new/root/.local/share/kactivitymanag
> erd/resources/test-backup/database-wal clone_offset=65536
> clone
> ./root-ro-new/root/.local/share/kactivitymanagerd/resources/working-backup/database-shm
> offset=4096 len=12288
> from=./root-ro-new/root/.local/share/kactivitymanager
> d/resources/test-backup/database-shm clone_offset=4096
> clone
> ./root-ro-new/root/.local/share/kactivitymanagerd/resources/working-backup/database-shm
> offset=16384 len=8192
> from=./root-ro-new/root/.local/share/kactivitymanager
> d/resources/test-backup/database-shm clone_offset=16384
> 

Good. So this confirms the root cause. To be absolutely sure you could 
also enable verbose logging in "btrfs receive" to get the exact point 
where it stops.

> This would also explain why it worked for the first boot (fresh
> profile, db-files are created) and fails the second time (db files are
> modified leading to clone).
> 

Correct. New files result in "btrfs send" instructions to create and 
fill in data, not to clone existing data.

>> It is definitely "btrfs receive" bug. Either it should fail from the
>> very beginning due to missing received_uuid or it should consistently
>> fall back to subvolume uuid in all cases.
> 
> Do you have any idea why the received_uuid is missing in the first place?
> 

Where should it come from? Received_uuid is set by "btrfs receive" on 
the target of replication and is always equal to UUID of the source 
subvolume data. But in your workflow when you sync in reverse direction 
the former source subvolumes are not the result of replication and so do 
not have received_uuid.

When you perform int/root-ro-new -> ext/root-ro-new, ext/root-ro-new 
gets received_uuid equal to int/root-ro-new UUID. But int/root-ro-new 
does not have any received_uuid because it was just created locally on int.

As Hugo mentioned, the current design of btrfs replication with 
send/receive is fundamentally unidirectional.

To allow bidirectional replication in the current design you must always 
have the same data with the same received_uuid on both sides. So 
replication will become something like

1. Start with initial replica having the same received_uuid (look 
carefully at your btrfs subvolume list output)

int/root-ro <-> ext/root-ro (*both* received_uuid == UUID1)

2. Do incremental update with initial replica as parent

btrfs send -p int/root-ro int/root-ro-new

int/root-ro-new (*empty* received_uuid) -> ext/root-ro-new 
(received_uuid == UUID2 == int/root-ro-new UUID)

3. Delete int/root-ro-new and perform reverse incremental sync

btrfs sub del int/root-ro-new
btrfs send -p ext/root-ro ext/root-ro-new

which results in

int/root-ro-new <-> ext/root-ro-new (*both* now have received_uuid == UUID2)

now you are on the square one and can again resync in every direction 
but must follow the same procedure every time.

It does mean extra time and bandwidth. Of course you can take the 
sledgehammer and simply manually set int/root-ro-new received_uuid to be 
equal its own UUID ... but I am not sure if this would not violate some 
of the current or future btrfs checks (there is no legitimate workflow 
where received_uuid would be equal to UUID).

> Should I file a bug-report against btrfs receive?
> 

I guess it never hurts, but I am not sure whether btrfs is using any 
formal bug tracker.

Thinking more about it - the fallback to subvolume UUID if received_uuid 
is missing is wrong. We have no way to verify that subvolume data was 
not really changed. It was one of the most common problem with 
send/receive - people flipped read-only -> read-write, made some 
changes, then flipped back read-write -> read-only and now source and 
target replicas were no more in sync and incremental stream created on 
one side could not be applied to another side.

Recent btrfs progs will clear received_uuid when subvolume is changed to 
read-write which offers some protection against such mishandling, but if 
we will fall back to plain UUID anyway, any bets are off.