Re: btrfs send and receive showing errors

["Randy Nürnberger" <ranuberger@posteo.de>]

[-- Attachment #1: Type: text/plain, Size: 6377 bytes --]

On Wed, Jan 4, 2023 at 14:41, Filipe Manana wrote:
> On Wed, Jan 4, 2023 at 1:05 PM Randy Nürnberger <ranuberger@posteo.de> wrote:
>> Hello,
>>
>> I’m in the process of copying a btrfs filesystem (a couple years old)
>> from one disk to another by using btrfs send and receive on all
>> snapshots. The snapshots were created by a tool every hour on the hour
>> as one backup measure and automatically removed as they became older.
>>
>> I got errors like the following and when I compare the copied snapshots
>> with the originals, some files are missing:
>>
>> ERROR: unlink █████ failed: No such file or directory
>> ERROR: link █████ -> █████ failed: No such file or directory
>> ERROR: utimes █████ failed: No such file or directory
>> ERROR: rmdir █████ failed: No such file or directory
>>
>> Is this a known bug and how can I help diagnosing and fixing this?
> So this is a problem caused by the sender issuing commands with outdated paths.
>
> First, try doing the send operation again with a 6.1 kernel - there
> was at least one fix here that may be relevant.

I tried again with the following kernel version and still got the same 
errors:
Linux arktos 6.1.0-0-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.2-1~exp1 
(2023-01-01) x86_64 GNU/Linux

>
> To actual debug things, in case it's not working with 6.1:
>
> 1) Show how you invoked send and receive. I.e. the full command lines
> for 'btrfs send ...' and 'btrfs receive ...'

Those are my full command lines:

btrfs send -p /mnt/randy/randy-snapshots/2022-01-29--07-00 
/mnt/randy/randy-snapshots/2022-02-27--10-00 | btrfs receive -vvv 
/mnt/intern/randy-snapshots/ 2>receive-1.txt

btrfs send -p /mnt/randy/randy-snapshots/2022-02-27--10-00 
/mnt/randy/randy-snapshots/2022-03-26--07-00 | btrfs receive -vvv 
/mnt/intern/randy-snapshots/ 2>receive-2.txt

>
> 2) Provide the whole output of 'btrfs receive' with the -vvv command
> line option.
>      This will reveal all path names, but it's necessary to debug things.
>      You've hidden path names above, so I suppose that's not acceptable for you.

At least I’m not comfortable sharing the file names on this public 
mailing list. I carefully tried to extract and redact what may be the 
relevant parts.

The second command line above is the first one that fails with the 
following error: “ERROR: unlink Hase/Fuchs/2022-02-23 Reh.odt failed: No 
such file or directory”.

This is the directory listing for the snapshot before said file was 
created (this snapshot can be copied without errors):

root@arktos /m/r/randy-snapshots# ls -alh 2022-01-29--07-00/Hase/Fuchs/
insgesamt 2,6M
drwxrws--- 1 randy randy  136 19. Dez 2021   ./
drwxrws--- 1 randy randy  134 24. Nov 2021   ../
-rw-rw---- 1 randy randy  38K  5. Mai 2021  '2021-05-01 Igel.odt'
-rw-rw---- 1 randy randy  16K 30. Sep 2021  '2021-09-30 Wolf.odt'
-rw-rw---- 1 randy randy 2,6M 19. Dez 2021  '2021-12-19 Wildschwein.pdf'

This is the directory listing for the snapshot after the file has been 
created (this snapshot can be copied without errors):

root@arktos /m/r/randy-snapshots# ls -alh 2022-02-27--10-00/Hase/Fuchs/
insgesamt 2,7M
drwxrws---  1 randy randy  178 27. Feb 2022   ./
drwxrws---  1 randy randy  134 24. Nov 2021   ../
-rw-rw----  1 randy randy  38K  5. Mai 2021  '2021-05-01 Igel.odt'
-rw-rw----  1 randy randy  16K 30. Sep 2021  '2021-09-30 Wolf.odt'
-rw-rw----  1 randy randy 2,6M 19. Dez 2021  '2021-12-19 Wildschwein.pdf'
-rw-rwx---+ 1 randy randy  42K 26. Feb 2022  '2022-02-23 Reh.odt'*

This is the directory listing for the snapshot after the file has been 
edited in LibreOffice and *renamed* (trying to copy this one fails):

root@arktos /m/r/randy-snapshots# ls -alh 2022-03-26--07-00/Hase/Fuchs/
insgesamt 2,6M
drwxrws--- 1 randy randy  178  5. Mär 2022   ./
drwxrws--- 1 randy randy  134 24. Nov 2021   ../
-rw-rw---- 1 randy randy  38K  5. Mai 2021  '2021-05-01 Igel.odt'
-rw-rw---- 1 randy randy  16K 30. Sep 2021  '2021-09-30 Wolf.odt'
-rw-rw---- 1 randy randy 2,6M 19. Dez 2021  '2021-12-19 Wildschwein.pdf'
-rw-rw---- 1 randy randy  18K  4. Mär 2022  '2022-03-03 Reh.odt'

I’ve attached the outputs of the commands above. Apparently ‘btrfs send’ 
instructs ‘btrfs receive’ to unlink the file ‘Hase/Fuchs/2022-02-23 
Reh.odt’ which *does* exist in the copied snapshot ‘2022-02-27--10-00’ 
and this fails for whatever reason.

# sha256sum 
/mnt/randy/randy-snapshots/2022-02-27--10-00/Hase/Fuchs/2022-02-23\ Reh.odt
09ab560f8e2d79e61d253550eda5f388aceddb1b51792e01e589e86a53cdd226

# sha256sum 
/mnt/intern/randy-snapshots/2022-02-27--10-00/Hase/Fuchs/2022-02-23\ 
Reh.odt
09ab560f8e2d79e61d253550eda5f388aceddb1b51792e01e589e86a53cdd226

>
> Thanks.
>
>>
>> # uname -a  # this is the kernel on which this originally happend
>> Linux arktos 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13)
>> x86_64 GNU/Linux
>>
>>
>> # uname -a  # I already retried everything on the latest Debian
>> backports kernel with the same results
>> Linux arktos 6.0.0-0.deb11.6-amd64 #1 SMP PREEMPT_DYNAMIC Debian
>> 6.0.12-1~bpo11+1 (2022-12-19) x86_64 GNU/Linux
>>
>> # btrfs --version
>> btrfs-progs v5.10.1
>>
>> # btrfs fi sh /mnt/randy  # this is the source
>> Label: none  uuid: 84bba855-578d-48db-80eb-49f1029c7543
>>           Total devices 2 FS bytes used 2.04TiB
>>           devid    1 size 4.00TiB used 2.05TiB path /dev/mapper/randy_1_crypt
>>           devid    2 size 4.00TiB used 2.05TiB path /dev/mapper/randy_2_crypt
>>
>> # btrfs fi df /mnt/randy
>> Data, RAID1: total=2.02TiB, used=2.02TiB
>> System, RAID1: total=8.00MiB, used=320.00KiB
>> Metadata, RAID1: total=25.00GiB, used=22.99GiB
>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>
>>
>> # btrfs fi sh /mnt/intern  # this is the target
>> Label: none  uuid: ebb94498-644c-42cd-892f-37886173523c
>>           Total devices 2 FS bytes used 1.91TiB
>>           devid    1 size 8.00TiB used 1.91TiB path
>> /dev/mapper/vg_arktos_hdd_b-lv_randy_1
>>           devid    2 size 8.00TiB used 1.91TiB path
>> /dev/mapper/vg_arktos_hdd_b-lv_randy_2
>>
>> # btrfs fi df /mnt/intern
>> Data, RAID1: total=1.89TiB, used=1.89TiB
>> System, RAID1: total=8.00MiB, used=288.00KiB
>> Metadata, RAID1: total=21.00GiB, used=20.76GiB
>> GlobalReserve, single: total=512.00MiB, used=0.00B

[-- Attachment #2: receive-1-redacted.txt --]
[-- Type: text/plain, Size: 750 bytes --]

receiving snapshot 2022-02-27--10-00 uuid=5b98e81e-0e9b-814a-b191-19fe36517d1c, ctransid=1060 parent_uuid=5b98e81e-0e9b-814a-b191-19fe36517d1c, parent_ctransid=810
[…]
utimes Hase/Fuchs
[…]
mkfile o1525604-1014-0
rename o1525604-1014-0 -> Hase/Fuchs/2022-02-23 Reh.odt
utimes Hase/Fuchs
set_xattr Hase/Fuchs/2022-02-23 Reh.odt - name=system.posix_acl_access data_len=52 data=\x02
set_xattr Hase/Fuchs/2022-02-23 Reh.odt - name=user.DOSATTRIB data_len=32 data=
write Hase/Fuchs/2022-02-23 Reh.odt - offset=0 length=42184
chown Hase/Fuchs/2022-02-23 Reh.odt - uid=3000, gid=3000
chmod Hase/Fuchs/2022-02-23 Reh.odt - mode=0670
utimes Hase/Fuchs/2022-02-23 Reh.odt
BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=5b98e81e-0e9b-814a-b191-19fe36517d1c, stransid=1060

[-- Attachment #3: receive-2-redacted.txt --]
[-- Type: text/plain, Size: 310 bytes --]

receiving snapshot 2022-03-26--07-00 uuid=5b98e81e-0e9b-814a-b191-19fe36517d1c, ctransid=1411 parent_uuid=5b98e81e-0e9b-814a-b191-19fe36517d1c, parent_ctransid=1060
[…]
utimes Hase/Fuchs
[…]
unlink Hase/Fuchs/2022-02-23 Reh.odt
ERROR: unlink Hase/Fuchs/2022-02-23 Reh.odt failed: No such file or directory