Some file lost when send back snapshot.

Some file lost when send back snapshot.

[MASAKI Haruka]

I send snapshot from backup volume, and send new snaopshot to backup volume,
and some file lost in received volume.

In detail:

1. Host A has new Btrfs volume and Host B have two backuped readonly subvolumes (190312-010100, 190315-001000).
2. Host B send 190315-001000 subvolume to Host A.
3. Host A completely received (all files exist.)
4. Host B took new readonly snapshot "back1" on 190315-001000 (all files exist.)
5. Host B send back1 with `-p 190315-001000` to Host A (some file lost.)
6. Host B took new read-writable snapshot "world" on back1 (all files exist.)
7. Host B modified world.
8. Host B took new readonly snapshot "back2" on world (all files exist.)
9. Host B send back2 with `-p 190315-001000` to Host A (some file lost.)
10. Deleted received snapshot and Host B send back2 with `-p back1` to Host A (some file lost.)

Kernel
5.0.5-1-MANJARO

Btrfs progs
btrfs-progs 4.20.2-1


What I should do?

Thanks.



Masaki Haruka

Re: Some file lost when send back snapshot.

[Andrei Borzenkov]

25.04.2019 16:34, MASAKI Haruka пишет:
> I send snapshot from backup volume, and send new snaopshot to backup volume,
> and some file lost in received volume.
> 
> In detail:
> 
> 1. Host A has new Btrfs volume and Host B have two backuped readonly subvolumes (190312-010100, 190315-001000).
> 2. Host B send 190315-001000 subvolume to Host A.
> 3. Host A completely received (all files exist.)
> 4. Host B took new readonly snapshot "back1" on 190315-001000 (all files exist.)
> 5. Host B send back1 with `-p 190315-001000` to Host A (some file lost.)
> 6. Host B took new read-writable snapshot "world" on back1 (all files exist.)
> 7. Host B modified world.
> 8. Host B took new readonly snapshot "back2" on world (all files exist.)
> 9. Host B send back2 with `-p 190315-001000` to Host A (some file lost.)
> 10. Deleted received snapshot and Host B send back2 with `-p back1` to Host A (some file lost.)
> 

Please show

1. Full protocol of commands you executed (commands and their output),
e.g. captured using script.
2. Commands used to determine missing files on both source and receive
sides.
3. Listing of subvolumes on both source and receive sides with

btrfs sub li -Rqu /mount/point

> Kernel
> 5.0.5-1-MANJARO
> 
> Btrfs progs
> btrfs-progs 4.20.2-1
> 
> 
> What I should do?
> 
> Thanks.
> 
> 
> 
> Masaki Haruka
> 

Re: Some file lost when send back snapshot.

[MASAKI Haruka]

Thank you for your reply.

# What does

@Host A ("lily" master, new volume)
aki@lily /mnt % socat TCP-LISTEN:25500 STDOUT | sudo btrfs receive -vv . 

@Host B ("daisy" backup, having volumes)
aki@daisy /HySlave % sudo btrfs send -v 190315-001000 | socat STDIN TCP:lily.local:25500 

( ...Transfared 11TB... )

@Host A
aki@lily /mnt % sudo btrfs subvolume snapshot 190315-001000 world
Create a snapshot of '190315-001000' in './world'
aki@lily /mnt % sudo btrfs subvolume snapshot -r world back1
Create a readonly snapshot of 'world' in './back1'

@Host A
aki@lily /mnt % sudo btrfs send -vv -p 190315-001000 back1 | socat STDIN tcp-connect:192.168.1.17:25555
At subvol back1
BTRFS_IOC_SEND returned 0
joining genl thread

@Host B
root@daisy /HySlave # socat TCP-LISTEN:25555 STDOUT | btrfs receive -vv .
At snapshot back1
receiving snapshot back1 uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, ctransid=2658 parent_uuid=9c10b77f-74fb-f94a-9a24-3c6d30c00ca2, parent_ctransid=2658
BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, stransid=2658

# Host A current

aki@lily /mnt % ls
190315-001000  back1  back2  world
aki@lily /mnt % sudo btrfs sub li -Rqu .
ID 258 gen 2698 top level 5 parent_uuid -                                    received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid bc90d05c-7430-7b42-b683-702d906bb993 path 190315-001000
ID 2139 gen 2696 top level 5 parent_uuid bc90d05c-7430-7b42-b683-702d906bb993 received_uuid -                                    uuid 23c5fede-a338-d14a-afcd-677db2e97637 path world
ID 2140 gen 2660 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid -                                    uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 path back1
ID 2142 gen 2672 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid -                                    uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b path back2


# Host B current

aki@daisy /HySlave % pwd
/HySlave
aki@daisy /HySlave % ls
190312-010100  190315-001000  back1  back2
[aki@daisy HySlave]$ sudo btrfs sub li -Rqu .
ID 256 gen 36238 top level 5 parent_uuid -                                    received_uuid -                                    uuid 4e42555d-a715-0e47-bd0e-85d6eb8911e0 path slave
ID 259 gen 35116 top level 256 parent_uuid -                                    received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 path 190312-010100
ID 3059 gen 7273 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 114cd74f-80d8-af44-a9d0-fd8ce3c991b4 path 190315-001000
ID 3072 gen 35112 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 uuid fe7281d1-8b3e-8f46-a0b4-cdc52a1bb7aa path back1
ID 3073 gen 36236 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b uuid 65206183-f20d-4648-83ab-e0485b16f7af path back2

# Files missing

aki@daisy /HySlave % ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CD 
Little_My_Maid  SMEE  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala

aki@daisy /HySlave % ls /HySlave/back1/local/pub/media/sound/notmusic_in_CD
Little_My_Maid  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala

aki@lily /mnt % ls /mnt/190315-001000/local/pub/media/sound/notmusic_in_CD
Little_My_Maid  SMEE  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala

ki@lily /mnt % ls /mnt/back1/local/pub/media/sound/notmusic_in_CD        
Little_My_Maid  SMEE  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala

aki@lily /mnt % ls /mnt/world/local/pub/media/sound/notmusic_in_CD 
Little_My_Maid  SMEE  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala


SMEE is missing.




Thanks.


On Thu, 25 Apr 2019 20:55:59 +0300
Andrei Borzenkov <arvidjaar@gmail.com> wrote:

> 25.04.2019 16:34, MASAKI Haruka пишет:
>  [...]  
> 
> Please show
> 
> 1. Full protocol of commands you executed (commands and their output),
> e.g. captured using script.
> 2. Commands used to determine missing files on both source and receive
> sides.
> 3. Listing of subvolumes on both source and receive sides with
> 
> btrfs sub li -Rqu /mount/point
> 
>  [...]  
> 

Re: Some file lost when send back snapshot.

[Andrei Borzenkov]

26.04.2019 7:50, MASAKI Haruka пишет:
> Thank you for your reply.
> 
> # What does
> 
> @Host A ("lily" master, new volume)
> aki@lily /mnt % socat TCP-LISTEN:25500 STDOUT | sudo btrfs receive -vv . 
> 
> @Host B ("daisy" backup, having volumes)
> aki@daisy /HySlave % sudo btrfs send -v 190315-001000 | socat STDIN TCP:lily.local:25500 
> 
> ( ...Transfared 11TB... )
> 
> @Host A
> aki@lily /mnt % sudo btrfs subvolume snapshot 190315-001000 world
> Create a snapshot of '190315-001000' in './world'
> aki@lily /mnt % sudo btrfs subvolume snapshot -r world back1
> Create a readonly snapshot of 'world' in './back1'
> 
> @Host A
> aki@lily /mnt % sudo btrfs send -vv -p 190315-001000 back1 | socat STDIN tcp-connect:192.168.1.17:25555
> At subvol back1
> BTRFS_IOC_SEND returned 0
> joining genl thread
> 
> @Host B
> root@daisy /HySlave # socat TCP-LISTEN:25555 STDOUT | btrfs receive -vv .
> At snapshot back1
> receiving snapshot back1 uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, ctransid=2658 parent_uuid=9c10b77f-74fb-f94a-9a24-3c6d30c00ca2, parent_ctransid=2658
> BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, stransid=2658
> 
> # Host A current
> 
> aki@lily /mnt % ls
> 190315-001000  back1  back2  world
> aki@lily /mnt % sudo btrfs sub li -Rqu .
> ID 258 gen 2698 top level 5 parent_uuid -                                    received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid bc90d05c-7430-7b42-b683-702d906bb993 path 190315-001000
> ID 2139 gen 2696 top level 5 parent_uuid bc90d05c-7430-7b42-b683-702d906bb993 received_uuid -                                    uuid 23c5fede-a338-d14a-afcd-677db2e97637 path world
> ID 2140 gen 2660 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid -                                    uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 path back1
> ID 2142 gen 2672 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid -                                    uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b path back2
> 
> 
> # Host B current
> 
> aki@daisy /HySlave % pwd
> /HySlave
> aki@daisy /HySlave % ls
> 190312-010100  190315-001000  back1  back2
> [aki@daisy HySlave]$ sudo btrfs sub li -Rqu .
> ID 256 gen 36238 top level 5 parent_uuid -                                    received_uuid -                                    uuid 4e42555d-a715-0e47-bd0e-85d6eb8911e0 path slave
> ID 259 gen 35116 top level 256 parent_uuid -                                    received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 path 190312-010100
> ID 3059 gen 7273 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 114cd74f-80d8-af44-a9d0-fd8ce3c991b4 path 190315-001000

You have two subvolumes with the same received_uuid. Is the content of
these subvolumes the same? Is "missing" file present in both subvolumes?

> ID 3072 gen 35112 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 uuid fe7281d1-8b3e-8f46-a0b4-cdc52a1bb7aa path back1
> ID 3073 gen 36236 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b uuid 65206183-f20d-4648-83ab-e0485b16f7af path back2
> 
> # Files missing
> 
> aki@daisy /HySlave % ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CD 
> Little_My_Maid  SMEE  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala
> 
> aki@daisy /HySlave % ls /HySlave/back1/local/pub/media/sound/notmusic_in_CD
> Little_My_Maid  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala
> 
> aki@lily /mnt % ls /mnt/190315-001000/local/pub/media/sound/notmusic_in_CD
> Little_My_Maid  SMEE  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala
> 
> ki@lily /mnt % ls /mnt/back1/local/pub/media/sound/notmusic_in_CD        
> Little_My_Maid  SMEE  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala
> 
> aki@lily /mnt % ls /mnt/world/local/pub/media/sound/notmusic_in_CD 
> Little_My_Maid  SMEE  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala
> 
> 
> SMEE is missing.
> 
> 
> 
> 
> Thanks.
> 
> 
> On Thu, 25 Apr 2019 20:55:59 +0300
> Andrei Borzenkov <arvidjaar@gmail.com> wrote:
> 
>> 25.04.2019 16:34, MASAKI Haruka пишет:
>>  [...]  
>>
>> Please show
>>
>> 1. Full protocol of commands you executed (commands and their output),
>> e.g. captured using script.
>> 2. Commands used to determine missing files on both source and receive
>> sides.
>> 3. Listing of subvolumes on both source and receive sides with
>>
>> btrfs sub li -Rqu /mount/point
>>
>>  [...]  
>>

Re: Some file lost when send back snapshot.

[MASAKI Haruka]

> You have two subvolumes with the same received_uuid. Is the content of
> these subvolumes the same? Is "missing" file present in both subvolumes?

No, each subvolume name (190312-010100, 190315-001000) is timestamp.
These snapshot volumes were created by job scheduler on original (already lost) volume, so 190315-001000 is modified from 190312-010100.

190315-001000 has lost file,
% ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CD/
Little_My_Maid  SMEE  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala

but 190312-010100 doesn't.
% ls /HySlave/190312-010100/local/pub/media/sound/notmusic_in_CD/
Little_My_Maid  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala




On Fri, 26 Apr 2019 20:30:07 +0300
Andrei Borzenkov <arvidjaar@gmail.com> wrote:

> 26.04.2019 7:50, MASAKI Haruka пишет:
> > Thank you for your reply.
> > 
> > # What does
> > 
> > @Host A ("lily" master, new volume)
> > aki@lily /mnt % socat TCP-LISTEN:25500 STDOUT | sudo btrfs receive -vv . 
> > 
> > @Host B ("daisy" backup, having volumes)
> > aki@daisy /HySlave % sudo btrfs send -v 190315-001000 | socat STDIN TCP:lily.local:25500 
> > 
> > ( ...Transfared 11TB... )
> > 
> > @Host A
> > aki@lily /mnt % sudo btrfs subvolume snapshot 190315-001000 world
> > Create a snapshot of '190315-001000' in './world'
> > aki@lily /mnt % sudo btrfs subvolume snapshot -r world back1
> > Create a readonly snapshot of 'world' in './back1'
> > 
> > @Host A
> > aki@lily /mnt % sudo btrfs send -vv -p 190315-001000 back1 | socat STDIN tcp-connect:192.168.1.17:25555
> > At subvol back1
> > BTRFS_IOC_SEND returned 0
> > joining genl thread
> > 
> > @Host B
> > root@daisy /HySlave # socat TCP-LISTEN:25555 STDOUT | btrfs receive -vv .
> > At snapshot back1
> > receiving snapshot back1 uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, ctransid=2658 parent_uuid=9c10b77f-74fb-f94a-9a24-3c6d30c00ca2, parent_ctransid=2658
> > BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, stransid=2658
> > 
> > # Host A current
> > 
> > aki@lily /mnt % ls
> > 190315-001000  back1  back2  world
> > aki@lily /mnt % sudo btrfs sub li -Rqu .
> > ID 258 gen 2698 top level 5 parent_uuid -                                    received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid bc90d05c-7430-7b42-b683-702d906bb993 path 190315-001000
> > ID 2139 gen 2696 top level 5 parent_uuid bc90d05c-7430-7b42-b683-702d906bb993 received_uuid -                                    uuid 23c5fede-a338-d14a-afcd-677db2e97637 path world
> > ID 2140 gen 2660 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid -                                    uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 path back1
> > ID 2142 gen 2672 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid -                                    uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b path back2
> > 
> > 
> > # Host B current
> > 
> > aki@daisy /HySlave % pwd
> > /HySlave
> > aki@daisy /HySlave % ls
> > 190312-010100  190315-001000  back1  back2
> > [aki@daisy HySlave]$ sudo btrfs sub li -Rqu .
> > ID 256 gen 36238 top level 5 parent_uuid -                                    received_uuid -                                    uuid 4e42555d-a715-0e47-bd0e-85d6eb8911e0 path slave
> > ID 259 gen 35116 top level 256 parent_uuid -                                    received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 path 190312-010100
> > ID 3059 gen 7273 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 114cd74f-80d8-af44-a9d0-fd8ce3c991b4 path 190315-001000
> 
> You have two subvolumes with the same received_uuid. Is the content of
> these subvolumes the same? Is "missing" file present in both subvolumes?
> 
> > ID 3072 gen 35112 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 uuid fe7281d1-8b3e-8f46-a0b4-cdc52a1bb7aa path back1
> > ID 3073 gen 36236 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b uuid 65206183-f20d-4648-83ab-e0485b16f7af path back2
> > 
> > # Files missing
> > 
> > aki@daisy /HySlave % ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CD 
> > Little_My_Maid  SMEE  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala
> > 
> > aki@daisy /HySlave % ls /HySlave/back1/local/pub/media/sound/notmusic_in_CD
> > Little_My_Maid  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala
> > 
> > aki@lily /mnt % ls /mnt/190315-001000/local/pub/media/sound/notmusic_in_CD
> > Little_My_Maid  SMEE  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala
> > 
> > ki@lily /mnt % ls /mnt/back1/local/pub/media/sound/notmusic_in_CD        
> > Little_My_Maid  SMEE  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala
> > 
> > aki@lily /mnt % ls /mnt/world/local/pub/media/sound/notmusic_in_CD 
> > Little_My_Maid  SMEE  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala
> > 
> > 
> > SMEE is missing.
> > 
> > 
> > 
> > 
> > Thanks.
> > 
> > 
> > On Thu, 25 Apr 2019 20:55:59 +0300
> > Andrei Borzenkov <arvidjaar@gmail.com> wrote:
> > 
> >> 25.04.2019 16:34, MASAKI Haruka пишет:
> >>  [...]  
> >>
> >> Please show
> >>
> >> 1. Full protocol of commands you executed (commands and their output),
> >> e.g. captured using script.
> >> 2. Commands used to determine missing files on both source and receive
> >> sides.
> >> 3. Listing of subvolumes on both source and receive sides with
> >>
> >> btrfs sub li -Rqu /mount/point
> >>
> >>  [...]  
> >>
> 

Re: Some file lost when send back snapshot.

[Andrei Borzenkov]

26.04.2019 21:12, MASAKI Haruka пишет:
>> You have two subvolumes with the same received_uuid. Is the content of
>> these subvolumes the same? Is "missing" file present in both subvolumes?
> 
> No, each subvolume name (190312-010100, 190315-001000) is timestamp.
> These snapshot volumes were created by job scheduler on original (already lost) volume, so 190315-001000 is modified from 190312-010100.
> 

Which is your problem. Two subvolumes with the same received_uuid are
expected to have exactly the same content (received_uuid can be viewed
as dataset UUID). You send incremental stream against base uuid
9c10b77f-74fb-f94a-9a24-3c6d30c00ca2; on receiving side btrfs receive
looks for matching base subvolume and happens to pick 190312-010100.

There were patches to automatically remove received_uuid when subvolume
is changed to read-write. I do not know whether they were applied.

btrfs send/receive expects subvolumes to remain read-only. As soon as
you start fiddling with read-only status you already lost. btrfs should
have never allowed to change read-only into read-write in the first place.

> 190315-001000 has lost file,
> % ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CD/
> Little_My_Maid  SMEE  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala
> 
> but 190312-010100 doesn't.
> % ls /HySlave/190312-010100/local/pub/media/sound/notmusic_in_CD/
> Little_My_Maid  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala
> 
> 
> 
> 
> On Fri, 26 Apr 2019 20:30:07 +0300
> Andrei Borzenkov <arvidjaar@gmail.com> wrote:
> 
>> 26.04.2019 7:50, MASAKI Haruka пишет:
>>> Thank you for your reply.
>>>
>>> # What does
>>>
>>> @Host A ("lily" master, new volume)
>>> aki@lily /mnt % socat TCP-LISTEN:25500 STDOUT | sudo btrfs receive -vv . 
>>>
>>> @Host B ("daisy" backup, having volumes)
>>> aki@daisy /HySlave % sudo btrfs send -v 190315-001000 | socat STDIN TCP:lily.local:25500 
>>>
>>> ( ...Transfared 11TB... )
>>>
>>> @Host A
>>> aki@lily /mnt % sudo btrfs subvolume snapshot 190315-001000 world
>>> Create a snapshot of '190315-001000' in './world'
>>> aki@lily /mnt % sudo btrfs subvolume snapshot -r world back1
>>> Create a readonly snapshot of 'world' in './back1'
>>>
>>> @Host A
>>> aki@lily /mnt % sudo btrfs send -vv -p 190315-001000 back1 | socat STDIN tcp-connect:192.168.1.17:25555
>>> At subvol back1
>>> BTRFS_IOC_SEND returned 0
>>> joining genl thread
>>>
>>> @Host B
>>> root@daisy /HySlave # socat TCP-LISTEN:25555 STDOUT | btrfs receive -vv .
>>> At snapshot back1
>>> receiving snapshot back1 uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, ctransid=2658 parent_uuid=9c10b77f-74fb-f94a-9a24-3c6d30c00ca2, parent_ctransid=2658
>>> BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, stransid=2658
>>>
>>> # Host A current
>>>
>>> aki@lily /mnt % ls
>>> 190315-001000  back1  back2  world
>>> aki@lily /mnt % sudo btrfs sub li -Rqu .
>>> ID 258 gen 2698 top level 5 parent_uuid -                                    received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid bc90d05c-7430-7b42-b683-702d906bb993 path 190315-001000
>>> ID 2139 gen 2696 top level 5 parent_uuid bc90d05c-7430-7b42-b683-702d906bb993 received_uuid -                                    uuid 23c5fede-a338-d14a-afcd-677db2e97637 path world
>>> ID 2140 gen 2660 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid -                                    uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 path back1
>>> ID 2142 gen 2672 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid -                                    uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b path back2
>>>
>>>
>>> # Host B current
>>>
>>> aki@daisy /HySlave % pwd
>>> /HySlave
>>> aki@daisy /HySlave % ls
>>> 190312-010100  190315-001000  back1  back2
>>> [aki@daisy HySlave]$ sudo btrfs sub li -Rqu .
>>> ID 256 gen 36238 top level 5 parent_uuid -                                    received_uuid -                                    uuid 4e42555d-a715-0e47-bd0e-85d6eb8911e0 path slave
>>> ID 259 gen 35116 top level 256 parent_uuid -                                    received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 path 190312-010100
>>> ID 3059 gen 7273 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 114cd74f-80d8-af44-a9d0-fd8ce3c991b4 path 190315-001000
>>
>> You have two subvolumes with the same received_uuid. Is the content of
>> these subvolumes the same? Is "missing" file present in both subvolumes?
>>
>>> ID 3072 gen 35112 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 uuid fe7281d1-8b3e-8f46-a0b4-cdc52a1bb7aa path back1
>>> ID 3073 gen 36236 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b uuid 65206183-f20d-4648-83ab-e0485b16f7af path back2
>>>
>>> # Files missing
>>>
>>> aki@daisy /HySlave % ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CD 
>>> Little_My_Maid  SMEE  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala
>>>
>>> aki@daisy /HySlave % ls /HySlave/back1/local/pub/media/sound/notmusic_in_CD
>>> Little_My_Maid  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala
>>>
>>> aki@lily /mnt % ls /mnt/190315-001000/local/pub/media/sound/notmusic_in_CD
>>> Little_My_Maid  SMEE  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala
>>>
>>> ki@lily /mnt % ls /mnt/back1/local/pub/media/sound/notmusic_in_CD        
>>> Little_My_Maid  SMEE  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala
>>>
>>> aki@lily /mnt % ls /mnt/world/local/pub/media/sound/notmusic_in_CD 
>>> Little_My_Maid  SMEE  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala
>>>
>>>
>>> SMEE is missing.
>>>
>>>
>>>
>>>
>>> Thanks.
>>>
>>>
>>> On Thu, 25 Apr 2019 20:55:59 +0300
>>> Andrei Borzenkov <arvidjaar@gmail.com> wrote:
>>>
>>>> 25.04.2019 16:34, MASAKI Haruka пишет:
>>>>  [...]  
>>>>
>>>> Please show
>>>>
>>>> 1. Full protocol of commands you executed (commands and their output),
>>>> e.g. captured using script.
>>>> 2. Commands used to determine missing files on both source and receive
>>>> sides.
>>>> 3. Listing of subvolumes on both source and receive sides with
>>>>
>>>> btrfs sub li -Rqu /mount/point
>>>>
>>>>  [...]  
>>>>
>>

Re: Some file lost when send back snapshot.

[MASAKI Haruka]

On Fri, 26 Apr 2019 21:27:52 +0300
Andrei Borzenkov <arvidjaar@gmail.com> wrote:

> 26.04.2019 21:12, MASAKI Haruka пишет:
> >> You have two subvolumes with the same received_uuid. Is the content of
> >> these subvolumes the same? Is "missing" file present in both subvolumes?
> > 
> > No, each subvolume name (190312-010100, 190315-001000) is timestamp.
> > These snapshot volumes were created by job scheduler on original (already lost) volume, so 190315-001000 is modified from 190312-010100.
> > 
> 
> Which is your problem. Two subvolumes with the same received_uuid are
> expected to have exactly the same content (received_uuid can be viewed
> as dataset UUID). You send incremental stream against base uuid
> 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2; on receiving side btrfs receive
> looks for matching base subvolume and happens to pick 190312-010100.
> 
> There were patches to automatically remove received_uuid when subvolume
> is changed to read-write. I do not know whether they were applied.
> 
> btrfs send/receive expects subvolumes to remain read-only. As soon as
> you start fiddling with read-only status you already lost. btrfs should
> have never allowed to change read-only into read-write in the first place.
> 

Thanks it was solved!

I remember change attributes on original volume.

I deleted 190312-010100 and send back1, it's OK.
I learned that shouldn't change subvolume read-only to read-write.

Thank you very much!!!



> > 190315-001000 has lost file,
> > % ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CD/
> > Little_My_Maid  SMEE  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala
> > 
> > but 190312-010100 doesn't.
> > % ls /HySlave/190312-010100/local/pub/media/sound/notmusic_in_CD/
> > Little_My_Maid  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala
> > 
> > 
> > 
> > 
> > On Fri, 26 Apr 2019 20:30:07 +0300
> > Andrei Borzenkov <arvidjaar@gmail.com> wrote:
> > 
> >> 26.04.2019 7:50, MASAKI Haruka пишет:
> >>> Thank you for your reply.
> >>>
> >>> # What does
> >>>
> >>> @Host A ("lily" master, new volume)
> >>> aki@lily /mnt % socat TCP-LISTEN:25500 STDOUT | sudo btrfs receive -vv . 
> >>>
> >>> @Host B ("daisy" backup, having volumes)
> >>> aki@daisy /HySlave % sudo btrfs send -v 190315-001000 | socat STDIN TCP:lily.local:25500 
> >>>
> >>> ( ...Transfared 11TB... )
> >>>
> >>> @Host A
> >>> aki@lily /mnt % sudo btrfs subvolume snapshot 190315-001000 world
> >>> Create a snapshot of '190315-001000' in './world'
> >>> aki@lily /mnt % sudo btrfs subvolume snapshot -r world back1
> >>> Create a readonly snapshot of 'world' in './back1'
> >>>
> >>> @Host A
> >>> aki@lily /mnt % sudo btrfs send -vv -p 190315-001000 back1 | socat STDIN tcp-connect:192.168.1.17:25555
> >>> At subvol back1
> >>> BTRFS_IOC_SEND returned 0
> >>> joining genl thread
> >>>
> >>> @Host B
> >>> root@daisy /HySlave # socat TCP-LISTEN:25555 STDOUT | btrfs receive -vv .
> >>> At snapshot back1
> >>> receiving snapshot back1 uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, ctransid=2658 parent_uuid=9c10b77f-74fb-f94a-9a24-3c6d30c00ca2, parent_ctransid=2658
> >>> BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=32dda1a2-07bd-684f-b773-c5b4cd5f29c1, stransid=2658
> >>>
> >>> # Host A current
> >>>
> >>> aki@lily /mnt % ls
> >>> 190315-001000  back1  back2  world
> >>> aki@lily /mnt % sudo btrfs sub li -Rqu .
> >>> ID 258 gen 2698 top level 5 parent_uuid -                                    received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid bc90d05c-7430-7b42-b683-702d906bb993 path 190315-001000
> >>> ID 2139 gen 2696 top level 5 parent_uuid bc90d05c-7430-7b42-b683-702d906bb993 received_uuid -                                    uuid 23c5fede-a338-d14a-afcd-677db2e97637 path world
> >>> ID 2140 gen 2660 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid -                                    uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 path back1
> >>> ID 2142 gen 2672 top level 5 parent_uuid 23c5fede-a338-d14a-afcd-677db2e97637 received_uuid -                                    uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b path back2
> >>>
> >>>
> >>> # Host B current
> >>>
> >>> aki@daisy /HySlave % pwd
> >>> /HySlave
> >>> aki@daisy /HySlave % ls
> >>> 190312-010100  190315-001000  back1  back2
> >>> [aki@daisy HySlave]$ sudo btrfs sub li -Rqu .
> >>> ID 256 gen 36238 top level 5 parent_uuid -                                    received_uuid -                                    uuid 4e42555d-a715-0e47-bd0e-85d6eb8911e0 path slave
> >>> ID 259 gen 35116 top level 256 parent_uuid -                                    received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 path 190312-010100
> >>> ID 3059 gen 7273 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid 9c10b77f-74fb-f94a-9a24-3c6d30c00ca2 uuid 114cd74f-80d8-af44-a9d0-fd8ce3c991b4 path 190315-001000
> >>
> >> You have two subvolumes with the same received_uuid. Is the content of
> >> these subvolumes the same? Is "missing" file present in both subvolumes?
> >>
> >>> ID 3072 gen 35112 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid 32dda1a2-07bd-684f-b773-c5b4cd5f29c1 uuid fe7281d1-8b3e-8f46-a0b4-cdc52a1bb7aa path back1
> >>> ID 3073 gen 36236 top level 256 parent_uuid 4a91d585-ab25-0c49-b183-ae951b5377e1 received_uuid a8e3e041-c1fc-e443-9baf-ab481ae32b8b uuid 65206183-f20d-4648-83ab-e0485b16f7af path back2
> >>>
> >>> # Files missing
> >>>
> >>> aki@daisy /HySlave % ls /HySlave/190315-001000/local/pub/media/sound/notmusic_in_CD 
> >>> Little_My_Maid  SMEE  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala
> >>>
> >>> aki@daisy /HySlave % ls /HySlave/back1/local/pub/media/sound/notmusic_in_CD
> >>> Little_My_Maid  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala
> >>>
> >>> aki@lily /mnt % ls /mnt/190315-001000/local/pub/media/sound/notmusic_in_CD
> >>> Little_My_Maid  SMEE  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala
> >>>
> >>> ki@lily /mnt % ls /mnt/back1/local/pub/media/sound/notmusic_in_CD        
> >>> Little_My_Maid  SMEE  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala
> >>>
> >>> aki@lily /mnt % ls /mnt/world/local/pub/media/sound/notmusic_in_CD 
> >>> Little_My_Maid  SMEE  el_louis  firstkiss  gakuensai  infinity-kid  kid-memories-off  ojyo_tokkyu  samusupi  wizards_harmony  yuukyuu-chara-seala
> >>>
> >>>
> >>> SMEE is missing.
> >>>
> >>>
> >>>
> >>>
> >>> Thanks.
> >>>
> >>>
> >>> On Thu, 25 Apr 2019 20:55:59 +0300
> >>> Andrei Borzenkov <arvidjaar@gmail.com> wrote:
> >>>
> >>>> 25.04.2019 16:34, MASAKI Haruka пишет:
> >>>>  [...]  
> >>>>
> >>>> Please show
> >>>>
> >>>> 1. Full protocol of commands you executed (commands and their output),
> >>>> e.g. captured using script.
> >>>> 2. Commands used to determine missing files on both source and receive
> >>>> sides.
> >>>> 3. Listing of subvolumes on both source and receive sides with
> >>>>
> >>>> btrfs sub li -Rqu /mount/point
> >>>>
> >>>>  [...]  
> >>>>
> >>
> 

Re: Some file lost when send back snapshot.

[Chris Murphy]

On Fri, Apr 26, 2019 at 12:27 PM Andrei Borzenkov <arvidjaar@gmail.com> wrote:
>
> btrfs send/receive expects subvolumes to remain read-only. As soon as
> you start fiddling with read-only status you already lost. btrfs should
> have never allowed to change read-only into read-write in the first place.

I'm in agreement, and would like to see 'btrfs property set <> ro
false> removed. It's confusing. Once a subvolume is read-only, it
should remain read-only, and if the use case requires rw, then just
snapshot it (default rw).


-- 
Chris Murphy