Re: [PATCH] btrfs: reject device with CHANGING_FSID_V2

[Anand Jain <anand.jain@oracle.com>]

On 19/09/2023 06:44, David Sterba wrote:
> On Wed, Aug 16, 2023 at 08:30:40PM +0800, Anand Jain wrote:
>> The BTRFS_SUPER_FLAG_CHANGING_FSID_V2 flag indicates a transient state
>> where the device in the userspace btrfstune -m|-M operation failed to
>> complete changing the fsid.
>>
>> This flag makes the kernel to automatically determine the other
>> partner devices to which a given device can be associated, based on the
>> fsid, metadata_uuid and generation values.
>>
>> btrfstune -m|M feature is especially useful in virtual cloud setups, where
>> compute instances (disk images) are quickly copied, fsid changed, and
>> launched. Given numerous disk images with the same metadata_uuid but
>> different fsid, there's no clear way a device can be correctly assembled
>> with the proper partners when the CHANGING_FSID_V2 flag is set. So, the
>> disk could be assembled incorrectly, as in the example below:
>>
>> Before this patch:
>>
>> Consider the following two filesystems:
>>     /dev/loop[2-3] are raw copies of /dev/loop[0-1] and the btrsftune -m
>> operation fails.
>>
>> In this scenario, as the /dev/loop0's fsid change is interrupted, and the
>> CHANGING_FSID_V2 flag is set as shown below.
>>
>>    $ p="device|devid|^metadata_uuid|^fsid|^incom|^generation|^flags"
>>
>>    $ btrfs inspect dump-super /dev/loop0 | egrep '$p'
>>    superblock: bytenr=65536, device=/dev/loop0
>>    flags			0x1000000001
>>    fsid			7d4b4b93-2b27-4432-b4e4-4be1fbccbd45
>>    metadata_uuid		bb040a9f-233a-4de2-ad84-49aa5a28059b
>>    generation		9
>>    num_devices		2
>>    incompat_flags	0x741
>>    dev_item.devid	1
>>
>>    $ btrfs inspect dump-super /dev/loop1 | egrep '$p'
>>    superblock: bytenr=65536, device=/dev/loop1
>>    flags			0x1
>>    fsid			11d2af4d-1b71-45a9-83f6-f2100766939d
>>    metadata_uuid		bb040a9f-233a-4de2-ad84-49aa5a28059b
>>    generation		10
>>    num_devices		2
>>    incompat_flags	0x741
>>    dev_item.devid	2
>>
>>    $ btrfs inspect dump-super /dev/loop2 | egrep '$p'
>>    superblock: bytenr=65536, device=/dev/loop2
>>    flags			0x1
>>    fsid			7d4b4b93-2b27-4432-b4e4-4be1fbccbd45
>>    metadata_uuid		bb040a9f-233a-4de2-ad84-49aa5a28059b
>>    generation		8
>>    num_devices		2
>>    incompat_flags	0x741
>>    dev_item.devid	1
>>
>>    $ btrfs inspect dump-super /dev/loop3 | egrep '$p'
>>    superblock: bytenr=65536, device=/dev/loop3
>>    flags			0x1
>>    fsid			7d4b4b93-2b27-4432-b4e4-4be1fbccbd45
>>    metadata_uuid		bb040a9f-233a-4de2-ad84-49aa5a28059b
>>    generation		8
>>    num_devices		2
>>    incompat_flags	0x741
>>    dev_item.devid	2
>>
>>
>> It is normal that some devices aren't instantly discovered during
>> system boot or iSCSI discovery. The controlled scan below demonstrates
>> this.
>>
>>    $ btrfs device scan --forget
>>    $ btrfs device scan /dev/loop0
>>    Scanning for btrfs filesystems on '/dev/loop0'
>>    $ mount /dev/loop3 /btrfs
>>    $ btrfs filesystem show -m
>>    Label: none  uuid: 7d4b4b93-2b27-4432-b4e4-4be1fbccbd45
>> 	Total devices 2 FS bytes used 144.00KiB
>> 	devid    1 size 300.00MiB used 48.00MiB path /dev/loop0
>> 	devid    2 size 300.00MiB used 40.00MiB path /dev/loop3
>>
>> /dev/loop0 and /dev/loop3 are incorrectly partnered.
>>
>> This kernel patch removes functions and code connected to the
>> CHANGING_FSID_V2 flag.
>>
>> With this patch, now devices with the CHANGING_FSID_V2 flag are rejected.
>> And its partner will fail to mount with the extra -o degraded option.
>>
>> Signed-off-by: Anand Jain <anand.jain@oracle.com>
>> ---
>> Moreover, a btrfs-progs patch (below) has eliminated the use of the
>> CHANGING_FSID_V2 flag entirely:
>>
>>     [PATCH RFC] btrfs-progs: btrfstune -m|M remove 2-stage commit
>>
>> And we solve the compatability concerns as below:
>>
>>    New-kernel new-progs - has no CHANGING_FSID_V2 flag.
>>    Old-kernel new-progs - has no CHANGING_FSID_V2 flag, kernel code unused.
>>    Old-kernel old-progs - bug may occur.
>>    New-kernel old-progs - Should use host with the newer btrfs-progs to fix.
>>
>> For legacy systems to help fix such a condition in the userspace instead
>> we have the below patchset which ports of kernel's CHANGING_FSID_V2 code.
>>
>>     [PATCH 00/16] btrfs-progs: recover from failed metadata_uuid
>>
>> And if it couldn't fix in some cases, users can use manually reunite,
>> with the patchset:
>>
>>     [PATCH 00/10] btrfs-progs: check and tune: add device and noscan options
>>
>>   fs/btrfs/disk-io.c |  10 ---
>>   fs/btrfs/volumes.c | 166 ++++-----------------------------------------
>>   fs/btrfs/volumes.h |   1 -
>>   3 files changed, 13 insertions(+), 164 deletions(-)
> 
> Please split the kernel patch in two, one rejecting the CHANGING_FSID_V2
> bit and then removing the unused code.

Yep. Will do.

> I think the scanning code still
> has to recognize the bit and skip the device, I haven't checked if
> remains like that after this patch.

Right. This patch does it. We only have to do it at device_list_add()
because mount, device-scan, and device-ready all rely on
device_list_add() to scan.

Thanks, Anand