Re: What mechanisms protect against split brain?

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

On 2022/6/8 18:58, Wang Yugui wrote:
> Hi,
>
>> On 2022/6/8 18:15, Wang Yugui wrote:
>>> Hi, Forza, Qu Wenruo
>>>
>>> I write a script to test RAID1 split brain base on Qu's work of raid5(*1)
>>> *1: https://lore.kernel.org/linux-btrfs/53f7bace2ac75d88ace42dd811d48b7912647301.1654672140.git.wqu@suse.com/T/#u
>>
>> No no no, that is not to address split brain, but mostly to drop cache
>> for recovery path to maximize the chance of recovery.
>>
>> It's not designed to solve split brain problem at all, it's just one
>> case of such problem.
>>
>> In fact, fully split brain (both have the same generation, but
>> experienced their own degraded mount) case can not be solved by btrfs
>> itself at all.
>>
>> Btrfs can only solve partial split brain case (one device has higher
>> generation, thus btrfs can still determine which copy is the correct one).
>>
>>>
>>> #!/bin/bash
>>> set -uxe -o pipefail
>>>
>>> mnt=/mnt/test
>>> dev1=/dev/vdb1
>>> dev2=/dev/vdb2
>>>
>>>     dmesg -C
>>>     mkdir -p $mnt
>>>
>>>     mkfs.btrfs -f -m raid1 -d raid1 $dev1 $dev2
>>>     mount $dev1 $mnt
>>>     xfs_io -f -c "pwrite -S 0xee 0 1M" $mnt/file1
>>>     sync
>>>     umount $mnt
>>>
>>>     btrfs dev scan -u $dev2
>>>     mount -o degraded $dev1 $mnt
>>>     #xfs_io -f -c "pwrite -S 0xff 0 128M" $mnt/file2
>>>     mkdir -p $mnt/branch1; /bin/cp -R /usr/bin $mnt/branch1 #complex than xfs_io
>>>     umount $mnt
>>>
>>>     btrfs dev scan
>>>     btrfs dev scan -u $dev1
>>>     mount -o degraded $dev2 $mnt
>>
>> Your case is the full split brain case.
>>
>> Not possible to solve.
>>
>> In fact, if you don't do the degraded mount on dev2, btrfs is completely
>> fine to resilve the fs without any problem.
>
> step1: we mark btrfs/RAID1 with degraded write as not-clean-RAID1.

Then when to clean?
Full scrub or some timing else?

> step2: in that state, we default try to read copy 0 of RAID1
> 	current pid based i/o patch select policy
>             preferred_mirror = first + (current->pid % num_stripes);

That's feasible, but still need an ondisk format change.

Furthermore, this idea can also be done by a more generic way,
write-intent bitmap.

In fact, DM layer uses this to speed up resilver, and handle split brain
cases.

With write-intent bitmap, every degraded write will leave the record in
the write-intent bitmap until properly resilvered.

Thanks,
Qu

>
> this idea seem to work?
>
> degraded RAID1 write is almost the same as full split brain?
>
> Best Regards
> Wang Yugui (wangyugui@e16-tech.com)
> 2022/06/08
>
>