Re: What mechanisms protect against split brain?

["Austin S. Hemmelgarn" <ahferroin7@gmail.com>]

On 08/06/2022 06.32, Qu Wenruo wrote:
> In fact, fully split brain (both have the same generation, but
> experienced their own degraded mount) case can not be solved by btrfs
> itself at all.
> 
> Btrfs can only solve partial split brain case (one device has higher
> generation, thus btrfs can still determine which copy is the correct one).
Of note, this is not unique to BTRFS. The quorum requirement that Ceph 
and many other distributed storage systems impose on writes exists to 
very specifically avoid this type of situation.
> 
>>
>> #!/bin/bash
>> set -uxe -o pipefail
>>
>> mnt=/mnt/test
>> dev1=/dev/vdb1
>> dev2=/dev/vdb2
>>
>>    dmesg -C
>>    mkdir -p $mnt
>>
>>    mkfs.btrfs -f -m raid1 -d raid1 $dev1 $dev2
>>    mount $dev1 $mnt
>>    xfs_io -f -c "pwrite -S 0xee 0 1M" $mnt/file1
>>    sync
>>    umount $mnt
>>
>>    btrfs dev scan -u $dev2
>>    mount -o degraded $dev1 $mnt
>>    #xfs_io -f -c "pwrite -S 0xff 0 128M" $mnt/file2
>>    mkdir -p $mnt/branch1; /bin/cp -R /usr/bin $mnt/branch1 #complex 
>> than xfs_io
>>    umount $mnt
>>
>>    btrfs dev scan
>>    btrfs dev scan -u $dev1
>>    mount -o degraded $dev2 $mnt
> 
> Your case is the full split brain case.
> 
> Not possible to solve.
> 
> In fact, if you don't do the degraded mount on dev2, btrfs is completely
> fine to resilve the fs without any problem.
> 
And this, in turn, is why BTRFS refuses to mount degraded without the 
user explicitly asking for it, and why having `degraded` in your mount 
options in `/etc/fstab` (or on the kernel command line) is so dangerous. 
There’s no way for BTRFS (or the block layer for that matter) to 
reliably differentiate between a missing device resulting from a device 
failure and a missing device resulting from other issues, and those 
other issues can easily result in one half of a two-device volume not 
being present for one boot, and the other half not being present on the 
next boot.