Re: raid1 root device with efi

[Goffredo Baroncelli <kreijack@libero.it>]

On 11/03/2024 15.34, Kai Krakow wrote:
> Hello!
> 
> Am So., 10. März 2024 um 19:18 Uhr schrieb Forza <forza@tnonline.net>:
>>
>>
>>
>> On 2024-03-08 22:58, Matt Zagrabelny wrote:
>>> On Fri, Mar 8, 2024 at 3:54 PM Matthew Warren
>>> <matthewwarren101010@gmail.com> wrote:
>>>>
>>>> On Fri, Mar 8, 2024 at 4:48 PM Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:
>>>>>
>>>>> Hi Qu and Matthew,
>>>>>
>>>>> On Fri, Mar 8, 2024 at 3:46 PM Matthew Warren
>>>>> <matthewwarren101010@gmail.com> wrote:
>>>>>>
>>>>>> On Fri, Mar 8, 2024 at 3:46 PM Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:
>>>>>>>
>>>>>>> Greetings,
>>>>>>>
>>>>>>> I've read some conflicting info online about the best way to have a
>>>>>>> raid1 btrfs root device.
> 
> I think the main issue here that leads to conflicting ideas is:
> 
> Grub records the locations (or extent index) of the boot files during
> re-configuration for non-trivial filesystems. If you later move the
> files, or need to switch to the mirror, it will no longer be able to
> read the boot files. Grub doesn't have a full btrfs implementation to
> read all the metadata, nor does it know or detect the member devices
> of the pool.

I don't think that what you describe is really accurate. Grub (in the NON uefi version)
stores some code in the first 2MB of the disk (this is one of the reason why fdisk by
default starts the first partition at the 1st MB of the disk). This code is mapped as
you wrote. And if you mess with this disk area grub gets confused.

And the btrfs grub module is stored in this area. After this module is loaded, grub
has a full access to a btrfs partition.

The fact in some condition grub is not able to access anymore to a btrfs filesystem
is more related to a not mature btrfs implementation in grub.

I am quite sure that grub access a btrfs filesystem dynamically, without using a
pre-stored table with the location of a file.

To verify that, try to access a random file or directory in a btrfs location (e.g.
ls /bin) that is not related to a 'boot' process.

> So in this context, it supports btrfs raid1 under certain
> conditions, if, and only if, just two devices are used, and the grub
> device remains the same. If you add a third device, both raid1 stripes
> for boot files may end up on devices of the pool that grub doesn't
> consider. As an example, bees is known to mess up grub boot on btrfs
> because it relocates the boot files without letting grub know:
> https://github.com/Zygo/bees/issues/249
> 
> I'd argue that grub can only boot reliably from single-device btrfs
> unless you move boot file extents without re-configuring it. Grub only
> has very basic support for btrfs.
> 
> mdadm for ESP is not supported for very similar reasons (because EFI
> doesn't open the filesystem read-only): It will break the mirror.
> 
> The best way, as outlined in the thread already, is two have two ESP,
> not put the kernel boot files in btrfs but in ESP instead, and adjust
> your kernel-install plugins to mirror the boot files to the other ESP
> partition.
> 
> Personally, I've got a USB stick where I keep a copy of my ESP created
> with major configuration changes (e.g. major kernel update, boot
> configuration changes), and the ESP is also included in my daily
> backup. I keep blank reserve partitions on all other devices which I
> can copy the ESP to in case of disaster. This serves an additional
> purpose of keeping some part of the devices trimmed for wear-leveling.
> 
> 
>>>>>>>
>>>>>>> I've got two disks, with identical partitioning and I tried the
>>>>>>> following scenario (call it scenario 1):
>>>>>>>
>>>>>>> partition 1: EFI
>>>>>>> partition 2: btrfs RAID1 (/)
>>>>>>>
>>>>>>> There are some docs that claim that the above is possible...
>>>>>>
>>>>>> This is definitely possible. I use it on both my server and desktop with GRUB.
>>>>>
>>>>> Are there any docs you follow for this setup?
>>>>>
>>>>> Thanks for the info!
>>>>>
>>>>> -m
>>>>
>>>> The main important thing is that mdadm has several metadata versions.
>>>> Versions 0.9 and 1.0 store the metadata at the end of the partition
>>>> which allows UEFI to think the filesystem is EFI rather than mdadm
>>>> raid.
>>>> https://raid.wiki.kernel.org/index.php/RAID_superblock_formats#Sub-versions_of_the_version-1_superblock
>>>>
>>>> I followed the arch wiki for setting it up, so here's what I followed.
>>>> https://wiki.archlinux.org/title/EFI_system_partition#ESP_on_software_RAID1
>>>
>>> Thanks for the hints. Hopefully there aren't any more unexpected issues.
>>>
>>> Cheers!
>>>
>>> -m
>>>
>>
>> An alternative to mdadm is to simply have separate ESP partitions on
>> each device. You can manually copy the contents between the two if you
>> were to update the EFI bootloader. This way you can keep the 'other' ESP
>> as backup during GRUB/EFI updates.
>>
>> This solution is what I use on one of my servers. GRUB2 supports Btrfs
>> RAID1 so you do not need to have the kernel and initramfs on the ESP,
>> though that works very well too.
>>
>> Good Luck!
>>
>> ~Forza
> 
> Regards,
> Kai
> 

-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5