grub_probe/grub-mkimage does not find all drives in BTRFS RAID1

grub_probe/grub-mkimage does not find all drives in BTRFS RAID1

[Matthew Hawn]

This is almost definitely a bug in GRUB, but I wanted to get the btrfs mailing list opinion first.

Symptoms:
I have a btrfs raid1 /boot and root filesystem.  Ever since I replaced a drive, when I run the grub utilities to create my grub.cfg and install to boot sector, it only recognizes one of the drives.

$ sudo grub-probe /boot/grub -t device
/dev/mapper/VG_BTRFS2-LV_BOOT2

$ sudo grub-probe /boot/grub -t bios_hints
lvmid/gEfhOx-J9hr-8tkA-OgjD-Aqqu-XR2T-sFB4me/oNnMDp-Rit5-P0qs-QZlf-bQQe-tZU7-Wwmz8z

This also prevents boot if the above drive is disconnected. Grub error in locating lvmid/gEfhOx-J9hr-8tkA-OgjD-Aqqu-XR2T-sFB4me/oNnMDp-Rit5-P0qs-QZlf-bQQe-tZU7-Wwmz8z

Boot works fine if both drives, or only the above drive is present.

Before drive replacement, the above commands returned both drives that were part of the RAID1 mirror.  I never tried booting with a device disconnected, but both showed up in my grub.cfg.   Replacement was not standard since the prior drive was developing bad sectors, but had not failed. Replacement was done by adding a third disk to the mirror, then removing the 1st disk.

Probable Cause:
To determine the boot drive, grub-probe and grub-mkimage make several ioctl  in osdep/linux/getroot.c: grub_find_root_devices_from_btrfs
A call to BTRFS_IOC_FS_INFO gets the max_id and num_devices.  It then iterates from 1 to max_id, calling BTRFS_IOC_DEV_INFO to get the path.  

For my system, max_id = 3 and num_devices = 2.  Requesting BTRFS_IOC_DEV_INFO for device 1 yields a "No Such Device".  Instead of continuing on to device 2 and 3 (which return without error), grub treats all ioctl errors as fatal, exits the btrfs specific code with a failure, then falls back to generic linux code that only detects the single drive. 

So, is this a grub bug? If so, any suggestions before I submit to the grub-bug list?  Also, as I wait until a fix is published (or I rebuild grub with my own patch), any ideas to workaround this?


Info:
Kernel:  4.15.0-12-generic #13-Ubuntu SMP (based on 4.15.7 mainline)
btrfs-progs: 4.15.1-1
Grub:  2.02~beta2-36ubuntu3.17 

$ btrfs fi show
Label: none  uuid: 84c8e78b-9d7f-4451-966d-3c25154e89b8
	Total devices 2 FS bytes used 22.16GiB
	devid    2 size 100.00GiB used 25.03GiB path /dev/mapper/VG_BTRFS2-LV_ROOT2
	devid    3 size 100.00GiB used 25.03GiB path /dev/mapper/VG_BTRFS3-LV_ROOT3


Label: none  uuid: 059ab98f-eb63-471d-b099-6561baf39040
	Total devices 2 FS bytes used 61.04GiB
	devid    2 size 200.00GiB used 62.03GiB path /dev/mapper/VG_BTRFS2-LV_HOME2
	devid    3 size 200.00GiB used 62.03GiB path /dev/mapper/VG_BTRFS3-LV_HOME3


Label: none  uuid: ffe8b1a0-030c-42c2-94f5-b7e8e54b1439
	Total devices 2 FS bytes used 342.04MiB
	devid    2 size 1.00GiB used 693.62MiB path /dev/mapper/VG_BTRFS2-LV_BOOT2
	devid    3 size 1.00GiB used 693.62MiB path /dev/mapper/VG_BTRFS3-LV_BOOT3

Re: grub_probe/grub-mkimage does not find all drives in BTRFS RAID1

[Duncan]

Matthew Hawn posted on Thu, 22 Mar 2018 00:13:38 +0000 as excerpted:

> This is almost definitely a bug in GRUB, but I wanted to get the btrfs
> mailing list opinion first.
> 
> Symptoms:
> I have a btrfs raid1 /boot and root filesystem.  Ever since I replaced a
> drive, when I run the grub utilities to create my grub.cfg and install
> to boot sector, it only recognizes one of the drives.

> So, is this a grub bug? If so, any suggestions before I submit to the
> grub-bug list?  Also, as I wait until a fix is published (or I rebuild
> grub with my own patch), any ideas to workaround this?

It does appear to be a grub bug, yes.  I'm not a dev, just a btrfs user 
and list regular, so won't comment further on that, but I do have a 
suggested workaround based on what I do here.

> Grub:  2.02~beta2-36ubuntu3.17

FWIW, 2.02 is out (and installed here on gentoo, gentoo git log says it 
was committed to the gentoo tree on 2017-04-27, and ftp.gnu.org has a 
date of 2017-04-26, so it's out nearing a year now, and gentoo picked it 
up right away =:^), but I don't know if that bug is fixed.

> $ btrfs fi show
> Label: none  uuid: 84c8e78b-9d7f-4451-966d-3c25154e89b8
> 	Total devices 2 FS bytes used 22.16GiB
> 	devid    2 size 100.00GiB used 25.03GiB
> path /dev/mapper/VG_BTRFS2-LV_ROOT2
> 	devid    3 size 100.00GiB used 25.03GiB
> path /dev/mapper/VG_BTRFS3-LV_ROOT3
> 
> 
> Label: none  uuid: 059ab98f-eb63-471d-b099-6561baf39040
> 	Total devices 2 FS bytes used 61.04GiB
> 	devid    2 size 200.00GiB used 62.03GiB 
> path /dev/mapper/VG_BTRFS2-LV_HOME2
> 	devid    3 size 200.00GiB used 62.03GiB
> path /dev/mapper/VG_BTRFS3-LV_HOME3
> 
> 
> Label: none  uuid: ffe8b1a0-030c-42c2-94f5-b7e8e54b1439
> 	Total devices 2 FS bytes used 342.04MiB
> 	devid    2 size 1.00GiB used 693.62MiB 
> path /dev/mapper/VG_BTRFS2-LV_BOOT2
> 	devid    3 size 1.00GiB used 693.62MiB
> path /dev/mapper/VG_BTRFS3-LV_BOOT3

Good, you have a separate dedicated boot.  That will make my suggestion 
/much/ easier. =:^)

OT: I still remember the big deal of my first GB drive upgrade, not even 
a full GiB, and get a bit of "historical vertigo" seeing "a whole gig!!" 
used as a dedicated /boot...


So my setup here now uses mostly btrfs raid1, with a separate /boot, but 
I came up with the general layout years ago, before btrfs, when I was 
using mdraid, then with grub1 (technically 0.97 with a bunch of patches 
after upstream abandoned further development, as they never released 
grub1, dumping it for what became grub2 before 1.0 release).

And the much simpler grub1 didn't really understand mdraid, tho it could 
work with mdraid1 simply because grub treated each mdraid1 device as a 
single device.  But you had to install grub1 to each of the mdraid1 
devices separately if you wanted to be able to boot any of them, as it 
didn't have grub2-core's ability to dynamically detect other devices to 
load further grub modules from.

Meanwhile, while I was running mdraid and additionally had a backup raid 
for most of my filesystems, including root, that I could boot to if my 
primary/working raid and filesystem wouldn't assemble and/or mount for 
whatever reason, because grub1 could only point to one /boot for its 
stage2, I couldn't have a backup /boot raid1 on the same set of physical 
devices as my primary, which meant that if something other than a bad 
device happened to that /boot that made it unbootable, say I fat-fingered 
a mkfs and overwrote /boot, or if a grub upgrade went bad, I was stuck.

But at the time I was using 4-way mdraid1, 4 mirrors, for my primary 
raid, and that gave me an idea.  Instead of the usual 4 mirrors for each 
filesystem and its backup, as I had for most filesystems, for /boot I did 
two, two-way raid1s, each on two two devices, with one of the two two-way 
raid1s being the primary /boot and the second being the backup.  That way 
I could have the BIOS boot selector default to one of the primaries, and 
could still have it select one of the secondaries if I needed to boot the 
backup grub and its stage2 located in the backup /boot raid.

That came in quite handy when I upgraded to grub2, since I could upgrade 
the backup /boot raid1 and its grub to grub2, then test and configure it 
to my satisfaction, all the while keeping the grub1 installed in my 
normal working copy raid1 /boot untouched until I was satisfied that the 
grub2 install on the backup was functional and configured as I wanted, 
and upgrading the working copy raid1 /boot to grub2 only after I knew 
things were working well.


Much later, after some hardware upgrades so I wasn't using the 4-way 
raid1 setup any longer, I switched to btrfs and btrfs raid1.  But I so 
well liked the concept of having a backup /boot so I didn't have to worry 
about bad upgrades or fat-fingering my /boot, that I kept it!

But in addition to two-way raid1 redundancy on multiple devices, btrfs 
has the dup mode, two-way dup redundancy on a single device, so that's 
what I do with my /boot and its backups on other devices now, instead of 
making them raid1s across multiple devices.

So while most of my filesystems and their backups are btrfs raid1 both 
data and metadata across two physical devices (with another pair of 
physical devices for the btrfs raid1 backups), /boot and its backups are 
all btrfs dup mixed-bg-mode (so data and metadata mixed, easier to work 
with on small filesystems), giving me one primary /boot and three 
backups, and I can still select which one to boot from the hardware/BIOS 
(legacy not EFI mode, tho I do use GPT and have EFI-boot partitions 
reserved in case I decide to switch to EFI at some point).


So my suggestion would be to do something similar, multiple /boot, one 
per device, one as the working copy and the other(s) as backups, instead 
of btrfs raid1 across multiple devices.  If you still want to take 
advantage of btrfs' ability to error-correct from a second copy if the 
first fails checksum, as I do, btrfs dup mode is useful, but regardless, 
you'll then have a backup in case the working /boot entirely fails.  Tho 
of course with dup mode you can only use a bit under half the capacity.

Your btrfs fi show says 342 MB used (as data) of the 1 GiB, so dup mode 
should be possible as you'd have a bit under 500 MiB capacity then.  Your 
individual devices say nearly 700 MiB each used, but with only 342 MiB of 
that as data, the rest is likely partially used chunks that a filtered 
balance can take care of.  A btrfs fi usage report would tell the details 
(or btrfs fi df, combined with the show you've already posted).  At a 
GiB, creating the filesystem as mixed-mode is also recommended, tho that 
does make a filtered balance a bit more of a hassle since you have to use 
the same filters for both data and metadata because they're the same 
chunks.

FWIW, I started out with 256 MiB /boot here, btrfs dup mode so ~ 100 MiB 
usable, but after ssd upgrades and redoing the layout, now use 512 MiB 
/boots, for 200+ MiB usable.  That's better.  Your 1 GiB doubles that, so 
should be no trouble at all, even with dup, unless you're storing way 
more in /boot than I do.  (Being gentoo I do configure and run a rather 
slimmer custom initramfs and monolithic kernel configured for only the 
hardware and dracut initr* modules I need, and a fatter generic initr* 
and kernel modules would likely need more space, but your show output 
says it's only using 342 MiB for data, so as I said your 1 GiB for ~500 
MiB usable in dup mode should be quite reasonable.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman