BTRFS partitioning scheme (was BTRFS with RAID1 cannot boot when removing drive)

BTRFS partitioning scheme (was BTRFS with RAID1 cannot boot when removing drive)

[Saint Germain]

Hello and thanks for your feedback !

Cc back to the mailing-list as it may be of interest here as well.

On 11 February 2014 16:11, Kyle Gates <kylegates@hotmail.com> wrote:
>> The big problem I currently have is that based on your input, I
>> hesitate a lot on my partitioning scheme: should I use a dedicated
>> /boot partition or should I have one global BTRFS partition ?
>> It is not very clear in the doc (a lof of people used a dedicated
>> /boot because at that time, grub couldn't natively boot BTRFS it
>> seems, but it has changed).
>> Could you recommend a partitioning scheme for a simple RAID1 with 2
>> identical hard drives (just for home computing, not business).
>
> I run a 1GiB RAID1 btrfs /boot in mixed mode with grub2 and gpt partitions.
> IIRC grub2 doesn't understand lzo compression nor subvolumes.
>

Well I did tried to read about this and ended up being confused
because development is so fast, documentation can become quickly
outdated.
It seems that grub can boot BTRFS /boot subvolumes:
https://bbs.archlinux.org/viewtopic.php?pid=1222358

However Chris Murphy has some problems a few months ago:
http://comments.gmane.org/gmane.comp.file-systems.btrfs/29140

So I still don't know if it is a good idea or not to have a BTRFS /boot ?
Of course the idea is that I would like to snapshot /boot and have it on RAID1.

To summarize, I think I have 3 options for partitioning (I am not
considering UEFI secure boot or swap):
1) grub, BTRFS partition (i.e. full disk in BTRFS), /boot inside BTRFS subvolume
2) grub, GPT partition, with (A) on sda1, and a BTRFS partition on
sda2, /boot inside BTRFS subvolume
3) grub, GPT partition, with (A) on sda1, /boot (ext4) on sda2, and a
BTRFS on sda3

(A) = BIOS Boot partition (1 MiB) or EFI System Partition (FAT32, 550MiB)

I don't really see the point of having UEFI/ESP if I don't use other
proprietary operating system, so I think I will go with (A) = BIOS
Boot partition except if there is someting I have missed.

Can someone recommend which one would be the most stable and easier to manage ?

Thanks in advance,

Re: BTRFS partitioning scheme (was BTRFS with RAID1 cannot boot when removing drive)

[Chris Murphy]

On Feb 11, 2014, at 10:02 AM, Saint Germain <saintger@gmail.com> wrote:

> Hello and thanks for your feedback !
> 
> Cc back to the mailing-list as it may be of interest here as well.
> 
> On 11 February 2014 16:11, Kyle Gates <kylegates@hotmail.com> wrote:
>>> The big problem I currently have is that based on your input, I
>>> hesitate a lot on my partitioning scheme: should I use a dedicated
>>> /boot partition or should I have one global BTRFS partition ?
>>> It is not very clear in the doc (a lof of people used a dedicated
>>> /boot because at that time, grub couldn't natively boot BTRFS it
>>> seems, but it has changed).
>>> Could you recommend a partitioning scheme for a simple RAID1 with 2
>>> identical hard drives (just for home computing, not business).
>> 
>> I run a 1GiB RAID1 btrfs /boot in mixed mode with grub2 and gpt partitions.
>> IIRC grub2 doesn't understand lzo compression nor subvolumes.

Any sufficiently recent version of GRUB2 understands both lzo, zlib. 

Grub doesn't directly understand the concept of subvolumes, but it treats them as directories, which is expected to work.

grub-mkconfig likewise is indirectly aware of subvolumes. If fstab indicates / is on Btrfs and there's a subvol= option, grub-mkconfig creates a rootflags=subvol= boot parameter in the resulting grub.cfg.


> Well I did tried to read about this and ended up being confused
> because development is so fast, documentation can become quickly
> outdated.
> It seems that grub can boot BTRFS /boot subvolumes:
> https://bbs.archlinux.org/viewtopic.php?pid=1222358

I think it's more accurate to say it can locate boot files on a subvolume by treating the subvolume as a directory. So as long as the path to those boot files is correct, it should work.

And yes it's true that distros are frequently very far behind with grub, and this poses problems. Fedora has been on grub 2.00 since release two years ago. Many other distros are still using pre-release grub2 versions 1.97 through 1.99 which isn't good. The distros should be pressured to move to grub 2.02, currently in beta, upon release. And I think it would be good for Btrfs testers to build grub 2.02 beta, and try to break it with various Btrfs configurations so that it can be better tested grub release.



> However Chris Murphy has some problems a few months ago:
> http://comments.gmane.org/gmane.comp.file-systems.btrfs/29140

The decision by Grub upstream is that Grub should always treat pathnames as starting from the top level of the file system, ID5, rather than starting from the user defined default subvolume. Otherwise, the user can instantly render their system unbootable just by changing the default subvolume.

subvol= option is intended to be read as starting from the top level of the file system regardless of the default subvolume. Probably more testing of this is needed, along with grub2, and other boot loaders.


> 
> So I still don't know if it is a good idea or not to have a BTRFS /boot ?
> Of course the idea is that I would like to snapshot /boot and have it on RAID1.

It's fine to do that, and it does work. You can also just have /boot as a directory within a root subvolume.


> 
> To summarize, I think I have 3 options for partitioning (I am not
> considering UEFI secure boot or swap):
> 1) grub, BTRFS partition (i.e. full disk in BTRFS), /boot inside BTRFS subvolume

This doesn't seem like a good idea for a boot drive to be without partitions.


> 2) grub, GPT partition, with (A) on sda1, and a BTRFS partition on
> sda2, /boot inside BTRFS subvolume
> 3) grub, GPT partition, with (A) on sda1, /boot (ext4) on sda2, and a
> BTRFS on sda3
> 
> (A) = BIOS Boot partition (1 MiB) or EFI System Partition (FAT32, 550MiB)
> 
> I don't really see the point of having UEFI/ESP if I don't use other
> proprietary operating system, so I think I will go with (A) = BIOS
> Boot partition except if there is someting I have missed.

You need to boot your system in UEFI and CSM-BIOS modes, and compare the dmesg for each. I'm finding it common the CSM limits power management, and relegates drives to IDE speeds rather than full SATA link speeds. Sometimes it's unavoidable to use the CSM if it has better overall behavior for your use case. I've found it to be lacking and have abandoned it. It's basically intended for booting Windows XP, right?

Chris Murphy

Re: BTRFS partitioning scheme (was BTRFS with RAID1 cannot boot when removing drive)

[Duncan]

Chris Murphy posted on Tue, 11 Feb 2014 11:15:56 -0700 as excerpted:

> The distros should be pressured to move to grub 2.02, currently in beta,
> upon release. And I think it would be good for Btrfs testers to build
> grub 2.02 beta, and try to break it with various Btrfs configurations so
> that it can be better tested grub release.

On the topic of grub minor update releases (and betas too), do you know 
where I can get a good grub-script-level sysadmin view of what has 
actually changed?

I'm running the betas, but a bit blind as the only update info I've been 
able to find is per-commit granularity, many commits per day, and I can't 
see the forest for all the trees!  I'd be far more comfortable with 
something rather more fuzzy than individual commits, but not so fuzzy and 
opaque as simply version number bumps.

This is of concern to me in part because I directly edit grub.cfg and 
includes, because the meta-level config "helpers" that are supposedly 
intended for "mere mortals" were far more fog than help to me, and I both 
couldn't get done what I wanted to do, and I needed to use the direct 
grub-script level commands anyway, and once I understood them, all the 
extra noise from the meta-level stuff was simply noisy cruft that 
obscured the real operation.  

Since I'm working at the direct grub-script level, I really need that 
level of detail on changes in updates as well.  Not the high-level-meta-
config stuff, not the per-commit stuff, something in between, analogous 
to the grub-script level but in a changelog.

And I need it for the betas as well as the full releases, since I'm 
running the betas.  Which means a changelog with the betas included so I 
could see what changed between betas as well as between general releases 
would be best.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: BTRFS partitioning scheme (was BTRFS with RAID1 cannot boot when removing drive)

[Saint Germain]

On 11 February 2014 19:15, Chris Murphy <lists@colorremedies.com> wrote:
>>
>> To summarize, I think I have 3 options for partitioning (I am not
>> considering UEFI secure boot or swap):
>> 1) grub, BTRFS partition (i.e. full disk in BTRFS), /boot inside BTRFS subvolume
>
> This doesn't seem like a good idea for a boot drive to be without partitions.
>
>
>> 2) grub, GPT partition, with (A) on sda1, and a BTRFS partition on
>> sda2, /boot inside BTRFS subvolume
>> 3) grub, GPT partition, with (A) on sda1, /boot (ext4) on sda2, and a
>> BTRFS on sda3
>>
>> (A) = BIOS Boot partition (1 MiB) or EFI System Partition (FAT32, 550MiB)
>>
>> I don't really see the point of having UEFI/ESP if I don't use other
>> proprietary operating system, so I think I will go with (A) = BIOS
>> Boot partition except if there is someting I have missed.
>
> You need to boot your system in UEFI and CSM-BIOS modes, and compare the dmesg for each. I'm finding it common the CSM limits power management, and relegates drives to IDE speeds rather than full SATA link speeds. Sometimes it's unavoidable to use the CSM if it has better overall behavior for your use case. I've found it to be lacking and have abandoned it. It's basically intended for booting Windows XP, right?
>

Ok based on your advices, here is what I have done so far to use UEFI
(remeber that the objective is to have a clean and simple BTRFS RAID1
install).

A) I start first with only one drive, I have gone with the following
partition scheme (Debian wheezy, kernel 3.12, grub 2.00, GPT partition
with parted):
sda1 = 1MiB BIOS Boot partition (no FS, "set 1 bios_grub on" with
parted to set the type)
sda2 = 550 MiB EFI System Partition (FAT32, "toggle 2 boot" with
parted to set the type),  mounted on /boot/efi
sda3 = 1 TiB root partition (BTRFS), mounted on /
sda4 = 6 GiB swap partition
(that way I should be able to be compatible with both CSM or UEFI)

B) normal Debian installation on sdas, activate the CSM on the
motherboard and reboot.

C) apt-get install grub-efi-amd64 and grub-install /dev/sda

And the problems begin:
1) grub-install doesn't give any error but using the --debug I can see
that it is not using EFI.
2) Ok I force with grub-install --target=x86_64-efi
--efi-directory=/boot/efi --bootloader-id=grub --recheck --debug
/dev/sda
3) This time something is generated in /boot/efi: /boot/efi/EFI/grub/grubx64.efi
4) Copy the file /boot/efi/EFI/grub/grubx64.efi to
/boot/efi/EFI/boot/bootx64.efi
5) Reboot and disable the CSM on the motherboard
6) No boot possible, I always go directly to the UEFI-BIOS

I am currently stuck there. I read a lot of conflicting advises which
doesn't work:
  - use "modprobe efivars" and efibootmgr: not possible because I have
not booted in EFI (chicken-egg problem)
  - use update-grub or use grub-mkconfig (to generate
/boot/efi/grub/grub.cfg): no results
  - other exotic commands...
So I will try to upgrade to grub 2.02beta (as recommender by Chris
Murphy) but I am not sure that it will help. If someone has some
Debian experience on this UEFI install, please don't hesitate to
propose solutions !

I will continue to document this "experience" (hope that it will be
useful for others), and hope to get to the point where I can have a
good system in BTRFS RAID1 mode.
You have to be very motivated to get into this, It is really a challenge ! ;-)

Re: BTRFS partitioning scheme (was BTRFS with RAID1 cannot boot when removing drive)

[Chris Murphy]

On Feb 12, 2014, at 10:13 AM, Saint Germain <saintger@gmail.com> wrote:

> On 11 February 2014 19:15, Chris Murphy <lists@colorremedies.com> wrote:
>>> 
>>> To summarize, I think I have 3 options for partitioning (I am not
>>> considering UEFI secure boot or swap):
>>> 1) grub, BTRFS partition (i.e. full disk in BTRFS), /boot inside BTRFS subvolume
>> 
>> This doesn't seem like a good idea for a boot drive to be without partitions.
>> 
>> 
>>> 2) grub, GPT partition, with (A) on sda1, and a BTRFS partition on
>>> sda2, /boot inside BTRFS subvolume
>>> 3) grub, GPT partition, with (A) on sda1, /boot (ext4) on sda2, and a
>>> BTRFS on sda3
>>> 
>>> (A) = BIOS Boot partition (1 MiB) or EFI System Partition (FAT32, 550MiB)
>>> 
>>> I don't really see the point of having UEFI/ESP if I don't use other
>>> proprietary operating system, so I think I will go with (A) = BIOS
>>> Boot partition except if there is someting I have missed.
>> 
>> You need to boot your system in UEFI and CSM-BIOS modes, and compare the dmesg for each. I'm finding it common the CSM limits power management, and relegates drives to IDE speeds rather than full SATA link speeds. Sometimes it's unavoidable to use the CSM if it has better overall behavior for your use case. I've found it to be lacking and have abandoned it. It's basically intended for booting Windows XP, right?
>> 
> 
> Ok based on your advices, here is what I have done so far to use UEFI
> (remeber that the objective is to have a clean and simple BTRFS RAID1
> install).
> 
> A) I start first with only one drive, I have gone with the following
> partition scheme (Debian wheezy, kernel 3.12, grub 2.00, GPT partition
> with parted):
> sda1 = 1MiB BIOS Boot partition (no FS, "set 1 bios_grub on" with
> parted to set the type)
> sda2 = 550 MiB EFI System Partition (FAT32, "toggle 2 boot" with
> parted to set the type),  mounted on /boot/efi
> sda3 = 1 TiB root partition (BTRFS), mounted on /
> sda4 = 6 GiB swap partition
> (that way I should be able to be compatible with both CSM or UEFI)
> 
> B) normal Debian installation on sdas, activate the CSM on the
> motherboard and reboot.
> 
> C) apt-get install grub-efi-amd64 and grub-install /dev/sda
> 
> And the problems begin:
> 1) grub-install doesn't give any error but using the --debug I can see
> that it is not using EFI.


Expected. If you boot with CSM enabled, you get a BIOS boot.



> 2) Ok I force with grub-install --target=x86_64-efi
> --efi-directory=/boot/efi --bootloader-id=grub --recheck --debug
> /dev/sda
> 3) This time something is generated in /boot/efi: /boot/efi/EFI/grub/grubx64.efi
> 4) Copy the file /boot/efi/EFI/grub/grubx64.efi to
> /boot/efi/EFI/boot/bootx64.efi

I wouldn't expect this grubx64.efi to work because efivars and access to NVRAM simply doesn't exist if you are CSM booted.

> 5) Reboot and disable the CSM on the motherboard
> 6) No boot possible, I always go directly to the UEFI-BIOS

Boot CSM mode, grub-install /dev/sda will install a BIOS grub into the BIOS Boot partition. It's fine for the fstab to still mount the ESP to /boot/efi - it's ignored.

Boot UEFI mode, grub-install will install a UEFI grub to the ESP.



> I am currently stuck there. I read a lot of conflicting advises which
> doesn't work:
>  - use "modprobe efivars" and efibootmgr: not possible because I have
> not booted in EFI (chicken-egg problem)
>  - use update-grub or use grub-mkconfig (to generate
> /boot/efi/grub/grub.cfg): no results
>  - other exotic commands…

Hmm I'm not certain whether the grub.cfg produced by grub-mkconfig is the same in UEFI vs CSM modes. Boot in both modes, create grub.cfg in each, then diff them.

However, I think grub.cfg on the ESP is a mistake because for bootable raid1 to be resilient in the face of one disk failure, you need some way to always keep each disk's ESP synchronized. It's better for he grub.cfg to be on Btrfs raid1 /boot/grub/grub.cfg. 

It's a separate topic for the grub list but I'm finding the idea of a persistently rw mounted ESP to be troubling. This isn't done on Windows or OS X, I don't know why this convention is used on Linux. The firmware doesn't mount it per se, but it's read only in any case, and the OS loader doesn't need the ESP mounted rw either, nor should it.

> 
> I will continue to document this "experience" (hope that it will be
> useful for others), and hope to get to the point where I can have a
> good system in BTRFS RAID1 mode.
> You have to be very motivated to get into this, It is really a challenge ! ;-)

Bootable raid1 on UEFI on grub-devel@
http://lists.gnu.org/archive/html/grub-devel/2014-02/msg00002.html

I think it's just not fully considered yet.



Chris Murphy

Re: BTRFS partitioning scheme (was BTRFS with RAID1 cannot boot when removing drive)

[Frank Kingswood]

On 12/02/14 17:13, Saint Germain wrote:
> Ok based on your advices, here is what I have done so far to use UEFI
> (remeber that the objective is to have a clean and simple BTRFS RAID1
> install).
>
> A) I start first with only one drive, I have gone with the following
> partition scheme (Debian wheezy, kernel 3.12, grub 2.00, GPT partition
> with parted):
> sda1 = 1MiB BIOS Boot partition (no FS, "set 1 bios_grub on" with
> parted to set the type)
> sda2 = 550 MiB EFI System Partition (FAT32, "toggle 2 boot" with
> parted to set the type),  mounted on /boot/efi

I'm curious, why so big? There's only one file of about 100kb there, and 
I was considering shrinking mine to the minimum possible (which seems to 
be about 33 MB).

> sda3 = 1 TiB root partition (BTRFS), mounted on /
> sda4 = 6 GiB swap partition
> (that way I should be able to be compatible with both CSM or UEFI)
>
> B) normal Debian installation on sdas, activate the CSM on the
> motherboard and reboot.
>
> C) apt-get install grub-efi-amd64 and grub-install /dev/sda
>
> And the problems begin:
> 1) grub-install doesn't give any error but using the --debug I can see
> that it is not using EFI.
> 2) Ok I force with grub-install --target=x86_64-efi
> --efi-directory=/boot/efi --bootloader-id=grub --recheck --debug
> /dev/sda
> 3) This time something is generated in /boot/efi: /boot/efi/EFI/grub/grubx64.efi
> 4) Copy the file /boot/efi/EFI/grub/grubx64.efi to
> /boot/efi/EFI/boot/bootx64.efi

                 ^^^^ is EFI/boot/ correct here?

If you're lucky then your BIOS will tell what path it will try to read 
for the boot code. For me that is /EFI/debian/grubx64.efi.

> 5) Reboot and disable the CSM on the motherboard
> 6) No boot possible, I always go directly to the UEFI-BIOS
>
> I am currently stuck there. I read a lot of conflicting advises which
> doesn't work:
>    - use "modprobe efivars" and efibootmgr: not possible because I have
> not booted in EFI (chicken-egg problem)

Not exactly. Boot in EFI mode into your favourite installer rescue mode, 
then chroot into the target filesystem and run efibootmgr there.

Frank

Re: BTRFS partitioning scheme (was BTRFS with RAID1 cannot boot when removing drive)

[Saint Germain]

On 13 February 2014 09:50, Frank Kingswood
<frank@kingswood-consulting.co.uk> wrote:
> On 12/02/14 17:13, Saint Germain wrote:
>>
>> Ok based on your advices, here is what I have done so far to use UEFI
>> (remeber that the objective is to have a clean and simple BTRFS RAID1
>> install).
>>
>> A) I start first with only one drive, I have gone with the following
>> partition scheme (Debian wheezy, kernel 3.12, grub 2.00, GPT partition
>> with parted):
>> sda1 = 1MiB BIOS Boot partition (no FS, "set 1 bios_grub on" with
>> parted to set the type)
>> sda2 = 550 MiB EFI System Partition (FAT32, "toggle 2 boot" with
>> parted to set the type),  mounted on /boot/efi
>
>
> I'm curious, why so big? There's only one file of about 100kb there, and I
> was considering shrinking mine to the minimum possible (which seems to be
> about 33 MB).
>

It is quite difficult to find reliable information on this whole UEFI
boot with linux (info you can find for sure, but which ones to follow
? there are so many different info out there).

So I don't know if this 550 MiB is an urban legend or not, but you can
find several people recommending it and the reason why:
http://askubuntu.com/questions/336439/any-problems-with-this-partition-scheme
http://askubuntu.com/questions/287441/different-uses-of-term-efi-partition
https://bbs.archlinux.org/viewtopic.php?pid=1306753
http://forums.gentoo.org/viewtopic-p-7352214.html

Other people recommend around 200-300 MiB, so I basically took the
upper limit to see what happen.
If you have more reliable info on the topic I would be interested !

>> sda3 = 1 TiB root partition (BTRFS), mounted on /
>> sda4 = 6 GiB swap partition
>> (that way I should be able to be compatible with both CSM or UEFI)
>>
>> B) normal Debian installation on sdas, activate the CSM on the
>> motherboard and reboot.
>>
>> C) apt-get install grub-efi-amd64 and grub-install /dev/sda
>>
>> And the problems begin:
>> 1) grub-install doesn't give any error but using the --debug I can see
>> that it is not using EFI.
>> 2) Ok I force with grub-install --target=x86_64-efi
>> --efi-directory=/boot/efi --bootloader-id=grub --recheck --debug
>> /dev/sda
>> 3) This time something is generated in /boot/efi:
>> /boot/efi/EFI/grub/grubx64.efi
>> 4) Copy the file /boot/efi/EFI/grub/grubx64.efi to
>> /boot/efi/EFI/boot/bootx64.efi
>
>
>                 ^^^^ is EFI/boot/ correct here?
>
> If you're lucky then your BIOS will tell what path it will try to read for
> the boot code. For me that is /EFI/debian/grubx64.efi.
>

I followed the advices here (first result on Google with "grub uefi debian"):
http://tanguy.ortolo.eu/blog/article51/debian-efi

>> 5) Reboot and disable the CSM on the motherboard
>> 6) No boot possible, I always go directly to the UEFI-BIOS
>>
>> I am currently stuck there. I read a lot of conflicting advises which
>> doesn't work:
>>    - use "modprobe efivars" and efibootmgr: not possible because I have
>> not booted in EFI (chicken-egg problem)
>
>
> Not exactly. Boot in EFI mode into your favourite installer rescue mode,
> then chroot into the target filesystem and run efibootmgr there.
>

In the end I managed to do it like this:
1) Make a USB stick with FAT32 partition
2) Install grub on it with:
grub-install --target=x86_64-efi --efi-directory=/media/usb0 --removable
3) Note on a paper the grub commands to start the kernel in /boot/grub/grub.cfg
3) Reboot, Disable CSM in the motherboard boot utility (BIOS?), Reboot
with the USB stick connected
4) Normally it should have started on the USB stick grub command-line
5) Enter the necessary command to start the kernel (if you have some
problem with video mode, use "insmod efi_gop")
6) Normally your operating system should start normally
7) Check that efibootmgr is installed and working (normally efivars
should be loaded in the modules already)
8) grub-install --efi-directory=/boot/efi --recheck --debug
(with the debug info you should see that it is using grub-efi and not grub-pc)
9) efibootmgr -c -d /dev/sda -p 2 -w -L "Debian (GRUB)" -l
'\EFI\Debian\grubx64.efi'
(replace -p 2 with yout correct ESP partition number)
10) Reboot and enjoy !

I made a lot of mistakes during these steps. The good thing is that
error are quire verbose, so you can easily see what is going wrong.
I hope that it will be easier for the next Debian user.

So now I can continue on this BTRFS RAID1 adventure... Let's see if my
setup is resilient to a hard drive failure.

Thanks for the help. Most comments here are quite on the spot and
reliable, that is very helpful !

Re: BTRFS partitioning scheme (was BTRFS with RAID1 cannot boot when removing drive)

[Chris Murphy]

On Feb 13, 2014, at 1:50 AM, Frank Kingswood <frank@kingswood-consulting.co.uk> wrote:

> On 12/02/14 17:13, Saint Germain wrote:
>> Ok based on your advices, here is what I have done so far to use UEFI
>> (remeber that the objective is to have a clean and simple BTRFS RAID1
>> install).
>> 
>> A) I start first with only one drive, I have gone with the following
>> partition scheme (Debian wheezy, kernel 3.12, grub 2.00, GPT partition
>> with parted):
>> sda1 = 1MiB BIOS Boot partition (no FS, "set 1 bios_grub on" with
>> parted to set the type)
>> sda2 = 550 MiB EFI System Partition (FAT32, "toggle 2 boot" with
>> parted to set the type),  mounted on /boot/efi
> 
> I'm curious, why so big? There's only one file of about 100kb there, and I was considering shrinking mine to the minimum possible (which seems to be about 33 MB).

I'm not sure what OS loader you're using but I haven't seen a grubx64.efi less than ~500KB. In general I'm seeing it at about 1MB. The Fedora grub-efi and shim packages as installed on the ESP take up 10MB. So 33MiB is a bit small, and if we were more conservative, we'd update the OS loader by writing the new one to a temp directory rather than overwriting existing. And then remove the old, and rename the new.

The UEFI spec says if the system partition is FAT, it should be FAT32. For removable media it's FAT12/FAT16. I don't know what tool the various distro installers are using, but at least on Fedora they are using mkdosfs, part of dosfstools. And its cutoff for making FAT16/FAT32 based on media size is 500MB unless otherwise specified, and the installer doesn't specify so actually by default Fedora system partitions are FAT16, to no obvious ill effect. But if you want a FAT32 ESP created by the installer the ESP needs to be 500MiB or 525MB. So 550MB is a reasonable number to make that happen.

If we were slightly smarter (and more A.R.), UEFI bugs aside, we'd put the ESP as the last partition on the disk rather than as the first and then honestly would we really care about consuming even 1GiB of the slowest part of a spinning disk? Or causing a bit of overprovisioning for SSD? No. It's probably a squeak of an improvement if anything.

For those who want to use gummiboot, it calls for the kernel and initramfs to be located on the ESP and is mounted at /boot rather than /boot/efi. So that's also a reason to make it bigger than usual.




>> sda3 = 1 TiB root partition (BTRFS), mounted on /
>> sda4 = 6 GiB swap partition
>> (that way I should be able to be compatible with both CSM or UEFI)
>> 
>> B) normal Debian installation on sdas, activate the CSM on the
>> motherboard and reboot.
>> 
>> C) apt-get install grub-efi-amd64 and grub-install /dev/sda
>> 
>> And the problems begin:
>> 1) grub-install doesn't give any error but using the --debug I can see
>> that it is not using EFI.
>> 2) Ok I force with grub-install --target=x86_64-efi
>> --efi-directory=/boot/efi --bootloader-id=grub --recheck --debug
>> /dev/sda
>> 3) This time something is generated in /boot/efi: /boot/efi/EFI/grub/grubx64.efi
>> 4) Copy the file /boot/efi/EFI/grub/grubx64.efi to
>> /boot/efi/EFI/boot/bootx64.efi
> 
>                ^^^^ is EFI/boot/ correct here?

If you want a fallback bootloader, yes.

> 
> If you're lucky then your BIOS will tell what path it will try to read for the boot code. For me that is /EFI/debian/grubx64.efi.

NVRAM is what does this. But if NVRAM becomes corrupt, or the entry is deleted for whatever reason, the proper fallback is boot<arch>.efi.



Chris Murphy

Re: BTRFS partitioning scheme (was BTRFS with RAID1 cannot boot when removing drive)

[Chris Murphy]

On Feb 13, 2014, at 3:03 AM, Saint Germain <saintger@gmail.com> wrote:

> On 13 February 2014 09:50, Frank Kingswood
> <frank@kingswood-consulting.co.uk> wrote:
>> On 12/02/14 17:13, Saint Germain wrote:
>>> 
>>> Ok based on your advices, here is what I have done so far to use UEFI
>>> (remeber that the objective is to have a clean and simple BTRFS RAID1
>>> install).
>>> 
>>> A) I start first with only one drive, I have gone with the following
>>> partition scheme (Debian wheezy, kernel 3.12, grub 2.00, GPT partition
>>> with parted):
>>> sda1 = 1MiB BIOS Boot partition (no FS, "set 1 bios_grub on" with
>>> parted to set the type)
>>> sda2 = 550 MiB EFI System Partition (FAT32, "toggle 2 boot" with
>>> parted to set the type),  mounted on /boot/efi
>> 
>> 
>> I'm curious, why so big? There's only one file of about 100kb there, and I
>> was considering shrinking mine to the minimum possible (which seems to be
>> about 33 MB).
>> 
> 
> It is quite difficult to find reliable information on this whole UEFI
> boot with linux (info you can find for sure, but which ones to follow
> ? there are so many different info out there).
> 
> So I don't know if this 550 MiB is an urban legend or not, but you can
> find several people recommending it and the reason why:
> http://askubuntu.com/questions/336439/any-problems-with-this-partition-scheme
> http://askubuntu.com/questions/287441/different-uses-of-term-efi-partition
> https://bbs.archlinux.org/viewtopic.php?pid=1306753
> http://forums.gentoo.org/viewtopic-p-7352214.html
> 
> Other people recommend around 200-300 MiB, so I basically took the
> upper limit to see what happen.
> If you have more reliable info on the topic I would be interested !

Draw straws, flip a coin, I wouldn't stress out about it that much. Considering it's effectively impossible to resize a partition 1 ESP once there's a partition 2 right behind it, I think 550MB is totally sane in particular if you plan on experimenting with different EFI OS Loaders, or multibooting since all OS Loaders go on the ESP. If it's a single OS only system, yeah sure you can get by with a smaller ESP.

And yes the UEFI boot thing is like playing jacks (knucklebones). Many variants.



> 
>>> sda3 = 1 TiB root partition (BTRFS), mounted on /
>>> sda4 = 6 GiB swap partition
>>> (that way I should be able to be compatible with both CSM or UEFI)
>>> 
>>> B) normal Debian installation on sdas, activate the CSM on the
>>> motherboard and reboot.
>>> 
>>> C) apt-get install grub-efi-amd64 and grub-install /dev/sda
>>> 
>>> And the problems begin:
>>> 1) grub-install doesn't give any error but using the --debug I can see
>>> that it is not using EFI.
>>> 2) Ok I force with grub-install --target=x86_64-efi
>>> --efi-directory=/boot/efi --bootloader-id=grub --recheck --debug
>>> /dev/sda
>>> 3) This time something is generated in /boot/efi:
>>> /boot/efi/EFI/grub/grubx64.efi
>>> 4) Copy the file /boot/efi/EFI/grub/grubx64.efi to
>>> /boot/efi/EFI/boot/bootx64.efi
>> 
>> 
>>                ^^^^ is EFI/boot/ correct here?
>> 
>> If you're lucky then your BIOS will tell what path it will try to read for
>> the boot code. For me that is /EFI/debian/grubx64.efi.
>> 
> 
> I followed the advices here (first result on Google with "grub uefi debian"):
> http://tanguy.ortolo.eu/blog/article51/debian-efi
> 
>>> 5) Reboot and disable the CSM on the motherboard
>>> 6) No boot possible, I always go directly to the UEFI-BIOS
>>> 
>>> I am currently stuck there. I read a lot of conflicting advises which
>>> doesn't work:
>>>   - use "modprobe efivars" and efibootmgr: not possible because I have
>>> not booted in EFI (chicken-egg problem)
>> 
>> 
>> Not exactly. Boot in EFI mode into your favourite installer rescue mode,
>> then chroot into the target filesystem and run efibootmgr there.
>> 
> 
> In the end I managed to do it like this:
> 1) Make a USB stick with FAT32 partition
> 2) Install grub on it with:
> grub-install --target=x86_64-efi --efi-directory=/media/usb0 --removable
> 3) Note on a paper the grub commands to start the kernel in /boot/grub/grub.cfg
> 3) Reboot, Disable CSM in the motherboard boot utility (BIOS?), Reboot
> with the USB stick connected
> 4) Normally it should have started on the USB stick grub command-line
> 5) Enter the necessary command to start the kernel (if you have some
> problem with video mode, use "insmod efi_gop")
> 6) Normally your operating system should start normally
> 7) Check that efibootmgr is installed and working (normally efivars
> should be loaded in the modules already)
> 8) grub-install --efi-directory=/boot/efi --recheck --debug
> (with the debug info you should see that it is using grub-efi and not grub-pc)
> 9) efibootmgr -c -d /dev/sda -p 2 -w -L "Debian (GRUB)" -l
> '\EFI\Debian\grubx64.efi'
> (replace -p 2 with yout correct ESP partition number)
> 10) Reboot and enjoy !

OK at least with GRUB 2.00 I never have to use any options with grub-install when installing to a chrooted system. It also even writes the proper entry into NVRAM for me, I don't have to use efibootmgr.

Also I've never had single \ work with efibootmgr from shell. I have to use \\. Try typing efibootmgr -v to see the actual entry you created and whether it has the \ in the path.

But one thing that explains why the UEFI bootloading stuff is confusing for you is that every distro keeps their own grub patches. So there is very possibly a lot of difference between the downstream grub behaviors, and upstream.


Chris Murphy

Re: BTRFS partitioning scheme (was BTRFS with RAID1 cannot boot when removing drive)

[Austin S Hemmelgarn]

On 2014-02-13 12:33, Chris Murphy wrote:
> 
> On Feb 13, 2014, at 1:50 AM, Frank Kingswood <frank@kingswood-consulting.co.uk> wrote:
> 
>> On 12/02/14 17:13, Saint Germain wrote:
>>> Ok based on your advices, here is what I have done so far to use UEFI
>>> (remeber that the objective is to have a clean and simple BTRFS RAID1
>>> install).
>>>
>>> A) I start first with only one drive, I have gone with the following
>>> partition scheme (Debian wheezy, kernel 3.12, grub 2.00, GPT partition
>>> with parted):
>>> sda1 = 1MiB BIOS Boot partition (no FS, "set 1 bios_grub on" with
>>> parted to set the type)
>>> sda2 = 550 MiB EFI System Partition (FAT32, "toggle 2 boot" with
>>> parted to set the type),  mounted on /boot/efi
>>
>> I'm curious, why so big? There's only one file of about 100kb there, and I was considering shrinking mine to the minimum possible (which seems to be about 33 MB).
> 
> I'm not sure what OS loader you're using but I haven't seen a grubx64.efi less than ~500KB. In general I'm seeing it at about 1MB. The Fedora grub-efi and shim packages as installed on the ESP take up 10MB. So 33MiB is a bit small, and if we were more conservative, we'd update the OS loader by writing the new one to a temp directory rather than overwriting existing. And then remove the old, and rename the new.
> 
> The UEFI spec says if the system partition is FAT, it should be FAT32. For removable media it's FAT12/FAT16. I don't know what tool the various distro installers are using, but at least on Fedora they are using mkdosfs, part of dosfstools. And its cutoff for making FAT16/FAT32 based on media size is 500MB unless otherwise specified, and the installer doesn't specify so actually by default Fedora system partitions are FAT16, to no obvious ill effect. But if you want a FAT32 ESP created by the installer the ESP needs to be 500MiB or 525MB. So 550MB is a reasonable number to make that happen.
> 
> If we were slightly smarter (and more A.R.), UEFI bugs aside, we'd put the ESP as the last partition on the disk rather than as the first and then honestly would we really care about consuming even 1GiB of the slowest part of a spinning disk? Or causing a bit of overprovisioning for SSD? No. It's probably a squeak of an improvement if anything.
> 
> For those who want to use gummiboot, it calls for the kernel and initramfs to be located on the ESP and is mounted at /boot rather than /boot/efi. So that's also a reason to make it bigger than usual.
> 
> 
> 
> 
>>> sda3 = 1 TiB root partition (BTRFS), mounted on /
>>> sda4 = 6 GiB swap partition
>>> (that way I should be able to be compatible with both CSM or UEFI)
>>>
>>> B) normal Debian installation on sdas, activate the CSM on the
>>> motherboard and reboot.
>>>
>>> C) apt-get install grub-efi-amd64 and grub-install /dev/sda
>>>
>>> And the problems begin:
>>> 1) grub-install doesn't give any error but using the --debug I can see
>>> that it is not using EFI.
>>> 2) Ok I force with grub-install --target=x86_64-efi
>>> --efi-directory=/boot/efi --bootloader-id=grub --recheck --debug
>>> /dev/sda
>>> 3) This time something is generated in /boot/efi: /boot/efi/EFI/grub/grubx64.efi
>>> 4) Copy the file /boot/efi/EFI/grub/grubx64.efi to
>>> /boot/efi/EFI/boot/bootx64.efi
>>
>>                ^^^^ is EFI/boot/ correct here?
> 
> If you want a fallback bootloader, yes.
> 
>>
>> If you're lucky then your BIOS will tell what path it will try to read for the boot code. For me that is /EFI/debian/grubx64.efi.
> 
> NVRAM is what does this. But if NVRAM becomes corrupt, or the entry is deleted for whatever reason, the proper fallback is boot<arch>.efi.

While this is what the UEFI spec says is supposed to be the fallback,
many systems don't actually look there unless the media is removable.
All of my UEFI systems instead look for Microsoft/Boot/bootmgfw.efi as
the fallback (Cause most x86 system designers don't care at all about
standards compliance as long as it will run windows).

Re: BTRFS partitioning scheme (was BTRFS with RAID1 cannot boot when removing drive)

[Chris Murphy]

On Feb 13, 2014, at 11:02 AM, Austin S Hemmelgarn <ahferroin7@gmail.com> wrote:
> 
> While this is what the UEFI spec says is supposed to be the fallback,
> many systems don't actually look there unless the media is removable.
> All of my UEFI systems instead look for Microsoft/Boot/bootmgfw.efi as
> the fallback (Cause most x86 system designers don't care at all about
> standards compliance as long as it will run windows).

True, and they should be filed as bugs with the manufacturer, citing the UEFI spec as being violated. There's not much else to do in such cases, while considering they have a different fallback location: Microsoft/Boot/bootmgfw.efi.

Nevertheless, so long as NVRAM is behaving and isn't pilfered for other purposes, then you can specify a fallback entry there.

Chris Murphy

Re: BTRFS partitioning scheme (was BTRFS with RAID1 cannot boot when removing drive)

[Saint Germain]

On Thu, 13 Feb 2014 10:43:08 -0700, Chris Murphy
<lists@colorremedies.com> wrote :
> >>> sda3 = 1 TiB root partition (BTRFS), mounted on /
> >>> sda4 = 6 GiB swap partition
> >>> (that way I should be able to be compatible with both CSM or UEFI)
> >>> 
> >>> B) normal Debian installation on sdas, activate the CSM on the
> >>> motherboard and reboot.
> >>> 
> >>> C) apt-get install grub-efi-amd64 and grub-install /dev/sda
> >>> 
> >>> And the problems begin:
> >>> 1) grub-install doesn't give any error but using the --debug I
> >>> can see that it is not using EFI.
> >>> 2) Ok I force with grub-install --target=x86_64-efi
> >>> --efi-directory=/boot/efi --bootloader-id=grub --recheck --debug
> >>> /dev/sda
> >>> 3) This time something is generated in /boot/efi:
> >>> /boot/efi/EFI/grub/grubx64.efi
> >>> 4) Copy the file /boot/efi/EFI/grub/grubx64.efi to
> >>> /boot/efi/EFI/boot/bootx64.efi
> >> 
> >> 
> >>                ^^^^ is EFI/boot/ correct here?
> >> 
> >> If you're lucky then your BIOS will tell what path it will try to
> >> read for the boot code. For me that is /EFI/debian/grubx64.efi.
> >> 
> > 
> > I followed the advices here (first result on Google with "grub uefi
> > debian"): http://tanguy.ortolo.eu/blog/article51/debian-efi
> > 
> >>> 5) Reboot and disable the CSM on the motherboard
> >>> 6) No boot possible, I always go directly to the UEFI-BIOS
> >>> 
> >>> I am currently stuck there. I read a lot of conflicting advises
> >>> which doesn't work:
> >>>   - use "modprobe efivars" and efibootmgr: not possible because I
> >>> have not booted in EFI (chicken-egg problem)
> >> 
> >> 
> >> Not exactly. Boot in EFI mode into your favourite installer rescue
> >> mode, then chroot into the target filesystem and run efibootmgr
> >> there.
> >> 
> > 
> > In the end I managed to do it like this:
> > 1) Make a USB stick with FAT32 partition
> > 2) Install grub on it with:
> > grub-install --target=x86_64-efi --efi-directory=/media/usb0
> > --removable 3) Note on a paper the grub commands to start the
> > kernel in /boot/grub/grub.cfg 3) Reboot, Disable CSM in the
> > motherboard boot utility (BIOS?), Reboot with the USB stick
> > connected 4) Normally it should have started on the USB stick grub
> > command-line 5) Enter the necessary command to start the kernel (if
> > you have some problem with video mode, use "insmod efi_gop")
> > 6) Normally your operating system should start normally
> > 7) Check that efibootmgr is installed and working (normally efivars
> > should be loaded in the modules already)
> > 8) grub-install --efi-directory=/boot/efi --recheck --debug
> > (with the debug info you should see that it is using grub-efi and
> > not grub-pc) 9) efibootmgr -c -d /dev/sda -p 2 -w -L "Debian
> > (GRUB)" -l '\EFI\Debian\grubx64.efi'
> > (replace -p 2 with yout correct ESP partition number)
> > 10) Reboot and enjoy !
> 
> OK at least with GRUB 2.00 I never have to use any options with
> grub-install when installing to a chrooted system. It also even
> writes the proper entry into NVRAM for me, I don't have to use
> efibootmgr.

Yes you are right, this is probably unnecessary (see below).

> 
> Also I've never had single \ work with efibootmgr from shell. I have
> to use \\. Try typing efibootmgr -v to see the actual entry you
> created and whether it has the \ in the path.
> 

Here is the output:
BootCurrent: 0001
Timeout: 1 seconds
BootOrder: 0001,0000
Boot0000* debian        HD(2,7d8,106430,5d012c09-b70d-4225-ae18-9831f997c493)File(\EFI\debian\grubx64.efi)
Boot0001* Debian (GRUB) HD(2,7d8,106430,5d012c09-b70d-4225-ae18-9831f997c493)File(\EFI\Debian\grubx64.efi)

Ah the joy of FAT32 and the case sensitivity !
So it seems that grub-install automatically install the correct entry
 and using efibootmgr was unnecessary.
However it seems that single '\' works.

> But one thing that explains why the UEFI bootloading stuff is
> confusing for you is that every distro keeps their own grub patches.
> So there is very possibly a lot of difference between the downstream
> grub behaviors, and upstream.
> 

Understood. That is why I took the step to describe what I did.
Perhaps it will be useful for others (most info on the topic was
not for Debian...).

Thanks again for your insights !

Re: BTRFS partitioning scheme (was BTRFS with RAID1 cannot boot when removing drive)

[Frank Kingswood]

On 13/02/14 18:02, Austin S Hemmelgarn wrote:
> On 2014-02-13 12:33, Chris Murphy wrote:
>>
>> On Feb 13, 2014, at 1:50 AM, Frank Kingswood <frank@kingswood-consulting.co.uk> wrote:
>>
>>> On 12/02/14 17:13, Saint Germain wrote:
>>>> Ok based on your advices, here is what I have done so far to use UEFI
>>>> (remeber that the objective is to have a clean and simple BTRFS RAID1
>>>> install).
>>>>
>>>> A) I start first with only one drive, I have gone with the following
>>>> partition scheme (Debian wheezy, kernel 3.12, grub 2.00, GPT partition
>>>> with parted):
>>>> sda1 = 1MiB BIOS Boot partition (no FS, "set 1 bios_grub on" with
>>>> parted to set the type)
>>>> sda2 = 550 MiB EFI System Partition (FAT32, "toggle 2 boot" with
>>>> parted to set the type),  mounted on /boot/efi
>>>
>>> I'm curious, why so big? There's only one file of about 100kb there, and I was considering shrinking mine to the minimum possible (which seems to be about 33 MB).
>>
>> I'm not sure what OS loader you're using but I haven't seen a grubx64.efi less than ~500KB. In general I'm seeing it at about 1MB. The Fedora grub-efi and shim packages as installed on the ESP take up 10MB. So 33MiB is a bit small, and if we were more conservative, we'd update the OS loader by writing the new one to a temp directory rather than overwriting existing. And then remove the old, and rename the new.
>>
>> The UEFI spec says if the system partition is FAT, it should be FAT32. For removable media it's FAT12/FAT16. I don't know what tool the various distro installers are using, but at least on Fedora they are using mkdosfs, part of dosfstools. And its cutoff for making FAT16/FAT32 based on media size is 500MB unless otherwise specified, and the installer doesn't specify so actually by default Fedora system partitions are FAT16, to no obvious ill effect. But if you want a FAT32 ESP created by the installer the ESP needs to be 500MiB or 525MB. So 550MB is a reasonable number to make that happen.

For the record, this works for me on a Dell laptop:

I have resized my EFI partition to 40 MB, reformatted it with
    mkfs.vfat -F 32 -n EFI /dev/sda1
and copied back the grubx64.efi file.

Note that mkvs.vfat gives "WARNING: Not enough clusters for a 32 bit 
FAT!" unless the file system is more than 32 MiB.

Because the GUID in the GPT partition table is modified by the 
repartitioning the BIOS must be updated to point to this file. That can 
be done with efibootmgr, or in my case it is also possible in the BIOS 
setup.

Frank