GRUB 2.12 release - update

GRUB 2.12 release - update

[Daniel Kiper]

Hi,

We are getting closer to the 2.12 release. Sadly we still do not have
many of important patch sets in the tree. So, I am going to spend more
time on reviews in the following weeks. Below you can find my list of
key patch sets which should land in the release:
  - Dynamic allocation of memory regions and IBM vTPM v2,
  - Unify ARM64 & RISC-V Linux Loader,
  - Add support for LoongArch,
  - plainmount: Support plain encryption mode,
  - Glenn's tests fixes.

Of course all patches which got my RB or are under review will be merged too.

There is also "nice to have" list but I do not consider lack of this
patch sets as release blockers:
  - Add support for EFI file system transposition,
  - term/serial: Add support for PCI serial devices,
  - Add MMIO serial ports support and SPCR detection,
  - Glenn's gdb patch set.

I hope I will be able to review and merge all patch sets from first list
during November. Then if everything goes well we will make code freeze
in December and release at the beginning of next year, preferably in January.

I am considering to not block merges for tests and documentation during
code freeze.

If you have any comments on that plan please drop me a line.

Daniel

Re: GRUB 2.12 release - update

[Pete Batard]

Hi Daniel,

Thanks for keeping us updated on your the release progress and release 
plans.

On 2022.10.26 15:52, Daniel Kiper wrote:
> We are getting closer to the 2.12 release. Sadly we still do not have
> many of important patch sets in the tree. So, I am going to spend more
> time on reviews in the following weeks. Below you can find my list of
> key patch sets which should land in the release:
>    - Dynamic allocation of memory regions and IBM vTPM v2,
>    - Unify ARM64 & RISC-V Linux Loader,
>    - Add support for LoongArch,
>    - plainmount: Support plain encryption mode,
>    - Glenn's tests fixes.
> 
> Of course all patches which got my RB or are under review will be merged too.
> 
> There is also "nice to have" list but I do not consider lack of this
> patch sets as release blockers:
>    - Add support for EFI file system transposition,

I'm afraid I will have to strongly disagree about this being a mere 
"nice to have".

Lack of support for EFI file system transposition needs to be considered 
as blocking as if grub-mkrescue was producing ISOs that *broke* the 
ability to write them in dd mode. In other words, if someone came to 
this list and told you that grub-mkrescue in its current state was 
unable to produce functional ISOHybrid images that can be copied in dd 
mode, I am pretty sure that it would be treated as a showstopper.

Well, it is exactly that. Except the showstopping happens with the 
alternative to dd mode.

It is that simple.

And it is exceedingly disheartening for folks who are ultimately 
advocating for the end-users' welfare (rather than the distro 
maintainer's welfare, because these two do not always entirely overlap) 
to see that, yet again, the ability to create ISOHybrid media using EFI 
file system transposition is being treated as a simple "nice to have" 
feature, instead of the "SHOULD have equal footing as dd" that it should 
arguably be viewed as.

In my submission from 2022-06-06, I have provided more than a dozen 
direct examples of how having a casual approach to EFI File system 
transposition is very negatively impacting end users, and I could easily 
pick a dozen more examples that intervened since I submitted the series. 
So I will kindly remind folks that we're not talking about an alleged or 
estimated negative impact here. The negative impact of not supporting 
EFI file system transposition is happening. It is affecting thousands of 
ISOHybrid end users. And the failure of grub-mkrescue to produce 
ISOHybrids that can be used in file system transposition mode is going 
to continue to be a major contributor to it.

So, let me go over the main points yet again, so that, maybe, it'll 
start to make sense why this issue needs to be treated as as the 
showstopper it actually is.

1. ISOHybrid should not be seen as nothing more than a "glorified dd 
image that can also burned to optical" (And I could, again, go over the 
may reasons that make ISOHybrid as dd less than the panacea that many 
proponents think it is, such as, for instance, the fact that it is 
factually impossible to create a *valid* GPT based media from an 
ISOhybrid using dd, on account that the backup GPT will never ever be 
written properly [*MUST* reside in the last 23 sectors of the disk], 
unless you use a media that is the exact same size as the ISO, which is 
never the case). Instead, users SHOULD be given the *choice* to create 
bootable media from ISOHybrid using either dd or EFI File System 
transposition with no nudging by the people who mastered the ISO for one 
or the other. But current grub-mkrescue is removing that *choice* from 
end-users, which is showstopper behaviour.

2. One of whole points of UEFI was to do away with the need to write 
boot media at the block level, to instead write it at the file system 
level (i.e. EFI file system transposition) because it was identified 
that writing anything at the block system level was (rightfully) 
ultimately problematic for end users. Yet what we are seeing is that 
Linux maintainers (and by extension GRUB maintainers) have done a 
complete 180 on that promise, by embracing the idea that users 
everywhere, including ones on platforms where it's super inconvenient 
(read Windows, but not only) should just use dd to write their boot 
media, thereby sticking to the old problematic write boot media at the 
block level, with little or no concern for file system transposition.

3. With GRUB releases being spaced about one year at best, if this is 
treated as "nice to have" and delayed, we're not going to see file 
system transposition being supported downstream for at least another 
year. But the problem is that we are seeing Linux distro maintainers, 
who were using a mix of Syslinux (for BIOS) and GRUB (for UEFI) [which 
would usually support or be salvageable when it comes to file system 
transposition] in the process of switching to GRUB exclusively, and 
switching to using a grub-mkrescue based process to do just that. This 
means that, if we don't get file system transposition included in the 
next release, we're going to get a full year or more of distros 
switching to unsalvageable grub-mkrescue and completely preventing 
end-users from using file system transposition to create their boot 
media, resulting in even more pain than what we're currently seeing. So 
please be very mindful that we are in an active transition phase (which 
is how I came to realise that we had a major problem on our hand, as it 
took a while for the grub-mkrescue limitation to percolate downstream up 
to the end-user,  otherwise I would have sent a patch much sooner) and 
that any delay we add in fixing this issue will result in more literal 
years of pain for end users. In other words, if we are actively looking 
up at what's coming up ahead, it should be clear that time to fix this 
is now rather than later (because, even if this gets integrated shortly 
after the next release, a lot of distros tend to stick with dot release 
with a few patches rather than rolling GRUB git, and they are unlikely 
to pick this, especially when GRUB is actively cementing the idea that 
EFI file system transposition is simply "nice to have" instead of the 
fundamental foundation of creating boot media that is should actually be 
seen as).

4. While full file system transposition support does also require the 
involvement of Linux distro maintainers (in its current state, a fixed 
grub-mkrescue alone is not enough as the ISO masterer must also make 
sure that the content of the efi.img is dusplicated onto the ISO file 
system), this last matter is something that can be easily salvaged by 
forward thinking applications (e.g. Rufus is smart enough to extract the 
content of efi.img for the user in case it wasn't duplicated on the 
ISO). But this cannot be salvaged at all, if grub-mkrescue remains in 
it's current state, which is why the grub-mkrescue issue needs to be 
addressed as broken behaviour that requires fixing for the next release.

So please please please!!, for the sake of the end-users who are --and 
will be-- creating GRUB based bootable media everywhere, do make sure 
that you consider these patches (or any updated version -- I'll be there 
to work with you on that if needed!) for integration in the next formal 
release. And do not dismiss EFI file system translation as "nice to 
have" when it is both a fundamental keystone of the boot media creation 
process, and GRUB should be seen at the very forefront of promoting it.

Thank you,

/Pete

>    - term/serial: Add support for PCI serial devices,
>    - Add MMIO serial ports support and SPCR detection,
>    - Glenn's gdb patch set.
> 
> I hope I will be able to review and merge all patch sets from first list
> during November. Then if everything goes well we will make code freeze
> in December and release at the beginning of next year, preferably in January.
> 
> I am considering to not block merges for tests and documentation during
> code freeze.
> 
> If you have any comments on that plan please drop me a line.
> 
> Daniel
> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel

Re: GRUB 2.12 release - update

[Daniel Kiper]

Hi Pete,

On Fri, Oct 28, 2022 at 01:42:39PM +0100, Pete Batard via Grub-devel wrote:
> Hi Daniel,
>
> Thanks for keeping us updated on your the release progress and release
> plans.
>
> On 2022.10.26 15:52, Daniel Kiper wrote:
> > We are getting closer to the 2.12 release. Sadly we still do not have
> > many of important patch sets in the tree. So, I am going to spend more
> > time on reviews in the following weeks. Below you can find my list of
> > key patch sets which should land in the release:
> >    - Dynamic allocation of memory regions and IBM vTPM v2,
> >    - Unify ARM64 & RISC-V Linux Loader,
> >    - Add support for LoongArch,
> >    - plainmount: Support plain encryption mode,
> >    - Glenn's tests fixes.
> >
> > Of course all patches which got my RB or are under review will be merged too.
> >
> > There is also "nice to have" list but I do not consider lack of this
> > patch sets as release blockers:
> >    - Add support for EFI file system transposition,
>
> I'm afraid I will have to strongly disagree about this being a mere "nice to
> have".
>
> Lack of support for EFI file system transposition needs to be considered as
> blocking as if grub-mkrescue was producing ISOs that *broke* the ability to
> write them in dd mode. In other words, if someone came to this list and told
> you that grub-mkrescue in its current state was unable to produce functional
> ISOHybrid images that can be copied in dd mode, I am pretty sure that it
> would be treated as a showstopper.
>
> Well, it is exactly that. Except the showstopping happens with the
> alternative to dd mode.
>
> It is that simple.
>
> And it is exceedingly disheartening for folks who are ultimately advocating
> for the end-users' welfare (rather than the distro maintainer's welfare,
> because these two do not always entirely overlap) to see that, yet again,
> the ability to create ISOHybrid media using EFI file system transposition is
> being treated as a simple "nice to have" feature, instead of the "SHOULD
> have equal footing as dd" that it should arguably be viewed as.
>
> In my submission from 2022-06-06, I have provided more than a dozen direct
> examples of how having a casual approach to EFI File system transposition is
> very negatively impacting end users, and I could easily pick a dozen more
> examples that intervened since I submitted the series. So I will kindly
> remind folks that we're not talking about an alleged or estimated negative
> impact here. The negative impact of not supporting EFI file system
> transposition is happening. It is affecting thousands of ISOHybrid end
> users. And the failure of grub-mkrescue to produce ISOHybrids that can be
> used in file system transposition mode is going to continue to be a major
> contributor to it.
>
> So, let me go over the main points yet again, so that, maybe, it'll start to
> make sense why this issue needs to be treated as as the showstopper it
> actually is.

[...]

Thank you for detailed explanation. I can consider your patches as more
than "nice to have" but it may mean we will have to delay code freeze
and maybe release date. To reduce impact on dates please be ready to
reply to my comments as quickly as possible when I start reviewing your
patches. Probably I will start doing that in a few weeks. I hope it
is OK for you...

Daniel

Re: GRUB 2.12 release - update

[Pete Batard]

Hi Daniel,

On 2022.11.03 17:59, Daniel Kiper wrote:
> Thank you for detailed explanation. I can consider your patches as more
> than "nice to have" but it may mean we will have to delay code freeze
> and maybe release date. To reduce impact on dates please be ready to
> reply to my comments as quickly as possible when I start reviewing your
> patches. Probably I will start doing that in a few weeks. I hope it
> is OK for you...

Thanks for reconsidering the priority. Much appreciated!

I'll be standing by for whenever you start reviewing. Feel free to also 
ping me privately if needed.

Regards,

/Pete

Re: GRUB 2.12 release - update

[Michael Chang]

Dear Daniel,

On Wed, Oct 26, 2022 at 04:52:09PM +0200, Daniel Kiper wrote:
> Hi,
> 
> We are getting closer to the 2.12 release. Sadly we still do not have
> many of important patch sets in the tree. So, I am going to spend more
> time on reviews in the following weeks. Below you can find my list of
> key patch sets which should land in the release:
>   - Dynamic allocation of memory regions and IBM vTPM v2,
>   - Unify ARM64 & RISC-V Linux Loader,
>   - Add support for LoongArch,
>   - plainmount: Support plain encryption mode,
>   - Glenn's tests fixes.
> 
> Of course all patches which got my RB or are under review will be merged too.
> 
> There is also "nice to have" list but I do not consider lack of this
> patch sets as release blockers:
>   - Add support for EFI file system transposition,
>   - term/serial: Add support for PCI serial devices,
>   - Add MMIO serial ports support and SPCR detection,
>   - Glenn's gdb patch set.
> 
> I hope I will be able to review and merge all patch sets from first list
> during November. Then if everything goes well we will make code freeze
> in December and release at the beginning of next year, preferably in January.
> 
> I am considering to not block merges for tests and documentation during
> code freeze.
> 
> If you have any comments on that plan please drop me a line.

Is there any chance to get argon2 support [1] in 2.12 ? Given the required
memory manager patch has been merged to allow allocating huge chunks of
data in heap, it seems all is ready to go now ?

This is the major feature brought by LUKS2 making it really a successor
to LUKS1 IMHO.

[1] https://lists.gnu.org/archive/html/grub-devel/2021-08/msg00027.html

Thanks,
Michael

> 
> Daniel
> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel

Re: GRUB 2.12 release - update

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 2603 bytes --]

On Tue, Nov 22, 2022 at 10:46:45AM +0800, Michael Chang via Grub-devel wrote:
> Dear Daniel,
> 
> On Wed, Oct 26, 2022 at 04:52:09PM +0200, Daniel Kiper wrote:
> > Hi,
> > 
> > We are getting closer to the 2.12 release. Sadly we still do not have
> > many of important patch sets in the tree. So, I am going to spend more
> > time on reviews in the following weeks. Below you can find my list of
> > key patch sets which should land in the release:
> >   - Dynamic allocation of memory regions and IBM vTPM v2,
> >   - Unify ARM64 & RISC-V Linux Loader,
> >   - Add support for LoongArch,
> >   - plainmount: Support plain encryption mode,
> >   - Glenn's tests fixes.
> > 
> > Of course all patches which got my RB or are under review will be merged too.
> > 
> > There is also "nice to have" list but I do not consider lack of this
> > patch sets as release blockers:
> >   - Add support for EFI file system transposition,
> >   - term/serial: Add support for PCI serial devices,
> >   - Add MMIO serial ports support and SPCR detection,
> >   - Glenn's gdb patch set.
> > 
> > I hope I will be able to review and merge all patch sets from first list
> > during November. Then if everything goes well we will make code freeze
> > in December and release at the beginning of next year, preferably in January.
> > 
> > I am considering to not block merges for tests and documentation during
> > code freeze.
> > 
> > If you have any comments on that plan please drop me a line.
> 
> Is there any chance to get argon2 support [1] in 2.12 ? Given the required
> memory manager patch has been merged to allow allocating huge chunks of
> data in heap, it seems all is ready to go now ?
> 
> This is the major feature brought by LUKS2 making it really a successor
> to LUKS1 IMHO.

No, I don't think this would be realistic. I'd love to finally implement
it, and theoretically-speaking the patches I sent a while ago would work
now with the changes in the memory manager. But meanwhile libgcrypt has
landed support for Argon2, so using its implementation instead of the
reference implementation is very much preferable. Unfortunately though,
the version of libgcrypt we have in our tree is ancient, and updating it
to a more recent version is a _major_ effort.

I have taken a look at this several times already, but each time I
quickly felt overwhelmed with the adjustments I'd have to do to do this.
I do plan to have another go at it eventually, but if anybody else feels
like updating our vendored libgcrypt version I wouldn't mind at all.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: GRUB 2.12 release - update

[Daniel Kiper]

Hi,

Quick update below...

On Wed, Oct 26, 2022 at 04:52:09PM +0200, Daniel Kiper wrote:
> Hi,
>
> We are getting closer to the 2.12 release. Sadly we still do not have
> many of important patch sets in the tree. So, I am going to spend more
> time on reviews in the following weeks. Below you can find my list of
> key patch sets which should land in the release:
>   - Dynamic allocation of memory regions and IBM vTPM v2,
>   - Unify ARM64 & RISC-V Linux Loader,
>   - Add support for LoongArch,
>   - plainmount: Support plain encryption mode,
>   - Glenn's tests fixes.
>
> Of course all patches which got my RB or are under review will be merged too.
>
> There is also "nice to have" list but I do not consider lack of this
> patch sets as release blockers:
>   - Add support for EFI file system transposition,
>   - term/serial: Add support for PCI serial devices,
>   - Add MMIO serial ports support and SPCR detection,
>   - Glenn's gdb patch set.
>
> I hope I will be able to review and merge all patch sets from first list
> during November. Then if everything goes well we will make code freeze
> in December and release at the beginning of next year, preferably in January.

Most of the patches from the list above are now under review or (almost)
ready for merging. I will push all patches which got my RB recently
after holidays.

(Sadly) we have to add UEFI NX patches to the list above due to
introduction of new requirements "for all applications to be signed by
the Microsoft third-party Unified Extensible Firmware Interface (UEFI)
Certificate Authority (CA)" [1]. I hope relevant patches will appear on
the grub-devel in January or February.

> I am considering to not block merges for tests and documentation during
> code freeze.

Nothing changes here...

Happy holidays,

Daniel

[1] https://techcommunity.microsoft.com/t5/hardware-dev-center/new-uefi-ca-memory-mitigation-requirements-for-signing/ba-p/3608714

Re: GRUB 2.12 release - update

[Daniel Kiper]

Hey,

Quick update...

On Thu, Dec 22, 2022 at 07:53:33PM +0100, Daniel Kiper wrote:
> Hi,
>
> Quick update below...
>
> On Wed, Oct 26, 2022 at 04:52:09PM +0200, Daniel Kiper wrote:
> > Hi,
> >
> > We are getting closer to the 2.12 release. Sadly we still do not have
> > many of important patch sets in the tree. So, I am going to spend more
> > time on reviews in the following weeks. Below you can find my list of
> > key patch sets which should land in the release:
> >   - Dynamic allocation of memory regions and IBM vTPM v2,
> >   - Unify ARM64 & RISC-V Linux Loader,
> >   - Add support for LoongArch,
> >   - plainmount: Support plain encryption mode,
> >   - Glenn's tests fixes.
> >
> > Of course all patches which got my RB or are under review will be merged too.
> >
> > There is also "nice to have" list but I do not consider lack of this
> > patch sets as release blockers:
> >   - Add support for EFI file system transposition,
> >   - term/serial: Add support for PCI serial devices,
> >   - Add MMIO serial ports support and SPCR detection,
> >   - Glenn's gdb patch set.
> >
> > I hope I will be able to review and merge all patch sets from first list
> > during November. Then if everything goes well we will make code freeze
> > in December and release at the beginning of next year, preferably in January.
>
> Most of the patches from the list above are now under review or (almost)
> ready for merging. I will push all patches which got my RB recently
> after holidays.
>
> (Sadly) we have to add UEFI NX patches to the list above due to
> introduction of new requirements "for all applications to be signed by
> the Microsoft third-party Unified Extensible Firmware Interface (UEFI)
> Certificate Authority (CA)" [1]. I hope relevant patches will appear on
> the grub-devel in January or February.

After discussing the UEFI NX support with distros GRUB maintainers I am
dropping the feature from 2.12 release requirement. Simply speaking it
is still not ready... :-(

Most of the patches from the list above are in upstream right now. I am
still working on getting some important features into 2.12. Though at
this point things look good and I am planning to do the code freeze in
May. So, if you think we should still merge some patches drop me a line
ASAP.

> > I am considering to not block merges for tests and documentation during
> > code freeze.
>
> Nothing changes here...

Still no changes here...

Expect more updates in May...

Daniel

> [1] https://techcommunity.microsoft.com/t5/hardware-dev-center/new-uefi-ca-memory-mitigation-requirements-for-signing/ba-p/3608714