Several GNU projects wondering about the reason for mformat partition table

Several GNU projects wondering about the reason for mformat partition table

[Thomas Schmitt]

Hi,

in the course of a problem search between GRUB, Guix, and xorriso we found
the trigger of a problem with some old Macbook EFI firmware in a gesture
of mformat.
Line 1375 of
  http://svn.savannah.gnu.org/viewvc/mtools/trunk/mformat.c?revision=506&view=markup
has

  /* install fake partition table pointing to itself */

Now we are wondering for which use case there is this MBR partition table
entry starting with LBA 0 and claiming the whole device.

The SVN at savannah shows that this gesture was introduced with revision 4
in may 2002 by "aknaff". My hope is that this was you and that you can tell
grub-devel (Cc'ed) more about the motivation.

---------------------------------------------------------------------------
In case you are curious about the motivation of this mail:

Our particular use case is to create a FAT filesystem image for use as EFI
system partition in ISO 9660 images on USB sticks.
grub-mkrescue runs mformat to create this image and then uses mcopy to
populate it with EFI start programs. See
  http://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkrescue.c#n812

The partition entry with start LBA 0 caused the EFI firmware to get stuck,
probably looping endlessly. Experiments with dd reveiled that the partition
entry was to blame, and in it the start LBA 0.

This is clearly a firmware bug and obviously fixed in later models of
Macbooks. Nevertheless we care for old oddball hardware.

So the question is which of these alternatives to choose:

- Keep the partition entry because its removal could break some other
  EFI firmware's boot process.

- Overwrite the partition table in bytes 446 to 509 of the mformat result
  by zeros before populating it with files.

- Use mformat option -k to avoid production of the partition table.

I personally have scruples to omit the other fields which get written
if option -k is not present. Is there anything written with (!keepBoot)
which we need for a vanilla FAT filesystem to be recognizable or usable ?

---------------------------------------------------------------------------

Have a nice day :)

Thomas

Re: Several GNU projects wondering about the reason for mformat partition table

[Mihai Moldovan]

[-- Attachment #1.1: Type: text/plain, Size: 1277 bytes --]

* On 5/20/19 6:52 PM, Thomas Schmitt wrote:
> in the course of a problem search between GRUB, Guix, and xorriso we found
> the trigger of a problem with some old Macbook EFI firmware in a gesture
> of mformat.
> Line 1375 of
>   http://svn.savannah.gnu.org/viewvc/mtools/trunk/mformat.c?revision=506&view=markup
> has
> 
>   /* install fake partition table pointing to itself */
> 
> Now we are wondering for which use case there is this MBR partition table
> entry starting with LBA 0 and claiming the whole device.
> 
> The SVN at savannah shows that this gesture was introduced with revision 4
> in may 2002 by "aknaff". My hope is that this was you and that you can tell
> grub-devel (Cc'ed) more about the motivation.

No authoritative answer from me, but if this is coupling GPT + MBR, creating a
MBR partition table from LBA 0 to the end of (reachable) device space (which is
not necessarily the actual device space/end for limiting factors such as disk
sector size [mis-]usage and the like) is a pretty common thing to do and called
"protective MBR". The rationale being to "trick" tools that are only MBR-aware
into thinking that the device is NOT unpartitioned and hence not erroring out or
offering to "correct the situation".



Mihai


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 899 bytes --]

Re: Several GNU projects wondering about the reason for mformat partition table

[Alain Knaff]

Hi,

On 20/05/2019 18:52, Thomas Schmitt wrote:
> Hi,
> 
> in the course of a problem search between GRUB, Guix, and xorriso we found
> the trigger of a problem with some old Macbook EFI firmware in a gesture
> of mformat.
> Line 1375 of
>   http://svn.savannah.gnu.org/viewvc/mtools/trunk/mformat.c?revision=506&view=markup
> has
> 
>   /* install fake partition table pointing to itself */
> 
> Now we are wondering for which use case there is this MBR partition table
> entry starting with LBA 0 and claiming the whole device.
> 
> The SVN at savannah shows that this gesture was introduced with revision 4
> in may 2002 by "aknaff". My hope is that this was you and that you can tell
> grub-devel (Cc'ed) more about the motivation.
> 

This was indeed me.

As far as I remember this was indeed intended as a convenience when
moving removable disks (Zip, Jaz or similar) from platforms that expect
the device to be partitions to platforms that expect it to be unpartitioned.

A platform that expects the disk to be unpartitioned would just ignore
this mini partition table.

A platform that expects the disk to be partitioned would see this
partition table, follow the "link" to the first partition, and then
again fall on the same sector and continue.

> ---------------------------------------------------------------------------
> In case you are curious about the motivation of this mail:
> 
> Our particular use case is to create a FAT filesystem image for use as EFI
> system partition in ISO 9660 images on USB sticks.
> grub-mkrescue runs mformat to create this image and then uses mcopy to
> populate it with EFI start programs. See
>   http://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkrescue.c#n812
> 
> The partition entry with start LBA 0 caused the EFI firmware to get stuck,
> probably looping endlessly. Experiments with dd reveiled that the partition
> entry was to blame, and in it the start LBA 0.
> 
> This is clearly a firmware bug and obviously fixed in later models of
> Macbooks. Nevertheless we care for old oddball hardware.
> 
> So the question is which of these alternatives to choose:
> 
> - Keep the partition entry because its removal could break some other
>   EFI firmware's boot process.
> 
> - Overwrite the partition table in bytes 446 to 509 of the mformat result
>   by zeros before populating it with files.
> 
> - Use mformat option -k to avoid production of the partition table.
> 
> I personally have scruples to omit the other fields which get written
> if option -k is not present. Is there anything written with (!keepBoot)
> which we need for a vanilla FAT filesystem to be recognizable or usable ?

In the next release I'll provide a new option to just skip this
partition table, but still initialize the other fields (jump at
beginning of boot sector, mformat "banner", 0xaa55 boot signature,
physdrive, mtools boot code, zeroing out unused bytes)

If all goes well, I expect to make this new release by the end of the month

> 
> ---------------------------------------------------------------------------
> 
> Have a nice day :)
> 
> Thomas
> 

Thanks, same to you.

Alain

Re: Several GNU projects wondering about the reason for mformat partition table

[Thomas Schmitt]

Hi,

thanks a lot for the background info.

Alain Knaff wrote:
> A platform that expects the disk to be unpartitioned would just ignore
> this mini partition table.

The EFI platform encounters the mformat-produced image in a partition.
Nearly all implementations indeed ignore the MBR-ish table in the
partition. But one wanted to be smart, obviously.
(The disk device, a USB stick, was partitioned by GPT.)

If you see any other potential problem with using a mformat image
as content of a partition, then please tell.


> In the next release I'll provide a new option to just skip this
> partition table [...]
>
> If all goes well, I expect to make this new release by the end of the month

Good move.

------------------------------------------------------------------------

I assume that grub-mkrescue will have to wait a while before it can
unconditionally use a new option of mformat, or that it will need a
function like check_xorriso()
  http://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkrescue.c#n316
which is used to look for a five year old option in the help text of xorriso:
  http://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkrescue.c#n608


The three alternatives for grub-mkrescue now look like this:

> > - Keep the partition entry because its removal could break some other
> >   EFI firmware's boot process.

No objections are known yet against partition table removal. Its purpose
has nothing to do with the use case as EFI System Partition.
I am out of ideas whom else to ask for an opinion.


So i think grub-mkrescue should dare to choose one of the other two:

> > - Overwrite the partition table in bytes 446 to 509 of the mformat result
> >   by zeros before populating it with files.

If i am not mistaken, then this is equivalent to the new option of mformat.
grub-mkrescue could anticipate this option by own postprocessing after the
mformat run.


> > - Use mformat option -k to avoid production of the partition table.

Alain Knaff wrote about the new option probably in contrast to -k:
> [...] but still initialize the other fields (jump at
> beginning of boot sector, mformat "banner", 0xaa55 boot signature,
> physdrive, mtools boot code, zeroing out unused bytes)

None of the items in this list looks like it would be necessary for a
partition image which gets created as new file. I assume that no non-zero
bytes can sneak in, if the image file does not exist before it gets
written by mformat.


Have a nice day :)

Thomas