grub-mkrescue does not boot on older Macs as USB stick

grub-mkrescue does not boot on older Macs as USB stick

[Andrei Borzenkov]

I tested stock grub-mkrescue on MacBook 3,1 (late 2007, Core 2 Duo) and
iMac 9.1 (2009, same CPU) as USB stick. Neither boots; moreover,
inserting USB stick when boot manager is shown hangs system (it stops
reacting to mouse/keyboard); if stick is inserted during boot it never
gets as far as even showing boot manager.

Alexander reported something similar for 2007 Macs.

Stick is not detected even when inserted under OS X. MacBook runs 10.7
and reports that USB stick is not readable and suggests to iniitialize
it; iMac runs 10.9 and simply ignores it (may be I do not know where to
look).

I tried two different USB and one of them boots fine here on Dell
Latitude E5450 as EFI so I do not think both of them are defect. Of
course, they are not of finest quality ...

Same image as CD boots (I tried only MacBook but do not have reasons to
think that iMac will not boot). It presents three choices - two EFI and
one Windows. Windows ends up in presenting 1, 2 menu (without any
additional explanation) and not accepting any input. This is known
problem with legacy El Torito boot. First EFI boots; checking $cmdpath
it boots from HFS+ partition. Second EFI does not boot and falls through
to default boot order. When I try to chainload bootx64.efi manually (I
have dual OS X/openSUSE on it) it actually boots fine. I /think/ it
tries to load \EFI\Boot\boot.efi which is 32 bit only. I need to test
with removed boot.efi and with fat image.

I do not know if mkrescue image ever booted as USB on Mac. Alexander
says it does on modern Mac; anyone ever tried it on earlier models?

grub-mkrescue does not boot on older Macs as USB stick

[Andrei Borzenkov]

I tested stock grub-mkrescue on MacBook 3,1 (late 2007, Core 2 Duo) and
iMac 9.1 (2009, same CPU) as USB stick. Neither boots; moreover,
inserting USB stick when boot manager is shown hangs system (it stops
reacting to mouse/keyboard); if stick is inserted during boot it never
gets as far as even showing boot manager.

Alexander reported something similar for 2007 Macs.

Stick is not detected even when inserted under OS X. MacBook runs 10.7
and reports that USB stick is not readable and suggests to iniitialize
it; iMac runs 10.9 and simply ignores it (may be I do not know where to
look).

I tried two different USB and one of them boots fine here on Dell
Latitude E5450 as EFI so I do not think both of them are defect. Of
course, they are not of finest quality ...

Same image as CD boots (I tried only MacBook but do not have reasons to
think that iMac will not boot). It presents three choices - two EFI and
one Windows. Windows ends up in presenting 1, 2 menu (without any
additional explanation) and not accepting any input. This is known
problem with legacy El Torito boot. First EFI boots; checking $cmdpath
it boots from HFS+ partition. Second EFI does not boot and falls through
to default boot order. When I try to chainload bootx64.efi manually (I
have dual OS X/openSUSE on it) it actually boots fine. I /think/ it
tries to load \EFI\Boot\boot.efi which is 32 bit only. I need to test
with removed boot.efi and with fat image.

I do not know if mkrescue image ever booted as USB on Mac. Alexander
says it does on modern Mac; anyone ever tried it on earlier models?

Re: grub-mkrescue does not boot on older Macs as USB stick

[Andrei Borzenkov]

22.12.2015 23:02, Andrei Borzenkov пишет:
> I tested stock grub-mkrescue on MacBook 3,1 (late 2007, Core 2 Duo) and
> iMac 9.1 (2009, same CPU) as USB stick. Neither boots; moreover,
> inserting USB stick when boot manager is shown hangs system (it stops
> reacting to mouse/keyboard); if stick is inserted during boot it never
> gets as far as even showing boot manager.
> 
> Alexander reported something similar for 2007 Macs.
> 
> Stick is not detected even when inserted under OS X. MacBook runs 10.7
> and reports that USB stick is not readable and suggests to iniitialize
> it; iMac runs 10.9 and simply ignores it (may be I do not know where to
> look).
> 
> I tried two different USB and one of them boots fine here on Dell
> Latitude E5450 as EFI so I do not think both of them are defect. Of
> course, they are not of finest quality ...
> 
> Same image as CD boots (I tried only MacBook but do not have reasons to
> think that iMac will not boot). It presents three choices - two EFI and
> one Windows. Windows ends up in presenting 1, 2 menu (without any
> additional explanation) and not accepting any input. This is known
> problem with legacy El Torito boot. First EFI boots; checking $cmdpath
> it boots from HFS+ partition. Second EFI does not boot and falls through
> to default boot order. When I try to chainload bootx64.efi manually (I
> have dual OS X/openSUSE on it) it actually boots fine. I /think/ it
> tries to load \EFI\Boot\boot.efi which is 32 bit only. I need to test
> with removed boot.efi and with fat image.
> 

It is red herring. Mac correctly loads bootx64.efi and does not need or
use startup.nsh (I do not think it even has EFI Shell which interprets
this file).

> I do not know if mkrescue image ever booted as USB on Mac. Alexander
> says it does on modern Mac; anyone ever tried it on earlier models?
> 

OK, it turned out to be not related to xorriso, but for the record ...

Mac does not like FAT created by mformat. I can reproduce it easily by
creating single MSDOS partition on USB stick of type 0xef size 2880K,
formatting it with same options as grub-mkrescue:

sudo mformat -C -f 2880 -L 16 -i /dev/sdb1 ::

and copying bootx64.efi into \EFI\BOOT on it.

The same partition formatted with mkfs -t fat works like a charm.

Both Porteus and openSUSE CD (which correctly boot) have ESP created by
mkfs.fat.

TBH I think that precreating FAT once and shipping image with GRUB is
the most simple solution. It can be empty, we can dd files later.

Ideas?

Re: [Bug-xorriso] grub-mkrescue does not boot on older Macs as USB stick

[Alexander E. Patrakov]

23.12.2015 23:44, Andrei Borzenkov пишет:
> OK, it turned out to be not related to xorriso, but for the record ...
>
> Mac does not like FAT created by mformat. I can reproduce it easily by
> creating single MSDOS partition on USB stick of type 0xef size 2880K,
> formatting it with same options as grub-mkrescue:
>
> sudo mformat -C -f 2880 -L 16 -i /dev/sdb1 ::
>
> and copying bootx64.efi into \EFI\BOOT on it.
>
> The same partition formatted with mkfs -t fat works like a charm.
>
> Both Porteus and openSUSE CD (which correctly boot) have ESP created by
> mkfs.fat.
>
> TBH I think that precreating FAT once and shipping image with GRUB is
> the most simple solution. It can be empty, we can dd files later.
>
> Ideas?
>

I have byte-compared the boot sectors (because that's the only different 
things in the filesystems).

So, here are the differences:

Bytes 0x03 .. 0x0a represent the OEM ID (software that created the 
filesystem). mformat: "MTOO4018", mkfs.vfat: "mkfs.fat". Should not be 
significant.

Bytes 0x11 .. 0x12: a WORD that represents the number of root directory 
entries. mformat: 240, mkfs.vfat: 224. This is settable via the -r 
parameter of mformat.

Bytes 0x27 .. 0x2a: a DWORD with the serial number of the filesystem. 
Should not be significant.

Bytes 0x3e .. 0x1fe: x86 bootstrap code, which is not executed and thus 
is likely not significant.

If you make the number of root directory entries match (using -r 14), 
then there appears a difference in bytes 0x16 .. 0x17, which form a WORD 
that means the number of sectors per FAT. mformat -r 14: 16, mkfs.vfat: 
9. But this is settable with -L.

So, could you please run this?

sudo mformat -C -f 2880 -L 9 -r 14 -i /dev/sdb1 ::

If this fails, try giving mformat a known-good x86 boot sector using the 
-B option.

-- 
Alexander E. Patrakov

Re: [Bug-xorriso] grub-mkrescue does not boot on older Macs as USB stick

[Andrei Borzenkov]

[-- Attachment #1: Type: text/plain, Size: 719 bytes --]

23.12.2015 22:37, Alexander E. Patrakov пишет:
> Bytes 0x3e .. 0x1fe: x86 bootstrap code, which is not executed and thus
> is likely not significant.
> 

Wanna bet?

Could you test if the attached patch works for you on newer Mac? Note,
that it still does not detect EFI bootable USB stick if it is inserted
before power on - only when you stop in boot manager and insert stick.

It also still does not boot from the second (unknown) EFI suggested for
CD. But this second choice is also present only when CD is inserted
before power on - it is not shown if CD is inserted in boot manager.

I also do not get "Windows" menu choice on USB. May be because there is
no active partition.

Merry Christmas to everybody!

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: mac-mformat.patch --]
[-- Type: text/x-patch; name="mac-mformat.patch", Size: 3125 bytes --]

From: Andrei Borzenkov <arvidjaar@gmail.com>
Subject: [PATCH] grub-mkrescue: overwrite mtools bootstrap code with one from dosfstools

With ESP created by mtools Mac boot manager hangs. Tested on MacBook 3,1
and iMac 9,1.  It was determined that using bootstrap code from dosfstools
avoids this.

This patch explicitly overwrites bootstrap code with dosfstools version.

---
 util/grub-mkrescue.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)

diff --git a/util/grub-mkrescue.c b/util/grub-mkrescue.c
index 4511826..66ae085 100644
--- a/util/grub-mkrescue.c
+++ b/util/grub-mkrescue.c
@@ -53,6 +53,35 @@ static char **xorriso_argv;
 static char *iso_uuid;
 static char *iso9660_dir;
 
+/* The following bootstrap code is from dosfstools distributed under
+   GPL Version 3 license.
+ */
+#define BOOTCODE_OFFSET         62
+#define BOOTCODE_SIZE           448
+
+static char dummy_boot_code[BOOTCODE_SIZE] =
+  "\x0e"                      /* push cs */
+  "\x1f"                      /* pop ds */
+  "\xbe\x5b\x7c"              /* mov si, offset message_txt */
+  /* write_msg: */
+  "\xac"                      /* lodsb */
+  "\x22\xc0"                  /* and al, al */
+  "\x74\x0b"                  /* jz key_press */
+  "\x56"                      /* push si */
+  "\xb4\x0e"                  /* mov ah, 0eh */
+  "\xbb\x07\x00"              /* mov bx, 0007h */
+  "\xcd\x10"                  /* int 10h */
+  "\x5e"                      /* pop si */
+  "\xeb\xf0"                  /* jmp write_msg */
+  /* key_press: */
+  "\x32\xe4"                  /* xor ah, ah */
+  "\xcd\x16"                  /* int 16h */
+  "\xcd\x19"                  /* int 19h */
+  "\xeb\xfe"                  /* foo: jmp foo */
+  /* message_txt: */
+  "This is not a bootable disk.  Please insert a bootable floppy and\r\n"
+  "press any key to try again ... \r\n";
+
 static void
 xorriso_push (const char *val)
 {
@@ -793,6 +822,29 @@ main (int argc, char *argv[])
       xorriso_push ("-efi-boot-part");
       xorriso_push ("--efi-boot-image");
 
+      /* Mac fails to boot from ESP created by mformat; it was found that using
+         different bootstrap code in bootblock avoids this problem.
+	 Tested on MacBook 3,1 (late 2007) and iMac 9.1 (2009).
+       */
+      {
+	int efiimgfatfd;
+
+	efiimgfatfd = grub_util_fd_open (efiimgfat, GRUB_UTIL_FD_O_WRONLY);
+	if (efiimgfatfd == -1)
+	  grub_util_error (_("cannot open `%s': %s"), efiimgfat,
+			   strerror (errno));
+
+	if (grub_util_fd_seek (efiimgfatfd, BOOTCODE_OFFSET) == -1)
+	  grub_util_error (_("cannot seek `%s': %s"), efiimgfat,
+			   strerror (errno));
+
+	if (grub_util_fd_write (efiimgfatfd, dummy_boot_code, BOOTCODE_SIZE) != BOOTCODE_SIZE)
+	  grub_util_error (_("cannot write to `%s': %s"), efiimgfat,
+			   strerror (errno));
+
+	close (efiimgfatfd);
+      }
+
       grub_util_unlink_recursive (efidir);
       free (efiimgfat);
       free (efidir_efi);
-- 
tg: (a9399f2..) t/mtools-bad-boot-sector (depends on: master)

Re: [Bug-xorriso] grub-mkrescue does not boot on older Macs as USB stick

[Alexander E. Patrakov]

(all of the testing below was actually performed by Robert Jones, I only 
built the ISOs)

25.12.2015 01:17, Andrei Borzenkov пишет:
> 23.12.2015 22:37, Alexander E. Patrakov пишет:
>> Bytes 0x3e .. 0x1fe: x86 bootstrap code, which is not executed and thus
>> is likely not significant.
>>
>
> Wanna bet?
>
> Could you test if the attached patch works for you on newer Mac? Note,
> that it still does not detect EFI bootable USB stick if it is inserted
> before power on - only when you stop in boot manager and insert stick.

I cannot confirm this finding. The ISOs made before the fix hang the 
boot manager if booted from a USB flash drive. The ISOs made with the 
fix boot correctly (and show exactly one "EFI Boot" entry) both when 
cold-plugged and when hot-plugged.

The testing was done with grub commit id a9399f2.

> It also still does not boot from the second (unknown) EFI suggested for
> CD. But this second choice is also present only when CD is inserted
> before power on - it is not shown if CD is inserted in boot manager.

Not tested.

> I also do not get "Windows" menu choice on USB. May be because there is
> no active partition.

Even if I add a one-sector bootable partition of type 0 (as done by 
xorriso 1.4.3 with -DLibisofs_protective_msdos_plus_boot_dummY 
-DLibisofs_pmpbd_on_lba0), it doesn't show "Windows" when booted from USB.

-- 
Alexander E. Patrakov

Re: [Bug-xorriso] grub-mkrescue does not boot on older Macs as USB stick

[Andrei Borzenkov]

25.12.2015 20:35, Alexander E. Patrakov пишет:
> (all of the testing below was actually performed by Robert Jones, I only
> built the ISOs)
> 
> 25.12.2015 01:17, Andrei Borzenkov пишет:
>> 23.12.2015 22:37, Alexander E. Patrakov пишет:
>>> Bytes 0x3e .. 0x1fe: x86 bootstrap code, which is not executed and thus
>>> is likely not significant.
>>>
>>
>> Wanna bet?
>>
>> Could you test if the attached patch works for you on newer Mac? Note,
>> that it still does not detect EFI bootable USB stick if it is inserted
>> before power on - only when you stop in boot manager and insert stick.
> 
> I cannot confirm this finding. 

It means fix does not work?

> The ISOs made before the fix hang the
> boot manager if booted from a USB flash drive. The ISOs made with the
> fix boot correctly (and show exactly one "EFI Boot" entry) both when
> cold-plugged and when hot-plugged.
> 

Please, PLEASE - always make clear, *where* you test it!

> The testing was done with grub commit id a9399f2.
> 
>> It also still does not boot from the second (unknown) EFI suggested for
>> CD. But this second choice is also present only when CD is inserted
>> before power on - it is not shown if CD is inserted in boot manager.
> 
> Not tested.
> 
>> I also do not get "Windows" menu choice on USB. May be because there is
>> no active partition.
> 
> Even if I add a one-sector bootable partition of type 0 (as done by
> xorriso 1.4.3 with -DLibisofs_protective_msdos_plus_boot_dummY
> -DLibisofs_pmpbd_on_lba0), it doesn't show "Windows" when booted from USB.
> 

But your screenshots you posted earlier had it. I wonder what is the
difference (not that I care about legacy boot, but the more we
understand the better).

Re: [Bug-xorriso] grub-mkrescue does not boot on older Macs as USB stick

[Alexander E. Patrakov]

25.12.2015 22:44, Andrei Borzenkov пишет:
> 25.12.2015 20:35, Alexander E. Patrakov пишет:
>> (all of the testing below was actually performed by Robert Jones, I only
>> built the ISOs)
>>
>> 25.12.2015 01:17, Andrei Borzenkov пишет:
>>> 23.12.2015 22:37, Alexander E. Patrakov пишет:
>>>> Bytes 0x3e .. 0x1fe: x86 bootstrap code, which is not executed and thus
>>>> is likely not significant.
>>>>
>>>
>>> Wanna bet?
>>>
>>> Could you test if the attached patch works for you on newer Mac? Note,
>>> that it still does not detect EFI bootable USB stick if it is inserted
>>> before power on - only when you stop in boot manager and insert stick.
>>
>> I cannot confirm this finding.
>
> It means fix does not work?

The fix works.

Contrary to your statement, the USB stick works if it is inserted before 
power on. It also works when you stop in boot manager and insert stick.

>
>> The ISOs made before the fix hang the
>> boot manager if booted from a USB flash drive. The ISOs made with the
>> fix boot correctly (and show exactly one "EFI Boot" entry) both when
>> cold-plugged and when hot-plugged.
>>
>
> Please, PLEASE - always make clear, *where* you test it!

The 2007 Mac.

>
>> The testing was done with grub commit id a9399f2.
>>
>>> It also still does not boot from the second (unknown) EFI suggested for
>>> CD. But this second choice is also present only when CD is inserted
>>> before power on - it is not shown if CD is inserted in boot manager.
>>
>> Not tested.
>>
>>> I also do not get "Windows" menu choice on USB. May be because there is
>>> no active partition.

Correction: this only applies to the 2007 Mac. Other models not tested.

>>
>> Even if I add a one-sector bootable partition of type 0 (as done by
>> xorriso 1.4.3 with -DLibisofs_protective_msdos_plus_boot_dummY
>> -DLibisofs_pmpbd_on_lba0), it doesn't show "Windows" when booted from USB.
>>
>
> But your screenshots you posted earlier had it. I wonder what is the
> difference (not that I care about legacy boot, but the more we
> understand the better).

Are you referring to screenshots at 
https://goo.gl/photos/GRS5khE8r7Xgat9L8 ? They are from the 2012 Mac 
model. There was another statement from me about the Windows boot 
option, and it also was about the 2012 model.

-- 
Alexander E. Patrakov