How important is the MBR partition offset of grub-mkrescue ?

How important is the MBR partition offset of grub-mkrescue ?

[Thomas Schmitt]

Hi,

i stumbled over a problem in the handling of grub-mkrescue
images on USB stick by Linux in /dev/disk/by-label.

The first MBR partition begins at 512-LBA 1 and thus is
not mountable as ISO image. I understand this is intentional.

ISO filesystems have a label (Volume Id) which gets used
for auto-mounting and shows up in modern Linuxes as
symbolic link in directory /dev/disk/by-label. 

But Linux (or its udev) insists in linking to /dev/sdX1
rather than to /dev/sdX, where the ISO filesystem is
mountable.
So this link cannot be used for mounting the ISO.

I hate automounters, but /dev/disk/by-label is also
a good way for a booting system to find its own
filesystem from where it stems.

----------------------------------------------------------

Is a workaround known on Linux side ? (E.g. is it a known
bug in udev configurations which can be fixed ?)

How important ist the partition start at LBA 1 in comparison
to this obvious drawback of grub-mkrescue images on Linux ?
(Start at LBA 0 solves the problem but makes the MBR part
of the first partition. I understand this is undesirable
somehow.)

Actually i wanted to show a system packager how to
use ./by-label instead of test-mounting all files in
/proc/partitions. Are there other methods which one
could use in the Linux initial RAM disk to find the
USB stick from where GRUB2 did boot ?


Have a nice day :)

Thomas

Re: How important is the MBR partition offset of grub-mkrescue ?

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 782 bytes --]

Hello, Thomas. Noce to hear from you.
> How important ist the partition start at LBA 1 in comparison
> to this obvious drawback of grub-mkrescue images on Linux ?
> (Start at LBA 0 solves the problem but makes the MBR part
> of the first partition. I understand this is undesirable
> somehow.)
LBA 0 is not an option. It will make hang, at least some versions OVMF,
quite possibly other firmwares as well. Nowadays what we have between 1
(inclusive) and isofs headers is all generated by xorriso (except on
sparc where we supply LBA 1)
It's possible to use another small value other than >1 in order to make
a set of headers at another position pointing to the same data.

Another workaround is to mount HFS+ filesystem which is already in a
partition (GPT one).



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 291 bytes --]

Re: How important is the MBR partition offset of grub-mkrescue ?

[Thomas Schmitt]

Hi,

> Noce to hear from you.

I am always watching here for any problems with ISOs.


> LBA 0 is not an option.

Thanks for confirming this (once again).


It turned out that /dev/sdb1 is link target in ./by-label
with any partition start LBA. Lack of brains.

So the last hope is the double superblock option of xorriso,
which to my knowledge was only tested with ISOLINUX up to now.
It seems to work with pure GRUB2 equipment too.

I repacked the ISO in question according to what i can
see in my olde copy of grub-mkrescue.
"$old_iso" is mounted at /mnt, modification-date was learned
from that image, /dvdbuffer is my local playground.
Option -partition_offset 16 is the new thing to be tested:

  xorriso -as mkisofs \
    -V "$version" \
    -o /dvdbuffer/"$version".iso \
    -graft-points \
    --modification-date=2013030413525500 \
    -b boot/grub/i386-pc/eltorito.img -no-emul-boot -boot-info-table \
    --embedded-boot "$old_iso" \
    --protective-msdos-label -r --sort-weight 0 / --sort-weight 1 /boot \
    -partition_offset 16 \
    /mnt

This yields an MBR (copied from $old_iso) with partition start
LBA 64 (16 blocks of 2048), two Primary Volume descriptors,
and two directory trees with the same file names but data block
addresses differing by 16.

The resulting image was put onto USB stick and DVD. Both boot fine
on my test machine.
Unlike with partition start at LBA 1, the USB stick can be mounted
as /dev/sdb and as /dev/sdb1.
(Simultaneously only by help of -o loop. Brain suffices to recognize
that /dev/sdb and /dev/sdb1 do overlap.)

My test machine boots via BIOS. The ISO image is only equipped
for BIOS, anyway. I have no means to test UEFI with partition
offset 16.

Does contemporary grub-mkrescue cause xorriso to produce GPT
for UEFI ? (That would be a new adventure with -partition_offset.)


If anybody has opportunity and curiosity:

It should be possible to append option -partition_offset 16
to the options of grub-mkrescue, so that it reaches xorriso
as one of the ${source} arguments.

There was a bug with -partition_offset with older versions
of xorriso, which caused Debian to stop using this option
in its amd64 UEFI capable images. So better get newest stable
xorriso for such a test. Currently this is:
  http://www.gnu.org/software/xorriso/xorriso-1.3.2.tar.gz


Theoretical problems:

El Torito booting from CD/DVD should not be influenced by
this unusual layout.  The ISO 9660 image beginning at LBA 0
is quite the same as without that option.

Nevertheless, there was a report that Apple "Snow Leopard"
refused to mount an ISO image with partition offset 16.
  http://lists.debian.org/debian-cd/2011/04/msg00032.html
  http://lists.debian.org/debian-cd/2011/04/msg00040.html
  http://lists.debian.org/debian-cd/2011/04/msg00042.html
Note well that this was not about booting.

Further, one never knows what the booting operating system
expects as layout of USB sticks. I consider the current layout
with LBA 1 to be more confusing. But that's only me ...


Have a nice day :)

Thomas

Re: How important is the MBR partition offset of grub-mkrescue ?

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 2757 bytes --]

> 
> This yields an MBR (copied from $old_iso) with partition start
> LBA 64 (16 blocks of 2048)
Hm, that would mean that some crapware like adobe might write between
MBR and first partition.
> 
> My test machine boots via BIOS. The ISO image is only equipped
> for BIOS, anyway. I have no means to test UEFI with partition
> offset 16.
> 
> Does contemporary grub-mkrescue cause xorriso to produce GPT
> for UEFI ? (That would be a new adventure with -partition_offset.)
> 
Yes, and we add HFS+ as well. Isn't this HFS+ catalog sufficient for the
problem at hand?
> 
> If anybody has opportunity and curiosity:
> 
> It should be possible to append option -partition_offset 16
> to the options of grub-mkrescue, so that it reaches xorriso
> as one of the ${source} arguments.
> 
> There was a bug with -partition_offset with older versions
> of xorriso, which caused Debian to stop using this option
> in its amd64 UEFI capable images. So better get newest stable
> xorriso for such a test. Currently this is:
>   http://www.gnu.org/software/xorriso/xorriso-1.3.2.tar.gz
>
Can we detect presence of this bug?

> 
> Theoretical problems:
> 
Additional problem you don't mention: consumption of space by additional
headers. We use xorriso for making floppies as well (and it works).
> El Torito booting from CD/DVD should not be influenced by
> this unusual layout.  The ISO 9660 image beginning at LBA 0
> is quite the same as without that option.
> 
> Nevertheless, there was a report that Apple "Snow Leopard"
> refused to mount an ISO image with partition offset 16.
>   http://lists.debian.org/debian-cd/2011/04/msg00032.html
>   http://lists.debian.org/debian-cd/2011/04/msg00040.html
>   http://lists.debian.org/debian-cd/2011/04/msg00042.html
> Note well that this was not about booting.
> 
But it will most likely swallow HFS+-hybrid disk without any problem.
Perhaps we should generate HFS+-hybrid even for BIOS-only CDs.
Another solution is to have a hybrid ISO + FAT or ISO + HFS+ + FAT layouts.
> Further, one never knows what the booting operating system
> expects as layout of USB sticks. I consider the current layout
> with LBA 1 to be more confusing. But that's only me ...
> 
The reason for the specified partition is twofold:
- Some computers check for presence of an active partition.
- To avoid some software overwriting the ISO
If user desires to have a partition for data he can always add a second
partition but it doesn't solve the problem of accessing ISO files.

> 
> Have a nice day :)
> 
> Thomas
> 
> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 291 bytes --]

Re: How important is the MBR partition offset of grub-mkrescue ?

[Thomas Schmitt]

Hi,

> Isn't this HFS+ catalog sufficient for the problem at hand?

I see the misleading link to /dev/sdb1 with Debian 6
and with a Debian 7 based distro. Dunno which udev rule
first looks at /dev/sdb for an ISO PVD and then links
./by-label to the first partition of that device.

The filesystem itself is not necessarily mounted.
Actually, the /dev/sdb1 with start at LBA 1 cannot be mounted.
Neither as ISO nor as HFS+ (if that was present). An offset
of 512 bytes would spoil any address that works in an ISO image.


> Can we detect presence of this bug?

It is a bug of the perceiving system. Linux specific. Lifetime
at least for Debian 6 and 7, i'd say. 

Put a grub-mkrescue generated ISO on a USB stick (e.g. /dev/sdb),
replug, and see what your Linux has in /dev/disk/by-label.
(If it has such a directory, at all.)


> Additional problem you don't mention: consumption of space by additional
> headers. We use xorriso for making floppies as well (and it works).

The feature is easy to add by user arguments of grub-mkrescue
anyway. No need for action inside grub-mkrescue.
I am mainly fishing for experience with pure GRUB2 boot setups
and Linux. Most distros let BIOS boot ISOLINUX.

Is there a distro which produces its ISO images with grub-mkrescue
and uses all or most of the BIOS, UEFI, and Apple features ?

As soon as i have a working ISO image, i can play with it.
I never learned to build operating systems or boot loaders myself.
It would be helpful to have a copy of the grub-mkrescue script
which was used for image production. I understand it evolves.


> Another solution is to have a hybrid ISO + FAT or ISO + HFS+ + FAT layouts.

But will this help against udev madness ?

Whatever. If i remember correctly, we still have open plans
about adding FAT to the super-hybrid layout.
See the very end of
  http://bazaar.launchpad.net/~libburnia-team/libisofs/scdbackup/view/head:/doc/boot_sectors.txt

(I see that i did not yet document the --grub2-boot-info
patching.)


Have a nice day :)

Thomas

Re: How important is the MBR partition offset of grub-mkrescue ?

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 2865 bytes --]

On 03.11.2013 16:21, Thomas Schmitt wrote:
> Hi,
> 
>> Isn't this HFS+ catalog sufficient for the problem at hand?
> 
> I see the misleading link to /dev/sdb1 with Debian 6
> and with a Debian 7 based distro. Dunno which udev rule
> first looks at /dev/sdb for an ISO PVD and then links
> ./by-label to the first partition of that device.
> 
> The filesystem itself is not necessarily mounted.
> Actually, the /dev/sdb1 with start at LBA 1 cannot be mounted.
> Neither as ISO nor as HFS+ (if that was present). An offset
> of 512 bytes would spoil any address that works in an ISO image.
> 
If GPT is present it takes precedence over MBR contents. And GPT would
point nicely to HFS+
> 
>> Can we detect presence of this bug?
> 
> It is a bug of the perceiving system. Linux specific. Lifetime
> at least for Debian 6 and 7, i'd say. 
> 
> Put a grub-mkrescue generated ISO on a USB stick (e.g. /dev/sdb),
> replug, and see what your Linux has in /dev/disk/by-label.
> (If it has such a directory, at all.)
> 
So it's bug in Debian, not in xorriso?
> 
>> Additional problem you don't mention: consumption of space by additional
>> headers. We use xorriso for making floppies as well (and it works).
> 
> The feature is easy to add by user arguments of grub-mkrescue
> anyway. No need for action inside grub-mkrescue.
> I am mainly fishing for experience with pure GRUB2 boot setups
> and Linux. Most distros let BIOS boot ISOLINUX.
> 
> Is there a distro which produces its ISO images with grub-mkrescue
> and uses all or most of the BIOS, UEFI, and Apple features ?
> 
> As soon as i have a working ISO image, i can play with it.
> I never learned to build operating systems or boot loaders myself.
> It would be helpful to have a copy of the grub-mkrescue script
> which was used for image production. I understand it evolves.
> 
It's in our git repo in util/ . There are 2 versions currently: sh one
and C one. It wasn't decided yet as to whether we'll switch to C one.
http://git.savannah.gnu.org/cgit/grub.git/tree/util?h=phcoder/install_c
grub-mkrescue.in is sh one and grub-mkrescue.c is C one.
> 
>> Another solution is to have a hybrid ISO + FAT or ISO + HFS+ + FAT layouts.
> 
> But will this help against udev madness ?
>
FAT would make it look like most common USB stick.

> Whatever. If i remember correctly, we still have open plans
> about adding FAT to the super-hybrid layout.
> See the very end of
>   http://bazaar.launchpad.net/~libburnia-team/libisofs/scdbackup/view/head:/doc/boot_sectors.txt
> 
> (I see that i did not yet document the --grub2-boot-info
> patching.)
> 
> 
> Have a nice day :)
> 
> Thomas
> 
> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 291 bytes --]

Re: How important is the MBR partition offset of grub-mkrescue ?

[Thomas Schmitt]

Hi,

> So it's bug in Debian, not in xorriso?

xorriso is not at fault. It only follows orders.


> It's in our git repo in util/

Yes, i know. But obviously there are older versions around.
For re-tinkering the image it would be helpful to know
the options used.

> grub-mkrescue.c is C one.

Oh. I was not aware of that one.

I just collected all xorrisofs options of
  http://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkrescue.in

Just stunning.
Has all this stuff actually been tested ?


> FAT would make it look like most common USB stick.

You mean a FAT partition with offset 512 bytes towards the
start of the ISO filesystem ?

But how to keep udev from recognizing the ISO PVD ?
(I think i need to find out by what udev rule this link
 gets created.)


Have a nice day :)

Thomas

Re: How important is the MBR partition offset of grub-mkrescue ?

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 860 bytes --]

> 
>> It's in our git repo in util/
> 
> Yes, i know. But obviously there are older versions around.
> For re-tinkering the image it would be helpful to know
> the options used.
> 
You can retrieve older versions from git history.
> 
> I just collected all xorrisofs options of
>   http://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkrescue.in
> 
> Just stunning.
> Has all this stuff actually been tested ?
> 
Yes, I've tsted mega-boot image manually an single-boot images are used
in automated testing system.
> 
>> FAT would make it look like most common USB stick.
> 
> You mean a FAT partition with offset 512 bytes towards the
> start of the ISO filesystem ?
>
Probably more, to avoid conflicting with APM and GPT

> But how to keep udev from recognizing the ISO PVD ?
My guess is that FAT rule would take precedence.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 291 bytes --]

Re: How important is the MBR partition offset of grub-mkrescue ?

[Thomas Schmitt]

Hi,

i believe to have found the udev rules in Debian 6 which
govern the population of /dev/disk/by-label.

File
  /lib/udev/rules.d/60-persistent-storage.rules
has
  # probe filesystem metadata of disks
  KERNEL!="sr*", IMPORT{program}="/sbin/blkid -o udev -p $tempnode"
  ...
  ENV{ID_FS_LABEL_ENC}=="?*",     ENV{ID_FS_USAGE}=="filesystem|other", \
          SYMLINK+="disk/by-label/$env{ID_FS_LABEL_ENC}"

I understand that if blkid sets variable ID_FS_LABEL_ENC to
a non-empty value, then this will become the link name in ./by-label.
The link target is the device that is being examined by the rule.

From a USB stick with partition start LBA 1, i get:

  $ /sbin/blkid -o udev -p /dev/sdb 
  ID_PART_TABLE_TYPE=dos
  ID_FS_LABEL=epidemic-4.1-b1-1-ts-amd64
  ID_FS_LABEL_ENC=epidemic-4.1-b1-1-ts-amd64
  ID_FS_TYPE=iso9660
  ID_FS_USAGE=filesystem
  $ /sbin/blkid -o udev -p /dev/sdb1
  $

So why the hell does /dev/sdb1 become link target ?
Its ID_FS_LABEL_ENC must be empty.

Any idea how to get a verbous log of these decisions ?


Have a nice day :)

Thomas

Re: How important is the MBR partition offset of grub-mkrescue ?

[Andrey Borzenkov]

В Sun, 03 Nov 2013 18:16:09 +0100
"Thomas Schmitt" <scdbackup@gmx.net> пишет:

> Hi,
> 
> i believe to have found the udev rules in Debian 6 which
> govern the population of /dev/disk/by-label.
> 
> File
>   /lib/udev/rules.d/60-persistent-storage.rules
> has
>   # probe filesystem metadata of disks
>   KERNEL!="sr*", IMPORT{program}="/sbin/blkid -o udev -p $tempnode"
>   ...
>   ENV{ID_FS_LABEL_ENC}=="?*",     ENV{ID_FS_USAGE}=="filesystem|other", \
>           SYMLINK+="disk/by-label/$env{ID_FS_LABEL_ENC}"
> 
> I understand that if blkid sets variable ID_FS_LABEL_ENC to
> a non-empty value, then this will become the link name in ./by-label.
> The link target is the device that is being examined by the rule.
> 
> From a USB stick with partition start LBA 1, i get:
> 
>   $ /sbin/blkid -o udev -p /dev/sdb 
>   ID_PART_TABLE_TYPE=dos
>   ID_FS_LABEL=epidemic-4.1-b1-1-ts-amd64
>   ID_FS_LABEL_ENC=epidemic-4.1-b1-1-ts-amd64
>   ID_FS_TYPE=iso9660
>   ID_FS_USAGE=filesystem
>   $ /sbin/blkid -o udev -p /dev/sdb1
>   $
> 
> So why the hell does /dev/sdb1 become link target ?
> Its ID_FS_LABEL_ENC must be empty.
> 
> Any idea how to get a verbous log of these decisions ?
> 

I confirm this. The culprit is this rule in 60-persistent-storage.rules:

# for partitions import parent information
ENV{DEVTYPE}=="partition", IMPORT{parent}="ID_*"

I'm not really sure how exactly to fix it. I.e. normally it is assumed
that device is either partitioned or not. Situation when we have
filesystem on a whole disk *and* individual partitions ... not sure.

I'm interested in which information actually needs to be imported from
parent. May be it should be less aggressive.

Re: How important is the MBR partition offset of grub-mkrescue ?

[Thomas Schmitt]

Hi,

Andrey Borzenkov:
> I confirm this. The culprit is this rule in 60-persistent-storage.rules:
> 
> # for partitions import parent information
> ENV{DEVTYPE}=="partition", IMPORT{parent}="ID_*"
>
> I'm not really sure how exactly to fix it. [...]
> I'm interested in which information actually needs to be
> imported from parent. May be it should be less aggressive.

In its broadness it looks like a mistake.
Bringing the filesystem label from the overall device to the
partition makes few sense.

I cannot spot a rule that would exclude DEVTYPE "disk" from
./by-label. So i assume that the partition link overwrote
the device link.

I am very unkeen with udev. Just imitating a skilled programmer
i think that this could be a fix:

ENV{DEVTYPE}=="partition",              IMPORT{parent}="ID_*", \
        ENV{ID_FS_LABEL}="" , ENV{ID_FS_LABEL_ENC}="" , \
        ENV{ID_FS_TYPE}="" , ENV{ID_FS_USAGE}=""

I am not sure whether ID_PART_TABLE_TYPE is intended to
propagate from storage device to partition. At least it
could make some sense.

Ok. Bold try. Adding the setters to the parent import.
Running udevadm control --reload-rules. Plugging stick
with partition start LBA 1.
Success !
  epidemic-4.1-b1-1-ts-amd64 -> ../../sdb

So it seems that really the "disk" link was overwritten by the
"partition" link.

This will still happen if grub-mkrescue was used with extra option
-partition_offset 16. But in that case it is intentional and
beautifying. (And even if it does not happen, it makes no difference.)


> I.e. normally it is assumed that device is either partitioned or not.
> Situation when we have filesystem on a whole disk *and* individual
> partitions ... not sure.

Obviously this was not foreseen when the rules were written.

Is there an upstream with whom one could discuss this issue
and who - in best case - could propagate an improvement
downstream ?

Can GRUB2 do advertising for above change in the rule set ?


Have a nice day :)

Thomas

Re: How important is the MBR partition offset of grub-mkrescue ?

[Andrey Borzenkov]

В Mon, 04 Nov 2013 15:03:45 +0100
"Thomas Schmitt" <scdbackup@gmx.net> пишет:

> Is there an upstream with whom one could discuss this issue

Yes. It was on Cc on my message; you removed it.

Re: How important is the MBR partition offset of grub-mkrescue ?

[Thomas Schmitt]

Hi,

> > Is there an upstream with whom one could discuss this issue
> Yes. It was on Cc on my message; you removed it.

Sorry. I did not notice and did not copy it.
I'll subscribe there and post a short version of my previous mail.


Have a nice day :)

Thomas