grub-mkrescue fails with HFS+ error possibly due to directory size

grub-mkrescue fails with HFS+ error possibly due to directory size

[Thomas Schmitt]

Hi,

i just got a bug report about Guix ISO image production failing with

  libisofs: FAILURE : HFS+ map nodes aren't implemented

The message stems from the HFS+ contribution to libisofs by Vladimir
Serbinenko in 2012. About the only program which asks libisofs to
produce HFS+ is grub-mkrescue, which to my knowledge is indeed used
by Guix.

I need a confirmation of my preliminary diagnosis and, if true, a decision
how important HFS+ is nowadays for grub-mkrescue.

I believe to see a limitation to 30720 files per directory, which would
become an official limitation of grub-mkrescue if HFS+ production shall
be continued for I386, X86_64, and POWERPC_IEEE1275.

------------------------------------------------------------------------

The code which issues the error is in
  https://dev.lovelyhq.com/libburnia/libisofs/raw/master/libisofs/hfsplus.c
line 1791:

    if (target->hfsp_nnodes > (cat_node_size - 0x100) * 8)
      {
        iso_msg_submit(target->image->id, ISO_MANGLE_TOO_MUCH_FILES, 0,
                        "HFS+ map nodes aren't implemented");
        ret = ISO_MANGLE_TOO_MUCH_FILES;
        goto ex;
      }

This looks not like a mangling problem but more like a plain overflow.

target->hfsp_nnodes is a counter which i have difficulties to understand:

      target->hfsp_levels[level].level_size++;
      target->hfsp_nnodes += target->hfsp_levels[level].level_size;

cat_node_size is set by

    target->hfsp_cat_node_size = 2 * target->opts->hfsp_block_size;
    ...
    cat_node_size = target->hfsp_cat_node_size;

target->opts->hfsp_block_size is most probably HFSPLUS_DEFAULT_BLOCK_SIZE
= 2048.

My rough theory is that a HFS+ directory cannot have more than
   (4096 - 256) * 8 = 30720
file names. There seems to be a workaround "HFS+ map nodes". But the HFS+
code in libisofs reports not to have it implemented.


Have a nice day :)

Thomas

Re: grub-mkrescue fails with HFS+ error possibly due to directory size

[Thomas Schmitt]

Hi,

(cc-ing bug-xorriso@gnu.org and the reporter of the problem.)

I now have the Guix ISO which fails when created by grub-mkrescue
with HFS+ tree.

To my newest theory it is not about the number of files in a directory
but about the total number of files in the tree and their name lengths.

I wonder how to describe this limit to the users of grub-mkrescue.
Maybe:

  grub-mkrescue for platforms I386_EFI, X86_64_EFI, and POWERPC_IEEE1275
  has a limit on the number of files multiplied by their average name
  length. Beginning with about 300,000 files of usual name length
  expect a xorriso error
    libisofs: FAILURE : HFS+ map nodes aren't implemented
  plus a rather misleading error message
    libisofs: FAILURE : Too much files to mangle, cannot guarantee
                        unique file names

  The limit can only be avoided by suppressing xorrisofs option -hfsplus.
  This can be done by using xorriso script
    frontend/grub-mkrescue-sed.sh
  with MKRESCUE_SED_MODE="mbr_only" or MKRESCUE_SED_MODE="gpt_appended".
  Another method is to add as last arguments of grub-mkrescue these two
    -- -hfsplus off
  in order to leave xorriso's mkisofs emulation mode and to disable HFS+
  production by a generic xorriso command.


I still hope for a clarifying comment by Vladimir Serbinenko.


----------------------------------------------------------------------
Reasoning:

Riddling what is overflowing in hfsplus.c i found some description of HFS+
in
  https://developer.apple.com/library/archive/technotes/tn/tn1150.html

Since hfsplus.c reports to need "map nodes" i assume that it is the
"header node" which contains a "map record". Map nodes would then
be data structures which contain more map records.
So for now i believe the overflow is in the "B-tree Map Record" of
the header node.
  "It is a bitmap that indicates which nodes in the B-tree are used
   and which are free. The bits are interpreted in the same way as the
   bits in the allocation file."

The number in  target->hfsp_nnodes  which causes the overflow is 35487.
This is not the number of files 434,920. The number in
  target->hfsp_nleafs
is 869842 which is (434920+1)*2.
The loop which accumulates the number of target->hfsp_nnodes iterates
over this number of 869842.

I guess that the 869842 leafs are planned to get stored in allocation
blocks of target->hfsp_cat_node_size which i now know is 4096.
Each leaf occupies 50 to 200 bytes in the allocation block.
This roughly matches the ratio of 35487 * 4096 / 869842  = 167.

So it seems to be about the number of files and the sum of their name
lengths. The size limit gets exceeded by about 19,525,632 bytes or
(estimated by above ratio) 116,920 files.


Have a nice day :)

Thomas

Re: grub-mkrescue fails with HFS+ error possibly due to directory size

[Thomas Schmitt]

Hi,

what shall be done about the failure of grub-mkrescue for I386_EFI,
X86_64_EFI, and POWERPC_IEEE1275 with very large input file trees ?
  https://lists.gnu.org/archive/html/grub-devel/2020-06/msg00062.html

libisofs now emits a better error message
  Too many files in HFS+ directory tree
instead of
  Too much files to mangle, cannot guarantee unique file names
man xorrisofs now says:
  WARNING:
  The  HFS+  implementation in libisofs has a limit of 125,829,120
  bytes for the size of the overall directory tree. This  suffices
  for about 300,000 files of normal name length. If the limit gets
  exceeded, a FAILURE event will be issued and the ISO  production
  will not happen.


But a solution of the problem itself is out of my reach.

Possible ways to go:

- Enhance libisofs to create "HFS+ map nodes" for more tree storage.
  (Would need help from Vladimir Serbinenko and substantial testing.)

- Disable -hfsplus in grub-mkrescue by default and have a new option to
  enable it for the antique Macs which need it.
  (It would be not enough for the user to just add xorrisofs option
   -hfsplus. Needed are the options pushed at
     http://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkrescue.c#n718
   up to line 731.)

- Document the problem for grub-mkrescue and add a new option of
  grub-mkrescue which suppresses xorrisofs option -hfsplus.

- Document for grub-mkrescue the existing workaround of adding as last
  arguments these three:
    -- -hfsplus off
  (Problem is that xorriso -as mkisofs cannot disable -hfsplus once it
   was given. But xorriso as itself can. "--" ends mkisofs emulation.)


Have a nice day :)

Thomas