[grub-fs-tester.in] zisofs test looks unsuitable

[grub-fs-tester.in] zisofs test looks unsuitable

[Thomas Schmitt]

Hi,

the xorriso run for testing zisofs in

  tests/util/grub-fs-tester.in

looks not like it would cause any zisofs compression in the ISO.

Line 1024:

      xorriso -compliance rec_mtime -set_filter_r --zisofs -- -zisofs default
              -as mkisofs ... some options ... --
              -set_filter_r --zisofs -- -zisofs default -add /="$MASTER" ;;

should probably become

      xorriso -compliance rec_mtime \
              -as mkisofs ... same options as above ... -- \
              -add /="$MASTER" -- \
              -zisofs default \
              -set_filter_r --zisofs / --
              ;;

In the current run, both times when "-set_filter_r --zisofs" is executed
in xorriso's native command mode, the emerging root directory is still empty.
So -set_filter_r cannot yet affect the files of "$MASTER" which get added
only later.

Command -zisofs sets parameters for -set_filter_r. So it must be executed
before any successful -set_filter_r. Only the lack of any installed
compression filters keeps the existing run from failing with
  libisofs: FAILURE : Cannot set global zisofs parameters while filters exist

Further:
Command -set_filter_r expects one or more paths in the ISO as parameters.
The empty path list defaults to "/" indeed, but that's not a documented
feature.
Command -add has a variable length parameter list and should thus be
terminated by '--'. In my proposal it has to be terminated because more
commands follow.


Experiment:

Currently the run is essentially like

  xorriso -compliance rec_mtime \
          -as mkisofs -o test.iso --graft-points -- \
          -add /=test -- -set_filter_r --zisofs / --

My test directory contains

  -rw-r--r-- 1 thomas thomas 51200 Aug 25 11:51 zeros
  -rw-r--r-- 1 thomas thomas     6 Aug 25 09:37 ÄÖÜß

No zisofs compression gets applied by above run to the files, as can be
seen by a subsequent run of

  xorriso -indev test.iso -find / -exec show_stream --

which reports

  '/zeros' < image:'/zeros'
  '/ÄÖÜß' < image:'/ÄÖÜß'

In contrast, my proposed xorriso run causes compression of file "zeros",
which then causes the xorriso run with -indev to install a decompression
filter to that file:

  '/zeros' < --zisofs-decode:pz:32k < image:'/zeros'
  '/ÄÖÜß' < image:'/ÄÖÜß'

The small "ÄÖÜß" staid uncompressed, because compression yielded no gain in
terms of 2048 byte blocks.


Have a nice day :)

Thomas

Re: [grub-fs-tester.in] zisofs test looks unsuitable

[Glenn Washburn]

On Wed, 25 Aug 2021 12:29:48 +0200
"Thomas Schmitt" <scdbackup@gmx.net> wrote:

> Hi,
> 
> the xorriso run for testing zisofs in
> 
>   tests/util/grub-fs-tester.in
> 
> looks not like it would cause any zisofs compression in the ISO.
> 
> Line 1024:
> 
>       xorriso -compliance rec_mtime -set_filter_r --zisofs -- -zisofs
> default -as mkisofs ... some options ... --
>               -set_filter_r --zisofs -- -zisofs default -add
> /="$MASTER" ;;
> 
> should probably become
> 
>       xorriso -compliance rec_mtime \
>               -as mkisofs ... same options as above ... -- \
>               -add /="$MASTER" -- \
>               -zisofs default \
>               -set_filter_r --zisofs / --
>               ;;
> 
> In the current run, both times when "-set_filter_r --zisofs" is
> executed in xorriso's native command mode, the emerging root
> directory is still empty. So -set_filter_r cannot yet affect the
> files of "$MASTER" which get added only later.
> 
> Command -zisofs sets parameters for -set_filter_r. So it must be
> executed before any successful -set_filter_r. Only the lack of any
> installed compression filters keeps the existing run from failing with
>   libisofs: FAILURE : Cannot set global zisofs parameters while
> filters exist
> 
> Further:
> Command -set_filter_r expects one or more paths in the ISO as
> parameters. The empty path list defaults to "/" indeed, but that's
> not a documented feature.
> Command -add has a variable length parameter list and should thus be
> terminated by '--'. In my proposal it has to be terminated because
> more commands follow.
> 
> 
> Experiment:
> 
> Currently the run is essentially like
> 
>   xorriso -compliance rec_mtime \
>           -as mkisofs -o test.iso --graft-points -- \
>           -add /=test -- -set_filter_r --zisofs / --
> 
> My test directory contains
> 
>   -rw-r--r-- 1 thomas thomas 51200 Aug 25 11:51 zeros
>   -rw-r--r-- 1 thomas thomas     6 Aug 25 09:37 ÄÖÜß
> 
> No zisofs compression gets applied by above run to the files, as can
> be seen by a subsequent run of
> 
>   xorriso -indev test.iso -find / -exec show_stream --
> 
> which reports
> 
>   '/zeros' < image:'/zeros'
>   '/ÄÖÜß' < image:'/ÄÖÜß'
> 
> In contrast, my proposed xorriso run causes compression of file
> "zeros", which then causes the xorriso run with -indev to install a
> decompression filter to that file:
> 
>   '/zeros' < --zisofs-decode:pz:32k < image:'/zeros'
>   '/ÄÖÜß' < image:'/ÄÖÜß'
> 
> The small "ÄÖÜß" staid uncompressed, because compression yielded no
> gain in terms of 2048 byte blocks.

If I'm not mistaken, there is currently no test which actually
exercises that code, ie no test runs "grub-fs-tester ziso9660". I think
it could be a good idea to have one though.

However, when attempting to run the test manually, I'm getting a
failure probably related to what you've found.

$ ./grub-fs-tester ziso9660
xorriso 1.5.2 : RockRidge filesystem manipulator, libburnia project.

Drive current: -outdev
'stdio:/tmp/tmp.82yFdjU5NO/ziso9660_512_512_1_0.img' Media current:
stdio file, overwriteable Media status : is blank
Media summary: 0 sessions, 0 data blocks, 0 data, 4621m free
xorriso : FAILURE : -volid: Text too long (1650 > 32)
xorriso : aborting : -abort_on 'FAILURE' encountered 'FAILURE'

I haven't dug very far, but I'm unsure if GRUB currently is supposed to
support this.

Glenn

Re: [grub-fs-tester.in] zisofs test looks unsuitable

[Thomas Schmitt]

Hi,

Glenn Washburn wrote:
> If I'm not mistaken, there is currently no test which actually
> exercises that code, ie no test runs "grub-fs-tester ziso9660". I think
> it could be a good idea to have one though.

So we should get this to work. zisofs can be helpful to squeeze more
operating system into a size limited ISO image.


> However, when attempting to run the test manually, I'm getting a
> failure probably related to what you've found.
> $ ./grub-fs-tester ziso9660
> [...]
> xorriso : FAILURE : -volid: Text too long (1650 > 32)

This is something different.
"-volid" is the native xorriso command that's behind mkisofs emulation
option -V. In
  https://git.savannah.gnu.org/cgit/grub.git/tree/tests/util/grub-fs-tester.in#n1024
i see

  -V "$FSLABEL"

which is also used with the case x"iso9660").

So i assume that it is not curbed to 32 bytes by the code after line 315

  # FS LIMITATION: afs and iso9660 label is at most 32 UTF-8 characters

It seems that line 317 should be extended by a new line 318

-                     | xiso9660_1999 | xrockridge_1999 | xrockridge_joliet_1999)
+                     | xiso9660_1999 | xrockridge_1999 | xrockridge_joliet_1999\
+                     | xziso9660)

Nitpicking:
The comment speaks of "32 UTF-8 characters" but it must be at most 32 bytes.
(Wisely the code does not truncate the long UTF-8 string but rather creates
a new string, which seems to have 31 bytes. I'm unsure whether my desktop
properly copies and pastes those [02F 99x] characters from line 318.)


Have a nice day :)

Thomas

Re: [grub-fs-tester.in] zisofs test looks unsuitable

[Thomas Schmitt]

Hi,

i managed to get the ziso9660 test running with actual compression.

  ./grub-fs-tester ziso9660

now produces an ISO image but seems not to be happy with it.

If i get it right then it sees the compressed size 1124519 of file "1.img"
but expects its uncompressed size 5242879.
Linux mount sees
  -rw-r--r-- 2 thomas thomas 5242879 Aug 26 21:52 1.img

So grub-fstest.c CMD_LS seems not yet ready for dealing with zisofs.

This is the change which brought me that far, including a draft for a
commit message:
============================================================================

The grub-fs-tester test "ziso9660" failed early with a xorriso error
because it was not in the list of tests which tolerate at most 32 bytes
of FSLABEL.
The xorriso run of ziso9660 did not actually compress the data files because
the xorriso command for installing zisofs filters was applied before any
files had populated the emerging ISO image.
So add "ziso9660" where other ISO 9660 tests get a shorter FSLABEL and
re-arrange the commands of the xorriso run to bring -add and -zisofs before
-set_filter_r.

Signed-off-by: Thomas Schmitt <scdbackup@gmx.net>

diff --git a/tests/util/grub-fs-tester.in b/tests/util/grub-fs-tester.in
index bfc425e1f..834c05728 100644
--- a/tests/util/grub-fs-tester.in
+++ b/tests/util/grub-fs-tester.in
@@ -314,7 +314,8 @@ for LOGSECSIZE in $(range "$MINLOGSECSIZE" "$MAXLOGSECSIZE" 1); do
 		    FSLABEL="grub_;/testéтi u😁莽茝кириrewfceniuewruevrewnuuireurevueurnievrewfne";;
  	            # FS LIMITATION: afs and iso9660 label is at most 32 UTF-8 characters
 		x"afs" | xiso9660 | xrockridge | xrockridge_joliet\
-                     | xiso9660_1999 | xrockridge_1999 | xrockridge_joliet_1999)
+		      | xiso9660_1999 | xrockridge_1999\
+		      | xrockridge_joliet_1999 | xziso9660)
 		     FSLABEL="gr_;/é莭莽😁кирит u";;
  	            # FS LIMITATION: bfs label is at most 32 UTF-8 characters
  	            # OS LIMITATION: bfs label can't contain ; or /
@@ -1020,7 +1021,7 @@ for LOGSECSIZE in $(range "$MINLOGSECSIZE" "$MAXLOGSECSIZE" 1); do
 		    (cd "$MASTER"; find . | cpio -o -H "$(echo ${fs} | sed 's@^cpio_@@')" > "${FSIMAGEP}0.img" ) ;;
 		x"ziso9660")
 		    FSUUID=$(date -u +%Y-%m-%d-%H-%M-%S-00);
-		    xorriso -compliance rec_mtime -set_filter_r --zisofs -- -zisofs default -as mkisofs -iso-level 3 -graft-points -R -J -joliet-long -V "$FSLABEL" --modification-date=$(echo ${FSUUID} | sed 's/-//g;') -o "${FSIMAGEP}0.img"  -- -set_filter_r --zisofs -- -zisofs default -add /="$MASTER" ;;
+		    xorriso -compliance rec_mtime -as mkisofs -iso-level 3 -graft-points -R -J -joliet-long -V "$FSLABEL" --modification-date=$(echo ${FSUUID} | sed 's/-//g;') -o "${FSIMAGEP}0.img" -- -add /="$MASTER" -- -zisofs default -set_filter_r --zisofs / -- ;;
 		x"iso9660")
 		    FSUUID=$(date -u +%Y-%m-%d-%H-%M-%S-00);
 		    xorriso --rockridge off -compliance rec_mtime -as mkisofs -iso-level 3 -graft-points -V "$FSLABEL" --modification-date=$(echo ${FSUUID} | sed 's/-//g;') -o "${FSIMAGEP}0.img" /="$MASTER"  ;;

============================================================================

Here is a run with this change:
============================================================================

$ ./grub-fs-tester ziso9660
GNU xorriso 1.5.4.pl02 : RockRidge filesystem manipulator, libburnia project.

Drive current: -outdev 'stdio:/tmp/tmp.iIXC5zIfEd/ziso9660_512_512_1_0.img'
Media current: stdio file, overwriteable
Media status : is blank
Media summary: 0 sessions, 0 data blocks, 0 data,  400g free
xorriso : WARNING : -volid text problematic as automatic mount point name
xorriso : WARNING : -volid text is too long for Joliet (31 > 16)
xorriso : WARNING : -volid text does not comply to ISO 9660 / ECMA 119 rules
Added to ISO image: directory '/'='/tmp/tmp.iIXC5zIfEd/master'
xorriso : UPDATE :     227 files added in 1 seconds
xorriso : UPDATE :      10 file filters processed in 1 seconds
libisofs: WARNING : Cannot add /longsym to Joliet tree. Symlinks can only be added to a Rock Ridge tree.
libisofs: WARNING : Cannot add /psym to Joliet tree. Symlinks can only be added to a Rock Ridge tree.
libisofs: WARNING : Cannot add /sdir/ssym to Joliet tree. Symlinks can only be added to a Rock Ridge tree.
libisofs: WARNING : Cannot add /sdir/usym to Joliet tree. Symlinks can only be added to a Rock Ridge tree.
libisofs: WARNING : Cannot add /sym to Joliet tree. Symlinks can only be added to a Rock Ridge tree.
libisofs: WARNING : Filename not suitable for Joliet character set UCS-2 : "éàèüöäëñкирилица莭莽茝Ελληνικά😁😜😒"
libisofs: WARNING : Filename not suitable for Joliet character set UCS-2 : "Ελληνικάкирилица😁😜😒éàèüöäëñ莭莽茝"
libisofs: WARNING : Cannot add /Ελληνικάкирилица😁😜😒éàèüöäëñ莭莽茝 to Joliet tree. Symlinks can only be added to a Rock Ridge tree.
xorriso : UPDATE : Writing:       1658s   32.2%   fifo   0%  buf  50%
libisofs: WARNING : Sum of filenames not suitable for Joliet character set UCS-2: 2
ISO image produced: 4978 sectors
Written to medium : 5152 sectors at LBA 32
Writing to 'stdio:/tmp/tmp.iIXC5zIfEd/ziso9660_512_512_1_0.img' completed successfully.

Device proc: Filesystem type procfs - Sector size 512B - Total size 0KiB
Device loop0: Filesystem type iso9660 - Label `gr_;/é莭莽😁кирит u' - Last modifition time 2021-08-26 19:17:46 Thursday, UUID 2021-08-26-19-17-46-00 - Sector size 512B - Total size 10368KiB
Device host: Filesystem type hostfs - Sector size 512B - Total size 0KiB

LIST FAIL
DIR          20210826191746 ./
DIR          20210826191746 ../
DIR          20210826191745 0/
1124311      20210826191746 1.img
257343       20210826191746 american-english
1124219      20210826191746 CaSe
1124683      20210826191746 cAsE
1124311      20210826191746 hard
1124653      20210826191746 longsym
1124353      20210826191746 psym
1124653      20210826191746 qwertzuiopasdfghjklyxcvbnm1234567890qwertzuiopasdfghjklyxcvbnm1234567890oiewqfiewioqoiqoiurqruewqoiuwoieoiiuewqroreqiufieiuwrnureweriuvceoiroiewqoiricdsalkcndsakfirefoiwqeoircorejwoijfreoijojoiewjfwnfcoirenfoirefnreoifenoiwfnoi
DIR          20210826191746 sdir/
1124311      20210826191746 sym
1124813      20210826191746 .?*\!"#%@$%&'()+ ,-.:;<=>^{_}[]`|~.
1124638      20210826191746 éàèüöäëñкирилица莭莽茝Ελληνικά😁😜😒
1124638      20210826191746 Ελληνικάкирилица😁😜😒éàèüöäëñ莭莽茝
total 31692
drwxr-xr-x 3 thomas thomas    4096 Aug 26 19:17 0
-rw-r--r-- 2 thomas thomas 5242879 Aug 26 19:17 1.img
-rw-r--r-- 1 thomas thomas 5242879 Aug 26 19:17 CaSe
-rw-r--r-- 1 thomas thomas  972398 Aug 26 19:17 american-english
-rw-r--r-- 1 thomas thomas 5242879 Aug 26 19:17 cAsE
-rw-r--r-- 2 thomas thomas 5242879 Aug 26 19:17 hard
lrwxrwxrwx 1 thomas thomas     227 Aug 26 19:17 longsym -> qwertzuiopasdfghjklyxcvbnm1234567890qwertzuiopasdfghjklyxcvbnm1234567890oiewqfiewioqoiqoiurqruewqoiuwoieoiiuewqroreqiufieiuwrnureweriuvceoiroiewqoiricdsalkcndsakfirefoiwqeoircorejwoijfreoijojoiewjfwnfcoirenfoirefnreoifenoiwfnoi
lrwxrwxrwx 1 thomas thomas     806 Aug 26 19:17 psym -> /0//1/2/3//4/5/6//7/8/9//10/11/12//13/14/15//16/17/18//19/20/21//22/23/24//25/26/27//28/29/30//31/32/33//34/35/36//37/38/39//40/41/42//43/44/45//46/47/48//49/50/51//52/53/54//55/56/57//58/59/60//61/62/63//64/65/66//67/68/69//70/71/72//73/74/75//76/77/78//79/80/81//82/83/84//85/86/87//88/89/90//91/92/93//94/95/96//97/98/99//100/101/102//103/104/105//106/107/108//109/110/111//112/113/114//115/116/117//118/119/120//121/122/123//124/125/126//127/128/129//130/131/132//133/134/135//136/137/138//139/140/141//142/143/144//145/146/147//148/149/150//151/152/153//154/155/156//157/158/159//160/161/162//163/164/165//166/167/168//169/170/171//172/173/174//175/176/177//178/179/180//181/182/183//184/185/186//187/188/189//190/191/192//193/194/195//196/197/198//199/200/201//202/203/204//205/206/207//208/209/p.img
-rw-r--r-- 1 thomas thomas 5242879 Aug 26 19:17 qwertzuiopasdfghjklyxcvbnm1234567890qwertzuiopasdfghjklyxcvbnm1234567890oiewqfiewioqoiqoiurqruewqoiuwoieoiiuewqroreqiufieiuwrnureweriuvceoiroiewqoiricdsalkcndsakfirefoiwqeoircorejwoijfreoijojoiewjfwnfcoirenfoirefnreoifenoiwfnoi
drwxr-xr-x 2 thomas thomas    4096 Aug 26 19:17 sdir
lrwxrwxrwx 1 thomas thomas       5 Aug 26 19:17 sym -> 1.img
-rw-r--r-- 1 thomas thomas 5242879 Aug 26 19:17 éàèüöäëñкирилица莭莽茝Ελληνικά😁😜
lrwxrwxrwx 1 thomas thomas      72 Aug 26 19:17 Ελληνικάкирилица😁😜😒éàèüöäëñ莭莽茝 -> éàèüöäëñкирилица莭莽茝Ελληνικά😁😜😒
$ echo $?
1

============================================================================


Have a nice day :)

Thomas

Re: [grub-fs-tester.in] zisofs test looks unsuitable

[Glenn Washburn]

On Thu, 26 Aug 2021 22:07:29 +0200
"Thomas Schmitt" <scdbackup@gmx.net> wrote:

> Hi,
> 
> i managed to get the ziso9660 test running with actual compression.
> 
>   ./grub-fs-tester ziso9660
> 
> now produces an ISO image but seems not to be happy with it.
> 
> If i get it right then it sees the compressed size 1124519 of file
> "1.img" but expects its uncompressed size 5242879.
> Linux mount sees
>   -rw-r--r-- 2 thomas thomas 5242879 Aug 26 21:52 1.img
> 
> So grub-fstest.c CMD_LS seems not yet ready for dealing with zisofs.

This confirms my suspicions. I think the changes to get the test
working are worthy of inclusion so that the tests are ready when this
feature gets implemented. I'm not familiar with that code nor the
format details, but it sounds like you might be able to determine if
its something somewhat easy to add to GRUB.

> This is the change which brought me that far, including a draft for a
> commit message:
> ============================================================================
> 
> The grub-fs-tester test "ziso9660" failed early with a xorriso error
> because it was not in the list of tests which tolerate at most 32
> bytes of FSLABEL.
> The xorriso run of ziso9660 did not actually compress the data files
> because the xorriso command for installing zisofs filters was applied
> before any files had populated the emerging ISO image.
> So add "ziso9660" where other ISO 9660 tests get a shorter FSLABEL and
> re-arrange the commands of the xorriso run to bring -add and -zisofs
> before -set_filter_r.
> 
> Signed-off-by: Thomas Schmitt <scdbackup@gmx.net>
> 
> diff --git a/tests/util/grub-fs-tester.in
> b/tests/util/grub-fs-tester.in index bfc425e1f..834c05728 100644
> --- a/tests/util/grub-fs-tester.in
> +++ b/tests/util/grub-fs-tester.in
> @@ -314,7 +314,8 @@ for LOGSECSIZE in $(range "$MINLOGSECSIZE"
> "$MAXLOGSECSIZE" 1); do FSLABEL="grub_;/testéтi
> u😁莽茝кириrewfceniuewruevrewnuuireurevueurnievrewfne";; # FS
> LIMITATION: afs and iso9660 label is at most 32 UTF-8 characters
> x"afs" | xiso9660 | xrockridge | xrockridge_joliet\
> -                     | xiso9660_1999 | xrockridge_1999 |
> xrockridge_joliet_1999)
> +		      | xiso9660_1999 | xrockridge_1999\
> +		      | xrockridge_joliet_1999 | xziso9660)
>  		     FSLABEL="gr_;/é莭莽😁кирит u";;
>   	            # FS LIMITATION: bfs label is at most 32 UTF-8
> characters # OS LIMITATION: bfs label can't contain ; or /
> @@ -1020,7 +1021,7 @@ for LOGSECSIZE in $(range "$MINLOGSECSIZE"
> "$MAXLOGSECSIZE" 1); do (cd "$MASTER"; find . | cpio -o -H "$(echo
> ${fs} | sed 's@^cpio_@@')" > "${FSIMAGEP}0.img" ) ;; x"ziso9660")
>  		    FSUUID=$(date -u +%Y-%m-%d-%H-%M-%S-00);
> -		    xorriso -compliance rec_mtime -set_filter_r
> --zisofs -- -zisofs default -as mkisofs -iso-level 3 -graft-points -R
> -J -joliet-long -V "$FSLABEL" --modification-date=$(echo ${FSUUID} |
> sed 's/-//g;') -o "${FSIMAGEP}0.img"  -- -set_filter_r --zisofs --
> -zisofs default -add /="$MASTER" ;;
> +		    xorriso -compliance rec_mtime -as mkisofs
> -iso-level 3 -graft-points -R -J -joliet-long -V "$FSLABEL"
> --modification-date=$(echo ${FSUUID} | sed 's/-//g;') -o
> "${FSIMAGEP}0.img" -- -add /="$MASTER" -- -zisofs default
> -set_filter_r --zisofs / -- ;; x"iso9660") FSUUID=$(date -u
> +%Y-%m-%d-%H-%M-%S-00); xorriso --rockridge off -compliance rec_mtime
> -as mkisofs -iso-level 3 -graft-points -V "$FSLABEL"
> --modification-date=$(echo ${FSUUID} | sed 's/-//g;') -o
> "${FSIMAGEP}0.img" /="$MASTER"  ;;
> 
> ============================================================================
> 
> Here is a run with this change:
> ============================================================================
> 
> $ ./grub-fs-tester ziso9660
> GNU xorriso 1.5.4.pl02 : RockRidge filesystem manipulator, libburnia
> project.
> 
> Drive current: -outdev
> 'stdio:/tmp/tmp.iIXC5zIfEd/ziso9660_512_512_1_0.img' Media current:
> stdio file, overwriteable Media status : is blank
> Media summary: 0 sessions, 0 data blocks, 0 data,  400g free
> xorriso : WARNING : -volid text problematic as automatic mount point
> name xorriso : WARNING : -volid text is too long for Joliet (31 > 16)
> xorriso : WARNING : -volid text does not comply to ISO 9660 / ECMA
> 119 rules Added to ISO image: directory
> '/'='/tmp/tmp.iIXC5zIfEd/master' xorriso : UPDATE :     227 files
> added in 1 seconds xorriso : UPDATE :      10 file filters processed
> in 1 seconds libisofs: WARNING : Cannot add /longsym to Joliet tree.
> Symlinks can only be added to a Rock Ridge tree. libisofs: WARNING :
> Cannot add /psym to Joliet tree. Symlinks can only be added to a Rock
> Ridge tree. libisofs: WARNING : Cannot add /sdir/ssym to Joliet tree.
> Symlinks can only be added to a Rock Ridge tree. libisofs: WARNING :
> Cannot add /sdir/usym to Joliet tree. Symlinks can only be added to a
> Rock Ridge tree. libisofs: WARNING : Cannot add /sym to Joliet tree.
> Symlinks can only be added to a Rock Ridge tree. libisofs: WARNING :
> Filename not suitable for Joliet character set UCS-2 :
> "éàèüöäëñкирилица莭莽茝Ελληνικά😁😜😒" libisofs: WARNING : Filename
> not suitable for Joliet character set UCS-2 :
> "Ελληνικάкирилица😁😜😒éàèüöäëñ莭莽茝" libisofs: WARNING : Cannot add
> /Ελληνικάкирилица😁😜😒éàèüöäëñ莭莽茝 to Joliet tree. Symlinks can
> only be added to a Rock Ridge tree. xorriso : UPDATE : Writing:
> 1658s   32.2%   fifo   0%  buf  50% libisofs: WARNING : Sum of
> filenames not suitable for Joliet character set UCS-2: 2 ISO image
> produced: 4978 sectors Written to medium : 5152 sectors at LBA 32
> Writing to 'stdio:/tmp/tmp.iIXC5zIfEd/ziso9660_512_512_1_0.img'
> completed successfully.

It seems to me that we should only be testing ziso9660 here, not joliet
or Rock Ridge extensions (although perhaps a complete testing of file
compression should include these tests??). So to get rid of these
warnings the I'm thinking we should not be building the iso with Joliet
or Rock Ridge extensions.

> Device proc: Filesystem type procfs - Sector size 512B - Total size
> 0KiB Device loop0: Filesystem type iso9660 - Label `gr_;/é莭莽😁кирит
> u' - Last modifition time 2021-08-26 19:17:46 Thursday, UUID
> 2021-08-26-19-17-46-00 - Sector size 512B - Total size 10368KiB
> Device host: Filesystem type hostfs - Sector size 512B - Total size
> 0KiB
> 
> LIST FAIL
> DIR          20210826191746 ./
> DIR          20210826191746 ../
> DIR          20210826191745 0/
> 1124311      20210826191746 1.img
> 257343       20210826191746 american-english
> 1124219      20210826191746 CaSe
> 1124683      20210826191746 cAsE
> 1124311      20210826191746 hard
> 1124653      20210826191746 longsym
> 1124353      20210826191746 psym
> 1124653      20210826191746
> qwertzuiopasdfghjklyxcvbnm1234567890qwertzuiopasdfghjklyxcvbnm1234567890oiewqfiewioqoiqoiurqruewqoiuwoieoiiuewqroreqiufieiuwrnureweriuvceoiroiewqoiricdsalkcndsakfirefoiwqeoircorejwoijfreoijojoiewjfwnfcoirenfoirefnreoifenoiwfnoi
> DIR          20210826191746 sdir/ 1124311      20210826191746 sym
> 1124813      20210826191746 .?*\!"#%@$%&'()+ ,-.:;<=>^{_}[]`|~.
> 1124638      20210826191746 éàèüöäëñкирилица莭莽茝Ελληνικά😁😜😒
> 1124638      20210826191746 Ελληνικάкирилица😁😜😒éàèüöäëñ莭莽茝
> total 31692
> drwxr-xr-x 3 thomas thomas    4096 Aug 26 19:17 0
> -rw-r--r-- 2 thomas thomas 5242879 Aug 26 19:17 1.img
> -rw-r--r-- 1 thomas thomas 5242879 Aug 26 19:17 CaSe
> -rw-r--r-- 1 thomas thomas  972398 Aug 26 19:17 american-english
> -rw-r--r-- 1 thomas thomas 5242879 Aug 26 19:17 cAsE
> -rw-r--r-- 2 thomas thomas 5242879 Aug 26 19:17 hard
> lrwxrwxrwx 1 thomas thomas     227 Aug 26 19:17 longsym ->
> qwertzuiopasdfghjklyxcvbnm1234567890qwertzuiopasdfghjklyxcvbnm1234567890oiewqfiewioqoiqoiurqruewqoiuwoieoiiuewqroreqiufieiuwrnureweriuvceoiroiewqoiricdsalkcndsakfirefoiwqeoircorejwoijfreoijojoiewjfwnfcoirenfoirefnreoifenoiwfnoi
> lrwxrwxrwx 1 thomas thomas     806 Aug 26 19:17 psym ->
> /0//1/2/3//4/5/6//7/8/9//10/11/12//13/14/15//16/17/18//19/20/21//22/23/24//25/26/27//28/29/30//31/32/33//34/35/36//37/38/39//40/41/42//43/44/45//46/47/48//49/50/51//52/53/54//55/56/57//58/59/60//61/62/63//64/65/66//67/68/69//70/71/72//73/74/75//76/77/78//79/80/81//82/83/84//85/86/87//88/89/90//91/92/93//94/95/96//97/98/99//100/101/102//103/104/105//106/107/108//109/110/111//112/113/114//115/116/117//118/119/120//121/122/123//124/125/126//127/128/129//130/131/132//133/134/135//136/137/138//139/140/141//142/143/144//145/146/147//148/149/150//151/152/153//154/155/156//157/158/159//160/161/162//163/164/165//166/167/168//169/170/171//172/173/174//175/176/177//178/179/180//181/182/183//184/185/186//187/188/189//190/191/192//193/194/195//196/197/198//199/200/201//202/203/204//205/206/207//208/209/p.img
> -rw-r--r-- 1 thomas thomas 5242879 Aug 26 19:17
> qwertzuiopasdfghjklyxcvbnm1234567890qwertzuiopasdfghjklyxcvbnm1234567890oiewqfiewioqoiqoiurqruewqoiuwoieoiiuewqroreqiufieiuwrnureweriuvceoiroiewqoiricdsalkcndsakfirefoiwqeoircorejwoijfreoijojoiewjfwnfcoirenfoirefnreoifenoiwfnoi
> drwxr-xr-x 2 thomas thomas    4096 Aug 26 19:17 sdir lrwxrwxrwx 1
> thomas thomas       5 Aug 26 19:17 sym -> 1.img -rw-r--r-- 1 thomas
> thomas 5242879 Aug 26 19:17 éàèüöäëñкирилица莭莽茝Ελληνικά😁😜
> lrwxrwxrwx 1 thomas thomas      72 Aug 26 19:17
> Ελληνικάкирилица😁😜😒éàèüöäëñ莭莽茝 ->
> éàèüöäëñкирилица莭莽茝Ελληνικά😁😜😒 $ echo $? 1
> 
> ============================================================================

Glenn

Re: [grub-fs-tester.in] zisofs test looks unsuitable

[Thomas Schmitt]

Hi.

Glenn Washburn wrote:
> I think the changes to get the test
> working are worthy of inclusion so that the tests are ready when this
> feature gets implemented.

Do you think my draft of a commit message is ok ?

Probably i should mention that this test will fail until zisofs is
implemented.


> I'm not familiar with that code nor the
> format details, but it sounds like you might be able to determine if
> its something somewhat easy to add to GRUB.

Well, i wrote an implementation as libisofs filter (which differs much
from a filesystem driver) and a documentation of the format:
  https://sources.debian.org/src/libisofs/1.5.2-1/doc/zisofs_format.txt
(Since noon our project web server is down. I need to pester the web
master to check which useless piece of software crashed it this time.)

Implementing SUSP entry "ZF" would be a substantial change to
grub-core/fs/iso9660.c including a new dependency on zlib.
The adventure would probably begin at function susp_iterate_dir()
which interprets some Rock Ridge SUSP entries.
Then i'd need to know which functions tell the file size and do the
reading of file content data and how to make the info from the ZF entry
known to them.

The Linux kernel has an implementation from the inventor, H.Peter Anvin.
(With a bug for machines with PAGE_SIZE > 32 KB
  https://lore.kernel.org/linux-scsi/20201120140633.1673-1-scdbackup@gmx.net/T/#u
and a dire need for a readahead implementation.)

And there is the successor format zisofs2, not yet in the kernel
  https://fossies.org/linux/misc/xorriso-1.5.4.pl02.tar.gz/xorriso-1.5.4/doc/zisofs2_format.txt?m=t
(Thanks to all the sites which store what i write.)
Last november a zisofs2 patch of ~200 lines enabled the then actual 5.10.0
release candidate to read the new format. But nearly a year of bitrot
already devalued my readahead patch. So my local test kernel is the only
one in the world which can read giant zisofs2 compressed files at high
speed.


Have a nice day :)

Thomas

Re: [grub-fs-tester.in] zisofs test looks unsuitable

[Glenn Washburn]

On Fri, 27 Aug 2021 00:14:20 +0200
"Thomas Schmitt" <scdbackup@gmx.net> wrote:

> Hi.
> 
> Glenn Washburn wrote:
> > I think the changes to get the test
> > working are worthy of inclusion so that the tests are ready when
> > this feature gets implemented.
> 
> Do you think my draft of a commit message is ok ?

Seems good to me.

> Probably i should mention that this test will fail until zisofs is
> implemented.

Yes, but as I noted before, the ziso9660 is not actually run by any
"make check" tests, which makes sense if its not actually implemented.
Also the ziso9660 fs test fails currently anyway, so I don't see a
problem here.

> > I'm not familiar with that code nor the
> > format details, but it sounds like you might be able to determine if
> > its something somewhat easy to add to GRUB.
> 
> Well, i wrote an implementation as libisofs filter (which differs much
> from a filesystem driver) and a documentation of the format:
>   https://sources.debian.org/src/libisofs/1.5.2-1/doc/zisofs_format.txt
> (Since noon our project web server is down. I need to pester the web
> master to check which useless piece of software crashed it this time.)
> 
> Implementing SUSP entry "ZF" would be a substantial change to
> grub-core/fs/iso9660.c including a new dependency on zlib.

There wouldn't be a new dependency on zlib. Grub already supports zlib
decompression using grub_zlib_decompress in grub-core/io/gzio.c. I
suspect this would work fine.

> The adventure would probably begin at function susp_iterate_dir()
> which interprets some Rock Ridge SUSP entries.
> Then i'd need to know which functions tell the file size and do the
> reading of file content data and how to make the info from the ZF
> entry known to them.

Functions for dealing with files can be found in include/grub/file.h
and grub-core/kern/file.c

> The Linux kernel has an implementation from the inventor, H.Peter
> Anvin. (With a bug for machines with PAGE_SIZE > 32 KB
>   https://lore.kernel.org/linux-scsi/20201120140633.1673-1-scdbackup@gmx.net/T/#u
> and a dire need for a readahead implementation.)
> 
> And there is the successor format zisofs2, not yet in the kernel
>   https://fossies.org/linux/misc/xorriso-1.5.4.pl02.tar.gz/xorriso-1.5.4/doc/zisofs2_format.txt?m=t
> (Thanks to all the sites which store what i write.)
> Last november a zisofs2 patch of ~200 lines enabled the then actual
> 5.10.0 release candidate to read the new format. But nearly a year of
> bitrot already devalued my readahead patch. So my local test kernel
> is the only one in the world which can read giant zisofs2 compressed
> files at high speed.

I'm sure support for the new format would be welcome as well and it
sounds like you're well-positioned to make that happen.

Glenn