[PATCH] xfstests: exclude hfs/hfsplus from generic/339 run

[PATCH] xfstests: exclude hfs/hfsplus from generic/339 run

[Viacheslav Dubeyko]

The dirhash_collide generates filenames from arbitrary random
bytes (only '.' and '/' are filtered out), which are not valid
UTF-8 in general. Catalog names on both classic HFS and HFS+ are
Unicode. Feeding random bytes causes fsck to report "Illegal name"
or catalog corruption that has nothing to do with directory
hash-collision handling, which is what this test is actually meant
to exercise.

This patch excludes HFS/HFS+ file systems from generic/339 run.

Signed-off-by: Viacheslav Dubeyko <slava@dubeyko.com>
---
 tests/generic/339 | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/tests/generic/339 b/tests/generic/339
index bf8f7729..5cc1086f 100755
--- a/tests/generic/339
+++ b/tests/generic/339
@@ -17,6 +17,15 @@ _begin_fstest auto dir
 _require_scratch
 _require_test_program "dirhash_collide"
 
+# dirhash_collide generates filenames from arbitrary random bytes (only
+# '.' and '/' are filtered out), which are not valid UTF-8 in general.
+# Catalog names on both classic HFS and HFS+ are Unicode. Feeding random
+# bytes causes fsck to report "Illegal name" / catalog corruption that has
+# nothing to do with directory hash-collision handling, which is what this
+# test is actually meant to exercise.
+_exclude_fs hfs
+_exclude_fs hfsplus
+
 _scratch_mkfs >>$seqres.full 2>&1
 _scratch_mount
 
-- 
2.43.0

Re: [PATCH] xfstests: exclude hfs/hfsplus from generic/339 run

[Christoph Hellwig]

On Wed, Sep 09, 2026 at 05:04:50PM -0700, Viacheslav Dubeyko wrote:
> The dirhash_collide generates filenames from arbitrary random
> bytes (only '.' and '/' are filtered out), which are not valid
> UTF-8 in general. Catalog names on both classic HFS and HFS+ are
> Unicode. Feeding random bytes causes fsck to report "Illegal name"
> or catalog corruption that has nothing to do with directory
> hash-collision handling, which is what this test is actually meant
> to exercise.

If user file name creation manage to trigger fsck complaints, you need
to righten the input checks in the file systems first to fix this.

We can then find a programmatic way to deal with these rejects in
xfstests, as ext4/f2fs in case insensiive mode probably have the same
issue.

Re: [PATCH] xfstests: exclude hfs/hfsplus from generic/339 run

[Theodore Tso]

On Thu, Sep 10, 2026 at 02:57:12AM -0500, Christoph Hellwig wrote:
> On Wed, Sep 09, 2026 at 05:04:50PM -0700, Viacheslav Dubeyko wrote:
> > The dirhash_collide generates filenames from arbitrary random
> > bytes (only '.' and '/' are filtered out), which are not valid
> > UTF-8 in general. Catalog names on both classic HFS and HFS+ are
> > Unicode. Feeding random bytes causes fsck to report "Illegal name"
> > or catalog corruption that has nothing to do with directory
> > hash-collision handling, which is what this test is actually meant
> > to exercise.

I'm curious --- does HFS/HFS+ reject unofficial Unicode characters
such as those used by Klingon?  (Klingon is an invented/artificial
language, so it's considered out of scope by Unicode; so Star Trek
enthusiasts are camping on an unofficial block of the Unicode code
set.)

Invalid Unicode characters could also happen because newer versions
of Unicode have added new emoji's, which might be added as support in
the mobile handset userspace, but since we aren't regularly updating
the Unicode tables, they might still be considered invalid if you
enable strict mode in ext4/f2fs.

> If user file name creation manage to trigger fsck complaints, you need
> to righten the input checks in the file systems first to fix this.
> 
> We can then find a programmatic way to deal with these rejects in
> xfstests, as ext4/f2fs in case insensiive mode probably have the same
> issue.

Well, strict mode isn't enabled by default, since users might want to
use Klingon or emoji's with a skin color that matches their own.  And
no one has been invested enough to continuously update our Unicode
tables, and add support in the userspace tooling for upgrading the
Unicode version in the file system superblock (which would then break
compatibility with older kernels --- for example, if they downgrade to
an older version of Android, etc.)

Isn't Unicode **fun**?  :-)

    	       	       	      	   	      - Ted

Re: [PATCH] xfstests: exclude hfs/hfsplus from generic/339 run

[Viacheslav Dubeyko]

On Thu, 2026-09-10 at 02:57 -0700, Christoph Hellwig wrote:
> On Wed, Sep 09, 2026 at 05:04:50PM -0700, Viacheslav Dubeyko wrote:
> > The dirhash_collide generates filenames from arbitrary random
> > bytes (only '.' and '/' are filtered out), which are not valid
> > UTF-8 in general. Catalog names on both classic HFS and HFS+ are
> > Unicode. Feeding random bytes causes fsck to report "Illegal name"
> > or catalog corruption that has nothing to do with directory
> > hash-collision handling, which is what this test is actually meant
> > to exercise.
> 
> If user file name creation manage to trigger fsck complaints, you
> need
> to righten the input checks in the file systems first to fix this.
> 
> We can then find a programmatic way to deal with these rejects in
> xfstests, as ext4/f2fs in case insensiive mode probably have the same
> issue.

I see your point. And I really like your suggestion. But what is your
vision of the proper way to check the validity of Unicode symbols?

Thanks,
Slava.

Re: [PATCH] xfstests: exclude hfs/hfsplus from generic/339 run

[Viacheslav Dubeyko]

On Thu, 2026-09-10 at 09:49 -0400, Theodore Tso wrote:
> On Thu, Sep 10, 2026 at 02:57:12AM -0500, Christoph Hellwig wrote:
> > On Wed, Sep 09, 2026 at 05:04:50PM -0700, Viacheslav Dubeyko wrote:
> > > The dirhash_collide generates filenames from arbitrary random
> > > bytes (only '.' and '/' are filtered out), which are not valid
> > > UTF-8 in general. Catalog names on both classic HFS and HFS+ are
> > > Unicode. Feeding random bytes causes fsck to report "Illegal
> > > name"
> > > or catalog corruption that has nothing to do with directory
> > > hash-collision handling, which is what this test is actually
> > > meant
> > > to exercise.
> 
> I'm curious --- does HFS/HFS+ reject unofficial Unicode characters
> such as those used by Klingon?  (Klingon is an invented/artificial
> language, so it's considered out of scope by Unicode; so Star Trek
> enthusiasts are camping on an unofficial block of the Unicode code
> set.)

I need to double check which particular Unicode characters fsck.hfsplus
treats as invalid one.

> 
> Invalid Unicode characters could also happen because newer versions
> of Unicode have added new emoji's, which might be added as support in
> the mobile handset userspace, but since we aren't regularly updating
> the Unicode tables, they might still be considered invalid if you
> enable strict mode in ext4/f2fs.
> 
> > If user file name creation manage to trigger fsck complaints, you
> > need
> > to righten the input checks in the file systems first to fix this.
> > 
> > We can then find a programmatic way to deal with these rejects in
> > xfstests, as ext4/f2fs in case insensiive mode probably have the
> > same
> > issue.
> 
> Well, strict mode isn't enabled by default, since users might want to
> use Klingon or emoji's with a skin color that matches their own.  And
> no one has been invested enough to continuously update our Unicode
> tables, and add support in the userspace tooling for upgrading the
> Unicode version in the file system superblock (which would then break
> compatibility with older kernels --- for example, if they downgrade
> to
> an older version of Android, etc.)
> 
> Isn't Unicode **fun**?  :-)

Could we have any generic way to detect that file system could not
support some Unicode characters?

Thanks,
Slava.

Re: [PATCH] xfstests: exclude hfs/hfsplus from generic/339 run

[Christoph Hellwig]

On Thu, Sep 10, 2026 at 01:13:12PM -0700, Viacheslav Dubeyko wrote:
> On Thu, 2026-09-10 at 02:57 -0700, Christoph Hellwig wrote:
> > On Wed, Sep 09, 2026 at 05:04:50PM -0700, Viacheslav Dubeyko wrote:
> > > The dirhash_collide generates filenames from arbitrary random
> > > bytes (only '.' and '/' are filtered out), which are not valid
> > > UTF-8 in general. Catalog names on both classic HFS and HFS+ are
> > > Unicode. Feeding random bytes causes fsck to report "Illegal name"
> > > or catalog corruption that has nothing to do with directory
> > > hash-collision handling, which is what this test is actually meant
> > > to exercise.
> > 
> > If user file name creation manage to trigger fsck complaints, you
> > need
> > to righten the input checks in the file systems first to fix this.
> > 
> > We can then find a programmatic way to deal with these rejects in
> > xfstests, as ext4/f2fs in case insensiive mode probably have the same
> > issue.
> 
> I see your point. And I really like your suggestion. But what is your
> vision of the proper way to check the validity of Unicode symbols?

I don't think validity of a unicode symbol matter directly here.  You
need to check what the Apple implementation accepts and/or what
Apple documents and follow that.

Re: [PATCH] xfstests: exclude hfs/hfsplus from generic/339 run

[Theodore Tso]

On Thu, Sep 10, 2026 at 02:14:08PM -0500, Viacheslav Dubeyko wrote:
> 
> Could we have any generic way to detect that file system could not
> support some Unicode characters?

We do have unicode tables which are optionally compiled in if the file
system requires it.  But those tables are from Unicode 12.1, and the
latest version of Unicode is 17.0.  Most of the new characters added
between Unicode 12.1 and Unicode 17.0 are for dead languages that
archeologists and academics care about, and new Emoji's.  So function
will let you know if a Unicode character was considered valid as of
2019.

You're asking a different question, though which is for hfs/hfsplus,
what character are deemed valid by fsck.hfs, right?  And as far as I
know, there is no way to answer that question in the hfs kernel code.
What you could do is to try creating a filename containing a
particular character, and see if the kernel rejects it.  Ext4 and f2fs
will reject such characters if and only if Unicode strict mode is
enbled.  But that's not going to help for hfs unless you add that
support into the kernel....

					- Ted

P.S.  A problem which we haven't really solved is how to add support
for multiple Unicode versions at the same time.  So if someone
imported Unicode 17.0 into the kernel, ext4 and f2fs really require
Unicode 12.1.  Short of supporting both Unicode tables in the sources,
someone would probably have to figure some mechanism (perhaps using
Agentic Engineering) to collapse the two Unicode tables into one, so
you can answer the question, is a particular Unicode valid in 12.1 vs
17.0, and handling the Unicode canonicalization rules for Unicode 12.1
and 17.0 based on some API.  This is the other reason why no one has
bothered updating Unicode, since it would be tricky.

P.P.S.  There is reason why, over a decade ago, when I was talking to
an internationalization expert at an Internet Engineering Task Force,
late at night, at a bar, the expert said jokingly (and when said
expert was very well lubricated), after reviewing all of the
engineering challeges of getting I18N correct, "It would be easier to
teach them all English."  :-)

Mercifully, we only need to solve a very tiny subset of the I18N
problem in the kernel.  :-)   :-)   :-)