Re: Questions about Unicode Normalization Form

["Theodore Ts'o" <tytso@mit.edu>]

On Sat, Apr 06, 2024 at 11:15:36PM +0800, HAN Yuwei wrote:
> 
> Sorry, I am not very familiar with Unicode nor kernel. Correct me if wrong.
> 
> As to what I have read, kernel seems like using NFD when processing all
> UTF-8 related string.
> If fs is using these helper function, then I can be sure kernel is applying
> NFD to every UTF-8 filenames.
> But I can't find any references to these helper function on Github mirror,
> how are they used by fs code?

For the most part, the kernel's file stysem code doesn't do anything
special for Unicode.  The exception is that the ext4 and f2fs file
systems can have an optional feature which is mostly only used by
Android systems to support case insensitive lookups.  This is called
the "casefold" feature, which is not enabled by default by most
desktop or server systems.

The casefold feature was developed because Android has a requirement
to support case-insensitive lookups, and it had to support Unicode
character sets (for example, XFS has support for case insensitive
lookups back from the Irix days, but it only supports ASCII), and the
alternative to adding support in the kernel for case fodling was this
terrible out-of-tree kernel module which use a file system wrapping
that was deadlock-prone (which is why the case-folding wrapfs would
never be accepted upstream; it was a trash fire).  Anyway, I got tired
of being asked to debug file system deadlocks which was not the VFS's
fault, but was rather caused by this terrible wrapfs kludge used by
Android, so I instigated proper case-folding support (ala Windows and
MacOS) for the file system types commonly used by Android, namely ext4
and f2fs.

So *if* you are using ext4 or f2fs, *and* the file system is specially
created with the file system feature flag "casefold", *and* the
directory has the casefold flag set, *then* the file system will
support case-preserving, case-insensitive lookups.  As a side effect
of using utf8_strcasecmp, it will also do string comparisons where
even if you have not normalized the file banes, so that the filename
contained some Unicode character, such as (for example) the NFC form
of the Anstrom Sign character (00C5), and you try to look it up using
the NFD form of the character (0041 030A), the lookup will succeed,
because we use utf8_strcasecmp().   However, this is *only* if case
folding is enabled, and in general, it isn't.

Aside from this exception (which as I said, is in general only enabled
for Android, because most other use cases such as for Desktop, Server,
etc. don't really care about MacOS / Windows style case insensitive
filename lookups), the Linux VFS in general treats UTF-8 characters as
null-terminated byte streams.  So the kernel doesn't validate to make
sure that a file name is composed of valid UTF-8 code points (e.g., so
we don't prohibit the use of Klingon characters which are not
recognized by the Unicode consortium), nor does the kernel do any kind
of Unicode normalization.  So for example, if casefolding is not
enabled, 0041 030A and 00C5 will be considered different, and kernel
will not force the NFC form (00C5) to the NFD form (0041 030A) or vice
versa.

Now, because the kernel tries very hard to be blissfully ignorant
about the nightmare which is I18N, it is up to the userspace Unicode
libraries to normalize strings before passing them to the kernel ---
either as data in text files, or as file names.  I am very glad that I
don't worry about whether the standard normalization form used by the
various GNOME, KDE, Unicode, etc., userspace libraries is NFD, NFC,
NFKD, or NFKC.  That's someone else's problem, and if you don't have
casefolding enabled, we will do the filename comparisons using the
strcmp() function.

Fundamentally, unicode and normalization is a userspace problem, not a
kernel problem, except when we don't have a choice (such as for casse
insensitive lookups).  And there we solve just the smallest part of
the problem, and make it userspace's problem for everything else.

Cheers,

					- Ted