Re: casefold is using unsuitable case mapping table

[Gabriel Krisman Bertazi <krisman@suse.de>]

Björn JACKE <bjacke@SerNet.DE> writes:

> It turns out though that the case insensitive feature is not usable because it
> does not match the case mapping tables that other operating systems use. More
> specifically, the german letter "ß" is treated as a case equivanten of "ss".
>
> There is an equivalent of "ß" and "ss in some other scopes, also AD LDAP treats
> them as an equivante. For systems that requires "lossless" case conversion
> however should not treat ß and ss as equivalent. This is also why a filesystem
> should never ever do that

Well, filesystems should never ever have filename encoding.  Once
they do, we have to make semantics decisions and they are all apparently
stupid to someone.  And any kind of Casefolding is an inherently lossy
operation in this sense, as multiple byte sequences will map to the
same file.

The big problem is that each of the big OS vendors chose specific
semantics of what to casefold.  APFS does NFD + full casefolding[1],
right?  except for "some code-points". I'm not sure what they do with ß,
tbh. I could never find any documentation on the specific code-points
they add/ignore.

In ext4, we decided to have no exceptions. Just do plain NFD + CF.  That
means we do C+F from the table below:

  https://www.unicode.org/Public/12.1.0/ucd/CaseFolding.txt

Which includes ß->SS.  We could argue forever whether that doesn't make
sense for language X, such as German.  I'm not a German speaker but
friends said it would be common to see straße uppercased to STRASSE there,
even though the 2017 agreement abolished it in favor of ẞ.  So what is
the right way?

My point is we can't rely fully on languages to argue the right
semantics.  There are no right semantics.  And Languages are also alive
and changing. There are many other examples where full casefold will
look stupid; for instance, one would argue we should also translate the
T column (i.e non-Turkish languages).  We can produce all sorts of
stupid examples with combining characters in Portuguese/Spanish too.
Linux is not broken beyond the fact the whole idea is broken.  These are
just the semantics we agreed were slightly less insane back in 2017
(considering we don't want to have locales).  And, apart from the
ignorable code points issue, I still think our implementation is
relatively sane.

> Since 2017 there is a well-defined uppercase version of the codepoint (U+00DF)
> of the "ß" letter in Unicode: U+1E9E, this could eventually be used but I
> haven't seen any filesystem using that so far. This would be a possible and
> lossless case equivalent, but well, that's actually another thing to
> discuss.
>
> The important point is to _not_ use the ß/ss case equicalent. The casefold
> feature is mainly useless otherwise.

It is not useless.  Android and Wine emulators have been using it just
fine for years.  We also cannot break compatibility for them.

> Can this be changed without causing too much hassle?

We attempted to do a much smaller change recently in commit
5c26d2f1d3f5, because we assumed no one would be trying to create files
with silly stuff like ZWSP (U+200B). Turns out there is a reasonable
use-case for that with Variation Selectors, and we had to revert it.  So
we need to be very careful with any changes here, so people don't lose
access to their files on a kernel update.  Even with that, more
casefolding flavor will cause all sorts of compatibility issues when
moving data across volumes, so I'd be very wary of having more than one
flavor.

What are the exact requirements for samba?  Do you only fold the C
column? Do you need stuff like compatibility normalization?

 [1] https://developer.apple.com/support/downloads/Apple-File-System-Reference.pdf

-- 
Gabriel Krisman Bertazi