Re: Unification of filesystem encoding options

["Theodore Y. Ts'o" <tytso@mit.edu>]

On Tue, Jan 07, 2020 at 06:38:42PM +0100, Pali Rohár wrote:
> Adding support for case-insensitivity into UTF-8 NLS encoding would mean
> to create completely new kernel NLS API (which would support variable
> length encodings) and rewrite all NLS filesystems to use this new API.
> Also all existing NLS encodings would be needed to port into this new
> API.
> 
> It is really something which have a value? Just because of UTF-8?
> 
> For me it looks like better option would be to remove UTF-8 NLS encoding
> as it is broken. Some filesystems already do not use NLS API for their
> UTF-8 support (e.g. vfat, udf or newly prepared exfat). And others could
> be modified/extended/fixed in similar way.

You didn't mention ext4 and f2fs, which is using the Unicode code in
fs/unicode for its case-folding and normalization support.  Ext4 and
f2fs only supports utf-8, so using the NLS API would have added no
value --- and it as you pointed out, the NLS API doesn't support
variable length encoding anyway.  In contrast the fs/unicode functions
have support for full Unicode case folding and normalization, and
currently has the latest Unicode 12.1 tables (released May 2019).

What I'd suggest is to create a new API, enhancing the functions in
fs/unicode, to support those file systems that need to deal with
UTF-16 and UTF-32 for their on-disk directory format, and that we
assume that for the most part, userspace *will* be using a UTF-8
encoding for the user<->kernel interface.  We can keep the existing
NLS interface and mount options for legacy support, but in my opinion
it's not worth the effort to try to do anything else.

					- Ted