man/man7/pathname.7: Correct handling of pathnames

man/man7/pathname.7: Correct handling of pathnames

[Alejandro Colomar]

[-- Attachment #1: Type: text/plain, Size: 2793 bytes --]

Hi Jason,

I think the recommendation to use the current locale for handling
pathnames isn't good.

If I use the C locale (and I do have systems with the C locale), then
programs running on that system would corrupt files that go through that
system.  Let's say you send me María.song, and I download it on a system
using the C locale.  Programs would fail to copy the file.

Instead, I think a good recommendation would be to behave in one of the
following ways:

-  Accept only the POSIX Portable Filename Character Set.
-  Assume UTF-8, but reject control characters.
-  Assume UTF-8.
-  Accept anything, but reject control characters.
-  Accept anything, just like the kernel.

The current locale should actively be ignored when handling pathnames.

I've modified the example in the manual page to use a filename that's
non-ASCII, to make it more interesting.  See how it fails:

	alx@devuan:~/tmp/gcc$ cat path.c 
	     #include <err.h>
	     #include <iconv.h>
	     #include <langinfo.h>
	     #include <locale.h>
	     #include <stdio.h>
	     #include <stdlib.h>
	     #include <uchar.h>

	     #define NELEMS(a)  (sizeof(a) / sizeof(a[0]))

	     int
	     main(void)
	     {
		 char      *locale_pathname;
		 char      *in, *out;
		 FILE      *fp;
		 size_t    size;
		 size_t    inbytes, outbytes;
		 iconv_t   cd;
		 char32_t  utf32_pathname[] = U"María";

		 if (setlocale(LC_ALL, "") == NULL)
		     err(EXIT_FAILURE, "setlocale");

		 size = NELEMS(utf32_pathname) * MB_CUR_MAX;
		 locale_pathname = malloc(size);
		 if (locale_pathname == NULL)
		     err(EXIT_FAILURE, "malloc");

		 cd = iconv_open(nl_langinfo(CODESET), "UTF-32");
		 if (cd == (iconv_t)-1)
		     err(EXIT_FAILURE, "iconv_open");

		 in = (char *) utf32_pathname;
		 inbytes = sizeof(utf32_pathname);
		 out = locale_pathname;
		 outbytes = size;
		 if (iconv(cd, &in, &inbytes, &out, &outbytes) == (size_t) -1)
		     err(EXIT_FAILURE, "iconv");

		 if (iconv_close(cd) == -1)
		     err(EXIT_FAILURE, "iconv_close");

		 fp = fopen(locale_pathname, "w");
		 if (fp == NULL)
		     err(EXIT_FAILURE, "fopen");

		 fputs("Hello, world!\n", fp);
		 if (fclose(fp) == EOF)
		     err(EXIT_FAILURE, "fclose");

		 free(locale_pathname);
		 exit(EXIT_SUCCESS);
	     }

	alx@devuan:~/tmp/gcc$ cc -Wall -Wextra path.c 
	alx@devuan:~/tmp/gcc$ ls
	a.out  path.c
	alx@devuan:~/tmp/gcc$ ./a.out ; echo $?
	0
	alx@devuan:~/tmp/gcc$ ls
	María  a.out  path.c
	alx@devuan:~/tmp/gcc$ cat María 
	Hello, world!
	alx@devuan:~/tmp/gcc$ LC_ALL=C ./a.out ; echo $?
	a.out: iconv: Invalid or incomplete multibyte or wide character
	1

What do you think?


Have a lovely day!
Alex

-- 
<https://www.alejandro-colomar.es/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: man/man7/pathname.7: Correct handling of pathnames

[Jason Yundt]

On Mon, Jan 27, 2025 at 12:22:49PM +0100, Alejandro Colomar wrote:
> Hi Jason,
> 
> I think the recommendation to use the current locale for handling
> pathnames isn't good.
> 
> If I use the C locale (and I do have systems with the C locale), then
> programs running on that system would corrupt files that go through that
> system.

I agree.  I think that this is just a limitation of the design of
UNIX-like systems.  As long as users are allowed to choose different
locale codesets, mojibake will always be possible.  Sometimes, you just
have to temporarily switch to a different locale in order to make things
work.  For example, I was trying to run some old Japanese software a
while ago, and I had to add a Shift-JIS locale to my system in order to
get it to work.

> Let's say you send me María.song, and I download it on a system
> using the C locale.  Programs would fail to copy the file.

Not necessarily.  pathname(7) says “Paths should be encoded and decoded
using the current locale’s codeset in order to help prevent mojibake.”
In many cases, you don’t need to encode or decode a pathname.  Here’s a
program that copies a file without encoding or decoding any pathnames:

    #include <stdio.h>


    int main(int argc, char *argv[]) {
        FILE *src = NULL, *dest = NULL;
        int exit_status = 0;
        if (argc != 3) {
            fprintf(stderr, "USAGE: %s <src> <dest>\n", argv[0]);
            exit_status = 1;
            goto clean_up;
        };

        // Everything in argv[] should already use the correct character encoding,
        // so we don’t need to do any encoding or decoding here.
        // See <https://sourceware.org/pipermail/libc-help/2024-August/006737.html>.
        src = fopen(argv[1], "r");
        if (src == NULL) {
            fprintf(stderr, "ERROR: Failed to open %s\n", argv[1]);
            exit_status = 1;
            goto clean_up;
        }
        dest = fopen(argv[2], "w");
        if (dest == NULL) {
            fprintf(stderr, "ERROR: Failed to open %s\n", argv[2]);
            exit_status = 1;
            goto clean_up;
        }
        int c;
        while((c = fgetc(src)) != EOF) {
            if (fputc(c, dest) == EOF) {
                fprintf(stderr, "ERROR: Error while writing to %s\n", argv[2]);
                exit_status = 1;
                goto clean_up;
            }
        }

    clean_up:
        if (src != NULL) {
            if (fclose(src) == EOF) {
                fprintf(stderr, "ERROR: Failed to close %s\n", argv[1]);
                exit_status = 1;
            }
        }
        if (dest != NULL) {
            if (fclose(dest) == EOF) {
                fprintf(stderr, "ERROR: Failed to close %s\n", argv[2]);
                exit_status = 1;
            }
        }
        return exit_status;
    }

> Instead, I think a good recommendation would be to behave in one of the
> following ways:
> 
> -  Accept only the POSIX Portable Filename Character Set.

This one isn’t quite a complete recommendation.  The POSIX Portable
Filename Character Set is just a character set.  It’s not a character
encoding.  If we go with this one, then we would need to say something
along the lines of “Encode and decode paths using ASCII and only accept
characters that are in the POSIX Protable Filename Character Set.”

> -  Assume UTF-8, but reject control characters.
> -  Assume UTF-8.

> -  Accept anything, but reject control characters.
> -  Accept anything, just like the kernel.

These last two also aren’t quite complete recommendations.  If a GUI
program wants to display a pathname on the screen, then what character
encoding should it use when decoding the bytes?

> The current locale should actively be ignored when handling pathnames.
> 
> I've modified the example in the manual page to use a filename that's
> non-ASCII, to make it more interesting.  See how it fails:
> 
> 	alx@devuan:~/tmp/gcc$ cat path.c 
> 	     #include <err.h>
> 	     #include <iconv.h>
> 	     #include <langinfo.h>
> 	     #include <locale.h>
> 	     #include <stdio.h>
> 	     #include <stdlib.h>
> 	     #include <uchar.h>
> 
> 	     #define NELEMS(a)  (sizeof(a) / sizeof(a[0]))
> 
> 	     int
> 	     main(void)
> 	     {
> 		 char      *locale_pathname;
> 		 char      *in, *out;
> 		 FILE      *fp;
> 		 size_t    size;
> 		 size_t    inbytes, outbytes;
> 		 iconv_t   cd;
> 		 char32_t  utf32_pathname[] = U"María";
> 
> 		 if (setlocale(LC_ALL, "") == NULL)
> 		     err(EXIT_FAILURE, "setlocale");
> 
> 		 size = NELEMS(utf32_pathname) * MB_CUR_MAX;
> 		 locale_pathname = malloc(size);
> 		 if (locale_pathname == NULL)
> 		     err(EXIT_FAILURE, "malloc");
> 
> 		 cd = iconv_open(nl_langinfo(CODESET), "UTF-32");
> 		 if (cd == (iconv_t)-1)
> 		     err(EXIT_FAILURE, "iconv_open");
> 
> 		 in = (char *) utf32_pathname;
> 		 inbytes = sizeof(utf32_pathname);
> 		 out = locale_pathname;
> 		 outbytes = size;
> 		 if (iconv(cd, &in, &inbytes, &out, &outbytes) == (size_t) -1)
> 		     err(EXIT_FAILURE, "iconv");
> 
> 		 if (iconv_close(cd) == -1)
> 		     err(EXIT_FAILURE, "iconv_close");
> 
> 		 fp = fopen(locale_pathname, "w");
> 		 if (fp == NULL)
> 		     err(EXIT_FAILURE, "fopen");
> 
> 		 fputs("Hello, world!\n", fp);
> 		 if (fclose(fp) == EOF)
> 		     err(EXIT_FAILURE, "fclose");
> 
> 		 free(locale_pathname);
> 		 exit(EXIT_SUCCESS);
> 	     }
> 
> 	alx@devuan:~/tmp/gcc$ cc -Wall -Wextra path.c 
> 	alx@devuan:~/tmp/gcc$ ls
> 	a.out  path.c
> 	alx@devuan:~/tmp/gcc$ ./a.out ; echo $?
> 	0
> 	alx@devuan:~/tmp/gcc$ ls
> 	María  a.out  path.c
> 	alx@devuan:~/tmp/gcc$ cat María 
> 	Hello, world!
> 	alx@devuan:~/tmp/gcc$ LC_ALL=C ./a.out ; echo $?
> 	a.out: iconv: Invalid or incomplete multibyte or wide character
> 	1
> 
> What do you think?

I honestly don’t know what the recommendation should be.  Here’s what I
would need to know in order to figure out what the recommendation should
be.  A while ago, I asked this question on the Unix & Linux Stack
Exchange [1]:

> What does a locale’s codeset get used for?
>
> According to glibc’s manual:
>
> > Most locale names follow XPG syntax and consist of up to four parts:
> >
> > ```
> > language[_territory[.codeset]][@modifier]
> > ```
>
> For example, you could have a locale named zh_CN.GB18030 which would
> use the GB 18030 character encoding, or you could have a locale named
> zh_CN.UTF-8 which would use the UTF-8 character encoding.
>
> Here’s where I’m confused: let’s say that I switch from a zh_CN.UTF-8
> locale to a zh_CN.GB18030 locale. In that situation, some things that
> used to be encoded in UTF-8 are now going to be encoded in GB 18030.
> Which things will now be encoded in GB 18030? Will stdin, stdout and
> stderr use GB 18030? What about argv? What about filesystem paths?
>
> Technically, a program can do whatever it wants and ignore the locale
> completely, but let’s assume that programs are doing the correct thing
> here. What is supposed to be encoded in GB 18030 if I use a
> zh_CN.GB18030 locale?

I didn’t get an answer to that question, so I asked it again on the
libc-help mailing list [2].  I got one response that was super helpful
[3].  That response clearly said that paths should be encoded using the
locale’s codeset.  If you think that answer was incorrect, then I would
like a very specific list of things that should and should not be
encoded using the locale’s codeset so that I can add it to the glibc
manual (and maybe the POSIX standard if I can figure out how to
contribute to that).

[1]: <https://unix.stackexchange.com/q/780404/316181>
[2]: <https://sourceware.org/pipermail/libc-help/2024-August/006736.html>
[3]: <https://sourceware.org/pipermail/libc-help/2024-August/006737.html>

> Have a lovely day!
> Alex
> 
> -- 
> <https://www.alejandro-colomar.es/>

Re: man/man7/pathname.7: Correct handling of pathnames

[Alejandro Colomar]

[-- Attachment #1: Type: text/plain, Size: 5334 bytes --]

Hi Jason, Florian,

On Mon, Jan 27, 2025 at 09:50:49AM -0500, Jason Yundt wrote:
> On Mon, Jan 27, 2025 at 12:22:49PM +0100, Alejandro Colomar wrote:
> > Hi Jason,
> > 
> > I think the recommendation to use the current locale for handling
> > pathnames isn't good.
> > 
> > If I use the C locale (and I do have systems with the C locale), then
> > programs running on that system would corrupt files that go through that
> > system.
> 
> I agree.  I think that this is just a limitation of the design of
> UNIX-like systems.  As long as users are allowed to choose different
> locale codesets, mojibake will always be possible.  Sometimes, you just
> have to temporarily switch to a different locale in order to make things
> work.  For example, I was trying to run some old Japanese software a
> while ago, and I had to add a Shift-JIS locale to my system in order to
> get it to work.
> 
> > Let's say you send me María.song, and I download it on a system
> > using the C locale.  Programs would fail to copy the file.
> 
> Not necessarily.  pathname(7) says “Paths should be encoded and decoded
> using the current locale’s codeset in order to help prevent mojibake.”
> In many cases, you don’t need to encode or decode a pathname.  Here’s a
> program that copies a file without encoding or decoding any pathnames:

Right.  But then, when do you need to do encoding?  Programs will either
receive the pathname from the command line, or read it from some file,
or create one of its own.

When creating a path of its own, it should restrict itself to the
Portable Filename Character Set, so encoding shouldn't be a problem.

When reading pathnames, they'll already be encoded suitably.

> > Instead, I think a good recommendation would be to behave in one of the
> > following ways:
> > 
> > -  Accept only the POSIX Portable Filename Character Set.
> 
> This one isn’t quite a complete recommendation.  The POSIX Portable
> Filename Character Set is just a character set.  It’s not a character
> encoding.  If we go with this one, then we would need to say something
> along the lines of “Encode and decode paths using ASCII and only accept
> characters that are in the POSIX Protable Filename Character Set.”
> 
> > -  Assume UTF-8, but reject control characters.
> > -  Assume UTF-8.
> 
> > -  Accept anything, but reject control characters.
> > -  Accept anything, just like the kernel.
> 
> These last two also aren’t quite complete recommendations.  If a GUI
> program wants to display a pathname on the screen, then what character
> encoding should it use when decoding the bytes?

Just print them as they got in.  No decoding.  Send the raw bytes to
write(2) or printf(3) or whatever.

> > The current locale should actively be ignored when handling pathnames.
> > 
> > I've modified the example in the manual page to use a filename that's
> > non-ASCII, to make it more interesting.  See how it fails:
> > 
> > 
> > What do you think?
> 
> I honestly don’t know what the recommendation should be.  Here’s what I
> would need to know in order to figure out what the recommendation should
> be.  A while ago, I asked this question on the Unix & Linux Stack
> Exchange [1]:
> 
> > What does a locale’s codeset get used for?
> >
> > According to glibc’s manual:
> >
> > > Most locale names follow XPG syntax and consist of up to four parts:
> > >
> > > ```
> > > language[_territory[.codeset]][@modifier]
> > > ```
> >
> > For example, you could have a locale named zh_CN.GB18030 which would
> > use the GB 18030 character encoding, or you could have a locale named
> > zh_CN.UTF-8 which would use the UTF-8 character encoding.
> >
> > Here’s where I’m confused: let’s say that I switch from a zh_CN.UTF-8
> > locale to a zh_CN.GB18030 locale. In that situation, some things that
> > used to be encoded in UTF-8 are now going to be encoded in GB 18030.
> > Which things will now be encoded in GB 18030? Will stdin, stdout and
> > stderr use GB 18030? What about argv? What about filesystem paths?
> >
> > Technically, a program can do whatever it wants and ignore the locale
> > completely, but let’s assume that programs are doing the correct thing
> > here. What is supposed to be encoded in GB 18030 if I use a
> > zh_CN.GB18030 locale?
> 
> I didn’t get an answer to that question, so I asked it again on the
> libc-help mailing list [2].  I got one response that was super helpful
> [3].  That response clearly said that paths should be encoded using the
> locale’s codeset.  If you think that answer was incorrect, then I would
> like a very specific list of things that should and should not be
> encoded using the locale’s codeset so that I can add it to the glibc
> manual (and maybe the POSIX standard if I can figure out how to
> contribute to that).
> 
> [1]: <https://unix.stackexchange.com/q/780404/316181>
> [2]: <https://sourceware.org/pipermail/libc-help/2024-August/006736.html>
> [3]: <https://sourceware.org/pipermail/libc-help/2024-August/006737.html>

Maybe Florian can comment.


Have a lovely day!
Alex

> 
> > Have a lovely day!
> > Alex
> > 
> > -- 
> > <https://www.alejandro-colomar.es/>

-- 
<https://www.alejandro-colomar.es/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: man/man7/pathname.7: Correct handling of pathnames

[Jason Yundt]

On Mon, Jan 27, 2025 at 04:53:10PM +0100, Alejandro Colomar wrote:
> Right.  But then, when do you need to do encoding?

Personally, my preference is that programs use the locale’s codeset
because I can override the locale codeset in the rare event that UTF-8
isn’t the correct option.  In my previous example, I was able to set the
LANG environment variable to jp_JP.SJIS so that I could run that old
software in an environment where pathnames were encoded in Shift-JIS.
If everything just always assumed a particular character encoding for
pathnames, then I wouldn’t have been able to do that.

That being said, I still don’t really know if that’s the best option.

> Programs will either receive the pathname from the command line, or
> read it from some file, or create one of its own.
> 
> When creating a path of its own, it should restrict itself to the
> Portable Filename Character Set, so encoding shouldn't be a problem.
> 
> When reading pathnames, they'll already be encoded suitably.
> 
> > > Instead, I think a good recommendation would be to behave in one of the
> > > following ways:
> > > 
> > > -  Accept only the POSIX Portable Filename Character Set.
> > 
> > This one isn’t quite a complete recommendation.  The POSIX Portable
> > Filename Character Set is just a character set.  It’s not a character
> > encoding.  If we go with this one, then we would need to say something
> > along the lines of “Encode and decode paths using ASCII and only accept
> > characters that are in the POSIX Protable Filename Character Set.”
> > 
> > > -  Assume UTF-8, but reject control characters.
> > > -  Assume UTF-8.
> > 
> > > -  Accept anything, but reject control characters.
> > > -  Accept anything, just like the kernel.
> > 
> > These last two also aren’t quite complete recommendations.  If a GUI
> > program wants to display a pathname on the screen, then what character
> > encoding should it use when decoding the bytes?
> 
> Just print them as they got in.  No decoding.  Send the raw bytes to
> write(2) or printf(3) or whatever.

I don’t think that printing is a good way for GUI applications to
display text.  I don’t normally run GUI applications in a terminal, so
I’m not normally able to see a GUI application’s stdout or stderr.  Most
of the GUI applications that I use display pathnames as part of a larger
window.  In order to do that, the GUI application needs to know which
characters the bytes in the pathname represent so that the GUI
application can draw those characters on the screen.

Re: man/man7/pathname.7: Correct handling of pathnames

[Alejandro Colomar]

[-- Attachment #1: Type: text/plain, Size: 2683 bytes --]

[CC += наб]

Hi Jason,

On Mon, Jan 27, 2025 at 12:14:43PM -0500, Jason Yundt wrote:
> On Mon, Jan 27, 2025 at 04:53:10PM +0100, Alejandro Colomar wrote:
> > Right.  But then, when do you need to do encoding?
> 
> Personally, my preference is that programs use the locale’s codeset
> because I can override the locale codeset in the rare event that UTF-8
> isn’t the correct option.  In my previous example, I was able to set the
> LANG environment variable to jp_JP.SJIS so that I could run that old
> software in an environment where pathnames were encoded in Shift-JIS.
> If everything just always assumed a particular character encoding for
> pathnames, then I wouldn’t have been able to do that.

But if the program handles arbitrary strings, just like the kernel does,
that would work too.

> > > > -  Accept anything, but reject control characters.
> > > > -  Accept anything, just like the kernel.
> > > 
> > > These last two also aren’t quite complete recommendations.  If a GUI
> > > program wants to display a pathname on the screen, then what character
> > > encoding should it use when decoding the bytes?
> > 
> > Just print them as they got in.  No decoding.  Send the raw bytes to
> > write(2) or printf(3) or whatever.
> 
> I don’t think that printing is a good way for GUI applications to
> display text.  I don’t normally run GUI applications in a terminal, so
> I’m not normally able to see a GUI application’s stdout or stderr.  Most
> of the GUI applications that I use display pathnames as part of a larger
> window.  In order to do that, the GUI application needs to know which
> characters the bytes in the pathname represent so that the GUI
> application can draw those characters on the screen.

I would do in a GUI exactly the same as what command-line programs do:
pass the raw string to whatever API prints them.  If the string makes
sense in the current locale, it will be shown nicely.  If it doesn't
make sense, it will display weird characters, but that's not a terrible
issue.  Just run again with the appropriate locale.

For example, in the git repository of the Linux man-pages project, there
are commits authored by наб <nabijaczleweli@nabijaczleweli.xyz>.  
Whenever I see the git-log(1) in one of my systems with the C locale, I
see weird characters.  I just need to re-run with the C.UTF-8 locale.

But it handles the bytes correctly, even if they don't make sense to the
system.  If git(1) failed whenever a string doesn't make sense in the
current locale, the repo would be corrupted sooner than later.


Cheers,
Alex

-- 
<https://www.alejandro-colomar.es/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: man/man7/pathname.7: Correct handling of pathnames

[наб]

[-- Attachment #1: Type: text/plain, Size: 1803 bytes --]

Skimming the thread: UNIX paths are sequences of non-NUL bytes.

It is never correct to expect to be able to have a (parse, unparse)
operation pair for which unparse(parse(x)) = x for path x.

It's obviously wrong to reject a pathname just because you dont like it.

Thus, when displaying a path, either (a) dump it directly to the output
(the user has configured their display device to understand the paths they use),
or if that's not possible (b) setlocale(LC_ALL, "") + mbrtowc() loop
and render the result (applying usual ?/� substitutions for mbrtowc()
errors makes sense here).

There are very few operations on paths that are actually reasonable
to do, ever; those are: appending stuff, prepending stuff
(this is just appending stuff with the arguments backwards),
and cleaving at /es;
the "stuff" better be copied whole-sale from some other path
or an unprocessed argument (or, sure, the PFCS).

If you're getting bytes to append to a path, do that directly.

If you're getting characters to append to a path,
then wctomb(3) is the only non-invalid solution,
since that (obviously) turns characters into bytes in the current
locale, which (ex def) is the operation desired.

I don't understand what the UTF-32 dance is supposed to be.

If you're recommending transcoding paths, don't.

To re-iterate: paths are not character sequences.
They do not represent characters.
You can't meaningfully coerce them thusly without loss of precision
(this is ok to do for display! and nothing else).
If at any point you find yourself turning wchar_t -> char
you are doing something wrong;
if you find yourself doing char -> wchar_t for anything beside display
you should probably reconsider.

This is different under Win32 of course. But that concerns us naught.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v1] man/man7/pathname.7: Pathnames are opaque C strings

[Alejandro Colomar]

[-- Attachment #1: Type: text/plain, Size: 6206 bytes --]

On Mon, Jan 27, 2025 at 07:27:59PM +0100, наб wrote:
> Skimming the thread: UNIX paths are sequences of non-NUL bytes.
>
> It is never correct to expect to be able to have a (parse, unparse)
> operation pair for which unparse(parse(x)) = x for path x.
>
> It's obviously wrong to reject a pathname just because you dont like it.
>
> Thus, when displaying a path, either (a) dump it directly to the output
> (the user has configured their display device to understand the paths they use),
> or if that's not possible (b) setlocale(LC_ALL, "") + mbrtowc() loop
> and render the result (applying usual ?/� substitutions for mbrtowc()
> errors makes sense here).
>
> There are very few operations on paths that are actually reasonable
> to do, ever; those are: appending stuff, prepending stuff
> (this is just appending stuff with the arguments backwards),
> and cleaving at /es;
> the "stuff" better be copied whole-sale from some other path
> or an unprocessed argument (or, sure, the PFCS).
>
> If you're getting bytes to append to a path, do that directly.
>
> If you're getting characters to append to a path,
> then wctomb(3) is the only non-invalid solution,
> since that (obviously) turns characters into bytes in the current
> locale, which (ex def) is the operation desired.
>
> I don't understand what the UTF-32 dance is supposed to be.
>
> If you're recommending transcoding paths, don't.
>
> To re-iterate: paths are not character sequences.
> They do not represent characters.
> You can't meaningfully coerce them thusly without loss of precision
> (this is ok to do for display! and nothing else).
> If at any point you find yourself turning wchar_t -> char
> you are doing something wrong;
> if you find yourself doing char -> wchar_t for anything beside display
> you should probably reconsider.
>
> This is different under Win32 of course. But that concerns us naught.

Suggested-by: наб <nabijaczleweli@nabijaczleweli.xyz>
Cc: Jason Yundt <jason@jasonyundt.email>
Cc: Florian Weimer <fweimer@redhat.com>
Cc: "G. Branden Robinson" <branden@debian.org>
Signed-off-by: Alejandro Colomar <alx@kernel.org>
---

Hi наб!

Thanks for the detailed response.  I applied this patch based on it.
Does it sound good to you?  Please review.

Have a lovely day!
Alex


 man/man7/pathname.7 | 87 ++-------------------------------------------
 1 file changed, 2 insertions(+), 85 deletions(-)

diff --git a/man/man7/pathname.7 b/man/man7/pathname.7
index 59650ef6e..996436606 100644
--- a/man/man7/pathname.7
+++ b/man/man7/pathname.7
@@ -17,7 +17,7 @@ .SH DESCRIPTION
 The kernel stores pathnames as C strings,
 that is,
 sequences of non-null bytes terminated by a null byte.
-The kernel has a few general rules that apply to all pathnames:
+There are a few general rules that apply to all pathnames:
 .IP \[bu] 3
 The last byte in the sequence needs to be a null byte.
 .IP \[bu]
@@ -59,17 +59,8 @@ .SH DESCRIPTION
 .P
 Some filesystems or APIs may apply further restrictions,
 such as requiring shorter filenames,
-or restricting the allowed characters in a filename.
+or restricting the allowed bytes in a filename.
 .P
-User-space programs treat pathnames differently.
-They typically expect pathnames to
-use a consistent character encoding.
-For maximum interoperability,
-programs should use
-.BR nl_langinfo (3)
-to determine the current locale's codeset.
-Pathnames should be encoded and decoded using the current locale's codeset
-in order to help prevent mojibake.
 For maximum interoperability,
 programs and users should also
 limit the characters that they use for their own pathnames to
@@ -77,83 +68,9 @@ .SH DESCRIPTION
 .UR https://pubs.opengroup.org/\:onlinepubs/\:9799919799/\:basedefs/\:V1_chap03.html#tag_03_265
 Portable Filename Character Set
 .UE .
-.SH EXAMPLES
-The following program demonstrates
-how to ensure that a pathname uses the proper encoding.
-The program starts with a UTF-32 encoded pathname.
-It then calls
-.BR nl_langinfo (3)
-in order to determine what the current locale's codeset is.
-After that, it uses
-.BR iconv (3)
-to convert the UTF-32-encoded pathname into a locale-codeset-encoded pathname.
-Finally,
-the program uses the locale-codeset-encoded pathname
-to create a file that contains the message \[lq]Hello, world!\[rq].
-.SS Program source
-.\" SRC BEGIN (pathname_encoding_example.c)
-.EX
-#include <err.h>
-#include <iconv.h>
-#include <langinfo.h>
-#include <locale.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <uchar.h>
-\&
-#define NELEMS(a)  (sizeof(a) / sizeof(a[0]))
-\&
-int
-main(void)
-{
-    char      *locale_pathname;
-    char      *in, *out;
-    FILE      *fp;
-    size_t    size;
-    size_t    inbytes, outbytes;
-    iconv_t   cd;
-    char32_t  utf32_pathname[] = U"María";
-\&
-    if (setlocale(LC_ALL, "") == NULL)
-        err(EXIT_FAILURE, "setlocale");
-\&
-    size = NELEMS(utf32_pathname) * MB_CUR_MAX;
-    locale_pathname = malloc(size);
-    if (locale_pathname == NULL)
-        err(EXIT_FAILURE, "malloc");
-\&
-    cd = iconv_open(nl_langinfo(CODESET), "UTF\-32");
-    if (cd == (iconv_t)\-1)
-        err(EXIT_FAILURE, "iconv_open");
-\&
-    in = (char *) utf32_pathname;
-    inbytes = sizeof(utf32_pathname);
-    out = locale_pathname;
-    outbytes = size;
-    if (iconv(cd, &in, &inbytes, &out, &outbytes) == (size_t) \-1)
-        err(EXIT_FAILURE, "iconv");
-\&
-    if (iconv_close(cd) == \-1)
-        err(EXIT_FAILURE, "iconv_close");
-\&
-    fp = fopen(locale_pathname, "w");
-    if (fp == NULL)
-        err(EXIT_FAILURE, "fopen");
-\&
-    fputs("Hello, world!\[rs]n", fp);
-    if (fclose(fp) == EOF)
-        err(EXIT_FAILURE, "fclose");
-\&
-    free(locale_pathname);
-    exit(EXIT_SUCCESS);
-}
-.EE
-.\" SRC END
 .SH SEE ALSO
 .BR limits.h (0p),
 .BR open (2),
 .BR fpathconf (3),
-.BR iconv (3),
-.BR nl_langinfo (3),
 .BR path_resolution (7),
 .BR mount (8)

Range-diff against v0:
-:  --------- > 1:  b9f5079f6 man/man7/pathname.7: Pathnames are opaque C strings
-- 
2.47.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v1] man/man7/pathname.7: Pathnames are opaque C strings

[наб]

[-- Attachment #1: Type: text/plain, Size: 388 bytes --]

On Mon, Jan 27, 2025 at 07:46:21PM +0100, Alejandro Colomar wrote:
> Thanks for the detailed response.  I applied this patch based on it.
> Does it sound good to you?  Please review.
Can't really find the base file for this, but the deletions look good.

>  man/man7/pathname.7 | 87 ++-------------------------------------------
>  1 file changed, 2 insertions(+), 85 deletions(-)

Best,

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v1] man/man7/pathname.7: Pathnames are opaque C strings

[Alejandro Colomar]

[-- Attachment #1: Type: text/plain, Size: 840 bytes --]

Hi,

On Mon, Jan 27, 2025 at 07:58:02PM +0100, наб wrote:
> On Mon, Jan 27, 2025 at 07:46:21PM +0100, Alejandro Colomar wrote:
> > Thanks for the detailed response.  I applied this patch based on it.
> > Does it sound good to you?  Please review.
> Can't really find the base file for this,

It's the contrib banch on my server:
<https://www.alejandro-colomar.es/src/alx/linux/man-pages/man-pages.git/log/?h=contrib>
<https://www.alejandro-colomar.es/src/alx/linux/man-pages/man-pages.git/commit/?h=contrib&id=b9f5079f6bdeb29594f8ac24bd2eb4289e562613>

> but the deletions look good.

Thanks!

Have a lovely night!
Alex

> 
> >  man/man7/pathname.7 | 87 ++-------------------------------------------
> >  1 file changed, 2 insertions(+), 85 deletions(-)
> 
> Best,



-- 
<https://www.alejandro-colomar.es/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v1] man/man7/pathname.7: Pathnames are opaque C strings

[наб]

[-- Attachment #1: Type: text/plain, Size: 764 bytes --]

On Mon, Jan 27, 2025 at 08:05:13PM +0100, Alejandro Colomar wrote:
> On Mon, Jan 27, 2025 at 07:58:02PM +0100, наб wrote:
> > On Mon, Jan 27, 2025 at 07:46:21PM +0100, Alejandro Colomar wrote:
> > > Thanks for the detailed response.  I applied this patch based on it.
> > > Does it sound good to you?  Please review.
> > Can't really find the base file for this,
> It's the contrib banch on my server:
> <https://www.alejandro-colomar.es/src/alx/linux/man-pages/man-pages.git/log/?h=contrib>
> <https://www.alejandro-colomar.es/src/alx/linux/man-pages/man-pages.git/commit/?h=contrib&id=b9f5079f6bdeb29594f8ac24bd2eb4289e562613>
Looks sane;
idk about "limits.h(0p)" (it's limits.h(7POSIX) for me),
but there aren't other limits.h pages so whatever.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v1] man/man7/pathname.7: Pathnames are opaque C strings

[Alejandro Colomar]

[-- Attachment #1: Type: text/plain, Size: 964 bytes --]

On Mon, Jan 27, 2025 at 08:10:10PM +0100, наб wrote:
> On Mon, Jan 27, 2025 at 08:05:13PM +0100, Alejandro Colomar wrote:
> > On Mon, Jan 27, 2025 at 07:58:02PM +0100, наб wrote:
> > > On Mon, Jan 27, 2025 at 07:46:21PM +0100, Alejandro Colomar wrote:
> > > > Thanks for the detailed response.  I applied this patch based on it.
> > > > Does it sound good to you?  Please review.
> > > Can't really find the base file for this,
> > It's the contrib banch on my server:
> > <https://www.alejandro-colomar.es/src/alx/linux/man-pages/man-pages.git/log/?h=contrib>
> > <https://www.alejandro-colomar.es/src/alx/linux/man-pages/man-pages.git/commit/?h=contrib&id=b9f5079f6bdeb29594f8ac24bd2eb4289e562613>
> Looks sane;
> idk about "limits.h(0p)" (it's limits.h(7POSIX) for me),
> but there aren't other limits.h pages so whatever.

Debian moves the 0p pages to 7POSIX.  Upstream, they're in 0p.  :-)



-- 
<https://www.alejandro-colomar.es/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: man/man7/pathname.7: Correct handling of pathnames

[Jason Yundt]

On Mon, Jan 27, 2025 at 06:37:40PM +0100, Alejandro Colomar wrote:
> [CC += наб]
> 
> Hi Jason,
> 
> On Mon, Jan 27, 2025 at 12:14:43PM -0500, Jason Yundt wrote:
> > On Mon, Jan 27, 2025 at 04:53:10PM +0100, Alejandro Colomar wrote:
> > > Right.  But then, when do you need to do encoding?
> > 
> > Personally, my preference is that programs use the locale’s codeset
> > because I can override the locale codeset in the rare event that UTF-8
> > isn’t the correct option.  In my previous example, I was able to set the
> > LANG environment variable to jp_JP.SJIS so that I could run that old
> > software in an environment where pathnames were encoded in Shift-JIS.
> > If everything just always assumed a particular character encoding for
> > pathnames, then I wouldn’t have been able to do that.
> 
> But if the program handles arbitrary strings, just like the kernel does,
> that would work too.
> 
> > > > > -  Accept anything, but reject control characters.
> > > > > -  Accept anything, just like the kernel.
> > > > 
> > > > These last two also aren’t quite complete recommendations.  If a GUI
> > > > program wants to display a pathname on the screen, then what character
> > > > encoding should it use when decoding the bytes?
> > > 
> > > Just print them as they got in.  No decoding.  Send the raw bytes to
> > > write(2) or printf(3) or whatever.
> > 
> > I don’t think that printing is a good way for GUI applications to
> > display text.  I don’t normally run GUI applications in a terminal, so
> > I’m not normally able to see a GUI application’s stdout or stderr.  Most
> > of the GUI applications that I use display pathnames as part of a larger
> > window.  In order to do that, the GUI application needs to know which
> > characters the bytes in the pathname represent so that the GUI
> > application can draw those characters on the screen.
> 
> I would do in a GUI exactly the same as what command-line programs do:
> pass the raw string to whatever API prints them.  If the string makes
> sense in the current locale, it will be shown nicely.  If it doesn't
> make sense, it will display weird characters, but that's not a terrible
> issue.  Just run again with the appropriate locale.

OK, but how does that API figure out what characters to display?  What
character encoding should that API use when drawing the characters?  I
think that it’s OK to replace the current recommendation, but
pathname(7) should really explain how such an API would figure out what
characters need to be drawn on the screen.

> For example, in the git repository of the Linux man-pages project, there
> are commits authored by наб <nabijaczleweli@nabijaczleweli.xyz>.  
> Whenever I see the git-log(1) in one of my systems with the C locale, I
> see weird characters.  I just need to re-run with the C.UTF-8 locale.
> 
> But it handles the bytes correctly, even if they don't make sense to the
> system.  If git(1) failed whenever a string doesn't make sense in the
> current locale, the repo would be corrupted sooner than later.

Re: man/man7/pathname.7: Correct handling of pathnames

[Alejandro Colomar]

[-- Attachment #1: Type: text/plain, Size: 1648 bytes --]

Hi Jason,

On Mon, Jan 27, 2025 at 06:07:30PM -0500, Jason Yundt wrote:
> > I would do in a GUI exactly the same as what command-line programs do:
> > pass the raw string to whatever API prints them.  If the string makes
> > sense in the current locale, it will be shown nicely.  If it doesn't
> > make sense, it will display weird characters, but that's not a terrible
> > issue.  Just run again with the appropriate locale.
> 
> OK, but how does that API figure out what characters to display?  What
> character encoding should that API use when drawing the characters?  I
> think that it???s OK to replace the current recommendation, but
> pathname(7) should really explain how such an API would figure out what
> characters need to be drawn on the screen.

That's not a pathname issue anymore.  It's just the issue of printing
bytes to a user.  I don't think pathname(7) should talk about how bytes
are shown to a user.

That wouldn't affect at all how applications handle files.

For example, I have just installed my new laptop (with the C locale),
and nab's name shows as ??????.  I expect a Japanese filename to be
shown similarly, although that depends on what each application wants to
do.  It doesn't really matter, since it's just a cosmetic issue.  The
string still contains the appropriate bytes, even if I can't read them
properly.  If I had a file called nab in cyrillic, I would expect ls(1)
to similarly show ??????, but internally just handle it well, because it
doesn't even look at the bytes; it just passes them through.


Have a lovely night!
Alex

-- 
<https://www.alejandro-colomar.es/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: man/man7/pathname.7: Correct handling of pathnames

[Jason Yundt]

On Tue, Jan 28, 2025 at 12:49:29AM +0100, Alejandro Colomar wrote:
> Hi Jason,
> 
> On Mon, Jan 27, 2025 at 06:07:30PM -0500, Jason Yundt wrote:
> > > I would do in a GUI exactly the same as what command-line programs do:
> > > pass the raw string to whatever API prints them.  If the string makes
> > > sense in the current locale, it will be shown nicely.  If it doesn't
> > > make sense, it will display weird characters, but that's not a terrible
> > > issue.  Just run again with the appropriate locale.
> > 
> > OK, but how does that API figure out what characters to display?  What
> > character encoding should that API use when drawing the characters?  I
> > think that it???s OK to replace the current recommendation, but
> > pathname(7) should really explain how such an API would figure out what
> > characters need to be drawn on the screen.
> 
> That's not a pathname issue anymore.  It's just the issue of printing
> bytes to a user.  I don't think pathname(7) should talk about how bytes
> are shown to a user.

Where should it be documented, then?

> That wouldn't affect at all how applications handle files.
> 
> For example, I have just installed my new laptop (with the C locale),
> and nab's name shows as ??????.  I expect a Japanese filename to be
> shown similarly, although that depends on what each application wants to
> do.  It doesn't really matter, since it's just a cosmetic issue.  The
> string still contains the appropriate bytes, even if I can't read them
> properly.  If I had a file called nab in cyrillic, I would expect ls(1)
> to similarly show ??????, but internally just handle it well, because it
> doesn't even look at the bytes; it just passes them through.
> 
> 
> Have a lovely night!
> Alex
> 
> -- 
> <https://www.alejandro-colomar.es/>

Re: man/man7/pathname.7: Correct handling of pathnames

[Alejandro Colomar]

[-- Attachment #1: Type: text/plain, Size: 1234 bytes --]

Hi Jason, Branden,

On Mon, Jan 27, 2025 at 10:06:52PM -0500, Jason Yundt wrote:
> > On Mon, Jan 27, 2025 at 06:07:30PM -0500, Jason Yundt wrote:
> > > > I would do in a GUI exactly the same as what command-line programs do:
> > > > pass the raw string to whatever API prints them.  If the string makes
> > > > sense in the current locale, it will be shown nicely.  If it doesn't
> > > > make sense, it will display weird characters, but that's not a terrible
> > > > issue.  Just run again with the appropriate locale.
> > > 
> > > OK, but how does that API figure out what characters to display?  What
> > > character encoding should that API use when drawing the characters?  I
> > > think that it???s OK to replace the current recommendation, but
> > > pathname(7) should really explain how such an API would figure out what
> > > characters need to be drawn on the screen.
> > 
> > That's not a pathname issue anymore.  It's just the issue of printing
> > bytes to a user.  I don't think pathname(7) should talk about how bytes
> > are shown to a user.
> 
> Where should it be documented, then?

I don't know.  Maybe Branden knows?


Have a lovely day!
Alex

-- 
<https://www.alejandro-colomar.es/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: man/man7/pathname.7: Correct handling of pathnames

[наб]

[-- Attachment #1: Type: text/plain, Size: 584 bytes --]

On Tue, Jan 28, 2025 at 11:17:59AM +0100, Alejandro Colomar wrote:
> On Mon, Jan 27, 2025 at 10:06:52PM -0500, Jason Yundt wrote:
> > > On Mon, Jan 27, 2025 at 06:07:30PM -0500, Jason Yundt wrote:
> > > That's not a pathname issue anymore.  It's just the issue of printing
> > > bytes to a user.  I don't think pathname(7) should talk about how bytes
> > > are shown to a user.
> > Where should it be documented, then?
> I don't know.  Maybe Branden knows?
I don't think this is ad rem to man-pages;
how you display a string falls squarely in the domain of the render toolkit.

Best,

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]