Are there users of ash's "pathopts"? Do other shells have such a thing?

Are there users of ash's "pathopts"? Do other shells have such a thing?

[Denys Vlasenko]

ash allows PATH to contain "%builtin" and "DIR%func"
pseudo-directories.

%builtin shows in what order builtins to be found
relative to searching directories for external commands.

DIR%func allows to have a directory of "auto-loadable" functions.
(dash git tree seems to have an example of it as src/funcs/*).

I tried a few searches on the internet and this feature
seems to be almost entirely not documented. I found exactly one
manpage documenting it:

https://www.unix.com/man-page/minix/1/ash/

     Path Search

        When locating a command, the shell first looks to see if it has a shell function by that name.  Then, if PATH does not contain an entry for
        "%builtin", it looks for a builtin command by that name.  Finally, it searches each entry in PATH in turn for the command.

        The value of the PATH variable should be a series of entries separated by colons.  Each entry consists of a directory name, or a  directory
        name followed by a flag beginning with a percent sign.  The current directory should be indicated by an empty directory name.

        If  no  percent	sign  is  present,  then  the entry causes the shell to search for the command in the specified directory.  If the flag is
        ``%builtin'' then the list of shell builtin commands is searched.  If the flag is ``%func'' then the directory is searched for a file which
        is read as input to the shell.  This file should define a function whose name is the name of the command being searched for.

Here is an example of a user having a problem because he has a PATH with
directory containing "GNU%2fLinux" string in its name:

https://unix.stackexchange.com/questions/126955/percent-in-path-environment-variable

So... maybe we can drop it?

I assume it's rarely (never?) used in the wild.
It interferes with valid directories with percents in names.
It's non-standard, and not a typical feature of other Bourne-like shells
   (maybe we (ash family) are the only ones?)
Code complication to support it is a chore.

Re: Are there users of ash's "pathopts"? Do other shells have such a thing?

[Harald van Dijk]

On 01/04/2023 19:46, Denys Vlasenko wrote:
> ash allows PATH to contain "%builtin" and "DIR%func"
> pseudo-directories.
> 
> %builtin shows in what order builtins to be found
> relative to searching directories for external commands.

Indeed. This is a desirable feature, regardless of whether the current 
implementation is the right way of doing it. POSIX intends for it to be 
possible to have a way to specify that a directory in PATH takes 
precedence over built-in commands. The way it tried to specify this was 
broken, and did not have the desired effect. The way it is implemented 
in dash using %builtin is one way of doing something that works.

I am not a fan of the precise way %builtin is implemented, and took that 
out in my fork, but do intend to make the feature available in *some* 
way again at some point and would suggest not just dropping it without a 
replacement in dash.

> DIR%func allows to have a directory of "auto-loadable" functions.
> (dash git tree seems to have an example of it as src/funcs/*).

When I used dash, I did use this feature. It is roughly equivalent to 
ksh's FPATH feature. In my own fork of dash I modified it to support 
that instead, and dropped %func since it no longer served a purpose, but 
if I were still using dash, I would still be using %func.

> I tried a few searches on the internet and this feature
> seems to be almost entirely not documented. I found exactly one
> manpage documenting it:
> 
> https://www.unix.com/man-page/minix/1/ash/
> 
>      Path Search
> 
>         When locating a command, the shell first looks to see if it has 
> a shell function by that name.  Then, if PATH does not contain an entry for
>         "%builtin", it looks for a builtin command by that name.  
> Finally, it searches each entry in PATH in turn for the command.
> 
>         The value of the PATH variable should be a series of entries 
> separated by colons.  Each entry consists of a directory name, or a  
> directory
>         name followed by a flag beginning with a percent sign.  The 
> current directory should be indicated by an empty directory name.
> 
>         If  no  percent    sign  is  present,  then  the entry causes 
> the shell to search for the command in the specified directory.  If the 
> flag is
>         ``%builtin'' then the list of shell builtin commands is 
> searched.  If the flag is ``%func'' then the directory is searched for a 
> file which
>         is read as input to the shell.  This file should define a 
> function whose name is the name of the command being searched for.
> 
> Here is an example of a user having a problem because he has a PATH with
> directory containing "GNU%2fLinux" string in its name:
> 
> https://unix.stackexchange.com/questions/126955/percent-in-path-environment-variable

This is unfortunate. It is well-known that arbitrary directory names 
cannot be used in PATH -- directory names containing : cannot be -- but 
it is less well-known that this extends to other directory names.

> So... maybe we can drop it?
> 
> I assume it's rarely (never?) used in the wild.
> It interferes with valid directories with percents in names.
> It's non-standard, and not a typical feature of other Bourne-like shells
>    (maybe we (ash family) are the only ones?)
> Code complication to support it is a chore.

It is not clear to me whether you mean "non-standard" as "conflicts with 
the standard", or "not required by the standard". It is an extension, 
but it is now a permitted extension: POSIX is being changed to permit 
this behaviour. In the latest POSIX draft, PATH lookup specifies:

   If PATH is unset or is set to null, or if a path prefix in PATH
   contains a <percent-sign> character ('%'), the path search is
   implementation-defined.

Cheers,
Harald van Dijk

Re: Are there users of ash's "pathopts"? Do other shells have such a thing?

[Denys Vlasenko]

On 4/1/23 21:13, Harald van Dijk wrote:
>> I assume it's rarely (never?) used in the wild.
>> It interferes with valid directories with percents in names.
>> It's non-standard, and not a typical feature of other Bourne-like shells
>>    (maybe we (ash family) are the only ones?)
>> Code complication to support it is a chore.
> 
> It is not clear to me whether you mean "non-standard" as "conflicts with the standard", or "not required by the standard".

"not required".

More importantly, no one else among shells seems to have such extension.
This means it's probably not terribly useful.

There is value in *not* having extensions "just because we can".
Extensions fragment user base.
IOW: extensions make sense when they are *useful*.

Shells already have way too much of featuritis.
Just now, I learned that CDPATH is a thing... oh god.
Yet another danger of subtle bugs in scripts when one
of their innocuous "mkdir subdir; cd subdir" suddenly
cd's somewhere else...
..and this "feature" seems to be in bash too,
and who knows where else.

Re: Are there users of ash's "pathopts"? Do other shells have such a thing?

[Lawrence Velázquez]

On Sat, Apr 1, 2023, at 4:49 PM, Denys Vlasenko wrote:
> Just now, I learned that CDPATH is a thing... oh god.
> Yet another danger of subtle bugs in scripts when one
> of their innocuous "mkdir subdir; cd subdir" suddenly
> cd's somewhere else...
> ..and this "feature" seems to be in bash too,
> and who knows where else.

At a minimum:
- traditional Bourne shells since 1983 [1]
- POSIX [2]
- busybox
- ksh and friends
- yash
- zsh

[1]: https://www.in-ulm.de/~mascheck/bourne/#svr1
[2]: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/cd.html

-- 
vq

Re: Are there users of ash's "pathopts"? Do other shells have such a thing?

[Sven Mascheck]

On 01/04/2023 19:46, Denys Vlasenko wrote:
> I tried a few searches on the internet and this feature
> seems to be almost entirely not documented. I found exactly one
> manpage documenting it:
>
> https://www.unix.com/man-page/minix/1/ash/
>
PS: interestingly, in 4.4BSD-Alpha ('92), this feature was removed from 
the man page.  sh had started aiming at POSIX and the man page was 
completely rewritten...

Minix-2 ash was taked from the 4.3BSD-Net/2 ('91), though, which still 
had the original Almquist man page, and so this is a rare place where 
the documentation survived.
-- 
https://www.in-ulm.de/~mascheck/various/ash/

Re: Are there users of ash's "pathopts"? Do other shells have such a thing?

[Denys Vlasenko]

On 4/1/23 20:46, Denys Vlasenko wrote:
> ash allows PATH to contain "%builtin" and "DIR%func"
> pseudo-directories.
> 
> %builtin shows in what order builtins to be found
> relative to searching directories for external commands.
> 
> DIR%func allows to have a directory of "auto-loadable" functions.
> (dash git tree seems to have an example of it as src/funcs/*).


On a related note, chkmail() used to iterate over MAILPATH
using padvance(), the function which detects "pathopts".

In commit a068bf7aa310e8d36ae11c2aec47af1446a18827
"exec: Stricter pathopt parsing"
padvance() was modified, and old behavior
is retained in the form padvance_magic(..., 2),
which treats any %text as pathopt (not only %builtin and %func).

chkmail() now uses padvance_magic(..., 2) call,
presumably as exact 1:1 equivalent of old code.

But it's not necessary, right? chkmail() does not actually uses
pathopts. It can just call padvance_magic(..., 0)
so as to ignore them.

If so, then code for padvance_magic(..., 2) is unused
and can be removed. Only "magic" values of 0 and 1 will be in use.

Re: Are there users of ash's "pathopts"? Do other shells have such a thing?

[Harald van Dijk]

On 03/04/2023 10:35, Denys Vlasenko wrote:
> But it's not necessary, right? chkmail() does not actually uses
> pathopts. It can just call padvance_magic(..., 0)
> so as to ignore them.

chkmail() does use pathopt. It is used to customise the message that 
gets printed:

		if (!changed && statb.st_mtime != *mtp) {
			outfmt(
				&errout, snlfmt,
				pathopt ? pathopt : "you have mail"
			);
		}

POSIX also specifies the use of '%' for this (although dash's 
implementation is not entirely compatible):

   MAILPATH
     Provide a list of pathnames and optional messages separated by
     <colon> characters. If this variable is set, the shell shall inform
     the user if any of the files named by the variable are created or if
     any of their modification times change. (See the preceding entry for
     MAIL for descriptions of mail arrival and user informing.) Each
     pathname can be followed by '%' and a string that shall be subjected
     to parameter expansion and written to standard error when the
     modification time changes. If a '%' character in the pathname is
     preceded by a <backslash>, it shall be treated as a literal '%' in
     the pathname. The default message is unspecified.

Cheers,
Harald van Dijk

Re: Are there users of ash's "pathopts"? Do other shells have such a thing?

[Denys Vlasenko]

On 4/3/23 13:35, Harald van Dijk wrote:
> On 03/04/2023 10:35, Denys Vlasenko wrote:
>> But it's not necessary, right? chkmail() does not actually uses
>> pathopts. It can just call padvance_magic(..., 0)
>> so as to ignore them.
> 
> chkmail() does use pathopt. It is used to customise the message that gets printed:
> 
>          if (!changed && statb.st_mtime != *mtp) {
>              outfmt(
>                  &errout, snlfmt,
>                  pathopt ? pathopt : "you have mail"
>              );
>          }

I missed this. Thank you.