Question on $IFS related differences (Was: Question on $@ vs $@$@)

Question on $IFS related differences (Was: Question on $@ vs $@$@)

[Steffen Nurpmeso]

Hello.

I am terribly sorry to be here once again; i already mentioned on
bug-bash@ the problem with IFS being reverse solidus that dash
(and myself) has (have).
I came back to this for my one again, and i stumble over one
additional difference in behaviour in between (myself and) bash,
dash and busybox sh.

We are still here, and this snippet gives identical results for
all, my MUA included,

  a() {
    echo $#,1=$1,2=$2,3=$3,4=$4,5=$5,6=$6,"$*",$*,
  }
  set -- '' '' ''
  IFS=' '; echo "$*"$* $*; a "$*"$* $*;unset IFS

ie, via "cat -vet":

    $
  1,1= ,2=,3=,4=,5=,6=,  , ,$

*However*, if we change IFS to : in the above:

  a() {
    echo $#,1=$1,2=$2,3=$3,4=$4,5=$5,6=$6,"$*",$*,
  }
  set -- '' '' ''
  IFS=':'; echo "$*"$* $*; a "$*"$* $*;unset IFS

then we get different behaviour for bash on the one side, dash and
busybox sh on the next, and myself on the third.  bash 5.2.32 says

  ::   $
  4,1=  ,2=,3=,4=,5=,6=,:::::,     ,$

whereas the ashs say

  ::$
  1,1=  ,2=,3=,4=,5=,6=,::,  ,$

and i say the same as for SPC except that : is in place of SPC:

  ::$
  1,1= ,2=,3=,4=,5=,6=,::, ,$

I really would like to finally get a nice mind map on what is
going on everywhere.

Thank you very much in advance,
Ciao (and good night),

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: Question on $IFS related differences (Was: Question on $@ vs $@$@)

[Oğuz]

On Wed, Sep 18, 2024 at 4:19 AM Steffen Nurpmeso <steffen@sdaoden.eu> wrote:
>

It boils down to this:

  f(){ echo $#;}; set "" "" ""; IFS=x; f $*

bash, NetBSD and FreeBSD sh, and ksh88 all agree and print 2. pdksh
prints 3 but mksh and oksh print 1. dash, ksh93, yash, and zsh print
0.

Re: Question on $IFS related differences (Was: Question on $@ vs $@$@)

[Greg Wooledge]

On Wed, Sep 18, 2024 at 08:05:10 +0300, Oğuz wrote:
> On Wed, Sep 18, 2024 at 4:19 AM Steffen Nurpmeso <steffen@sdaoden.eu> wrote:
> >
> 
> It boils down to this:
> 
>   f(){ echo $#;}; set "" "" ""; IFS=x; f $*
> 
> bash, NetBSD and FreeBSD sh, and ksh88 all agree and print 2. pdksh
> prints 3 but mksh and oksh print 1. dash, ksh93, yash, and zsh print
> 0.

At the risk of sounding like a broken record, using an unquoted $* or $@
in a context where word splitting occurs is just *begging* for trouble.
Please don't do this in your scripts.  All of these implementation
differences and possible bugs will just stop mattering, if you stop
using questionable shell features.

If you want to pass along your positional parameters to a function,
use "$@" with quotes.  This will pass each parameter as a separate
argument to the function, with no modifications.  It should work in
every post-Bourne shell (if it doesn't, that's a bug).  This is almost
always what you want.

If you want to join all of your positional parameters together into
a single string, use "$*" with quotes.  The first character of IFS
will be inserted between each pair of parameters.  This is sometimes
useful when writing messages to log files, or to produce a simple
row of delimited fields (not a full-blown CSV file, though).

Re: Question on $IFS related differences (Was: Question on $@ vs $@$@)

[Chet Ramey]

On 9/18/24 1:05 AM, Oğuz wrote:
> On Wed, Sep 18, 2024 at 4:19 AM Steffen Nurpmeso <steffen@sdaoden.eu> wrote:
>>
> 
> It boils down to this:
> 
>    f(){ echo $#;}; set "" "" ""; IFS=x; f $*
> 
> bash, NetBSD and FreeBSD sh, and ksh88 all agree and print 2. pdksh
> prints 3 but mksh and oksh print 1. dash, ksh93, yash, and zsh print
> 0.

It's an implementation difference.

The shells that print `0' treat the `may' as `shall' in this sentence from

https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V3_chap02.html#tag_19_05_02

"When the expansion occurs in a context where field splitting will be
performed, any empty fields may be discarded and each of the non-empty
fields shall be further split as described in 2.6.5 Field Splitting."

So they immediately expand the $* to nothing.

The shells that print `2' don't immediately discard the empty fields and
separate them with the first character of IFS. After quote removal and
removing the empty fields, since the individual words in $* are subject
to additional expansion, you're left with `xx'. When you split that,
you get two empty fields.

I'm not sure what's going on with the shells that print `1' unless they
somehow don't use the first character of IFS to separate the arguments
and let multiple sequential delimiters terminate the first field.

I assume that pdksh just expands each positional parameter to a quoted
empty string and passes them all to f.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

Re: Question on $IFS related differences (Was: Question on $@ vs $@$@)

[Steffen Nurpmeso]

Chet Ramey wrote in
 <b72edbc9-d336-4391-98a9-37c670afa435@case.edu>:
 |On 9/18/24 1:05 AM, Oğuz wrote:
 |> On Wed, Sep 18, 2024 at 4:19 AM Steffen Nurpmeso <steffen@sdaoden.eu> \
 |> wrote:
 |> It boils down to this:
 |> 
 |>    f(){ echo $#;}; set "" "" ""; IFS=x; f $*
 |> 
 |> bash, NetBSD and FreeBSD sh, and ksh88 all agree and print 2. pdksh
 |> prints 3 but mksh and oksh print 1. dash, ksh93, yash, and zsh print
 |> 0.
 |
 |It's an implementation difference.
 |
 |The shells that print `0' treat the `may' as `shall' in this sentence from
 |
 |https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V3_chap02.htm\
 |l#tag_19_05_02
 |
 |"When the expansion occurs in a context where field splitting will be
 |performed, any empty fields may be discarded and each of the non-empty
 |fields shall be further split as described in 2.6.5 Field Splitting."
 |
 |So they immediately expand the $* to nothing.
 |
 |The shells that print `2' don't immediately discard the empty fields and
 |separate them with the first character of IFS. After quote removal and
 |removing the empty fields, since the individual words in $* are subject
 |to additional expansion, you're left with `xx'. When you split that,
 |you get two empty fields.

Thank you for this analysis.
(Here it is just a shell expander, ie, it acts like a format
string expander, and these steps you include, namely "quote
removal" etc, are nothing i can or want to mimic.)

 |I'm not sure what's going on with the shells that print `1' unless they
 |somehow don't use the first character of IFS to separate the arguments
 |and let multiple sequential delimiters terminate the first field.
 |
 |I assume that pdksh just expands each positional parameter to a quoted
 |empty string and passes them all to f.

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: Question on $IFS related differences (Was: Question on $@ vs $@$@)

[Steffen Nurpmeso]

Oğuz wrote in
 <CAH7i3Lq9xQnnkAO7cd5XZEsV4jY+Va6uWqjw3yaMi=A3mdh83g@mail.gmail.com>:
 |On Wed, Sep 18, 2024 at 4:19 AM Steffen Nurpmeso <steffen@sdaoden.eu> \
 |wrote:
 |>
 |
 |It boils down to this:

In this case, ok .. i have not spent time trying to vaporise it.
I haved added this standalone thing to my tests.

 |  f(){ echo $#;}; set "" "" ""; IFS=x; f $*
 |
 |bash, NetBSD and FreeBSD sh, and ksh88 all agree and print 2. pdksh
 |prints 3 but mksh and oksh print 1. dash, ksh93, yash, and zsh print
 |0.

I am totally surprised they all swallow this code, there is no
separating whitespace in between the ; and the } which closes the
function -- isn't that invalid syntax even??

 --End of <CAH7i3Lq9xQnnkAO7cd5XZEsV4jY+Va6uWqjw3yaMi=A3mdh83g@mail.gmail\
 .com>

Thanks a lot for the impetus!

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)