Parameter expansion fails when assigning a local variable

Parameter expansion fails when assigning a local variable

[Jesper Bengtsson]

[-- Attachment #1: Type: text/plain, Size: 592 bytes --]

Hi,

I've found a problem with expanding $@.
If I declare a local variable and assigns the expanded positional parameters:
        local v="$@"
It fails with something like:
        local: 20: \/: bad variable name

If I split declaration and assignment into two lines it will work:
        local v
        v="$@"

I've tried this on dash 0.5.5.1 and 0.5.3 and both fail in the same way.
I've also tried on bash and ash (Busybox) and both handles declaration and assignment on one line.

I'm attaching a small script that shows the problem.

Best regards,
Jesper Bengtsson

[-- Attachment #2: dash-arg-bug.sh --]
[-- Type: application/octet-stream, Size: 371 bytes --]

#!/bin/dash

set -e
#set -x
#set -v

func() {
	local v="$@"	# unexpected behaviour == bug
	# v="$@"	# expected behaviour
	# local v	# also expected behaviour with declaration
	# v="$@"	# and asignment on separate lines

	echo "func: v='$v', \$1='$1', \$2='$2', \$3='$3'" >&2

	echo "$v"
}

xyz=" 	  123	 	xyz	\\/ 	"
echo "in:  '$xyz'"
abc=$(func $xyz)
echo "out: '$abc'"

Re: Parameter expansion fails when assigning a local variable

[Herbert Xu]

Jesper Bengtsson <jesper.bengtsson@axis.com> wrote:
> 
> I've found a problem with expanding $@.
> If I declare a local variable and assigns the expanded positional parameters:
>        local v="$@"
> It fails with something like:
>        local: 20: \/: bad variable name

This is expected behaviour.  local is just like any other utility
in how it handled parameters.  So "$@" will have been split before
it reaches local.  To get what you want, you should use "$*".
 
> I've tried this on dash 0.5.5.1 and 0.5.3 and both fail in the same way.
> I've also tried on bash and ash (Busybox) and both handles declaration and assignment on one line.

Bash and older versions of dash (like the one in Busybox) has a
hack that disables field splitting for local and certain other
commands.  This is not guaranteed by POSIX.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Parameter expansion fails when assigning a local variable

[Cristian Ionescu-Idbohrn]

On Mon, 22 Jun 2009, Herbert Xu wrote:

> Jesper Bengtsson <jesper.bengtsson@axis.com> wrote:
> >
> > I've found a problem with expanding $@.
> > If I declare a local variable and assigns the expanded positional
> > parameters:
> >        local v="$@"
> > It fails with something like:
> >        local: 20: \/: bad variable name
>
> This is expected behaviour.

Hmm...  Why is it so?  Because it's the only "natural" expectation?

> local is just like any other utility

Utility?  Not built-in?  Documentation refers to it as command.  Same
thing?

> in how it handled parameters.  So "$@" will have been split before
> it reaches local.  To get what you want, you should use "$*".

But wait a second.  Shouldn't quotes '"' around $@ protect the whole $@?
It does so in different other context.  Why not here?

> > I've tried this on dash 0.5.5.1 and 0.5.3 and both fail in the same way.
> > I've also tried on bash and ash (Busybox) and both handles declaration
> > and assignment on one line.
>
> Bash and older versions of dash (like the one in Busybox) has a hack

Hack.  Is that a hack?  It is not expected behaviour.  It's a hack?
Even newer/recent/latest versions of (busybox) ash have that "hack".
And dash claims to be the only POSIX "true" implementation?

> that disables field splitting for local and certain other
> commands.  This is not guaranteed by POSIX.

Sure.  That's what it looks like.  POSIX doesn't even mention it, does it?
Still.  Quotes around $@ should protect it, shouldn't it?

	local v="$@"

should expand to:

	v="'arg1' 'arg2' ..."

shouldn't it?

There's som parallel unnatural behaviour I noticed:

	local var=$(some forked thing here)

$(...)
^^   ^ doesn't seem to protect the resulting string. There seems to be a
difference between:

	local h=$(grep --help)

and:

	local h="$(grep --help)"

Can anything motivate that?  Is that POSIX too?


Cheers,

-- 
Cristian

Re: Parameter expansion fails when assigning a local variable

[Herbert Xu]

On Sun, Jun 28, 2009 at 11:39:16PM +0200, Cristian Ionescu-Idbohrn wrote:
>
> But wait a second.  Shouldn't quotes '"' around $@ protect the whole $@?
> It does so in different other context.  Why not here?

The whole point of $@ is to retain field splitting within double
quotes:

2.5.2      Special Parameters
	   Listed below are the special parameters and the values to
	   which they shall expand. Only the values of the special
	   parameters are listed; see Section 2.6 (on page 2238) for
	   a detailed summary
	   of all the stages involved in expanding words.
	   @    Expands to the positional parameters, starting from
		one. When the expansion occurs within double-quotes,
		and where field splitting (see Section 2.6.5 (on
		page 2244)) is performed, each positional parameter
		shall expand as a separate field, with the provision
		that the expansion of the first parameter shall
		still be joined with the beginning part of the
		original word (assuming that the expanded parameter
		was embedded within a word), and the expansion of
		the last parameter shall still be joined with the
		last part of the original word. If there are no
		positional parameters, the expansion of '@' shall
		generate zero fields, even when '@' is double-quoted.

> There's som parallel unnatural behaviour I noticed:
> 
> 	local var=$(some forked thing here)
> 
> $(...)
> ^^   ^ doesn't seem to protect the resulting string. There seems to be a
> difference between:
> 
> 	local h=$(grep --help)
> 
> and:
> 
> 	local h="$(grep --help)"
> 
> Can anything motivate that?  Is that POSIX too?

As I said local is identical to any other utility in this respect,
including shell functions.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Parameter expansion fails when assigning a local variable

[Cristian Ionescu-Idbohrn]

On Mon, 29 Jun 2009, Herbert Xu wrote:

> On Sun, Jun 28, 2009 at 11:39:16PM +0200, Cristian Ionescu-Idbohrn wrote:
> >
> > But wait a second.  Shouldn't quotes '"' around $@ protect the whole
> > $@? It does so in different other context.  Why not here?
>
> The whole point of $@ is to retain field splitting within double
> quotes:

Sure.  Quoted, like "$@".  I've no problem with that.

> > There's some parallel unnatural behaviour I noticed:
> >
> > 	local var=$(some forked thing here)
> >
> > $(...)
> > ^^   ^ doesn't seem to protect the resulting string. There seems to
> > be a difference between:
> >
> > 	local h=$(grep --help)
> >
> > and:
> >
> > 	local h="$(grep --help)"
> >
> > Can anything motivate that?  Is that POSIX too?
>
> As I said local is identical to any other utility in this respect,
> including shell functions.

Ok.  You're telling us how it's implemented in dash.  Other shells do it
different.  Are you implying dash is the only shell which does that thing
right?  Even though it is unnatural?

Let's try something else.  busybox ash and bash are doing the right thing.
dash and zsh got it wrong.  What do people think?

Please consider this example script:

---8<---
#!/bin/dash

set -e
#set -x

func() {
	local i=0 v xx="$@"

	for v; do
		i=$(($i + 1))
		echo "func arg. $i: '$v'"
	done
	echo "inside func: NOONE_SHOULD_TOUCH_THIS=$NOONE_SHOULD_TOUCH_THIS"
}

NOONE_SHOULD_TOUCH_THIS=protected
echo "main, bef. func: NOONE_SHOULD_TOUCH_THIS=$NOONE_SHOULD_TOUCH_THIS"
func "$@"
echo "main, aft. func: NOONE_SHOULD_TOUCH_THIS=$NOONE_SHOULD_TOUCH_THIS"

exit 0
---8<---

and run the script like this:

	$ <script path> abc NOONE_SHOULD_TOUCH_THIS=overwritten

Is it the intended behaviour?  Should one be able to overwrite script
function variables from command line?


Cheers,

-- 
Cristian

Re: Parameter expansion fails when assigning a local variable

[Herbert Xu]

On Mon, Jun 29, 2009 at 07:10:29PM +0200, Cristian Ionescu-Idbohrn wrote:
>
> Please consider this example script:
> 
> ---8<---
> #!/bin/dash
> 
> set -e
> #set -x
> 
> func() {
> 	local i=0 v xx="$@"
> 
> 	for v; do
> 		i=$(($i + 1))
> 		echo "func arg. $i: '$v'"
> 	done
> 	echo "inside func: NOONE_SHOULD_TOUCH_THIS=$NOONE_SHOULD_TOUCH_THIS"
> }
> 
> NOONE_SHOULD_TOUCH_THIS=protected
> echo "main, bef. func: NOONE_SHOULD_TOUCH_THIS=$NOONE_SHOULD_TOUCH_THIS"
> func "$@"
> echo "main, aft. func: NOONE_SHOULD_TOUCH_THIS=$NOONE_SHOULD_TOUCH_THIS"
> 
> exit 0
> ---8<---
> 
> and run the script like this:
> 
> 	$ <script path> abc NOONE_SHOULD_TOUCH_THIS=overwritten
> 
> Is it the intended behaviour?  Should one be able to overwrite script
> function variables from command line?

As I said, you should be using "$*" instead of "$@" which is
designed exactly for this purpose and is completely portable.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Parameter expansion fails when assigning a local variable

[Cristian Ionescu-Idbohrn]

On Tue, 30 Jun 2009, Herbert Xu wrote:

> On Mon, Jun 29, 2009 at 07:10:29PM +0200, Cristian Ionescu-Idbohrn wrote:
> >
> > Please consider this example script:
> >
> > ---8<---
> > #!/bin/dash
> >
> > set -e
> > #set -x
> >
> > func() {
> > 	local i=0 v xx="$@"
> >
> > 	for v; do
> > 		i=$(($i + 1))
> > 		echo "func arg. $i: '$v'"
> > 	done
> > 	echo "inside func: NOONE_SHOULD_TOUCH_THIS=$NOONE_SHOULD_TOUCH_THIS"
> > }
> >
> > NOONE_SHOULD_TOUCH_THIS=protected
> > echo "main, bef. func: NOONE_SHOULD_TOUCH_THIS=$NOONE_SHOULD_TOUCH_THIS"
> > func "$@"
> > echo "main, aft. func: NOONE_SHOULD_TOUCH_THIS=$NOONE_SHOULD_TOUCH_THIS"
> >
> > exit 0
> > ---8<---
> >
> > and run the script like this:
> >
> > 	$ <script path> abc NOONE_SHOULD_TOUCH_THIS=overwritten
> >
> > Is it the intended behaviour?  Should one be able to overwrite script
> > function variables from command line?
>
> As I said, you should be using "$*" instead of "$@" which is
> designed exactly for this purpose and is completely portable.

Sure I will.  But that won't make the misfeature go away :(

And, by the way, I think the bug should probably be classified as a design
flaw and a security bug.  Reason is it has undesired side effects which
may lead to overwriting unrelated local variables and may have other
serious implications.


Cheers,

-- 
Cristian

Re: Parameter expansion fails when assigning a local variable

[Oleg Verych]

Hallo.

2009/6/30, Cristian Ionescu-Idbohrn <cristian.ionescu-idbohrn@axis.com>:
[...]
> Sure I will.  But that won't make the misfeature go away :(
>
> And, by the way, I think the bug should probably be classified as a design
> flaw and a security bug.  Reason is it has undesired side effects which
> may lead to overwriting unrelated local variables and may have other
> serious implications.

Before you, guys, write such things, you'd better read something heavy
about /bin/shELL.

Please:

UNIX SHELL Quote Tutorial
Written by Bruce Barnett
http://www.grymoire.com/Unix/Quote.html

Sh - the Bourne Shell
http://www.grymoire.com/Unix/Sh.html

Thanks.
-- 
sed 'sed && sh + olecom = love'  <<  ''
-o--=O`C
 #oo'L O
<___=E M

Re: Parameter expansion fails when assigning a local variable

[Cristian Ionescu-Idbohrn]

On Wed, 1 Jul 2009, Oleg Verych wrote:

> Before you, guys, write such things, you'd better read something heavy
> about /bin/shELL.
>
> Please:
>
> UNIX SHELL Quote Tutorial
> Written by Bruce Barnett
> http://www.grymoire.com/Unix/Quote.html
>
> Sh - the Bourne Shell
> http://www.grymoire.com/Unix/Sh.html

Is this just good advice (thanks for that, by the way)?
Any particular part of the tutorial, you, advertise that offers some sort
of recipe on to how to solve the problem at hand (read it but did not
find anything relevant referring to this particular dash bug)?

> Thanks.

You're welcome.


Cheers,

-- 
Cristian

Re: Parameter expansion fails when assigning a local variable

[Oleg Verych]

2009/7/2 Cristian Ionescu-Idbohrn
[...]
>> UNIX SHELL Quote Tutorial
>> Written by Bruce Barnett
>> http://www.grymoire.com/Unix/Quote.html
>>
>> Sh - the Bourne Shell
>> http://www.grymoire.com/Unix/Sh.html
>
> Is this just good advice (thanks for that, by the way)?

You're welcome. But somehow it doesn't work. You have problem of not
understanding, how shell works. This is not a bug in dash.

> Any particular part of the tutorial, you, advertise that offers some sort
> of recipe on to how to solve the problem at hand (read it but did not
> find anything relevant referring to this particular dash bug)?

<http://www.grymoire.com/Unix/Sh.html>:
(optional)* Shell basics
(optional)* Meta-characters and Filename expansion
* Finding the executable
* Quoting with the Bourne Shell
(optional)* Nested quotations
(optional)* Strong versus weak quoting
(optional)* Quoting over several lines
(optional)* Mixing quotation marks
(optional)* Quotes within quotes - take two
(optional)* Placing variables within strings
* Variables
* A subtle point

"Whitespace (spaces, tabs or newlines) terminate the value." Please,
do not confuse 'whitespace' on input (script itself) with 'whitespace'
in the variable. For shell this

>        v="$@"

must be the same as
v=$*
v="$*"
v=$@
because whitespace inside parameters of $* or $@ does not "terminate
the value". Thus no need of quoting of anything, like in case of usual
command processing (* Finding the executable).

So why do you need `v=$*` anyway? Do you understand, that "$@" is
useful only in parameter list *for* a command ('executable' in the
tutorial), so it gets all parameters whitespace-untouched by the
shell?

> I've found a problem with expanding $@.
> If I declare a local variable and assigns the expanded positional parameters:
>        local v="$@"

Here *is* such case -- a command gets executed with a parameter list
handled by the shell, so goto "* Finding the executable", please, to
find out once again, what it does there.

Finally, find some info about why some shells, but not classic
`/bin/sh` or dash, have `[[` command with some special subtle
functionality, altering "* Finding the executable". Maybe this will
make the point clear.
-- 
sed 'sed && sh + olecom = love'  <<  ''
-o--=O`C
 #oo'L O
<___=E M