Quoted closing brace in variable default expansion

Quoted closing brace in variable default expansion

[Harald van Dijk]

Hi all,

It's probably best to start with the example:

   $ bash -c 'echo "${x:-"}"}"'
   }
   $ ksh -c 'echo "${x:-"}"}"'
   }
   $ dash -c 'echo "${x:-"}"}"'
   dash: 1: Syntax error: Unterminated quoted string
   $ busybox sh -c 'echo "${x:-"}"}"'
   sh: syntax error: unterminated quoted string

It looks like dash and other ash derivatives stop the expansion with the 
first }, instead of the first unquoted }. I'm getting confused trying to 
figure out whether this is a bug in dash or in the script relying on it.

A slightly modified example:

   $ dash -c 'echo "${x-"_{x}_"}"'
   dash: 1: Syntax error: Unterminated quoted string
   $ dash -c 'echo ${x-"_{x}_"}'
   _{x_}

I don't understand how the last line works. The } is printed after the 
_, so the } that is printed must be the final }, and the } directly 
following the x stops the variable default. But this would lead to two 
double quote characters which have no way of matching up, yet no syntax 
error is given.

Could you clarify?

Cheers,
Harald

Re: Quoted closing brace in variable default expansion

[Jonathan Nieder]

Hi Herald,

Harald van Dijk wrote:

>   $ ksh -c 'echo "${x:-"}"}"'
>   }
>   $ dash -c 'echo "${x:-"}"}"'
>   dash: 1: Syntax error: Unterminated quoted string
>   $ busybox sh -c 'echo "${x:-"}"}"'
>   sh: syntax error: unterminated quoted string
> 
> It looks like dash and other ash derivatives stop the expansion with
> the first }, instead of the first unquoted }. I'm getting confused
> trying to figure out whether this is a bug in dash or in the script
> relying on it.

The answer depends on how portable the script is meant to be.  If
the goal is to be portable to shells implementing future versions of
the POSIX standard, there seems to be have been an interpretation[1]
approved for the next major revision:

 http://austingroupbugs.net/view.php?id=221
 note #399

which would make the example nonconformant because there are an odd
number of unescaped double-quotes before the first unescaped closing
brace.

See http://thread.gmane.org/gmane.comp.shells.dash/262/focus=263 for
a nice summary (thanks, Jilles!).

Hope that helps,
Jonathan

Re: Quoted closing brace in variable default expansion

[Harald van Dijk]

On 13/11/10 17:41, Jonathan Nieder wrote:
> Harald van Dijk wrote:
>>    $ dash -c 'echo "${x:-"}"}"'
>>    dash: 1: Syntax error: Unterminated quoted string
>
>   http://austingroupbugs.net/view.php?id=221
>   note #399

Thanks, that helps to clarify. Unfortunately, the valid alternative 
given there:

   sh -c 'echo "${x:-\}}"'

seemingly incorrectly prints \} with dash and with bash, so that is a 
non-option at this time for me.

Re: Quoted closing brace in variable default expansion

[Jilles Tjoelker]

On Sat, Nov 13, 2010 at 10:41:47AM -0600, Jonathan Nieder wrote:
> Harald van Dijk wrote:

> >   $ ksh -c 'echo "${x:-"}"}"'
> >   }
> >   $ dash -c 'echo "${x:-"}"}"'
> >   dash: 1: Syntax error: Unterminated quoted string
> >   $ busybox sh -c 'echo "${x:-"}"}"'
> >   sh: syntax error: unterminated quoted string

> > It looks like dash and other ash derivatives stop the expansion with
> > the first }, instead of the first unquoted }. I'm getting confused
> > trying to figure out whether this is a bug in dash or in the script
> > relying on it.

> The answer depends on how portable the script is meant to be.  If
> the goal is to be portable to shells implementing future versions of
> the POSIX standard, there seems to be have been an interpretation[1]
> approved for the next major revision:

>  http://austingroupbugs.net/view.php?id=221
>  note #399

> which would make the example nonconformant because there are an odd
> number of unescaped double-quotes before the first unescaped closing
> brace.

Correct.

> See http://thread.gmane.org/gmane.comp.shells.dash/262/focus=263 for
> a nice summary (thanks, Jilles!).

Note that I have changed FreeBSD 9 sh since I wrote that. To preserve
sanity in expand.c, a substitution now ends with the same quoting level
as it started. Closing braces that do not conform to this are taken
literally. Making the parole configure script do what the author
expected was a bonus.

The change was needed to implement word splitting in WORD in ${v+WORD}.
For that, expand.c needs to know the quoted state of any character
of a word in ${v+-word}. Like NetBSD sh and dash have done, I added a
new magic character CTLQUOTEEND that shows the end of the quoted part
and interpreted CTLQUOTEMARK as the start. If I now have something like
  ${v-abc${vv-def"ghi}jkl"}
I would have to pass through the fact that the quotes were opened in
${vv-def"ghi} to the outer substitution, and I did not want that. The
change avoids it.

FreeBSD 9 sh is now very close to ksh93 in treatment of POSIX
substitutions and I am happy with this. (One of the deliberate
differences is that the above incorrect construct is flagged as an error
(missing '}') while ksh93 treats it as an incomplete command.)

As for the portable alternative, split up the command. Either put the
'}'-containing thing in a variable and use that as alternative or use an
explicit conditional. The Autoconf documentation has more information,
for example.

-- 
Jilles Tjoelker

Re: Quoted closing brace in variable default expansion

[Herbert Xu]

Harald van Dijk <harald@gigawatt.nl> wrote:
>
> Thanks, that helps to clarify. Unfortunately, the valid alternative 
> given there:
> 
>   sh -c 'echo "${x:-\}}"'
> 
> seemingly incorrectly prints \} with dash and with bash, so that is a 
> non-option at this time for me.

If you need to print anything involving a backspace you should
use printf and not echo.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Quoted closing brace in variable default expansion

[Harald van Dijk]

On 19/11/10 14:53, Herbert Xu wrote:
> Harald van Dijk<harald@gigawatt.nl>  wrote:
>>    sh -c 'echo "${x:-\}}"'
>
> If you need to print anything involving a backspace you should
> use printf and not echo.

It's true that backslashes should not be passed to echo, but assuming 
unset or empty x, "${x:-\}}" should expand to "}", and that is all echo 
should see.

Cheers,
Harald

Re: Quoted closing brace in variable default expansion

[Cristian Ionescu-Idbohrn]

On Sat, 20 Nov 2010, Harald van Dijk wrote:
> On 19/11/10 14:53, Herbert Xu wrote:
> > Harald van Dijk<harald@gigawatt.nl>  wrote:
> > >    sh -c 'echo "${x:-\}}"'
> >
> > If you need to print anything involving a backspace you should
> > use printf and not echo.
>
> It's true that backslashes should not be passed to echo, but assuming
> unset or empty x, "${x:-\}}" should expand to "}", and that is all echo
> should see.

You seem to have a point there:

,----[ http://opengroup.org/onlinepubs/007908799/xcu/chap2.html ]
| single-quotes cannot be used to quote the "}" within ${...}; for
| example:
|
| unset bar
| foo="${bar-'}'}"
|
| is invalid because the ${...} substitution contains an unpaired
| unescaped single-quote. The backslash can be used to escape the "}" in
| this example to achieve the desired result:
|
| unset bar
| foo="${bar-\}}"
`----


Cheers,

-- 
Cristian

[PATCH] [PARSER] Remove backslash before } in double-quotes in variable

[Jilles Tjoelker]

The backslash prevents the closing brace from terminating the
substitution, therefore it should be removed.

FreeBSD sh test expansion/plus-minus2.0 starts working, no other tests
are affected.

Example:
  printf "%s\n" ${$+\}} ${$+"\}"} "${$+\}}"
should print } three times, without backslashes.
---
 src/parser.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/src/parser.c b/src/parser.c
index be20ff7..25f22fd 100644
--- a/src/parser.c
+++ b/src/parser.c
@@ -916,6 +916,9 @@ readtoken1(int firstc, char const *syntax, char *eofmark, int striptabs)
 						c != '$' && (
 							c != '"' ||
 							eofmark != NULL
+						) && (
+							c != '}' ||
+							varnest == 0
 						)
 					) {
 						USTPUTC('\\', out);
-- 
1.7.3.2

Re: [PATCH] [PARSER] Remove backslash before } in double-quotes in variable

[Jonathan Nieder]

Hi,

Jilles Tjoelker wrote:

> The backslash prevents the closing brace from terminating the
> substitution, therefore it should be removed.

Thanks!

>   printf "%s\n" ${$+\}} ${$+"\}"} "${$+\}}"
> should print } three times, without backslashes.

Thanks especially for the example.  The standard makes these all
clear, even the second one (which motivates the check against varnest
rather than dqvarnest).  For what it's worth, I like the patch.

Regards,
Jonathan

Re: [PATCH] [PARSER] Remove backslash before } in double-quotes in variable

[Harald van Dijk]

On 21/11/10 14:41, Jilles Tjoelker wrote:
> The backslash prevents the closing brace from terminating the
> substitution, therefore it should be removed.

Thanks, that works great.

> Example:
>    printf "%s\n" ${$+\}} ${$+"\}"} "${$+\}}"
> should print } three times, without backslashes.

Some tests with different a few shells:

                                                     "${$+'\}'}"
                                              "${$+"\}"}"
                                       "${$+\}}"
                                ${$+'\}'}
                         ${$+"\}"}
                  ${$+\}}

bash           | }      \}     \}     \}     }      '\}'
pdksh          | }      \}     \}     }      \}     \}
ksh            | }      }      \}     }      }      '}'
dash (current) | }      \}     \}     \}     \}     '\}'
dash (patched) | }      }      \}     }      }      '}'

The one thing that I was a bit confused about was the difference between 
${$+'\}'} and ${$+"\}"}, but I do think that is correct: the backslash 
is used to see that the immediately following } is not a terminator, but 
the special rule allowing backslash before } to be removed in 
double-quoted strings in parameter expansions does not apply (there is 
no double-quoted string), so the backslash appears in the output as 
well. And since the last form uses a double-quoted string, the backslash 
gets removed there.

Cheers,
Harald

Re: [PATCH] [PARSER] Remove backslash before } in double-quotes in variable

[Herbert Xu]

On Sun, Nov 21, 2010 at 02:42:22PM +0100, Jilles Tjoelker wrote:
> The backslash prevents the closing brace from terminating the
> substitution, therefore it should be removed.
> 
> FreeBSD sh test expansion/plus-minus2.0 starts working, no other tests
> are affected.
> 
> Example:
>   printf "%s\n" ${$+\}} ${$+"\}"} "${$+\}}"
> should print } three times, without backslashes.

I agree with respect to the last one, but not the middle one.

${$+"\}"} should be identical in behaviour to "\}", which with
all shells I have access to produces a brace with a backslash.

So please update your patch so that we do not regress on the
second one.

Thanks,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH] [PARSER] Remove backslash before } in double-quotes in variable

[Jilles Tjoelker]

On Thu, Mar 10, 2011 at 04:59:27PM +0800, Herbert Xu wrote:
> On Sun, Nov 21, 2010 at 02:42:22PM +0100, Jilles Tjoelker wrote:
> > The backslash prevents the closing brace from terminating the
> > substitution, therefore it should be removed.
> > 
> > FreeBSD sh test expansion/plus-minus2.0 starts working, no other tests
> > are affected.
> > 
> > Example:
> >   printf "%s\n" ${$+\}} ${$+"\}"} "${$+\}}"
> > should print } three times, without backslashes.

> I agree with respect to the last one, but not the middle one.

> ${$+"\}"} should be identical in behaviour to "\}", which with
> all shells I have access to produces a brace with a backslash.

> So please update your patch so that we do not regress on the
> second one.

Meh. Now that I read http://austingroupbugs.net/view.php?id=221#c399
again, that particular bit is not so clear anymore. :(
That change only seems to affect parameter expansions inside
double-quotes, not double-quotes inside parameter expansions.

On the other hand, removing the backslash makes some sense because
${$+"}"} is unspecified (even though it expands to } in many shells). In
dash in particular it has an undesirable effect. When I wrote the code
to remove the backslash in FreeBSD sh, the first } ended the
substitution which as a result ended double-quoted although it started
unquoted; I changed this later so that the first } is literal, but did
not change the treatment of \} in that case.

My patch for dash goes further than what I did in FreeBSD in that it
also affects the four varieties of parameter expansion for substring
processing (#/##/%/%%). I think that is wrong and the FreeBSD behaviour
is correct. In the standard as modified by the aforementioned
interpretation, ${v#"}"} and ${v#'}'} seem valid (the pattern is a
literal closing brace). Therefore there seems no reason to remove the
backslash in ${v#"\}"}, even though ksh93 does it.

Distinguishing #/##/%/%% from the others at parse time is not possible
with the way dash's parser.c currently works. So a simple patch would
attempt to change \} inside "${...}" only (test dqvarnest instead of
varnest). I think that does not affect the end result for #/##/%/%% so
perhaps it can be committed as an small improvement.

At some point a larger change is probably necessary to fix things like
"${v#'a'}" which should trim the letter a, not single-quote a
single-quote. This works in FreeBSD 9 sh but I think its
Bourne/Korn-like approach to POSIX-unspecified constructs like
"${v+"what*is*this"}", which generates pathnames, just like the original
ash and older FreeBSD code, may not be appreciated in dash.

Some examples:

FreeBSD 9 sh does this:

% sh
$ v='a\}b'
$ echo ${v%"\}"*}
a
$ echo ${v+"\}"}
}

while dash with my original patch and ksh93 do this:

% ksh93
$ v='a\}b'
$ echo ${v%"\}"*}
a\
$ echo ${v+"\}"}
}

-- 
Jilles Tjoelker