t0008 test fails with ksh

t0008 test fails with ksh

[Armin Kunaschik]

Hello,

in t0008 I see tests fails with
not ok 374 - --stdin -v
#
#               expect_from_stdin <expected-verbose &&
#               test_check_ignore "-v --stdin" <stdin
#
[....]
not ok 381 - --stdin from subdirectory
#
#               expect_from_stdin <expected-default &&
#               (
#                       cd a &&
#                       test_check_ignore "--stdin" <../stdin
#               )
#
not ok 382 - --stdin from subdirectory with -v
#
#               expect_from_stdin <expected-verbose &&
#               (
#                       cd a &&
#                       test_check_ignore "--stdin -v" <../stdin
#               )
#
not ok 383 - --stdin from subdirectory with -v -n
#
#               expect_from_stdin <expected-all &&
#               (
#                       cd a &&
#                       test_check_ignore "--stdin -v -n" <../stdin
#               )
#

The reason seems that the snippet
cat <<-EOF >expected-all
        .gitignore:1:one        ../one
        ::      ../not-ignored
        .gitignore:1:one        one
        ::      not-ignored
        a/b/.gitignore:8:!on*   b/on
        a/b/.gitignore:8:!on*   b/one
        a/b/.gitignore:8:!on*   b/one one
        a/b/.gitignore:8:!on*   b/one two
        a/b/.gitignore:8:!on*   "b/one\"three"
        a/b/.gitignore:9:!two   b/two
        ::      b/not-ignored
        a/.gitignore:1:two*     b/twooo
        $global_excludes:2:!globaltwo   ../globaltwo
        $global_excludes:2:!globaltwo   globaltwo
        $global_excludes:2:!globaltwo   b/globaltwo
        $global_excludes:2:!globaltwo   ../b/globaltwo
        ::      c/not-ignored
EOF

behaves differently in bash and in ksh.
        a/b/.gitignore:8:!on*   "b/one\"three" comes out unmodified
with bash but with ksh it becomes
        a/b/.gitignore:8:!on*   "b/one"three"
I'm not sure what shell is "wrong" here, but when I modify the line to
        a/b/.gitignore:8:!on*   "b/one\\"three"
both shells generate the "right" output:
        a/b/.gitignore:8:!on*   "b/one\"three"

>From what I've learned I'd expect the double backslash to be the
"right" way to escape one
backslash. Do you agree or am I wrong?

This snippet appears twice in this test, generates expected-all and
expected-verbose.

Armin

Re: t0008 test fails with ksh

[Jeff King]

On Thu, May 12, 2016 at 07:53:28PM +0200, Armin Kunaschik wrote:

> The reason seems that the snippet
> cat <<-EOF >expected-all
>         .gitignore:1:one        ../one
>         ::      ../not-ignored
>         .gitignore:1:one        one
>         ::      not-ignored
>         a/b/.gitignore:8:!on*   b/on
>         a/b/.gitignore:8:!on*   b/one
>         a/b/.gitignore:8:!on*   b/one one
>         a/b/.gitignore:8:!on*   b/one two
>         a/b/.gitignore:8:!on*   "b/one\"three"
>         a/b/.gitignore:9:!two   b/two
>         ::      b/not-ignored
>         a/.gitignore:1:two*     b/twooo
>         $global_excludes:2:!globaltwo   ../globaltwo
>         $global_excludes:2:!globaltwo   globaltwo
>         $global_excludes:2:!globaltwo   b/globaltwo
>         $global_excludes:2:!globaltwo   ../b/globaltwo
>         ::      c/not-ignored
> EOF
> 
> behaves differently in bash and in ksh.
>         a/b/.gitignore:8:!on*   "b/one\"three" comes out unmodified
> with bash but with ksh it becomes
>         a/b/.gitignore:8:!on*   "b/one"three"
> I'm not sure what shell is "wrong" here, but when I modify the line to
>         a/b/.gitignore:8:!on*   "b/one\\"three"
> both shells generate the "right" output:
>         a/b/.gitignore:8:!on*   "b/one\"three"
> 
> From what I've learned I'd expect the double backslash to be the
> "right" way to escape one
> backslash. Do you agree or am I wrong?
> 
> This snippet appears twice in this test, generates expected-all and
> expected-verbose.

I think either is reasonable (there is no need to backslash-escape a
double-quote inside a here-doc, but one assumes that backslash would
generally have its usual behavior). I'm not quite sure how to interpret
POSIX here (see below), but it seems clear that spelling it with two
backslashes as you suggest is the best bet.

For the curious, here's what POSIX says (in the section on
here-documents; "word" here refers to the EOF word):

  If no characters in word are quoted, all lines of the here-document
  shall be expanded for parameter expansion, command substitution, and
  arithmetic expansion. In this case, the backslash in the input behaves
  as the backslash inside double-quotes (see Double-Quotes). However,
  the double-quote character ( '"' ) shall not be treated specially
  within a here-document, except when the double-quote appears within
  "$()", "``", or "${}".

So OK, that sounds like ksh is doing the right thing. But what's that
"specially" in the last sentence? Do they just mean it doesn't start a
quoted section as it would elsewhere? Or do they mean in the section on
Double-Quotes, when they say:

  \
      The backslash shall retain its special meaning as an escape
      character (see Escape Character (Backslash)) only when followed by
      one of the following characters when considered special:

              $   `   "   \   <newline>

Since the double-quote isn't special here, does that mean that backslash
shouldn't retain its special meaning in this case?  That would make bash
right.

Anyway, it doesn't really matter what the standard says. We can spell
this in a less ambiguous way, and it does not hurt too much to do so.

-Peff

Re: t0008 test fails with ksh

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

>   ... However,
>   the double-quote character ( '"' ) shall not be treated specially
>   within a here-document, except when the double-quote appears within
>   "$()", "``", or "${}".
>
> So OK, that sounds like ksh is doing the right thing. But what's that
> "specially" in the last sentence?

I would say: Just like \X is passed thru as-is without losing \, \"
is passed thru without losing \, because " is not special, just like
X is not special.

> Anyway, it doesn't really matter what the standard says. We can spell
> this in a less ambiguous way, and it does not hurt too much to do so.

Yes.

Re: t0008 test fails with ksh

[Jeff King]

On Fri, May 20, 2016 at 09:03:31AM -0700, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> >   ... However,
> >   the double-quote character ( '"' ) shall not be treated specially
> >   within a here-document, except when the double-quote appears within
> >   "$()", "``", or "${}".
> >
> > So OK, that sounds like ksh is doing the right thing. But what's that
> > "specially" in the last sentence?
> 
> I would say: Just like \X is passed thru as-is without losing \, \"
> is passed thru without losing \, because " is not special, just like
> X is not special.

Yeah, that was my reading, too (sorry, my question was mostly
rhetorical; imagine me reading it in an exaggerated surprised "woah,
what's this?" voice. :) ).

-Peff