local var=$(cat) only reads one line

local var=$(cat) only reads one line

[Jim Pryor]

Hi, I'm not a subscriber to this list and am not sure where's the best
place to report this bug. A cursory search didn't find other reports of
the same issue; but I can imagine there may well be such. If so, excuse
the noise. I'm just trying to help.

I notice the following issue in the version of dash that's bundled in a
recent release of finnix (not sure which one, but the kernel version is
3.0.6, so it's probably finnix 103, released 23 October 2011). I also
see it in the FreeBSD sh, from a FreeBSD-9 release candidate I compiled
in January. I know that's not dash, but I understand the codebases are
closely related. Neither of these are my active systems; hence the fuzzy
details.

Here is a testcase:

#!/bin/dash

test1() {
    local IN=$(cat)
    printf "test1 <%s>\n" "$IN"
}

test2() {
    local IN="$(cat)"
    printf "test2 <%s>\n" "$IN"
}

test3() {
    IN=$(cat)
    printf "test3 <%s>\n" "$IN"
}

test4() {
    IN=$(cat)
    printf "test4 <%s>\n" "$IN"
}

MSG=$(printf "abc\ndef\nghi")

printf "%s" "$MSG" | test1
printf "%s" "$MSG" | test2
printf "%s" "$MSG" | test3
unset IN
printf "%s" "$MSG" | test4

#######

The weird bit only shows up in test1: IN will only be assigned the first
line of stdin.

Hope this helps someone.
-- 
dubiousjim@gmail.com

Re: local var=$(cat) only reads one line

[Jim Pryor]

On Sun, Apr 8, 2012, at 06:09 PM, Jim Pryor wrote:
> test3() {
>     IN=$(cat)
>     printf "test3 <%s>\n" "$IN"
> }
> 
> test4() {
>     IN=$(cat)
>     printf "test4 <%s>\n" "$IN"
> }

test4 was supposed to use quotes: "$(cat)". It works fine either way.
This also works fine:

test1a() {
    local IN
    IN=$(cat)
    printf "test1a <%s>\n" "$IN"
}

It's only this which fails:

test1a() {
    local IN=$(cat)
    printf "test1 <%s>\n" "$IN"
}

I've also reported to FreeBSD:
<http://www.freebsd.org/cgi/query-pr.cgi?pr=166771>
-- 
dubiousjim@gmail.com

Re: local var=$(cat) only reads one line

[Jim Pryor]

Hi Jilles, thanks for your reply. Indeed now I remember having read
somewhere that you couldn't portably rely on assignments after "local"
to suppress word expansion in the same way that ordinary assignments do.
I had forgotten that. I hope shell implementations evolve towards more
consistency about this. In the meantime, it's easy enough to get
portable behavior if you remember the issue.

On another note, I was reading in this mailing list about dash's
treatment of:
    FOO=value shell_function $args
    printf "%s\n" "$FOO" # prints 'value'

The other discussion points out that this surprising behavior is in fact
mandated by POSIX.

I notice however in the dash.1 manpage distributed with finnix 103, the
following:

     When a shell function is executed, all of the shell positional
     parameters
     (except $0, which remains unchanged) are set to the arguments of
     the
     shell function.  The variables which are explicitly placed in the
     envi-
     ronment of the command (by placing assignments to them before the
     func-
     tion name) ***are made local to the function and are set to the
     values
     given.***  Then the command given in the function definition is
     executed.
     The positional parameters are restored to their original values
     when the
     command completes.  This all occurs within the current shell.

Presumably this manpage should be updated to track the current behavior
of dash. I'm not sure whether it comes from the dash source distro, or
was added in by the finnix maintainers.

-- 
dubiousjim@gmail.com

Re: local var=$(cat) only reads one line

[Jilles Tjoelker]

On Sun, Apr 08, 2012 at 06:09:06PM -0400, Jim Pryor wrote:
> Hi, I'm not a subscriber to this list and am not sure where's the best
> place to report this bug. A cursory search didn't find other reports
> of the same issue; but I can imagine there may well be such. If so,
> excuse the noise. I'm just trying to help.

> I notice the following issue in the version of dash that's bundled in
> a recent release of finnix (not sure which one, but the kernel version
> is 3.0.6, so it's probably finnix 103, released 23 October 2011). I
> also see it in the FreeBSD sh, from a FreeBSD-9 release candidate I
> compiled in January. I know that's not dash, but I understand the
> codebases are closely related. Neither of these are my active systems;
> hence the fuzzy details.

> Here is a testcase:

> test1() {
>     local IN=$(cat)
>     printf "test1 <%s>\n" "$IN"
> }

> MSG=$(printf "abc\ndef\nghi")

> printf "%s" "$MSG" | test1

> The weird bit only shows up in test1: IN will only be assigned the first
> line of stdin.

The cause is that the local command expands to
  local IN=abc def ghi
and therefore it sets IN to less than what you expect and also makes two
more variables local.

Now, local is not in POSIX, but export and readonly are and have exactly
the same issue.

Reading POSIX.1-2008 (and also older versions) literally, this way of
splitting is correct. However, ksh93, bash and pdksh think differently,
doing what your script expects. Also, the examples section of export has
  export PATH=/local/bin:$PATH
which assumes that either the value will not contain IFS characters or
metacharacters or it will not be split regardless.

Dash had this behaviour for some time but it was dropped again because
of inconsistencies between implementations.

However, there appears to be some sort of agreement on how it should
work now:
http://austingroupbugs.net/view.php?id=351
(scroll down to note 0000943).

-- 
Jilles Tjoelker