Feature request: set -o pipefail

Feature request: set -o pipefail

[Peter Jakobi]

Hi,

[Note:  I'm not on the list, so put jakobi@acm.org in cc: if I need to
reply. thanx, Peter]

Feature request:

implement  set  -o pipefail. This will allow the user to  request  the
return  code  to be the one of the first pipe command with a  non-zero
error. 

The  lack  of  which  can be quite painful in  scripting,  even  if  a
bourne-shell/posix subset would otherwise be quite sufficient.


Advantages:

This  allows use of pipes with proper error detection, which is  quite
helpful  for  init  scripts / cron or  generally  administration-style
scripts which exec shell commands from shell/perl/python/... scripts.


Usage case: 

have  a  command  like mv -i act on user input, while  keeping  output
(stdout+err) in a log AND returning mv's error code (or tee's if mv is
ok). in a perl script, this would be something like:

   system "set -o pipefail; mv -i foo bar </dev/tty 2>&1 | tee -a LOG";

Part  of  the problem is that the shell used for open/system is  often
hard-coded in the interpreter to be /bin/sh.


Work aound:

   1. as root: 

   restore /bin/sh to point to a more complete shell of bourne-ancestry.
   With the subsequent larger set of library dependencies, ...

   or

   2. non-root:

   first detect a suitable shell:
   # detect likely shells in path and /etc/shells
   # for each shell, try $tmp=`set -o pipefail 2>&1`
   # choose one that doesn't set a non-zero errror in $?.
   # if all else fails, try defaulting to bash or ksh

   system $shell, "-c", 
          'set -o pipefail; mv -i "$@" </dev/tty 2>&1 | tee -a LOG', 
          "world's most painful trivial system command",
          "foo", "bar";

btw,  wrappers  like script might help, besides being a pain  wrt  LOG
management.  But there's the slight of script -c false returning true.
Which makes e.g. script impossible to use.

-- 
cu
Peter
jakobi@acm.org

Re: Feature request: set -o pipefail

[Eric Blake]

According to Peter Jakobi on 8/7/2009 7:18 AM:
> implement  set  -o pipefail. This will allow the user to  request  the
> return  code  to be the one of the first pipe command with a  non-zero
> error. 

That just seems like bloat to me.  Not to mention that it can lead to
surprising results due to SIGPIPE (that is, there are cases where 'command
| head' should be considered successful, even though command exited from
SIGPIPE, where using bash's 'set -o pipefail' will make it look like a
failure).

> The  lack  of  which  can be quite painful in  scripting,  even  if  a
> bourne-shell/posix subset would otherwise be quite sufficient.

You can generally achieve what you want with just the posix subset, which
will make your code more portable in the long run.  Here's one idea:

>    system "set -o pipefail; mv -i foo bar </dev/tty 2>&1 | tee -a LOG";

system "exec 3>&1; s=$(exec 4>&1 >&3; { mv -i foo bar </dev/tty 2>&1; echo
$? >&4; } | tee -a LOG) && exit $s"

Basically, save the original stdout in fd 3, then use a command
substitution where we save the substitution's stdout in fd 4, restore the
original stdout, and perform the pipe.  Then, by modifying the first
command of the pipe to feed the result of the command substitution, we are
now able to guarantee non-zero status if either (or both) the mv or tee fail.

-- 
Don't work too hard, make some time for fun as well!

Eric Blake             ebb9@byu.net

Re: Feature request: set -o pipefail

[Peter Jakobi]

On Wed, Aug 12, 2009 at 06:51:21AM -0600, Eric Blake wrote:

Rehi Eric & *,

[please Cc if you want me to read it :). thx, Peter]

> According to Peter Jakobi on 8/7/2009 7:18 AM:
> > implement  set  -o pipefail. This will allow the user to  request  the
> > return  code  to be the one of the first pipe command with a  non-zero
> > error. 
> 
> That just seems like bloat to me.  Not to mention that it can lead to
> surprising results due to SIGPIPE (that is, there are cases where 'command
> | head' should be considered successful, even though command exited from
> SIGPIPE, where using bash's 'set -o pipefail' will make it look like a
> failure).

Please  don't assume that I want pipefail to be always on: 

There won't be any surprises at all if it is a 'set -o' option and off
by default.


Wrt  bloat,  I  sincerely hope that my request won't double  the  code
size.  I've  even gladly skip on implementing the return  code  status
array  :). I wouldn't have made the suggestion if I'd assume it to  be
bloat or introduce tons of extra code...


As  for sigpipe, of course, it SHOULD occur. And if pipefail is on, it
should be reported even from non-tail stages of the pipe.

And  it's  easy  enough to ignore, it you know that  your  list  might
contain  processes  that may legally create ignorable SIGPIPE:

   (cat LARGEFILE | head -1;true)  | cat
   cat LARGEFILE | ( perl -0pe : | head -1) | cat 

will do quite nicely. Even dd will do in a pinch:

   # gnu dd *bs options are slightly counter-intuitive: 
   # both bs and one of ibs/obs required)
   cat LARGEFILE | ( dd bs=500M obs=1 | head -1) | cat 


> > The  lack  of  which  can be quite painful in  scripting,  even  if  a
> > bourne-shell/posix subset would otherwise be quite sufficient.
> 
> system "exec 3>&1; s=$(exec 4>&1 >&3; { mv -i foo bar </dev/tty 2>&1; echo
> $? >&4; } | tee -a LOG) && exit $s"

Yes.  Oleg pointed me to the even more complete sample of  pipestatus,
which made for an interesting read (thanx Oleg!).

But  that's exactly why I wrote _painful_ (and even pipestatus isn't a
general workaround).

Consider  larger  system-level  scripts,  init.d-scripts,  daemons  or
scripts e.g. in perl using system().


Given  that  Debian/Ubuntu wants to use dash for /bin/sh,  this  issue
gains  some importance, unless you want me and anyone at larger  shops
to  replace  /bin/sh being dash again with the previously  used  bash.
Which  would  be  a shame, given the push the dash project  gained  by
Debian's recent move.

-- 
cu
Peter
jakobi@acm.org

Re: Feature request: set -o pipefail

[Oleg Verych]

>> system "exec 3>&1; s=$(exec 4>&1 >&3; { mv -i foo bar </dev/tty 2>&1; echo
>> $? >&4; } | tee -a LOG) && exit $s"
>
> Yes.  Oleg pointed me to the even more complete sample of  pipestatus,
> which made for an interesting read (thanx Oleg!).

You are welcome. Links were:

* comp.unix.shell FAQ - Answers to Frequently Asked Questions Part 2.
http://groups.google.com/group/comp.unix.shell/browse_thread/thread/5075fe6c19ddabb9/7a08dffe06316a29?pli=1

* UNIX Power Tools (47.2.1.4 More Elaborate Combinations)
http://unix.org.ua/orelly/unix/upt/ch47_02.htm

[...]
> Consider  larger  system-level  scripts,  init.d-scripts,  daemons  or
> scripts e.g. in perl using system().

Just for ref, some of my pipeless, undebianaized, small, reliable,
informative init.d scripts:

* `pure-ftpd+` replacement for #1 overbloated deb package
* `bind9+` using multiple configs and chroot (hope stuff like
CVE-2009-0696 https://www.isc.org/node/474 does nothing serious,
written before 2009-07-28 btw)
* ssh+ to have multiple sshd servers as simple as creating
sshd_config+$reason files. E.g. to have some loginless dummies for
those dudes, who like to f*ck ssh on the Internet.

== /etc/init.d/pure-ftpd+
#!/bin/sh

set -e

[ "$*" ] || exec echo "
Usage: $0 [&|] {start, stop, who}
"
trap "echo '
Unexpected Script Error! Use \`/bin/sh -x $0\` to trace it.
'" 0

for p in "$@"
do case "$p" in
start)
PFD_AUTH='/var/run/pure-ftpd-auth'
env -i "LANG=$LANG" /bin/sh <<__
pure-authd "-s$PFD_AUTH" "-r/srv/.ftp/pure-ftpd-auth"&
# tls, ports, no dot files read, no dns, extauth, broken clients fixes
pure-ftpd --tls=1 -U 337:007 -p 2010:65535 -X -x -H "-lextauth:$PFD_AUTH" -b&
__
echo 'Waiting for FTP server.'
while sleep 1
do ps h -C pure-ftpd,pure-authd -ouid,pid,cmd && break
done >/dev/null 2>&1
# show daemons
ps -C pure-ftpd,pure-authd -ouid,pid,cmd
;;
stop)
# stop all daemons, ignoring errors
while ps h -C pure-ftpd,pure-authd -opid
do kill -TERM `ps h -C pure-ftpd,pure-authd -opid`
   sleep 1
done >/dev/null 2>&1 || :
;;
who)
# show who's running
ps -C pure-ftpd,pure-authd -f || :
;;
esac
echo "$p"
done
trap "" 0

== /etc/init.d/bind9+
#!/bin/sh

# config directory: '/etc/bind/'
#                   every directory here is a config setup for every
#                   network interface (or IP address) on a server
# chroot directory: '/var/cache/bind-chroot/'
#                   this script will copy all config directories for all
#                   network interfaces here, and will run named daemon
#                   chroot()ed here in a config directory
##
## send comments to <olecom@gmail.com> ##

set -e

[ "$*" ] || exec echo "Usage: $0 [&|] {start, stop, reload, who}"
trap "echo '
Unexpected Script Error!. Use \`/bin/sh -x $0\` to trace it.
'" 0

for p in "$@"
do case "$p" in
start)

# set up file creation mask for 'bind:root' as '|r--|rw-|---|'
# we are root, and we will create config files there, so we
# must have write access (for while)

umask 0217
# create working directory, where `bind` will chroot() to
[ -d '/var/cache/bind-chroot' ] || mkdir -m 770 -p '/var/cache/bind-chroot'

# going into config directory
cd /etc/bind
# for every directory (with per network interface config)
# copy config files into chroot, exec a daemon
for d in *
do [ -d "$d" ] && (
    # go into a config
    cd "$d"
    # create chroot for it
    d="/var/cache/bind-chroot/$d"
    mkdir -m 770 -p "$d"
    chown bind:root "$d"
    # copy config files there
    for f in *
    do  dd <"$f">"$d/$f"
    done
    # go there
    cd "$d"
    # secure files
    chown bind:root *
    # drop write permission for root group for newly created files
    umask 0117
    # run `named` here as user 'bind'; it will chroot() here (in "$d").
    exec env -i /usr/sbin/named -4 -c named.conf -d0 -t"$d" -ubind
)
done >/dev/null 2>&1
echo 'Waiting for BIND9 server(s).'
while sleep 1
do ps h -C named -ouid,pid,cmd && break
done >/dev/null 2>&1
# show daemons
ps -C named -ouid,pid,cmd
;;
stop)
# stop all named daemons, ignoring errors
while ps h -C named -opid
do kill -TERM `ps h -C named -opid`
   sleep 1
done >/dev/null 2>&1 || :
;;
reload)
# NOTE: config files are being re-read from the chroot directory
#       in the cache /var/cache/bind-chroot/$config
kill -HUP `ps h -C named -opid`
;;
who)
# show who's running
ps -C named -f || :
;;
esac
echo "$p"
done
trap "" 0

== /etc/init.d/ssh+
#!/bin/sh

set -e

[ "$*" ] || exec echo "
Usage: $0 [&|] {start, stop, reload, who, rr (remote-restart; uses at(1))}"
trap "echo '
Unexpected Script Error! Use \`/bin/sh -x $0\` to trace it.
'" 0
for p in "$@"
do case "$p" in
start)
# Create the PrivSep empty dir if necessary
[ -d /var/run/sshd ] || {
	mkdir /var/run/sshd
	chmod 0750 /var/run/sshd
}
# dummy
env -i "LANG=$LANG" SSHD_OOM_ADJUST=0 /usr/sbin/sshd
# special
cd /etc/ssh
for c in sshd_config+*
do env -i "LANG=$LANG" SSHD_OOM_ADJUST=-17 /usr/sbin/sshd -f "/etc/ssh/$c"
done
echo 'Waiting for SSH servers.'
while sleep 1
do ps h -C sshd -ouid,pid,cmd && break
done >/dev/null 2>&1
# show daemons
ps -C sshd -ouid,pid,cmd
;;
reload)
kill -HUP `ps h -C sshd -opid`
;;
stop)
# kill daemons, ignoring errors
while ps h -C sshd -opid
do kill -TERM `ps h -C sshd -opid`
   sleep 1
done >/dev/null 2>&1 || :
;;
who)
# show who's running
ps -C sshd -f || :
;;
rr)
hash at && {
p='remote-restart within one minute'
at now + 1 minute <<'_'
/etc/init.d/ssh+ stop start
_
} || p="remote-restart doesn't work"
;;
esac
echo "$p"
done
trap "" 0