test -nt/-ot behavior

test -nt/-ot behavior

[Guido Berhoerster]

Hello,

The behavior of the test builtin's -nt and -ot operators differs
in an incompatible way from the behavior established by the
KornShell.

The -nt, -ot, and -ef operators of test originated from the
KornShell, they exist at least since ksh88. Their specified
behavior is as follows:

* file1 -nt file2 evaluates to true if only file1 exists or file1
  is newer than file2
* file1 -ot file2 evaluates to true if only file2 exists or file1
  is older than file2

This behavior is matched by the test builtin that is part of
bash, ksh93, pdksh 5.2.14, and the test utility that comes with
GNU coreutils.

Then there is the pdksh's test builtin in versions < 5.2.14 where
the -nt and -ot operators behaved differently until they got
"fixed" (it is listed in the changelog under bug fixes) in 1999
just before the last release 5.2.14. Here the behavior was as
follows:

* file1 -nt file2 evaluates to true if both file1 and file2 exist
  and file1 is newer than file2
* file1 -ot file2 evaluates to true if both file1 and file2 exist
  and file1 is older than file2

It has to be noted that the documented behavior of test in ksh93,
pdksh 5.2.14, bash, and GNU coreutils is not entirely correct,
rather than checking for the existence of a file they only check
if a stat on the file succeeds.

According to the posting of the ash sources on comp.sources.unix
in 1989 ash originally included a combined test/expr builtin and
a standalone version of the test command (which did not contain
-nt, -ot, and -ef). NetBSD apparently went with the latter as a
standalone utility and when ash was first ported to Linux in 1993
the "buggy" test builtin from pdksh was used (see the ash-0.2
source) and I suppose it was adopted for the later port that
became dash.  In 1994 NetBSD replaced its test utility with the
"buggy" version from pdksh, from there it went into OpenBSD and
in 1999 it was imported into FreeBSD.  The change that pdksh made
in 1999 was however never merged back into the standalone test
utility included in the BSDs or the builtin test contained in the
Linux ports of ash, however NetBSD and OpenBSD both contain ksh
versions derived from pdksh 5.2.14 in their base system resulting
in inconsistent behavior between /bin/test and the ksh builtin.

It should be noted that pdksh contains many other subtle
differences and is not really compatible to either ksh88 or
ksh93, the behavior introduced by the test builtin of earlier
pdksh versions is likely due to a misunderstanding of ksh88's
specified behavior which was later fixed.  So IMHO the
consequence for ash should be to either merge the fix from pdksh
or to drop -nt, -ot, and -ef altogether, I would prefer that
latter since I think silently changing the behavior in an
incompatible way is bad practice and these features are not
specified in POSIX anyway.

-- 
Guido Berhoerster

Re: test -nt/-ot behavior

[Oleg Verych]

2010/2/21, Guido Berhoerster <guido+kernel.org@berhoerster.name>:
> So IMHO the consequence for ash should be to either merge the fix from pdksh
> or to drop -nt, -ot, and -ef altogether,

> I would prefer that
> latter since I think silently changing the behavior in an
> incompatible way is bad practice and these features are not
> specified in POSIX anyway.

Me is for the first + `sed` script to add `test -e &&` stuff in
bug-oriented scripts, for bug compatibility.

Breaking stuff is not building useful features.
> --
> Guido Berhoerster
-- 
sed 'sed && sh + olecom = love'  <<  ''
-o--=O`C
 #oo'L O
<___=E M

Re: test -nt/-ot behavior

[Herbert Xu]

Guido Berhoerster <guido+kernel.org@berhoerster.name> wrote:
> 
> It should be noted that pdksh contains many other subtle
> differences and is not really compatible to either ksh88 or
> ksh93, the behavior introduced by the test builtin of earlier
> pdksh versions is likely due to a misunderstanding of ksh88's
> specified behavior which was later fixed.  So IMHO the
> consequence for ash should be to either merge the fix from pdksh
> or to drop -nt, -ot, and -ef altogether, I would prefer that
> latter since I think silently changing the behavior in an
> incompatible way is bad practice and these features are not
> specified in POSIX anyway.

Well, as you say the behaviour is not specified by POSIX so dash
isn't doing anything wrong per se.  As this has been the documented
behaviour within dash since forever I'm reluctant to change it or
for that matter, removing the feature entirely.

I think fixing scripts that rely on this for files that do not exist
would be the best option.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: test -nt/-ot behavior

[Guido Berhoerster]

* Herbert Xu <herbert@gondor.apana.org.au> [2010-02-22 07:50]:
> Guido Berhoerster <guido+kernel.org@berhoerster.name> wrote:
> > 
> > It should be noted that pdksh contains many other subtle
> > differences and is not really compatible to either ksh88 or
> > ksh93, the behavior introduced by the test builtin of earlier
> > pdksh versions is likely due to a misunderstanding of ksh88's
> > specified behavior which was later fixed.  So IMHO the
> > consequence for ash should be to either merge the fix from pdksh
> > or to drop -nt, -ot, and -ef altogether, I would prefer that
> > latter since I think silently changing the behavior in an
> > incompatible way is bad practice and these features are not
> > specified in POSIX anyway.
> 
> Well, as you say the behaviour is not specified by POSIX so dash
> isn't doing anything wrong per se.  As this has been the documented
> behaviour within dash since forever I'm reluctant to change it or
> for that matter, removing the feature entirely.

It is not doing something "wrong", but my point (and that of the
original reporter of
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=558989) is that
it is diverging from established behavior in widely used shells
such a ksh88, ksh93, bash, pdksh, and GNU userland, as I said
this was even fixed in pdksh where it originally came from.  I
don't know how many /bin/dash scripts there are which rely on
this behavior from test -nt/-ot but it can't be that many and
dash has only been called das since 2002.  On the other hand dash
is being widely used as /bin/sh, e.g. in Ubuntu, Debian, and
hopefully soon openSUSE.

Changing the behavior as pdksh did after ten years is certainly
not a good practice but removing it will lead to an clear error
message rather than unexpected behavior which would be an
advantage when dealing with scripts expecting -nt/-ot but using
/bin/sh. For this reason alone I'm inclined to patch this out on
openSUSE.

If dash aims to be a POSIX compliant /bin/sh which is "as small
as possible", why does it need test -nt/-ot/-ef at all?  BTW, why
did you go with pdksh's test instead of the one coming with ash?

-- 
Guido Berhoerster

Re: test -nt/-ot behavior

[Herbert Xu]

On Mon, Feb 22, 2010 at 11:33:53AM +0100, Guido Berhoerster wrote:
>
> It is not doing something "wrong", but my point (and that of the
> original reporter of
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=558989) is that
> it is diverging from established behavior in widely used shells
> such a ksh88, ksh93, bash, pdksh, and GNU userland, as I said
> this was even fixed in pdksh where it originally came from.  I
> don't know how many /bin/dash scripts there are which rely on
> this behavior from test -nt/-ot but it can't be that many and
> dash has only been called das since 2002.  On the other hand dash
> is being widely used as /bin/sh, e.g. in Ubuntu, Debian, and
> hopefully soon openSUSE.

Actually dash has been around since 1997, prior to 2002 it was
known in Debian as ash.  So this behaviour has been around for
more than a decade.
 
> If dash aims to be a POSIX compliant /bin/sh which is "as small
> as possible", why does it need test -nt/-ot/-ef at all?  BTW, why
> did you go with pdksh's test instead of the one coming with ash?

While dash does try to be minimal, existing features are not removed
unless there is a very good reason.

As to your second question, this behaviour is inherited from BSD
which is where dash came from originally.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt