Re: [PATCH v2] xfstests: update xfs/096 for new behaviour

[Dave Chinner <david@fromorbit.com>]

On Thu, Jun 30, 2016 at 02:54:59PM +0800, Eryu Guan wrote:
> On Wed, Jun 29, 2016 at 12:18:55PM +0200, Jan Tulak wrote:
> > Because we recently changed how mkfs behaves when it gets incorrect/invalid
> > values, update the expected output to reflect the current status.
> > However, keep also compatibility with the old version.
> > 
> > Signed-off-by: Jan Tulak <jtulak@redhat.com>
> > ---
> > CHANGE: added compatibility for the old xfsprogs.
> 
> Sorry for the late response, because I was lost on this :)
> 
> Hi Dave - what's the rule/policy of maintaining the backword
> compatibility in fstests?

We try to ensure that tests that work/pass on old versions of
utilities continue to do so, even as the newer code changes. If the
new code changes too much, then we can either stop running the test
on older code, or we fork the test for the new code....

> I know that efforts have been made to make
> sure new changes don't break old binaries, but is that a must or a
> best-to-have? And what do you think about the xfsprogs version
> comparing? (I'm OK with it :-))

We've tried to avoid using version numbers for comparisons, because
that becomes a downward spiral into a mess. Instead, we have
gone down the path of testing for supported features in binaries and
filesystems, not checking version numbers. i.e. we don't care about
the version number - we care about the feature that the binary
provides. Those checks are self documenting - the test tells use
what it requires which something that version number checks do not
explain at all.

In this case, we have a change in a binary that turns warnings into
errors or issues errors rather than silently ignores what the user
asked for and uses defaults. We already filter out anything relevant
from the result to support all the changes in binary output since
the test was introduced, so we really can't tell if the value
substitution behaviour has changed anymore. IOWs, this test really
isn't serving much purpose as a regression test anymore.

>From that perspective, I'd say we either remove it or we stop trying
to update it further by adding a new requires check for an old mkfs
binary that silently accepts invalid log stripe unit sizes. i.e.
don't add version number checks, add a feature check so that it only
runs on old mkfs binaries but not new ones. e.g.
_require_mkfs_accept_invalid_log_sunit()

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com