Re: [PATCH] fstest: CrashMonkey tests ported to xfstest

[Dave Chinner <david@fromorbit.com>]

On Wed, Nov 07, 2018 at 01:09:22PM +1100, Dave Chinner wrote:
> To put it in other words, developers need tests focussed on finding
> bugs quickly, not regression tests that provide the core
> requirements of integration and release testing. The development
> testing phase is all about finding fast and effciently.

To emphasis my point of having tests and tools capable of finding
new bugs, I noticed yesterday that fstress and fsx didn't support
copy_file_range, and fsx doesn't support clone/dedupe_file_range
either. Darrick added them overnight.

fsx as run by generic/263 takes *32* operations to find a data
corruption with copy_file_range on XFS. Even changing it to do
buffered IO instead of direct IO, it only takes ~600 operations to
fail with a different data corruption.

That's *at least* two previously unknown bugs exposed in under a
second of runtime.

That's the sort of tooling we need - we don't need hundreds of tests
that are scripted reproducers of fixed problems, we need tools that
exercise boundary conditions and corner cases in ways that are
likely to expose incorrect behaviour. Tools that do these things
quickly and in a reproducable manner are worth their weight in
gold...

IMO, Quality Engineering is not just about writing regression tests
to keep out known bugs - it's most important function is developing
and refining new testing tools to find bugs that have escaped
detection with existing testing methods and tools. If test engineers
can find new bugs, software engineers can fix them. That's really
the ultimate goal here - to find bugs and fix them before users are
exposed to them...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com