Re: [RFC: kdevops] Standardizing on failure rate nomenclature for expunges

[Matthew Wilcox <willy@infradead.org>]

On Thu, May 19, 2022 at 07:24:50PM +0800, Zorro Lang wrote:
> Yes, we talked about this, but if I don't rememeber wrong, I recommended each
> downstream testers maintain their own "testing data/config", likes exclude
> list, failed ratio, known failures etc. I think they're not suitable to be
> fixed in the mainline fstests.

This assumes a certain level of expertise, which is a barrier to entry.

For someone who wants to check "Did my patch to filesystem Y that I have
never touched before break anything?", having non-deterministic tests
run by default is bad.

As an example, run xfstests against jfs.  Hundreds of failures, including
some very scary-looking assertion failures from the page allocator.
They're (mostly) harmless in fact, just being a memory leak, but it
makes xfstests useless for this scenario.

Even for well-maintained filesystems like xfs which is regularly tested,
I expect generic/270 and a few others to fail.  They just do, and they're
not an indication that *I* broke anything.

By all means, we want to keep tests around which have failures, but
they need to be restricted to people who have a level of expertise and
interest in fixing long-standing problems, not people who are looking
for regressions.