Re: [ANN] oscheck: wrapper for fstests check.sh - tracking and working with baselines

["Luis R. Chamberlain" <mcgrof@kernel.org>]

The other thing I forgot to mention is annotations for failure rates.

For now I'll just indicate it as a ratio ratio as as part of the ending
comment for an expunge entry. For instance I just ran into a failure
with a set of stable patches for generic/475, however after re-running
the test it succeeded. So the test does not fail always. I'd like to
annotate this.

I now have to go and re-test with the oscheck/naggy-check.sh a few
times as follows:

./naggy-check.sh -s xfs_nocrc_512 -f generic/475

This will run the test in a loop until it fails, and I can use this
as an initial failure rate hint to the tester. If I wanted to get
more accurate I could use an average of few runs with naggy-check.sh.

In practice I typically am only sure a test succeeds if it passes at
least 1000 times (one can use -c 1000 on naggy-check.sh).

I will also then have to go back to the stable kernel without patches
and verify at least one failure was visible before. This will confirm
this is indeed not a regression. Likewise I'll have to test this also
with the other sections and see if this is also observed there.

  Luis