From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Return-Path: Date: Fri, 13 Jul 2018 10:46:20 -0700 From: "Luis R. Chamberlain" To: "Luis R. Chamberlain" Cc: Amir Goldstein , Jeff Mahoney , Linux FS Devel , xfs , fstests , Sasha Levin , Sasha Levin , Valentin Rothberg Subject: Re: [ANN] oscheck: wrapper for fstests check.sh - tracking and working with baselines Message-ID: <20180713174620.GA12723@garbanzo.do-not-panic.com> References: <20180713164420.GA3620@garbanzo.do-not-panic.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180713164420.GA3620@garbanzo.do-not-panic.com> List-ID: The other thing I forgot to mention is annotations for failure rates. For now I'll just indicate it as a ratio ratio as as part of the ending comment for an expunge entry. For instance I just ran into a failure with a set of stable patches for generic/475, however after re-running the test it succeeded. So the test does not fail always. I'd like to annotate this. I now have to go and re-test with the oscheck/naggy-check.sh a few times as follows: ./naggy-check.sh -s xfs_nocrc_512 -f generic/475 This will run the test in a loop until it fails, and I can use this as an initial failure rate hint to the tester. If I wanted to get more accurate I could use an average of few runs with naggy-check.sh. In practice I typically am only sure a test succeeds if it passes at least 1000 times (one can use -c 1000 on naggy-check.sh). I will also then have to go back to the stable kernel without patches and verify at least one failure was visible before. This will confirm this is indeed not a regression. Likewise I'll have to test this also with the other sections and see if this is also observed there. Luis