Re: Submitting patches to xfstests based on OSDI '18 paper (CrashMonkey)

[Dave Chinner <david@fromorbit.com>]

On Sun, Oct 21, 2018 at 05:52:21PM -0500, Jayashree Mohan wrote:
> > How long does each test case take to run?
> All the tests would touch 4 files at most or write 32-64KB of data to
> a file, starting from a empty file system image (hence very minimal
> running time). We are not going to bring CrashMonkey in the loop - we
> will port the generated tests to xfstest using dm-flakey (like
> generic/498 [1], which was submitted in response to a bug found by
> CrashMonkey in btrfs). Hence, each test should take the same time as
> that of current crash-consistency tests in the xfstest suite (for
> example, similar to generic/034 which takes about a second to run).
> 
> > And note, by the way, that
> > by default we automatically run fsck on the test device after each
> > test.  So number one, if you use the test device, you don't need to
> > worry about running fsck explicitly; the xfstests check script will do
> > that, and fail the test if the file system is corrupted --- and number
> > two, this will influence whether which groups each test should be
> > assigned.
> 
> Noted. Since we will be writing the out-file (checker) manually, will
> ensure that checks only the content/metadata.
> 
> > See the file xfstests-dev/tests/generic/group to see how groups get
> > assigned to tests.  I suppose all of the crashmonkey tests should be
> > assigned to a new group, say, "crashmonkey".  Whether or not they
> > should get assigned to the "auto" or "quick" group is a different
> > question.  Note that if running these tests will signicantly increase
> > the test run time of smoke tests and even the full "automatic"
> > regression tests, there may be some resistence in adding all of these
> > tests to the "auto" or "quick" groups.  Or even if you do, many file
> > system developers may choose to exclude all tests from the
> > "crashmonkey" group because if a 15 minute smoke test suddenly gets
> > extended to take 6 hours, developers are wont to get.... cranky.  :-)
> 
> It makes sense to add it to a new group as you suggest, and
> considering a second to run each test, it should take around 5 minutes
> to run this batch of CrashMonkey tests. Once the tests cases are
> ready, we can give you a better estimate of total time spent on the
> newly added tests.

Add the time between tests - fsck checks, scrub, etc, and that can
easily add another 10s per test.

Hence I'd strongly encourage you to batch similar tests into a
single xfstest test so that we're not needlessly adding 300x15s to
every test run because of the per-test external overhead.....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com