Re: Submitting patches to xfstests based on OSDI '18 paper (CrashMonkey)

["Theodore Y. Ts'o" <tytso@mit.edu>]

On Sun, Oct 21, 2018 at 10:48:56AM -0500, Jayashree Mohan wrote:
> Hi Eryu,
> 
> Thanks for the response! Will follow the template you suggested for writing
>  test cases.
> 
> CrashMonkey generates 300 test cases, which we’ll convert to xfstest.
> Should we submit each test case as a different patch, or say 10 test cases
> per patch, or all of them in a single patch ? Let us know if there’s a
> preference.

How long does each test case take to run?  And note, by the way, that
by default we automatically run fsck on the test device after each
test.  So number one, if you use the test device, you don't need to
worry about running fsck explicitly; the xfstests check script will do
that, and fail the test if the file system is corrupted --- and number
two, this will influence whether which groups each test should be
assigned.

See the file xfstests-dev/tests/generic/group to see how groups get
assigned to tests.  I suppose all of the crashmonkey tests should be
assigned to a new group, say, "crashmonkey".  Whether or not they
should get assigned to the "auto" or "quick" group is a different
question.  Note that if running these tests will signicantly increase
the test run time of smoke tests and even the full "automatic"
regression tests, there may be some resistence in adding all of these
tests to the "auto" or "quick" groups.  Or even if you do, many file
system developers may choose to exclude all tests from the
"crashmonkey" group because if a 15 minute smoke test suddenly gets
extended to take 6 hours, developers are wont to get.... cranky.  :-)

						- Ted