Re: OpenPosix test case mmap_11-4 fails in ext4 filesystem

[Theodore Ts'o <tytso@mit.edu>]

On Thu, May 22, 2014 at 12:45:05PM +0200, chrubis@suse.cz wrote:
> Right, sorting out right testcases is not easy task. But given that the
> runtime is not that long we can do rough selection and end up with
> something that runs in about 10 or 20 minutes and covers all possibly
> relevant testcases.

Something to keep in mind is that for at least some file system
developers, such as myself, we end up running certain tests multiple
times, so we can get test coverage for different file system mount
options, block sizes, file system features, etc.  Currently, I have
ten or so different configurations for my full test run.

So if the bulk of the 20 minutes are tests for which there is
duplicate coverage with xfstests, or which is stuff that is only
testing behaviour implemented in the VFS, that's actually 3-4 extra
hours of testing as far as I'm concerned.  Hence, the longer and the
longer the tests take, the less often I will run them --- and the
effect is not linear.

So failing to eliminate tests which are not relevant is not cost-free.
In some cases, the costs of winnowing out non-relevant tests may not
be worth the effort.  If it's only an extra 5 seconds, I probably
wouldn't care.  Although if you multiply the time saved by the number
of file system developers, and the number of full test runs that they
likely will be running during the development cycle, spending a few
minutes to disable a non-relevant test starts becoming net-positive.

> To be frank the reputation is one thing I would really like to fix.
> Nothing is more frustrating than spending years fixing code and still
> being ignored because of the past you haven't been even involved in.

Well, to the extent that many years ago, I was employed by one of the
companies had a well-meaning, but ultimately counterproductive
contribution to LTP, by putting relatively junior test engineers that
didn't understand the kernel code all that deeply, and was given the
mandate to increase the code coverage percentages as the only figure
of merit, I was actually somewhat involved, if only indirectly.

Unfortunately, I wasn't able to effect change by advocating strongly
enough for a more productive approach (which would have required
putting in far more senior, and thus more expensive and harder to
allocate engineering talent), although I did help eventually help by
arguing that ceasing support for what I considered to be a deeply
dysfunctional effort as the best thing that particular company could
have done at that point.  :-(

So I really have to commend Xiaoguang Wang for having done a large
amount of analysis when submitting the report about the LTP test
failure.  That's significantly different from my previous experience
interacting with previous people working on the LTP project, and it's
precisely that sort of work and analysis which I'm sure will help turn
around LTP's reputation.

Cheers,

						- Ted