Re: Testmgr fuzz testing

[Eric Biggers <ebiggers@kernel.org>]

Hi Pascal,

On Mon, Jul 22, 2019 at 10:27:22AM +0000, Pascal Van Leeuwen wrote:
> Eric,
> 
> While fixing some issues in the inside-secure driver reported by the fuzz, I noticed that the
> results are actually not repeatable: odds are  high that on the next run, the error case is 
> actually not hit anymore since they're typically very specific  corner cases.
> 
> There's 2 problems with that:
> a) Without repeatability, I cannot verify whether my fix actually worked. In fact, I cannot
> even verify with any certainty that any modification I do won't fail somewhere else :-(
> b) Odds are very significant that important corner cases are not hit by the fuzzing
> 
> Issue a) is usually solved by making the random generation deterministic, i.e. ensure
> you seed it with a known constant and pull the random numbers strictly sequentially.
> (you may or may not add the *option* to  pull the seed from some true random source)
> 
> Issue b) would be best solved by splitting the fuzz testing into two parts, a (properly
> constrained!) random part and a part with fixed known corner cases where you use
> constant parameters (like lengths and such) but depend on the generic implementation
> for the actual vector generation (as specifications usually don't provide vectors for
> all interesting corner cases but we consider the generic implementation to be correct) 
> 

Sure, it's not always repeatable, but that's the nature of fuzz testing.  We
*could* start with a constant seed, but then all the random numbers would change
every time anyone made any minor change to the fuzz tests anyway.

In my experience the bugs found by the fuzz tests tend to be found within a
couple hundred iterations, so are seen within a few boots at most with the
default fuzz_iterations=100, and are "always" seen with fuzz_iterations=1000.
Raising fuzz_iterations to 10000 didn't find anything else.

If you find otherwise and come across some really rare case, you should either
add a real test vector (i.e. not part of the fuzz tests) for it, or you should
update the fuzz tests to generate the case more often so that it's likely to be
hit with the default fuzz_iterations=100.  I don't think it's necessary to split
the fuzz testing into 2 parts; instead we just need to boost the probability of
generating known edge cases (e.g. see what generate_random_bytes() already does).

- Eric