Re: automated test? (was Re: Linux 2.6.17.7)

Matthias Andree
2006-07-25T21:20:08Z

On Tue, 25 Jul 2006, Arjan van de Ven wrote:

> well you can do such a thing withing statistical bounds; however... if
> the patch already is in -git (as is -stable policy normally).. it should
> have been found there already...

The sad facts I learned from Debian bug #212762 (not kernel related) that
culminated in CVE-2005-2335 (remote root exploit against older
fetchmail) and from various qmail bugs Guninski discovered:

- a bug need not necessarily be found soon after introduction

- a bug report may not convey the hint "look at this NOW, the shit
already hit the fan"
(sorry, I meant to write: look NOW, it's urgent and important)

- an automated test to catch non-trivial mistakes is non-trivial in
itself, and - what I've seen with another project I was involved with,
and more often than I found amusing - is that the test itself can be
buggy causing bogus results.

That doesn't mean I object to automated tests, but "it should have been
found by now" (because the source is open, someone could have tested it,
whatever) just doesn't work.

--
Matthias Andree