Re: Reporting bugs and bisection

[Andrew Morton <akpm@linux-foundation.org>]

On Mon, 14 Apr 2008 07:43:49 -0700 Arjan van de Ven <arjan@infradead.org> wrote:

> On Mon, 14 Apr 2008 01:04:12 -0700
> > 
> > The steps to be taken are:
> > 
> > a) agree that we have a problem
> > 
> 
> 
> I for one do not agree that we have a problem.
> 
> Based on actual data on oopses (which very clearly excludes other kinds of bugs, so I know I only see part of the story)
> we are doing reasonably well. Lets look at the 2.6.25 cycle. 
> We got a total of roughly 2700 reports of oopses/warn_ons from users. (This may sound high to those of you only reading
> lkml, but this includes automatically collected oopses from Fedora 9 beta testers).
> Out of these 2700, the top 20 issues account for 75% of the total reports.
> 
> Out of these 20 issues, 9 were from still out of tree drivers (wireless.git and drm.git included in F9). These were
> caught before they even got close to mainline.
> The remaining 11 issues can be split in
> 1) The ones we caught and fixed
> 2) TCP/IP warnings that DaveM and co are chasing down hard (but have trouble finding reproducers)
> 3) An EXT3 bug that in theory can cause data corruption, but in practice seems to happen after you yank out a USB stick
>   with an EXT3 filesystem on (so it can't corrupt the disk data). Ted is working on this
> 4) A bug (double free) that hits in the skb layer, probably caused by a bug in the ipv4 code
>    (a first analysis + potential patch was mailed to netdev this weekend)
> 5) sysfs "existing file added" warning, mostly in the USB stack
>    (gregkh claims he fixed this recently, I'm not entirely sure he got all cases)
> 
> And when I look beyond the first 20, the same pattern arises, we fixed the majority of the issues before -rc9.
> At position 25 we have less than 20 reports per bug. At position 35 we have less than 10 reports per bug. 
> At position 50 we have less than 5 reports per bug. Conclusion there: the bugs people actually hit fall of dramatically;
> there's a core set of issues that gets hit a lot, the rest quickly gets reduced to noise levels.
> 
> 
> To me this does not sound like we have a huge quality problem because
> 1) The distribution of the bugs is such that there is a relatively small set of core issues
>    that are widely hit, and then there's a near exponential drop after that
> 2) We are fixing the important bugs by and large before they hit a release
>    (important as defined by the number of people actually hitting the bug)
> 
> 
>  
> I'll be writing a report with more details about this soon with more analysis and statistics
> (I'll be looking at more detail around the top 25 issues, when they got introduced, when they got fixed etc)

Well OK.  But I don't think we can generalise from oops-causing bugs all
the way to all bugs.  Very few bugs actually cause oopses, and oopses tend
to be the thing which developers will zoom in on and pay attention to.

If we had metrics on "time goes backwards" or anything containing "ASUS",
things might be different.