Number of bugs - statistics

Number of bugs - statistics

[Bosko Radivojevic]

Hi!

Is there any kind of analysis of number of (reported or resolved) bugs
through time in Linux kernel floating around? I'm trying to convince
my colleagues that newer kernel version does not mean more bugs than
the previous 'well tested' ones.

Any kind of (research) reports or papers that address this issue is
more than welcome.

Thanks!

Sincerely,
Bosko

Re: Number of bugs - statistics

[Adrian Bunk]

On Thu, May 22, 2008 at 02:28:17PM +0200, Bosko Radivojevic wrote:

> Hi!

Hi Bosko!

> Is there any kind of analysis of number of (reported or resolved) bugs
> through time in Linux kernel floating around? I'm trying to convince
> my colleagues that newer kernel version does not mean more bugs than
> the previous 'well tested' ones.

We definitely have many regressions (something that previously worked
does no longer work) in each kernel.

> Any kind of (research) reports or papers that address this issue is
> more than welcome.

Any such reports or papers would anyway be flawed since we have no data 
one could use for doing serious statistics.

> Thanks!
> 
> Sincerely,
> Bosko

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Number of bugs - statistics

[Arjan van de Ven]

On Thu, 22 May 2008 17:41:14 +0300
Adrian Bunk <bunk@kernel.org> wrote:

> On Thu, May 22, 2008 at 02:28:17PM +0200, Bosko Radivojevic wrote:
> 
> > Hi!
> 
> Hi Bosko!
> 
> > Is there any kind of analysis of number of (reported or resolved)
> > bugs through time in Linux kernel floating around? I'm trying to
> > convince my colleagues that newer kernel version does not mean more
> > bugs than the previous 'well tested' ones.
> 
> We definitely have many regressions (something that previously worked
> does no longer work) in each kernel.

... and many of those regressions are things people are unlikely to
hit. And we fix many long standing bugs as well.

Maybe some other data:
* The incoming rate of ACPI bugs has been pretty much flat the last 3
years, while the number of Linux (and ACPI) users has grown
significantly. The number of unfixed bugs has more than halved, from
over 200 to well below 100.
* The SCSI maintainer also reports that he sees flat to declining bug
rates; again with the increase in Linux user base that is a good sign


> 
> > Any kind of (research) reports or papers that address this issue is
> > more than welcome.
> 
> Any such reports or papers would anyway be flawed since we have no
> data one could use for doing serious statistics.

We have data for 2.6.25 at least, on which we can and do serious
statistics. Unfortunately we don't have that for older kernels.
What we have for older kernels only comes from lkml reports (which
isn't very representative), but that shows that the bug report pattern
is a bit spikey, in that kernels that get picked up by popular
distributions get more reports than those that don't (no big surprise,
those just have a much larger user base), but otherwise there's no up
or down trend to be seen there.


> 
> > Thanks!
> > 
> > Sincerely,
> > Bosko
> 
> cu
> Adrian
> 

Re: Number of bugs - statistics

[Natalie Protasevich]

On Thu, May 22, 2008 at 7:51 AM, Arjan van de Ven <arjan@infradead.org> wrote:
> On Thu, 22 May 2008 17:41:14 +0300
> Adrian Bunk <bunk@kernel.org> wrote:
>
>> On Thu, May 22, 2008 at 02:28:17PM +0200, Bosko Radivojevic wrote:
>>
>> > Hi!
>>
>> Hi Bosko!
>>
>> > Is there any kind of analysis of number of (reported or resolved)
>> > bugs through time in Linux kernel floating around? I'm trying to
>> > convince my colleagues that newer kernel version does not mean more
>> > bugs than the previous 'well tested' ones.
>>
>> We definitely have many regressions (something that previously worked
>> does no longer work) in each kernel.
>
> ... and many of those regressions are things people are unlikely to
> hit. And we fix many long standing bugs as well.
>
> Maybe some other data:
> * The incoming rate of ACPI bugs has been pretty much flat the last 3
> years, while the number of Linux (and ACPI) users has grown
> significantly. The number of unfixed bugs has more than halved, from
> over 200 to well below 100.
> * The SCSI maintainer also reports that he sees flat to declining bug
> rates; again with the increase in Linux user base that is a good sign
>
>
>>
>> > Any kind of (research) reports or papers that address this issue is
>> > more than welcome.
>>
>> Any such reports or papers would anyway be flawed since we have no
>> data one could use for doing serious statistics.
>
> We have data for 2.6.25 at least, on which we can and do serious
> statistics. Unfortunately we don't have that for older kernels.
> What we have for older kernels only comes from lkml reports (which
> isn't very representative), but that shows that the bug report pattern
> is a bit spikey, in that kernels that get picked up by popular
> distributions get more reports than those that don't (no big surprise,
> those just have a much larger user base), but otherwise there's no up
> or down trend to be seen there.
>

Majority of old bugs that are still open are of "hard to reproduce"
category. That means you are not very likely to hit them. Those can
be: legacy devices failing, relatively rare hardware, code clean-ups
etc. You can't find  really critical long standing bugs in the old
backlog. It doesn't mean old bugs don't get fixed, rather get less
priority that severe bugs and regressions.
 With emphasys that was put on cleaning up bugs by Andrew and other
developers, bugs are taken even more seriously than before. New bugs,
regressions are being tracked aggressively, so they don't stay open
for long time. You can search for Rafael's regression reports and
sample  time line of a reported regression: they are short lived bugs.
Overall, each kernel release has great deal of improvement over older
ones, by nature of Linux. Many distros now tend to stay as close with
new releases of main line as possible. This seems to be the best way
to get all the bug fixes, optimizations, and features.
Regards,
 --Natalie
>
>>
>> > Thanks!
>> >
>> > Sincerely,
>> > Bosko
>>
>> cu
>> Adrian
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: Number of bugs - statistics

[Adrian Bunk]

On Thu, May 22, 2008 at 07:51:35AM -0700, Arjan van de Ven wrote:
> On Thu, 22 May 2008 17:41:14 +0300
> Adrian Bunk <bunk@kernel.org> wrote:
> 
> > On Thu, May 22, 2008 at 02:28:17PM +0200, Bosko Radivojevic wrote:
> > 
> > > Hi!
> > 
> > Hi Bosko!
> > 
> > > Is there any kind of analysis of number of (reported or resolved)
> > > bugs through time in Linux kernel floating around? I'm trying to
> > > convince my colleagues that newer kernel version does not mean more
> > > bugs than the previous 'well tested' ones.
> > 
> > We definitely have many regressions (something that previously worked
> > does no longer work) in each kernel.
> 
> ... and many of those regressions are things people are unlikely to
> hit. And we fix many long standing bugs as well.
> 
> Maybe some other data:
> * The incoming rate of ACPI bugs has been pretty much flat the last 3
> years, while the number of Linux (and ACPI) users has grown
> significantly. The number of unfixed bugs has more than halved, from
> over 200 to well below 100.
> * The SCSI maintainer also reports that he sees flat to declining bug
> rates; again with the increase in Linux user base that is a good sign

Andrew sees and handles the majority of incoming bug reports.
Ask him whether he agress with this.

And ALSA alone has 2000 open bug reports, which makes the open ACPI or 
SCSI bug numbers relatively irrelevant in any "number of bugs" 
statistics.

> > > Any kind of (research) reports or papers that address this issue is
> > > more than welcome.
> > 
> > Any such reports or papers would anyway be flawed since we have no
> > data one could use for doing serious statistics.
> 
> We have data for 2.6.25 at least, on which we can and do serious
> statistics.
>...

Where do we have data for 2.6.25 covering all kinds of bugs?

Regressions reported before 2.6.25 and Oops'es etc. are only a small 
part of the picture.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Number of bugs - statistics

[Natalie Protasevich]

On Thu, May 22, 2008 at 8:54 AM, Adrian Bunk <bunk@kernel.org> wrote:
> On Thu, May 22, 2008 at 07:51:35AM -0700, Arjan van de Ven wrote:
>> On Thu, 22 May 2008 17:41:14 +0300
>> Adrian Bunk <bunk@kernel.org> wrote:
>>
>> > On Thu, May 22, 2008 at 02:28:17PM +0200, Bosko Radivojevic wrote:
>> >
>> > > Hi!
>> >
>> > Hi Bosko!
>> >
>> > > Is there any kind of analysis of number of (reported or resolved)
>> > > bugs through time in Linux kernel floating around? I'm trying to
>> > > convince my colleagues that newer kernel version does not mean more
>> > > bugs than the previous 'well tested' ones.
>> >
>> > We definitely have many regressions (something that previously worked
>> > does no longer work) in each kernel.
>>
>> ... and many of those regressions are things people are unlikely to
>> hit. And we fix many long standing bugs as well.
>>
>> Maybe some other data:
>> * The incoming rate of ACPI bugs has been pretty much flat the last 3
>> years, while the number of Linux (and ACPI) users has grown
>> significantly. The number of unfixed bugs has more than halved, from
>> over 200 to well below 100.
>> * The SCSI maintainer also reports that he sees flat to declining bug
>> rates; again with the increase in Linux user base that is a good sign
>
> Andrew sees and handles the majority of incoming bug reports.
> Ask him whether he agress with this.
>
> And ALSA alone has 2000 open bug reports, which makes the open ACPI or
> SCSI bug numbers relatively irrelevant in any "number of bugs"
> statistics.
>
>> > > Any kind of (research) reports or papers that address this issue is
>> > > more than welcome.
>> >
>> > Any such reports or papers would anyway be flawed since we have no
>> > data one could use for doing serious statistics.
>>
>> We have data for 2.6.25 at least, on which we can and do serious
>> statistics.
>>...
>
> Where do we have data for 2.6.25 covering all kinds of bugs?
>
> Regressions reported before 2.6.25 and Oops'es etc. are only a small
> part of the picture.

There is no unified tracking mechanism as of now, this is a big
problem. There are some projects that have ideas about universal bug
tracking, for example some people from KDE team expressed their ideas,
I'm planning to get back with them.
It would be great if all reports from known and unknown bugzillas were
piped into one place and sorted out, by various criteria: ALSA bugs
are numerous, which is not important for most enterprise server users
who would completely disregard this category, whereas desktop users
will probably concentrate on those more than any other. So the answer
to question about kernel stability would be more adequate depending on
who's asking.

--Natalie

> cu
> Adrian
>
> --
>
>       "Is there not promise of rain?" Ling Tan asked suddenly out
>        of the darkness. There had been need of rain for many days.
>       "Only a promise," Lao Er said.
>                                       Pearl S. Buck - Dragon Seed
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: Number of bugs - statistics

[Stefan Richter]

Adrian Bunk wrote:
> On Thu, May 22, 2008 at 07:51:35AM -0700, Arjan van de Ven wrote:
>> Maybe some other data:
>> * The incoming rate of ACPI bugs has been pretty much flat the last 3
>> years, while the number of Linux (and ACPI) users has grown
>> significantly. The number of unfixed bugs has more than halved, from
>> over 200 to well below 100.
>> * The SCSI maintainer also reports that he sees flat to declining bug
>> rates; again with the increase in Linux user base that is a good sign
> 
> Andrew sees and handles the majority of incoming bug reports.
> Ask him whether he agress with this.
> 
> And ALSA alone has 2000 open bug reports, which makes the open ACPI or 
> SCSI bug numbers relatively irrelevant in any "number of bugs" 
> statistics.

Central facilities like ACPI and SCSI having relatively few [and
according to Arjan a decreasing number of] reported unfixed bugs is...
irrelevant?
-- 
Stefan Richter
-=====-==--- -=-= =-==-
http://arcgraph.de/sr/

Re: Number of bugs - statistics

[Bart Van Assche]

On Thu, May 22, 2008 at 2:28 PM, Bosko Radivojevic
<bosko.radivojevic@gmail.com> wrote:
> Is there any kind of analysis of number of (reported or resolved) bugs
> through time in Linux kernel floating around? I'm trying to convince
> my colleagues that newer kernel version does not mean more bugs than
> the previous 'well tested' ones.

The following summary of a speech by Andrew Morton contains good
arguments as why you should use a recent kernel:
http://lwn.net/Articles/278647/

Bart.

Re: Number of bugs - statistics

[Adrian Bunk]

On Thu, May 22, 2008 at 09:20:46AM -0700, Natalie Protasevich wrote:
> On Thu, May 22, 2008 at 8:54 AM, Adrian Bunk <bunk@kernel.org> wrote:
> > On Thu, May 22, 2008 at 07:51:35AM -0700, Arjan van de Ven wrote:
> >> On Thu, 22 May 2008 17:41:14 +0300
> >> Adrian Bunk <bunk@kernel.org> wrote:
> >>
> >> > On Thu, May 22, 2008 at 02:28:17PM +0200, Bosko Radivojevic wrote:
> >> >
> >> > > Hi!
> >> >
> >> > Hi Bosko!
> >> >
> >> > > Is there any kind of analysis of number of (reported or resolved)
> >> > > bugs through time in Linux kernel floating around? I'm trying to
> >> > > convince my colleagues that newer kernel version does not mean more
> >> > > bugs than the previous 'well tested' ones.
> >> >
> >> > We definitely have many regressions (something that previously worked
> >> > does no longer work) in each kernel.
> >>
> >> ... and many of those regressions are things people are unlikely to
> >> hit. And we fix many long standing bugs as well.
> >>
> >> Maybe some other data:
> >> * The incoming rate of ACPI bugs has been pretty much flat the last 3
> >> years, while the number of Linux (and ACPI) users has grown
> >> significantly. The number of unfixed bugs has more than halved, from
> >> over 200 to well below 100.
> >> * The SCSI maintainer also reports that he sees flat to declining bug
> >> rates; again with the increase in Linux user base that is a good sign
> >
> > Andrew sees and handles the majority of incoming bug reports.
> > Ask him whether he agress with this.
> >
> > And ALSA alone has 2000 open bug reports, which makes the open ACPI or
> > SCSI bug numbers relatively irrelevant in any "number of bugs"
> > statistics.
> >
> >> > > Any kind of (research) reports or papers that address this issue is
> >> > > more than welcome.
> >> >
> >> > Any such reports or papers would anyway be flawed since we have no
> >> > data one could use for doing serious statistics.
> >>
> >> We have data for 2.6.25 at least, on which we can and do serious
> >> statistics.
> >>...
> >
> > Where do we have data for 2.6.25 covering all kinds of bugs?
> >
> > Regressions reported before 2.6.25 and Oops'es etc. are only a small
> > part of the picture.
> 
> There is no unified tracking mechanism as of now, this is a big
> problem. There are some projects that have ideas about universal bug
> tracking, for example some people from KDE team expressed their ideas,
> I'm planning to get back with them.
> It would be great if all reports from known and unknown bugzillas were
> piped into one place and sorted out,

Not everyone uses Bugzilla (e.g. ALSA uses Mantis).

And the majority of bug reports might still go only to mailing lists and 
not into any bugtracker at all. As long as this happens the data is 
simply not available.

> by various criteria: ALSA bugs
> are numerous, which is not important for most enterprise server users
> who would completely disregard this category, whereas desktop users
> will probably concentrate on those more than any other.

The majority of machines running Linux most likely runs with ARM CPUs.

Show me any public source you want to use for getting serious data for 
this area.

> So the answer
> to question about kernel stability would be more adequate depending on
> who's asking.

You must not confuse "number of reported bugs" with "kernel stability" - 
these two can be quite decoupled.

> --Natalie

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Number of bugs - statistics

[Natalie Protasevich]

On Thu, May 22, 2008 at 9:50 AM, Adrian Bunk <bunk@kernel.org> wrote:
> On Thu, May 22, 2008 at 09:20:46AM -0700, Natalie Protasevich wrote:
>> On Thu, May 22, 2008 at 8:54 AM, Adrian Bunk <bunk@kernel.org> wrote:
>> > On Thu, May 22, 2008 at 07:51:35AM -0700, Arjan van de Ven wrote:
>> >> On Thu, 22 May 2008 17:41:14 +0300
>> >> Adrian Bunk <bunk@kernel.org> wrote:
>> >>
>> >> > On Thu, May 22, 2008 at 02:28:17PM +0200, Bosko Radivojevic wrote:
>> >> >
>> >> > > Hi!
>> >> >
>> >> > Hi Bosko!
>> >> >
>> >> > > Is there any kind of analysis of number of (reported or resolved)
>> >> > > bugs through time in Linux kernel floating around? I'm trying to
>> >> > > convince my colleagues that newer kernel version does not mean more
>> >> > > bugs than the previous 'well tested' ones.
>> >> >
>> >> > We definitely have many regressions (something that previously worked
>> >> > does no longer work) in each kernel.
>> >>
>> >> ... and many of those regressions are things people are unlikely to
>> >> hit. And we fix many long standing bugs as well.
>> >>
>> >> Maybe some other data:
>> >> * The incoming rate of ACPI bugs has been pretty much flat the last 3
>> >> years, while the number of Linux (and ACPI) users has grown
>> >> significantly. The number of unfixed bugs has more than halved, from
>> >> over 200 to well below 100.
>> >> * The SCSI maintainer also reports that he sees flat to declining bug
>> >> rates; again with the increase in Linux user base that is a good sign
>> >
>> > Andrew sees and handles the majority of incoming bug reports.
>> > Ask him whether he agress with this.
>> >
>> > And ALSA alone has 2000 open bug reports, which makes the open ACPI or
>> > SCSI bug numbers relatively irrelevant in any "number of bugs"
>> > statistics.
>> >
>> >> > > Any kind of (research) reports or papers that address this issue is
>> >> > > more than welcome.
>> >> >
>> >> > Any such reports or papers would anyway be flawed since we have no
>> >> > data one could use for doing serious statistics.
>> >>
>> >> We have data for 2.6.25 at least, on which we can and do serious
>> >> statistics.
>> >>...
>> >
>> > Where do we have data for 2.6.25 covering all kinds of bugs?
>> >
>> > Regressions reported before 2.6.25 and Oops'es etc. are only a small
>> > part of the picture.
>>
>> There is no unified tracking mechanism as of now, this is a big
>> problem. There are some projects that have ideas about universal bug
>> tracking, for example some people from KDE team expressed their ideas,
>> I'm planning to get back with them.
>> It would be great if all reports from known and unknown bugzillas were
>> piped into one place and sorted out,
>
> Not everyone uses Bugzilla (e.g. ALSA uses Mantis).
>
> And the majority of bug reports might still go only to mailing lists and
> not into any bugtracker at all. As long as this happens the data is
> simply not available.

True. It was argued that mailing list is more effective that bugzilla
as a collaboration mechanism while working on a bug. But  if it was a
way to record it into bugzilla "seemlessly" (as Andrew tried to
enforce, by asking those on the list to add CC to bugme-deamon) this
would be perfect.
Another one to implement...
>

>> by various criteria: ALSA bugs
>> are numerous, which is not important for most enterprise server users
>> who would completely disregard this category, whereas desktop users
>> will probably concentrate on those more than any other.
>
> The majority of machines running Linux most likely runs with ARM CPUs.
>
> Show me any public source you want to use for getting serious data for
> this area.

There is no one, except dispersed reports and ... appropriate
implementation involving search engine comes to mind.
>
>> So the answer
>> to question about kernel stability would be more adequate depending on
>> who's asking.
>
> You must not confuse "number of reported bugs" with "kernel stability" -
> these two can be quite decoupled.

Number of existing, proven and prioritized  bugs maybe better to say.
I think there is a correlation.
--Natalie
>
>> --Natalie
>
> cu
> Adrian
>
> --
>
>       "Is there not promise of rain?" Ling Tan asked suddenly out
>        of the darkness. There had been need of rain for many days.
>       "Only a promise," Lao Er said.
>                                       Pearl S. Buck - Dragon Seed
>
>

Re: Number of bugs - statistics

[Adrian Bunk]

On Thu, May 22, 2008 at 06:22:57PM +0200, Stefan Richter wrote:
> Adrian Bunk wrote:
> > On Thu, May 22, 2008 at 07:51:35AM -0700, Arjan van de Ven wrote:
> >> Maybe some other data:
> >> * The incoming rate of ACPI bugs has been pretty much flat the last 3
> >> years, while the number of Linux (and ACPI) users has grown
> >> significantly. The number of unfixed bugs has more than halved, from
> >> over 200 to well below 100.
> >> * The SCSI maintainer also reports that he sees flat to declining bug
> >> rates; again with the increase in Linux user base that is a good sign
> > 
> > Andrew sees and handles the majority of incoming bug reports.
> > Ask him whether he agress with this.
> > 
> > And ALSA alone has 2000 open bug reports, which makes the open ACPI or 
> > SCSI bug numbers relatively irrelevant in any "number of bugs" 
> > statistics.
> 
> Central facilities like ACPI and SCSI having relatively few [and
> according to Arjan a decreasing number of] reported unfixed bugs is...
> irrelevant?

Yes, _in any "number of bugs" statistics_ that is irrelevant.

You seem to confuse "number of bugs" with "stability".
As Arjan often emphasizes this is wrong.

We do not even have good data for "number of bug reports".
And getting data for "stability" would be a quite hard task.

> Stefan Richter

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Number of bugs - statistics

[Adrian Bunk]

On Thu, May 22, 2008 at 10:08:49AM -0700, Natalie Protasevich wrote:
> On Thu, May 22, 2008 at 9:50 AM, Adrian Bunk <bunk@kernel.org> wrote:
>...
> >> by various criteria: ALSA bugs
> >> are numerous, which is not important for most enterprise server users
> >> who would completely disregard this category, whereas desktop users
> >> will probably concentrate on those more than any other.
> >
> > The majority of machines running Linux most likely runs with ARM CPUs.
> >
> > Show me any public source you want to use for getting serious data for
> > this area.
> 
> There is no one, except dispersed reports and ... appropriate
> implementation involving search engine comes to mind.

Much of this information will never be published at any place where 
Google could find it.

> >> So the answer
> >> to question about kernel stability would be more adequate depending on
> >> who's asking.
> >
> > You must not confuse "number of reported bugs" with "kernel stability" -
> > these two can be quite decoupled.
> 
> Number of existing, proven and prioritized  bugs maybe better to say.
> I think there is a correlation.

Some correlaton.

If people are aware of the weaknesses and the many obvious ways to 
manipulate such statistics.

> --Natalie

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Number of bugs - statistics

[Stefan Richter]

Adrian Bunk wrote:
> On Thu, May 22, 2008 at 06:22:57PM +0200, Stefan Richter wrote:
>> Adrian Bunk wrote:
>>> And ALSA alone has 2000 open bug reports, which makes the open ACPI or 
>>> SCSI bug numbers relatively irrelevant in any "number of bugs" 
>>> statistics.
>> Central facilities like ACPI and SCSI having relatively few [and
>> according to Arjan a decreasing number of] reported unfixed bugs is...
>> irrelevant?
> 
> Yes, _in any "number of bugs" statistics_ that is irrelevant.

Well, yes, in any "simplistic total number of bugs without any frame of 
reference what is actually counted" statistics it is indeed irrelevant.

[...]
> We do not even have good data for "number of bug reports".

That's right.  We don't have comprehensive data, and it is difficult to 
draw conclusions from the data that we have.
-- 
Stefan Richter
-=====-==--- -=-= =-==-
http://arcgraph.de/sr/

Re: Number of bugs - statistics

[Natalie Protasevich]

On Thu, May 22, 2008 at 10:33 AM, Adrian Bunk <bunk@kernel.org> wrote:
> On Thu, May 22, 2008 at 10:08:49AM -0700, Natalie Protasevich wrote:
>> On Thu, May 22, 2008 at 9:50 AM, Adrian Bunk <bunk@kernel.org> wrote:
>>...
>> >> by various criteria: ALSA bugs
>> >> are numerous, which is not important for most enterprise server users
>> >> who would completely disregard this category, whereas desktop users
>> >> will probably concentrate on those more than any other.
>> >
>> > The majority of machines running Linux most likely runs with ARM CPUs.
>> >
>> > Show me any public source you want to use for getting serious data for
>> > this area.
>>
>> There is no one, except dispersed reports and ... appropriate
>> implementation involving search engine comes to mind.
>
> Much of this information will never be published at any place where
> Google could find it.

But Adrian, there will be a chain reaction. That's why Andrew
encourages interested parties from industry share code (which will
mean bug reports). This is gradual process, way of collaboration and
open source is being more and more accepted and even urged.
>
>> >> So the answer
>> >> to question about kernel stability would be more adequate depending on
>> >> who's asking.
>> >
>> > You must not confuse "number of reported bugs" with "kernel stability" -
>> > these two can be quite decoupled.
>>
>> Number of existing, proven and prioritized  bugs maybe better to say.
>> I think there is a correlation.
>
> Some correlaton.
>
> If people are aware of the weaknesses and the many obvious ways to
> manipulate such statistics.

... which needs to start and take its own development cycle. Then it
will become more sophisticated.
>
>> --Natalie
>
> cu
> Adrian
>
> --
>
>       "Is there not promise of rain?" Ling Tan asked suddenly out
>        of the darkness. There had been need of rain for many days.
>       "Only a promise," Lao Er said.
>                                       Pearl S. Buck - Dragon Seed
>
>

Re: Number of bugs - statistics

[Adrian Bunk]

On Thu, May 22, 2008 at 10:51:49AM -0700, Natalie Protasevich wrote:
> On Thu, May 22, 2008 at 10:33 AM, Adrian Bunk <bunk@kernel.org> wrote:
> > On Thu, May 22, 2008 at 10:08:49AM -0700, Natalie Protasevich wrote:
> >> On Thu, May 22, 2008 at 9:50 AM, Adrian Bunk <bunk@kernel.org> wrote:
> >>...
> >> >> by various criteria: ALSA bugs
> >> >> are numerous, which is not important for most enterprise server users
> >> >> who would completely disregard this category, whereas desktop users
> >> >> will probably concentrate on those more than any other.
> >> >
> >> > The majority of machines running Linux most likely runs with ARM CPUs.
> >> >
> >> > Show me any public source you want to use for getting serious data for
> >> > this area.
> >>
> >> There is no one, except dispersed reports and ... appropriate
> >> implementation involving search engine comes to mind.
> >
> > Much of this information will never be published at any place where
> > Google could find it.
> 
> But Adrian, there will be a chain reaction. That's why Andrew
> encourages interested parties from industry share code (which will
> mean bug reports). This is gradual process, way of collaboration and
> open source is being more and more accepted and even urged.

That's a completely different thing.

Example for what I'm talking about:

Customer says:
We know $driver in the kernel for their device is buggy, but they are 
the only ones manufacturing this kind of device we need.

Customer uses the device as part of some machines he sells.

Neither the manufactorer nor the customer do want to _ever_ become 
publically known that there are bugs in the device.

And that's not a made-up example, I know names for manufactorer and 
customer for such cases.

> >> >> So the answer
> >> >> to question about kernel stability would be more adequate depending on
> >> >> who's asking.
> >> >
> >> > You must not confuse "number of reported bugs" with "kernel stability" -
> >> > these two can be quite decoupled.
> >>
> >> Number of existing, proven and prioritized  bugs maybe better to say.
> >> I think there is a correlation.
> >
> > Some correlaton.
> >
> > If people are aware of the weaknesses and the many obvious ways to
> > manipulate such statistics.
> 
> ... which needs to start and take its own development cycle. Then it
> will become more sophisticated.

We do not need more sophisticated statistics, we need to find ways to 
get the bugs fixed.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Number of bugs - statistics

[Adrian Bunk]

On Thu, May 22, 2008 at 07:45:40PM +0200, Stefan Richter wrote:
> Adrian Bunk wrote:
>> On Thu, May 22, 2008 at 06:22:57PM +0200, Stefan Richter wrote:
>>> Adrian Bunk wrote:
>>>> And ALSA alone has 2000 open bug reports, which makes the open ACPI 
>>>> or SCSI bug numbers relatively irrelevant in any "number of bugs"  
>>>> statistics.
>>> Central facilities like ACPI and SCSI having relatively few [and
>>> according to Arjan a decreasing number of] reported unfixed bugs is...
>>> irrelevant?
>>
>> Yes, _in any "number of bugs" statistics_ that is irrelevant.
>
> Well, yes, in any "simplistic total number of bugs without any frame of  
> reference what is actually counted" statistics it is indeed irrelevant.

But this kind of statistics is the only one we could realistically ever 
get.

There is no simple and correct way to e.g. say whether some ACPI bug is 
more or less bad than some ALSA bug or which of them might affect more 
people.

> [...]
>> We do not even have good data for "number of bug reports".
>
> That's right.  We don't have comprehensive data, and it is difficult to  
> draw conclusions from the data that we have.
> 
> Stefan Richter

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Number of bugs - statistics

[Arjan van de Ven]

On Thu, 22 May 2008 19:50:08 +0300
Adrian Bunk <bunk@kernel.org> wrote:

> > by various criteria: ALSA bugs
> > are numerous, which is not important for most enterprise server
> > users who would completely disregard this category, whereas desktop
> > users will probably concentrate on those more than any other.
> 
> The majority of machines running Linux most likely runs with ARM CPUs.

and on a 2.4 kernel, and with likely with some binary modules at that.
it's cool to brag about market share, but these boxes unfortunately
don't have any value for Linux development.

> 
> Show me any public source you want to use for getting serious data
> for this area.

You know, and I hate to say this, I'm getting a bit tired of your
attitude of just throwing up your arms at the problem and at the same
time trying to discredit anything and anybody who tries to improve the
situation. Really, it's not helping nor does it improve ANYTHING. 

I gave you real data before based on first hand experience from
maintainers. We now have crash data, and we have data from every place
in the kernel that dumps a backtrace. We track regressions and we can
show that we attack the majority of those very agressively, and Linus
has delayed releases for serious regressions. Just throwing your hands
in the air and saying "it's hard don't even try and I don't want you to
try" doesn't cut it.

Are we perfect? No. Can we do better? Absolutely. Should we try to do
better? YES PLEASE. Lets get constructive and try to make things better.

In the past, a few simple things improved the situation, such as
printk'ing the kernel version in the oops, as well as (longer back)
kksymoops. Or think of things like lockdep, or the pagealloc-debug
stuff. Going forward there are things we can do to improve things in the
kernel, even if you don't have hardware. We need to make the kernel
more selfdiagnosing. Detecting the problem, and if it's likely a
driver/hw interaction bug, sticking in a WARN_ON(). That's the sort of
thing even starting kernel developers can do to help improve the
quality even wihtout hardware access, because it helps us improve the
quality of bugreports, and it improves our ability to find patterns and
group duplicates into a top 10. 

Adrian, how about working on that sort of thing rather than keep saying
"impossible, we're doomed"... pretty please?

Re: Number of bugs - statistics

[Adrian Bunk]

On Thu, May 22, 2008 at 03:18:34PM -0700, Arjan van de Ven wrote:
> On Thu, 22 May 2008 19:50:08 +0300
> Adrian Bunk <bunk@kernel.org> wrote:
> 
> > > by various criteria: ALSA bugs
> > > are numerous, which is not important for most enterprise server
> > > users who would completely disregard this category, whereas desktop
> > > users will probably concentrate on those more than any other.
> > 
> > The majority of machines running Linux most likely runs with ARM CPUs.
> 
> and on a 2.4 kernel,
>...

Do you have any serious numbers for that?

All the embedded stuff I've recently worked with had 2.6.2x kernels, 
and if you have good statistics how many percent of all ARM users and
in which areas still use kernel 2.4 I'd be very interested in them.

> > Show me any public source you want to use for getting serious data
> > for this area.
> 
> You know, and I hate to say this, I'm getting a bit tired of your
> attitude of just throwing up your arms at the problem and at the same
> time trying to discredit anything and anybody who tries to improve the
> situation. Really, it's not helping nor does it improve ANYTHING. 
> 
> I gave you real data before based on first hand experience from
> maintainers. We now have crash data, and we have data from every place
> in the kernel that dumps a backtrace. We track regressions and we can
> show that we attack the majority of those very agressively, and Linus
> has delayed releases for serious regressions. Just throwing your hands
> in the air and saying "it's hard don't even try and I don't want you to
> try" doesn't cut it.
> 
> Are we perfect? No. Can we do better? Absolutely. Should we try to do
> better? YES PLEASE. Lets get constructive and try to make things better.
> 
> In the past, a few simple things improved the situation, such as
> printk'ing the kernel version in the oops, as well as (longer back)
> kksymoops. Or think of things like lockdep, or the pagealloc-debug
> stuff. Going forward there are things we can do to improve things in the
> kernel, even if you don't have hardware. We need to make the kernel
> more selfdiagnosing. Detecting the problem, and if it's likely a
> driver/hw interaction bug, sticking in a WARN_ON(). That's the sort of
> thing even starting kernel developers can do to help improve the
> quality even wihtout hardware access, because it helps us improve the
> quality of bugreports, and it improves our ability to find patterns and
> group duplicates into a top 10. 
> 
> Adrian, how about working on that sort of thing rather than keep saying
> "impossible, we're doomed"... pretty please?

No disagreement that we can and should do much better on diagnosing and
fixing bugs (and I do appreciate your work in this area). And this is
neither impossible nor are we not doomed here.

But getting _statistics_ that would allow to reasonably express stuff 
like "stability of the kernel" in one number is nearly impossible: 
Including distribution bugs we are talking about a five digit number of 
open bug reports, and getting meaningful data from them is *ahem* hard.

And in case anyone says "not forwarded distribution bugs aren't our 
problem":
That's a nice example, since in this case a distribuion suddenly 
starting to agressively forwarding their bugs might contribute to
making the kernel better, but might also totally ruin the numbers in
any statistics.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Number of bugs - statistics

[James Courtier-Dutton]

Adrian Bunk wrote:
> On Thu, May 22, 2008 at 09:20:46AM -0700, Natalie Protasevich wrote:
>   
>> On Thu, May 22, 2008 at 8:54 AM, Adrian Bunk <bunk@kernel.org> wrote:
>>     
>>> Andrew sees and handles the majority of incoming bug reports.
>>> Ask him whether he agress with this.
>>>
>>> And ALSA alone has 2000 open bug reports, which makes the open ACPI or
>>> SCSI bug numbers relatively irrelevant in any "number of bugs"
>>> statistics.
>>>
>>>       
> Not everyone uses Bugzilla (e.g. ALSA uses Mantis).
>
> And the majority of bug reports might still go only to mailing lists and 
> not into any bugtracker at all. As long as this happens the data is 
> simply not available.
>
>   
>> by various criteria: ALSA bugs
>> are numerous, which is not important for most enterprise server users
>> who would completely disregard this category, whereas desktop users
>> will probably concentrate on those more than any other.
>>     
>
>   
Although ALSA might have a lot of bug reports. Any bugs that affect 
kernel stability are fixed quickly.
The rest of the ALSA bugs are more along the lines of "This feature on 
this particular sound card variant does not seem to work". This 
generally implies un-implemented features that uses now want. Some times 
the fix is simply adding that variant to the quirks list and finding out 
which quirk should be applied.
So, once one removes the "un-implemented features" category from the 
ALSA bug list, one would expect the remaining number to be low.

Re: Number of bugs - statistics

[Arjan van de Ven]

On Fri, 23 May 2008 12:09:40 +0300
Adrian Bunk <bunk@kernel.org> wrote:

> 
> And in case anyone says "not forwarded distribution bugs aren't our 
> problem":
> That's a nice example, since in this case a distribuion suddenly 
> starting to agressively forwarding their bugs might contribute to
> making the kernel better, but might also totally ruin the numbers in
> any statistics.

Fedora gets less than 1000 bugs per release (so 6 month window).
That's... peanuts.
(Also remember that distribution bugs have a real percentage of "please
help me" support request bugs, so the actual nr of bugs is quite
lower. The number is also before duplicates are marked).

We are tracking over 3000 bugs per WEEK via other channels.

So yes I will say that distribution bugs, while nice if we get
visibility in the good ones, aren't the end-all thing.

Re: Number of bugs - statistics

[Takashi Iwai]

At Fri, 23 May 2008 13:35:03 +0300,
James Courtier-Dutton wrote:
> 
> Adrian Bunk wrote:
> > On Thu, May 22, 2008 at 09:20:46AM -0700, Natalie Protasevich wrote:
> >   
> >> On Thu, May 22, 2008 at 8:54 AM, Adrian Bunk <bunk@kernel.org> wrote:
> >>     
> >>> Andrew sees and handles the majority of incoming bug reports.
> >>> Ask him whether he agress with this.
> >>>
> >>> And ALSA alone has 2000 open bug reports, which makes the open ACPI or
> >>> SCSI bug numbers relatively irrelevant in any "number of bugs"
> >>> statistics.
> >>>
> >>>       
> > Not everyone uses Bugzilla (e.g. ALSA uses Mantis).
> >
> > And the majority of bug reports might still go only to mailing lists and 
> > not into any bugtracker at all. As long as this happens the data is 
> > simply not available.
> >
> >   
> >> by various criteria: ALSA bugs
> >> are numerous, which is not important for most enterprise server users
> >> who would completely disregard this category, whereas desktop users
> >> will probably concentrate on those more than any other.
> >>     
> >
> >   
> Although ALSA might have a lot of bug reports. Any bugs that affect 
> kernel stability are fixed quickly.
> The rest of the ALSA bugs are more along the lines of "This feature on 
> this particular sound card variant does not seem to work". This 
> generally implies un-implemented features that uses now want. Some times 
> the fix is simply adding that variant to the quirks list and finding out 
> which quirk should be applied.
> So, once one removes the "un-implemented features" category from the 
> ALSA bug list, one would expect the remaining number to be low.

... and a normal screening would reduce the amount of bugs to 1/4, I
guess.  But, we have a serious problem regarding man power in this
area, unfortunately.


Takashi

Re: Number of bugs - statistics

[Adrian Bunk]

On Fri, May 23, 2008 at 07:11:57AM -0700, Arjan van de Ven wrote:
> On Fri, 23 May 2008 12:09:40 +0300
> Adrian Bunk <bunk@kernel.org> wrote:
> 
> > 
> > And in case anyone says "not forwarded distribution bugs aren't our 
> > problem":
> > That's a nice example, since in this case a distribuion suddenly 
> > starting to agressively forwarding their bugs might contribute to
> > making the kernel better, but might also totally ruin the numbers in
> > any statistics.
> 
> Fedora gets less than 1000 bugs per release (so 6 month window).
> That's... peanuts.

2000 kernel bugs per year?

Then Fedora alone gets roughly as many kernel bugs as the kernel 
Bugzilla.

>...
> We are tracking over 3000 bugs per WEEK via other channels.
>...

What exactly are you measuring with this "3000 bugs" number?

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Number of bugs - statistics

[Arjan van de Ven]

On Fri, 23 May 2008 19:50:55 +0300
Adrian Bunk <bunk@kernel.org> wrote:

> On Fri, May 23, 2008 at 07:11:57AM -0700, Arjan van de Ven wrote:
> > On Fri, 23 May 2008 12:09:40 +0300
> > Adrian Bunk <bunk@kernel.org> wrote:
> > 
> > > 
> > > And in case anyone says "not forwarded distribution bugs aren't
> > > our problem":
> > > That's a nice example, since in this case a distribuion suddenly 
> > > starting to agressively forwarding their bugs might contribute to
> > > making the kernel better, but might also totally ruin the numbers
> > > in any statistics.
> > 
> > Fedora gets less than 1000 bugs per release (so 6 month window).
> > That's... peanuts.
> 
> 2000 kernel bugs per year?

it's going down; 750-ish for the last few releases

(see http://kernelslacker.livejournal.com/116718.html )
> 
> Then Fedora alone gets roughly as many kernel bugs as the kernel 
> Bugzilla.

but kernel bugzilla is "only" a last resort, while fedora bugzilla is a
first point of contact. I'd rate a kernel.org bugzilla bug to be 3x a
fedora bz bug in terms of value (eg less dupes, it's not support
requests but actual bugs etc etc).

> 
> >...
> > We are tracking over 3000 bugs per WEEK via other channels.
> >...
> 
> What exactly are you measuring with this "3000 bugs" number?

Oopses and places where the kernel goes WARN_ON().
(the oops:warning ratio is roughly 1:2 to 1:3)

Re: Number of bugs - statistics

[Adrian Bunk]

On Fri, May 23, 2008 at 11:29:08AM -0700, Arjan van de Ven wrote:
> On Fri, 23 May 2008 19:50:55 +0300
> Adrian Bunk <bunk@kernel.org> wrote:
> 
> > On Fri, May 23, 2008 at 07:11:57AM -0700, Arjan van de Ven wrote:
> > > On Fri, 23 May 2008 12:09:40 +0300
> > > Adrian Bunk <bunk@kernel.org> wrote:
> > > 
> > > > 
> > > > And in case anyone says "not forwarded distribution bugs aren't
> > > > our problem":
> > > > That's a nice example, since in this case a distribuion suddenly 
> > > > starting to agressively forwarding their bugs might contribute to
> > > > making the kernel better, but might also totally ruin the numbers
> > > > in any statistics.
> > > 
> > > Fedora gets less than 1000 bugs per release (so 6 month window).
> > > That's... peanuts.
> > 
> > 2000 kernel bugs per year?
> 
> it's going down; 750-ish for the last few releases
> 
> (see http://kernelslacker.livejournal.com/116718.html )

It's a bit suspicious that the "Total number of bugs filed across all 
packages" also went down by quite some amount.

Are recent Fedora kernels less buggy or do less people file bugs since 
everyone now uses Ubuntu?

>...
> > > We are tracking over 3000 bugs per WEEK via other channels.
> > >...
> > 
> > What exactly are you measuring with this "3000 bugs" number?
> 
> Oopses and places where the kernel goes WARN_ON().
> (the oops:warning ratio is roughly 1:2 to 1:3)

Counting 2248 times the same WARN_ON triggered as 2248 distinct bugs is 
complete bullshit.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Number of bugs - statistics

[Dave Jones]

On Fri, May 23, 2008 at 11:31:12PM +0300, Adrian Bunk wrote:
 > On Fri, May 23, 2008 at 11:29:08AM -0700, Arjan van de Ven wrote:
 > > On Fri, 23 May 2008 19:50:55 +0300
 > > Adrian Bunk <bunk@kernel.org> wrote:
 > > 
 > > > On Fri, May 23, 2008 at 07:11:57AM -0700, Arjan van de Ven wrote:
 > > > > On Fri, 23 May 2008 12:09:40 +0300
 > > > > Adrian Bunk <bunk@kernel.org> wrote:
 > > > > 
 > > > > > 
 > > > > > And in case anyone says "not forwarded distribution bugs aren't
 > > > > > our problem":
 > > > > > That's a nice example, since in this case a distribuion suddenly 
 > > > > > starting to agressively forwarding their bugs might contribute to
 > > > > > making the kernel better, but might also totally ruin the numbers
 > > > > > in any statistics.
 > > > > 
 > > > > Fedora gets less than 1000 bugs per release (so 6 month window).
 > > > > That's... peanuts.
 > > > 
 > > > 2000 kernel bugs per year?
 > > 
 > > it's going down; 750-ish for the last few releases
 > > 
 > > (see http://kernelslacker.livejournal.com/116718.html )
 > 
 > It's a bit suspicious that the "Total number of bugs filed across all 
 > packages" also went down by quite some amount.

The last two releases hadn't reached end-of-life yet. I just updated
that entry, with the latest numbers for F7/F8, and threw in F9 too.

      total    kernel
F7    5534     724
F8    7589     902
F9    4624     292

Given F9 has only been out a few weeks, that's quite high.  As more
people move from F7 -> F8 as F7 gets closer to end of life, the bug
count there is shifting too.
(for point of reference, F7 is on 2.6.24, F8 is about to get a .25 update,
 F9 was released with .25)

 > Are recent Fedora kernels less buggy or do less people file bugs since 
 > everyone now uses Ubuntu?

My majority of my job is still spending reading and dealing with bugzilla.
I think if there were a noticable drop-off, I'd have more time to be
sitting around eating cheetos or doing something else.  That hasn't happened.

	Dave

-- 
http://www.codemonkey.org.uk