How to reduce the number of open kernel bugs

How to reduce the number of open kernel bugs

[Adrian Bunk]

Just seen in the kernel Bugzilla (driver anonymized to foobar):


Submitter [opens bug]:

Hardware Environment: See my other reports

This just happens while I were on surfing in the internet:
[ WARN_ON() traces involving driver foobar ]


Adrian [adds driver maintainer to Cc]


Maintainer [closes bug]:

Your hardware is seriously broken.


Submitter [reopens bug]:

Thanks for your answer, but I think that my hardware isn't broken, 
because all my reported issues don't occur on Windows and also not with 
foobaz.


Maintainer [closes bug]:

Then something in the chipset of PCI bus driver is broken. Also note the 
APIC errors. This certainly is not a foobar bug. The driver is just 
unable to access the card through the PCI bus, so it throws errors.


Submitter:

Interesting.

The APIC CPU errors are there since I've got the laptop, with and 
without foobar ;)


Maintainer:

Yeah, in any case. I cannot fix it, since it's not a bug in the fobar
code. Please reopen a new bug and CC the architecture or PCI maintainer 
or whatever person related to the bus, chipset or CPU, if you think the 
foobar device still works. If the foobar hardware got corrupted, you 
already know what to do...





I am well aware that loud flames are often the only working way of 
communication in Linux kernel development, but we mustn't communicate 
this way with bug submitters.


cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: How to reduce the number of open kernel bugs

[Parag Warudkar]

Adrian Bunk <bunk <at> kernel.org> writes:

> Maintainer:
>
> Yeah, in any case. I cannot fix it, since it's not a bug in the fobar
> code. Please reopen a new bug and CC the architecture or PCI maintainer
> or whatever person related to the bus, chipset or CPU, if you think the
> foobar device still works. If the foobar hardware got corrupted, you
> already know what to do...
>
> I am well aware that loud flames are often the only working way of
> communication in Linux kernel development, but we mustn't communicate
> this way with bug submitters.
>

Actually this way of _communication_ is better because the maintainer has -

a) at least seen the bug
b) made it clear upfront that he/she is not in a position to fix it and
c) not inflicted a huge amount of follow up work for the reporter
while giving no hope that it will be fixed.

There is not much you can do if the maintainer feels he/she can't do
anything - apart from fixing it yourself which has its limits.
So the best that can be done is to communicate it clearly - that
happens in this case.

Compare the above to -

* Reporter reports a bug.

* People in position to fix it  have no care for it and they show it
(By not even looking at it).

* Significant amount of time passes with no activity.

* Reporter receives a automated mail (bonus insult) - New release is
available, please retest with new release and open a new bug if the
problem still exists.

* Bug is closed

* Bug still exists in next release - reporter knows only when he/she
gets a chance to upgrade to next release (not always easy/possible).

Parag

Re: How to reduce the number of open kernel bugs

[Adrian Bunk]

On Fri, May 02, 2008 at 10:42:21AM -0400, Parag Warudkar wrote:
> Adrian Bunk <bunk <at> kernel.org> writes:
> 
> > Maintainer:
> >
> > Yeah, in any case. I cannot fix it, since it's not a bug in the fobar
> > code. Please reopen a new bug and CC the architecture or PCI maintainer
> > or whatever person related to the bus, chipset or CPU, if you think the
> > foobar device still works. If the foobar hardware got corrupted, you
> > already know what to do...
> >
> > I am well aware that loud flames are often the only working way of
> > communication in Linux kernel development, but we mustn't communicate
> > this way with bug submitters.
> >
> 
> Actually this way of _communication_ is better because the maintainer has -

When immediately closing a new bug with a comment only consisting of 
"Your hardware is seriously broken." it's borderline whether you can 
call this "communication" - especially if the hardware works under 
Windows or with a different driver.

> a) at least seen the bug
> b) made it clear upfront that he/she is not in a position to fix it and
> c) not inflicted a huge amount of follow up work for the reporter
> while giving no hope that it will be fixed.

And gave the bug reporter (who might have spent some time on writing a 
proper bug report) the clear impression his bug reports are unwanted.

> There is not much you can do if the maintainer feels he/she can't do
> anything - apart from fixing it yourself which has its limits.
> So the best that can be done is to communicate it clearly - that
> happens in this case.
>...

He can ask for the output of "dmesg" and then reassign the bug to the 
correct people.

Different to immediately closing the bug with the comment
"Your hardware is seriously broken." this can result in the
bug actually being fixed.

> Parag

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: How to reduce the number of open kernel bugs

[Daniel Hazelton]

On Friday 02 May 2008 10:42:21 Parag Warudkar wrote:
> Adrian Bunk <bunk <at> kernel.org> writes:
> > Maintainer:
> >
> > Yeah, in any case. I cannot fix it, since it's not a bug in the fobar
> > code. Please reopen a new bug and CC the architecture or PCI maintainer
> > or whatever person related to the bus, chipset or CPU, if you think the
> > foobar device still works. If the foobar hardware got corrupted, you
> > already know what to do...
> >
> > I am well aware that loud flames are often the only working way of
> > communication in Linux kernel development, but we mustn't communicate
> > this way with bug submitters.
>
> Actually this way of _communication_ is better because the maintainer has -
>
> a) at least seen the bug
> b) made it clear upfront that he/she is not in a position to fix it and
> c) not inflicted a huge amount of follow up work for the reporter
> while giving no hope that it will be fixed.
>
> There is not much you can do if the maintainer feels he/she can't do
> anything - apart from fixing it yourself which has its limits.
> So the best that can be done is to communicate it clearly - that
> happens in this case.

With the above behavior, someone is reporting "I can use OS's A, B and C with 
no problems - the hardware works perfectly. When I try to use Linux X.Y.Z the 
driver faults and makes things unusable." and the maintainer is saying "The 
driver is fine. I didn't write code that has any bugs in it. The problem is 
your hardware is broken."

That kind of response is irresponsible and idiotic. If the hardware works 
perfectly in every other OS - and possibly even previous versions of Linux - 
then obviously the hardware isn't broken. This is what Adrian was pointing 
out and is exactly what shouldn't be happening.

DRH

-- 
Dialup is like pissing through a pipette. Slow and excruciatingly painful.

Re: How to reduce the number of open kernel bugs

[Andi Kleen]

Daniel Hazelton <dhazelton@enter.net> writes:
>  If the hardware works 
> perfectly in every other OS

It depends. There used to be the famous case (not longer true now) that 
older Linux allocated memory from the top of the memory down and other OS
generally allocated it the from bottom to up and when some of your top
memory was broken Linux would often hit problems where other OS did not.

Also there can be the case when some OS use hardware quite differently
or different hardware features than other. Standard case for example
used to be that there were a few platforms where the SMM code had 64bit
bugs and of course you would only hit them when running a 64bit OS which
was Linux. 

A modern OS is a very complicated system with tens of millions of code
lines and you can't assume they all program the hardware in the same way.

Simple truths are often wrong.

 - and possibly even previous versions of Linux - 

Now that is a more interesting case. But regressions are always taken
seriously by all maintainers I know.

-Andi

Re: How to reduce the number of open kernel bugs

[Parag Warudkar]

On Fri, May 2, 2008 at 12:29 PM, Daniel Hazelton <dhazelton@enter.net> wrote:
This is what Adrian was pointing
> ... out and is exactly what shouldn't be happening.
>

I wasn't arguing that it is what should be happening - I was just
pointing out that there are worse things than that routinely happen in
distro bugzillas!

Also I feel that if maintainer refuse to fix something - that's a dead
end as far as that particular bug is considered. We can try and track
such bugs in different STATUS but there would have to be people
interested in digging through such bugs and willing to fix it after
the maintainer has given up. Which is possible but not likely.

Parag

Re: How to reduce the number of open kernel bugs

[Daniel Hazelton]

On Friday 02 May 2008 13:27:04 Andi Kleen wrote:
> Daniel Hazelton <dhazelton@enter.net> writes:
> >  If the hardware works
> > perfectly in every other OS
>
> It depends. There used to be the famous case (not longer true now) that
> older Linux allocated memory from the top of the memory down and other OS
> generally allocated it the from bottom to up and when some of your top
> memory was broken Linux would often hit problems where other OS did not.

True.

> Also there can be the case when some OS use hardware quite differently
> or different hardware features than other. Standard case for example
> used to be that there were a few platforms where the SMM code had 64bit
> bugs and of course you would only hit them when running a 64bit OS which
> was Linux.

Sadly things like this will continue as long as there is a single OS that has 
more than a 70% market share of the common desktop market.

> A modern OS is a very complicated system with tens of millions of code
> lines and you can't assume they all program the hardware in the same way.
>
> Simple truths are often wrong.

Yes, but if the hardware *ISN'T* broken, then claiming it is and closing out a 
bug is the wrong thing to do. The right thing to do would be to find out why 
the hardware isn't working correctly for the OS/driver combination in 
question and, if possible, fix the driver.

>  - and possibly even previous versions of Linux -
>
> Now that is a more interesting case. But regressions are always taken
> seriously by all maintainers I know.
>

Look to the bug-report posted (cleaned of evidence that would incriminate any 
single person) by (IIRC) Adrian. The guy says "It worked with X and works in 
Windows, but it now fails with message Z."

Maintainer says "Your hardware is broken", closes the bug and then goes on to 
say "If the hardware isn't broken, talk to the people in charge of A and B 
because my code isn't buggy."

That is idiotic and immature.

DRH

-- 
Dialup is like pissing through a pipette. Slow and excruciatingly painful.

Re: How to reduce the number of open kernel bugs

[Daniel Hazelton]

On Friday 02 May 2008 13:30:23 Parag Warudkar wrote:
> On Fri, May 2, 2008 at 12:29 PM, Daniel Hazelton <dhazelton@enter.net>
> wrote: This is what Adrian was pointing
>
> > ... out and is exactly what shouldn't be happening.
>
> I wasn't arguing that it is what should be happening - I was just
> pointing out that there are worse things than that routinely happen in
> distro bugzillas!

I have to admit that this is the truth. I've seen it myself. (Hell, I won't 
touch a bug-tracking system for that reason.)

> Also I feel that if maintainer refuse to fix something - that's a dead
> end as far as that particular bug is considered. We can try and track
> such bugs in different STATUS but there would have to be people
> interested in digging through such bugs and willing to fix it after
> the maintainer has given up. Which is possible but not likely.
>
> Parag

And I have to agree here. But there should be something that can be done 
to "coax" the maintainers that pull that kind of immature crap into acting 
mature and responsible.

DRH

-- 
Dialup is like pissing through a pipette. Slow and excruciatingly painful.

Re: How to reduce the number of open kernel bugs

[Rafael J. Wysocki]

On Friday, 2 of May 2008, Daniel Hazelton wrote:
> On Friday 02 May 2008 13:30:23 Parag Warudkar wrote:
> > On Fri, May 2, 2008 at 12:29 PM, Daniel Hazelton <dhazelton@enter.net>
> > wrote: This is what Adrian was pointing
> >
> > > ... out and is exactly what shouldn't be happening.
> >
> > I wasn't arguing that it is what should be happening - I was just
> > pointing out that there are worse things than that routinely happen in
> > distro bugzillas!
> 
> I have to admit that this is the truth. I've seen it myself. (Hell, I won't 
> touch a bug-tracking system for that reason.)
> 
> > Also I feel that if maintainer refuse to fix something - that's a dead
> > end as far as that particular bug is considered. We can try and track
> > such bugs in different STATUS but there would have to be people
> > interested in digging through such bugs and willing to fix it after
> > the maintainer has given up. Which is possible but not likely.
> >
> > Parag
> 
> And I have to agree here. But there should be something that can be done 
> to "coax" the maintainers that pull that kind of immature crap into acting 
> mature and responsible.

Well, how exactly would you want to do that?

Rafael

Re: How to reduce the number of open kernel bugs

[Adrian Bunk]

On Fri, May 02, 2008 at 01:30:23PM -0400, Parag Warudkar wrote:
> On Fri, May 2, 2008 at 12:29 PM, Daniel Hazelton <dhazelton@enter.net> wrote:
> This is what Adrian was pointing
> > ... out and is exactly what shouldn't be happening.
> >
> 
> I wasn't arguing that it is what should be happening - I was just
> pointing out that there are worse things than that routinely happen in
> distro bugzillas!

And the point is?

If I would kill a man it wouldn't become better by pointing out that 
other people were responsible for the killing of several million men.

> Also I feel that if maintainer refuse to fix something - that's a dead
> end as far as that particular bug is considered. We can try and track
> such bugs in different STATUS but there would have to be people
> interested in digging through such bugs and willing to fix it after
> the maintainer has given up. Which is possible but not likely.


You completely miss the point.


>From what I forwarded it's clear that we have a bug somewhere in the 
kernel.

And the maintainer might well be right that it's not in his driver.

Which means to reassign the bug to the maintainer of the area this bug 
is in.


And in any case, being more friendly to a bug submitter.


> Parag

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: How to reduce the number of open kernel bugs

[Andi Kleen]

"Rafael J. Wysocki" <rjw@sisk.pl> writes:
>> 
>> And I have to agree here. But there should be something that can be done 
>> to "coax" the maintainers that pull that kind of immature crap into acting 
>> mature and responsible.
>
> Well, how exactly would you want to do that?

He could start to fix a few such bugs he's describing (usually with
very vague reports) by himself. Or instead of fixing them just trying
to get all the information out of the reporter. Then he would realize
what the the difference between theory and practice is.

-Andi

Re: How to reduce the number of open kernel bugs

[Parag Warudkar]

On Fri, May 2, 2008 at 2:01 PM, Adrian Bunk <bunk@kernel.org> wrote:
> On Fri, May 02, 2008 at 01:30:23PM -0400, Parag Warudkar wrote:
>  > I wasn't arguing that it is what should be happening - I was just
>  > pointing out that there are worse things than that routinely happen in
>  > distro bugzillas!
>
>  And the point is?
>

Point being that no one can force anyone to do anything in the setting
in which kernel developers work.
Distros have been doing this since ever and people have no choice but
to move on.
(Everybody works out of their own (or employer's) interest and
judgement - so telling some one to act otherwise will not be a whole
lot productive.)

>
>  You completely miss the point.

I am not - my point is that you can't do  much about it that will go
long way in resolving the issue.

>  And in any case, being more friendly to a bug submitter.

Well again -  you can't force people to do something like that. In
your example it would amount to the maintainer going against his/her
own judgement - however wrong it is to you and me, to the person who
can actually fix it - it's not a possibility. So you have to just
accept it and reassign it to some one else who is interested/capable -
for that I suggested tracking such bugs under a new STATUS.

Parag

Re: How to reduce the number of open kernel bugs

[Daniel Hazelton]

On Friday 02 May 2008 14:18:03 Andi Kleen wrote:
> "Rafael J. Wysocki" <rjw@sisk.pl> writes:
> >> And I have to agree here. But there should be something that can be done
> >> to "coax" the maintainers that pull that kind of immature crap into
> >> acting mature and responsible.
> >
> > Well, how exactly would you want to do that?
>
> He could start to fix a few such bugs he's describing (usually with
> very vague reports) by himself. Or instead of fixing them just trying
> to get all the information out of the reporter. Then he would realize
> what the the difference between theory and practice is.
>
> -Andi

Andi (and everyone else) note that the key word in what I said is "Should".  
IE: there is nothing that can be done, but I feel that the opposite should be 
true.

DRH

-- 
Dialup is like pissing through a pipette. Slow and excruciatingly painful.

Re: How to reduce the number of open kernel bugs

[Daniel Hazelton]

On Friday 02 May 2008 14:01:51 Adrian Bunk wrote:
> On Fri, May 02, 2008 at 01:30:23PM -0400, Parag Warudkar wrote:
> > On Fri, May 2, 2008 at 12:29 PM, Daniel Hazelton <dhazelton@enter.net>
> > wrote: This is what Adrian was pointing
<snip>
>
> From what I forwarded it's clear that we have a bug somewhere in the
> kernel.
>
> And the maintainer might well be right that it's not in his driver.

This is something that can't be known without enough information from the 
person reporting the bug. From what you posted of the bugzilla entry it's 
unclear whether the reporter did post more than his simple description or 
not.

However, the point is well made and I, at least, agree. (whatever that may be 
worth - I've never managed to finish a patch for anything without seeing the 
equivalent show up here later)

> Which means to reassign the bug to the maintainer of the area this bug
> is in.

Yep. Closing it with a comment of "broken hardware" is an incorrect response. 
Doubly or trebly so if the bugzilla entry didn't contain enough information 
to even determine whether the bug was in the driver or not.

DRH

-- 
Dialup is like pissing through a pipette. Slow and excruciatingly painful.

Re: How to reduce the number of open kernel bugs

[SL Baur]

On 5/2/08, Adrian Bunk <bunk@kernel.org> wrote:
> Just seen in the kernel Bugzilla (driver anonymized to foobar):

 ... An all too common sequence

This is the kind of stuff that naturally happens when you tie performance
to the progression of bug reports through bug tracking systems.  It's
human nature.

It's not good, but it's human nature.  This situation is *not* unique to Linux.

-sb (My day job is the care and feeding of a Fortune 50 company's internal bug
tracking system)

Re: How to reduce the number of open kernel bugs

[Adrian Bunk]

On Sat, May 03, 2008 at 03:13:21AM -0700, SL Baur wrote:
> On 5/2/08, Adrian Bunk <bunk@kernel.org> wrote:
> > Just seen in the kernel Bugzilla (driver anonymized to foobar):
> 
>  ... An all too common sequence
> 
> This is the kind of stuff that naturally happens when you tie performance
> to the progression of bug reports through bug tracking systems.  It's
> human nature.

That's a problem elsewhere, but not the problem here.

As far as I know this maintainer developed and maintaines this driver as 
a hobby.

> It's not good, but it's human nature.  This situation is *not* unique to Linux.
> 
> -sb (My day job is the care and feeding of a Fortune 50 company's internal bug
> tracking system)

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: How to reduce the number of open kernel bugs

[SL Baur]

On 5/3/08, Adrian Bunk <bunk@kernel.org> wrote:

> That's a problem elsewhere, but not the problem here.
>
>  As far as I know this maintainer developed and maintaines this driver as
>  a hobby.

I'm not sure what point you're trying to make.  The point I was
trying to make was against the folks who were suggesting
adding certain metrics to Linux kernel patching.  The biggest
number of reversions script that was just posted pointed at
Linus and Andrew as the biggest villains, or that's how I read
it and that is clearly bogus.

I have had similar experiences with people *paid* to handle
bug reports, requests, etc. and had eerily similar dialogs.  Why
should a volunteer working in his spare time behave any
differently?

What am I missing?

-sb

Re: How to reduce the number of open kernel bugs

[Stefan Richter]

Adrian Bunk wrote:
> On Sat, May 03, 2008 at 03:13:21AM -0700, SL Baur wrote:
>> On 5/2/08, Adrian Bunk <bunk@kernel.org> wrote:
>>> Just seen in the kernel Bugzilla (driver anonymized to foobar):
>>  ... An all too common sequence
>>
>> This is the kind of stuff that naturally happens when you tie performance
>> to the progression of bug reports through bug tracking systems.  It's
>> human nature.
> 
> That's a problem elsewhere, but not the problem here.
> 
> As far as I know this maintainer developed and maintaines this driver as 
> a hobby.
> 
>> It's not good, but it's human nature.  This situation is *not* unique to Linux.
>>
>> -sb (My day job is the care and feeding of a Fortune 50 company's internal bug
>> tracking system)

Adrian, what is it that you criticize?  Was it an impolite tone in the 
maintainer's responses, or do you believe that the maintainer lied to 
the reporter?

Or did the maintainer miss to do something which he would be able to and 
had the resources to do?

Or did the maintainer refuse to fulfill an obligation?

If it is one of the latter points, is it that the maintainer failed, 
according to your observation, to explain in more detail why he closed 
the bug early, in a way that a person without deep technical insight 
into the problem domain can parse?  Or to ask for information which 
would bring the bug forward?  Or to analyze the problem deeper on his 
own?  Or use the bugzilla system in a more sophisticated way (reassign, 
set status, change title... instead of closing the bug and requiring the 
reporter to create a different bug entry)?

Not as an excuse for anything, but to bring in another perspective, let 
me remind that especially maintainers who work in spare time often 
handle bugs at daytimes when the ability to analyze technical or 
procedural problems is reduced.  That's because there is little choice 
when to work on what.  (People who work professionally in this area 
hopefully do so under proper working conditions.)

Or a maintainer may not be fluent in the most efficient use of the bug 
tracker, or bug handling in general.  If so, then there need to be 
persons who assist in the bug handling.  Ideally the maintainer will 
gather experience in effective bug handling over time.  (Again, people 
who do this professionally hopefully received the necessary training. 
Also, an employer who is interested in good relationships with customers 
takes care that his contact staff is socially skilled and motivated.)

If you want maintainers to change their behavior, what are you going to 
do?  Raise awareness (like you apparently intended to by starting this 
thread and a number of other of your postings)?  Do you want to motivate 
maintainers in a negative way (make them feel guilty) or in some 
positive way?  Do you want to or have ideas how to improve process 
skills of the maintainers?  Are you going to find ways to lower 
maintainers' workload?

If you feel that obligations haven't been fulfilled in the case which 
you presented:  What is founding those obligations?

Lastly, what do you think does the number of open kernel bugs (or rather 
the inverse of it) mean to people?  Is it something like having an 
expensive looking car parked in front of the house?  I would say 
bugzilla.kernel.org is too obscure and cryptic to be able to fulfill 
that kind of role.
-- 
Stefan Richter
-=====-==--- -=-= ---==
http://arcgraph.de/sr/

Re: How to reduce the number of open kernel bugs

[Stefan Richter]

I wrote:
> Are you going to find ways to lower maintainers' workload?

PS:  Of course I am well aware that you indeed already do extensive work 
with bug triage, ask reporters for information etc..  Thanks BTW.
-- 
Stefan Richter
-=====-==--- -=-= ---==
http://arcgraph.de/sr/

Re: How to reduce the number of open kernel bugs

[Adrian Bunk]

On Sat, May 03, 2008 at 05:43:25PM +0200, Stefan Richter wrote:
>
> Adrian, what is it that you criticize?  Was it an impolite tone in the  
> maintainer's responses, or do you believe that the maintainer lied to  
> the reporter?
> 
> Or did the maintainer miss to do something which he would be able to and  
> had the resources to do?
>
> Or did the maintainer refuse to fulfill an obligation?
> 
> If it is one of the latter points, is it that the maintainer failed,  
> according to your observation, to explain in more detail why he closed  
> the bug early, in a way that a person without deep technical insight  
> into the problem domain can parse?  Or to ask for information which  
> would bring the bug forward?  Or to analyze the problem deeper on his  
> own?  Or use the bugzilla system in a more sophisticated way (reassign,  
> set status, change title... instead of closing the bug and requiring the  
> reporter to create a different bug entry)?
>...

The only thing that was really bad was the tone.

When anyone says "he could anyway not done anything about the bug" 
that's not true since he e.g. noted himself APIC errors in the report,
and could have reassigned the bug accordingly. Or noted it in the 
bug - when I touch a bug (as I did with this one before) I'm putting 
myself to the Cc, and I read what happens later in the bug.

But that's not the problem. The problem was the (repeated) closing of a 
bug in a way that expresses to a submitter that bug reports were 
unwanted.

> Stefan Richter

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed