Tracking and crediting bug reporters

Tracking and crediting bug reporters

[Jonathan Corbet]

Several members of the Linux Foundation's Technical Advisory Board recently
got together with Andrew Morton to talk about kernel quality issues.  One
of the things which came out of that meeting was a desire to improve
incentives for people who report bugs.  Clearly, actually fixing those bugs
would qualify; nobody has lost sight of that.  But it was suggested that
the creation and publication of statistics on bug reporting would also
help.

One way to do this might be for Andrew (being the only one who actually
reads every message posted on the list) to keep a spreadsheet along with
everything else he does.  That idea did not go over very well.

So here's what we would like to try instead.  Whenever somebody sends up a
patch fixing a reported bug, the name of the person who reported the bug
would be immortalized with this tag:

	Reported-by: A. Bug Reporter <email@goes.here>

In particular, reporters who work with the developers toward the resolution
of the bug should be thanked in this way.  If we wanted to take things
further, perhaps we could add a Bisected-by: tag for really hard-core
helpers. 

If these tags go into the commit messages in any sort of consistent way, it
should be possible generate the usual sort of statistics from them.  I'll
then happily publicize them next to the traditional lists of people who are
adding new bugs.  The result will certainly be fame, fortune, and job
offers for the people at the top of the list.  Or something like that.

If the rest of the community is agreeable, it would be nice to make an
immediate start on this; it's not yet too late to get reasonable data for
the 2.6.26 kernel, and to have the habits well ingrained for 2.6.27.

Thoughts?

jon

Re: Tracking and crediting bug reporters

[Cyrill Gorcunov]

[Jonathan Corbet - Mon, May 12, 2008 at 11:27:52AM -0600]
| Several members of the Linux Foundation's Technical Advisory Board recently
| got together with Andrew Morton to talk about kernel quality issues.  One
| of the things which came out of that meeting was a desire to improve
| incentives for people who report bugs.  Clearly, actually fixing those bugs
| would qualify; nobody has lost sight of that.  But it was suggested that
| the creation and publication of statistics on bug reporting would also
| help.
| 
| One way to do this might be for Andrew (being the only one who actually
| reads every message posted on the list) to keep a spreadsheet along with
| everything else he does.  That idea did not go over very well.
| 
| So here's what we would like to try instead.  Whenever somebody sends up a
| patch fixing a reported bug, the name of the person who reported the bug
| would be immortalized with this tag:
| 
| 	Reported-by: A. Bug Reporter <email@goes.here>
| 
| In particular, reporters who work with the developers toward the resolution
| of the bug should be thanked in this way.  If we wanted to take things
| further, perhaps we could add a Bisected-by: tag for really hard-core
| helpers. 
| 
| If these tags go into the commit messages in any sort of consistent way, it
| should be possible generate the usual sort of statistics from them.  I'll
| then happily publicize them next to the traditional lists of people who are
| adding new bugs.  The result will certainly be fame, fortune, and job
| offers for the people at the top of the list.  Or something like that.
| 
| If the rest of the community is agreeable, it would be nice to make an
| immediate start on this; it's not yet too late to get reasonable data for
| the 2.6.26 kernel, and to have the habits well ingrained for 2.6.27.
| 
| Thoughts?
| 
| jon
| 

If my opinion does worth somehow - I'm absolutely agree!

(btw, it seems we forget the main tag ever was - Bug-made-by: ;)

		- Cyrill -

Re: Tracking and crediting bug reporters

[Alan Cox]

> If the rest of the community is agreeable, it would be nice to make an
> immediate start on this; it's not yet too late to get reasonable data for
> the 2.6.26 kernel, and to have the habits well ingrained for 2.6.27.
> 
> Thoughts?

I cannot add a tag with a third party personal information upon it
without their permission nor can anyone else in a part of the world with
any vaguely resembling privacy laws. So we need to document that it is
added with permission of the reporter only.

Alan

Re: Tracking and crediting bug reporters

[David Miller]

From: Alan Cox <alan@lxorguk.ukuu.org.uk>
Date: Mon, 12 May 2008 19:20:19 +0100

> > If the rest of the community is agreeable, it would be nice to make an
> > immediate start on this; it's not yet too late to get reasonable data for
> > the 2.6.26 kernel, and to have the habits well ingrained for 2.6.27.
> > 
> > Thoughts?
> 
> I cannot add a tag with a third party personal information upon it
> without their permission nor can anyone else in a part of the world with
> any vaguely resembling privacy laws. So we need to document that it is
> added with permission of the reporter only.

This is why I personally do not accept private bug reports or kernel
questions, I always direct them to post to the appropriate public
mailing list and CC: me if appropriate.

That solves the whole "privacy" issue quite neatly.

Re: Tracking and crediting bug reporters

[Alan Cox]

> This is why I personally do not accept private bug reports or kernel
> questions, I always direct them to post to the appropriate public
> mailing list and CC: me if appropriate.
> 
> That solves the whole "privacy" issue quite neatly.

Not really. About half the bugs I deal with and fix are from customers of
Red Hat who have differing expectations.

Alan

Re: Tracking and crediting bug reporters

[Andi Kleen]

Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

>> This is why I personally do not accept private bug reports or kernel
>> questions, I always direct them to post to the appropriate public
>> mailing list and CC: me if appropriate.
>> 
>> That solves the whole "privacy" issue quite neatly.
>
> Not really. About half the bugs I deal with and fix are from customers of
> Red Hat who have differing expectations.

Those would just default to not being credited. Where is the problem?

-Andi

Re: Tracking and crediting bug reporters

[Alan Cox]

> Those would just default to not being credited. Where is the problem?

We need to be sure that people understand that when submitting patches
they must not put Discovered-by: lines in without asking. This is
essentially a documentation matter which is why I directed the original
reply to Jonathan.

Alan

Re: Tracking and crediting bug reporters

[Jeff Garzik]

Alan Cox wrote:
>> Those would just default to not being credited. Where is the problem?
> 
> We need to be sure that people understand that when submitting patches
> they must not put Discovered-by: lines in without asking. This is
> essentially a documentation matter which is why I directed the original
> reply to Jonathan.

I tend to put the credit in the patch description text, /without/ an 
email address by default, since I don't feel as comfortable stamping 
users' email addresses into the kernel changelog without their 
permission, and (I'm lazy) asking permission is another email 
round-trip.  So my format is

	blah blah blah patch description blah blah blah
	blah blah blah patch description blah blah blah
	blah blah blah patch description blah blah blah

	Bug found by Lucifer Smith.

	Signed-off-by: Tejun <tejun@something.com>
	Signed-off-by: Jeff Garzik <me@somethingelse.com>

Regards,

	Jeff

Re: Tracking and crediting bug reporters

[Jiri Slaby]

On 05/12/2008 08:20 PM, Alan Cox wrote:
>> If the rest of the community is agreeable, it would be nice to make an
>> immediate start on this; it's not yet too late to get reasonable data for
>> the 2.6.26 kernel, and to have the habits well ingrained for 2.6.27.
>>
>> Thoughts?
> 
> I cannot add a tag with a third party personal information upon it
> without their permission nor can anyone else in a part of the world with
> any vaguely resembling privacy laws.

IANAL. Unless it's a public information, no? (As reports are.)

Otherwise we would be in a big trouble, since we do this as far as can git 
history tell at least.

Re: Tracking and crediting bug reporters

[Alan Cox]

> > I cannot add a tag with a third party personal information upon it
> > without their permission nor can anyone else in a part of the world with
> > any vaguely resembling privacy laws.
> 
> IANAL. Unless it's a public information, no? (As reports are.)

If they made it public fine - although building a database of that
requires care. That was my point. You need specific permission of the bug
reporter. Given bug reporters may have personal reasons (including 'works
at Microsoft') and business ones (from 'company policy' to 'three letter
agency') for not wanting the information public.

> Otherwise we would be in a big trouble, since we do this as far as can git 
> history tell at least.

No: The Signed-off-by in GIT references the 1.1 contributor agreement
which specifically deals with that case.

Re: Tracking and crediting bug reporters

[Rafael J. Wysocki]

On Monday, 12 of May 2008, Alan Cox wrote:
> > > I cannot add a tag with a third party personal information upon it
> > > without their permission nor can anyone else in a part of the world with
> > > any vaguely resembling privacy laws.
> > 
> > IANAL. Unless it's a public information, no? (As reports are.)
> 
> If they made it public fine - although building a database of that
> requires care. That was my point. You need specific permission of the bug
> reporter. Given bug reporters may have personal reasons (including 'works
> at Microsoft') and business ones (from 'company policy' to 'three letter
> agency') for not wanting the information public.

Well, don't the mailing list archives build such data bases already?

By posting a report to a mailing list the archives of which are publicly
available, and the LKML is one of those, you make your name and email address
information publicly available and recorded forever anyway, in the context of
the report.

Thanks,
Rafael

Re: Tracking and crediting bug reporters

[Rik van Riel]

On Mon, 12 May 2008 11:27:52 -0600
Jonathan Corbet <corbet@lwn.net> wrote:

> So here's what we would like to try instead.  Whenever somebody sends up a
> patch fixing a reported bug, the name of the person who reported the bug
> would be immortalized with this tag:
> 
> 	Reported-by: A. Bug Reporter <email@goes.here>

I like this idea.  If people help with kernel development, they deserve kudos.

-- 
All Rights Reversed

Re: Tracking and crediting bug reporters

[Rafael J. Wysocki]

On Monday, 12 of May 2008, Jonathan Corbet wrote:
> Several members of the Linux Foundation's Technical Advisory Board recently
> got together with Andrew Morton to talk about kernel quality issues.  One
> of the things which came out of that meeting was a desire to improve
> incentives for people who report bugs.  Clearly, actually fixing those bugs
> would qualify; nobody has lost sight of that.  But it was suggested that
> the creation and publication of statistics on bug reporting would also
> help.
> 
> One way to do this might be for Andrew (being the only one who actually
> reads every message posted on the list) to keep a spreadsheet along with
> everything else he does.  That idea did not go over very well.
> 
> So here's what we would like to try instead.  Whenever somebody sends up a
> patch fixing a reported bug, the name of the person who reported the bug
> would be immortalized with this tag:
> 
> 	Reported-by: A. Bug Reporter <email@goes.here>
> 
> In particular, reporters who work with the developers toward the resolution
> of the bug should be thanked in this way.  If we wanted to take things
> further, perhaps we could add a Bisected-by: tag for really hard-core
> helpers. 
> 
> If these tags go into the commit messages in any sort of consistent way, it
> should be possible generate the usual sort of statistics from them.  I'll
> then happily publicize them next to the traditional lists of people who are
> adding new bugs.  The result will certainly be fame, fortune, and job
> offers for the people at the top of the list.  Or something like that.
> 
> If the rest of the community is agreeable, it would be nice to make an
> immediate start on this; it's not yet too late to get reasonable data for
> the 2.6.26 kernel, and to have the habits well ingrained for 2.6.27.
> 
> Thoughts?

I like the idea.

Thanks,
Rafael

Re: Tracking and crediting bug reporters

[Adrian Bunk]

On Mon, May 12, 2008 at 11:27:52AM -0600, Jonathan Corbet wrote:
>...
> So here's what we would like to try instead.  Whenever somebody sends up a
> patch fixing a reported bug, the name of the person who reported the bug
> would be immortalized with this tag:
> 
> 	Reported-by: A. Bug Reporter <email@goes.here>
> 
> In particular, reporters who work with the developers toward the resolution
> of the bug should be thanked in this way.  If we wanted to take things
> further, perhaps we could add a Bisected-by: tag for really hard-core
> helpers. 

I assume it will be valid for one person to have two or all three tags 
in one commit (not uncommon for some of the stuff I'm doing)?

> If these tags go into the commit messages in any sort of consistent way, it
> should be possible generate the usual sort of statistics from them.  I'll
> then happily publicize them next to the traditional lists of people who are
> adding new bugs.  The result will certainly be fame, fortune, and job
> offers for the people at the top of the list.  Or something like that.
> 
> If the rest of the community is agreeable, it would be nice to make an
> immediate start on this; it's not yet too late to get reasonable data for
> the 2.6.26 kernel, and to have the habits well ingrained for 2.6.27.
>...

In which format do you want data for commits already in 2.6.26?

> jon

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Tracking and crediting bug reporters

[Jonathan Corbet]

[Operating in delayed response mode, sorry...]

Adrian Bunk <bunk@kernel.org> wrote:

> In which format do you want data for commits already in 2.6.26?

I hadn't thought about trying to backfill data for existing commits;
that sounds like a fair amount of work for coverage which could still be
somewhat spotty.  If you have the data at hand, though, then a simple
file of the form

	commit-id  full-name <email>

could be dealt with.  But is it really worth the effort?

Thanks,

jon

Re: Tracking and crediting bug reporters

[Adrian Bunk]

On Fri, May 16, 2008 at 10:35:43AM -0600, Jonathan Corbet wrote:
> [Operating in delayed response mode, sorry...]
> 
> Adrian Bunk <bunk@kernel.org> wrote:
> 
> > In which format do you want data for commits already in 2.6.26?
> 
> I hadn't thought about trying to backfill data for existing commits;
> that sounds like a fair amount of work for coverage which could still be
> somewhat spotty.  If you have the data at hand, though, then a simple
> file of the form
> 
> 	commit-id  full-name <email>

How to indicate whether it's a Reported-by or a Bisected-by?

> could be dealt with. But is it really worth the effort?

Not sure whether you don't consider it important that the data you 
publish is correct, but I will contribute the data I know about.

> Thanks,
> 
> jon

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Tracking and crediting bug reporters

[Rene Herman]

On 18-05-08 00:53, Adrian Bunk wrote:

> On Fri, May 16, 2008 at 10:35:43AM -0600, Jonathan Corbet wrote:
>> [Operating in delayed response mode, sorry...]
>>
>> Adrian Bunk <bunk@kernel.org> wrote:
>>
>>> In which format do you want data for commits already in 2.6.26?
>> I hadn't thought about trying to backfill data for existing commits;
>> that sounds like a fair amount of work for coverage which could still be
>> somewhat spotty.  If you have the data at hand, though, then a simple
>> file of the form
>>
>> 	commit-id  full-name <email>
> 
> How to indicate whether it's a Reported-by or a Bisected-by?

Literal Bisected-by might not make for a very good tag. Bisection is a 
way to pin down a bug; just one of them. So say person A reports and 
person B says "Hey, I know what that is. Try reverting foo." B then 
doesn't fit the tag while the only difference is that B was clever 
enough to not need a bisection. So if this is about giving credit; 
Tracked-down-by maybe? Something else?

Rene.

Re: Tracking and crediting bug reporters

[Jonathan Corbet]

Adrian Bunk <bunk@kernel.org> wrote:

> How to indicate whether it's a Reported-by or a Bisected-by?

Bisected-by was meant as a silly joke; I think that would be taking
things a bit too far.

Thanks,

jon

Re: Tracking and crediting bug reporters

[Rene Herman]

On 12-05-08 19:27, Jonathan Corbet wrote:

> So here's what we would like to try instead.  Whenever somebody sends up a
> patch fixing a reported bug, the name of the person who reported the bug
> would be immortalized with this tag:
> 
> 	Reported-by: A. Bug Reporter <email@goes.here>
> 
> In particular, reporters who work with the developers toward the resolution
> of the bug should be thanked in this way.  If we wanted to take things
> further, perhaps we could add a Bisected-by: tag for really hard-core
> helpers.

Ingo Molnar recently already used the latter:

http://marc.info/?l=linux-kernel&m=121032355203154&w=4

and a reply of mine consisted of now also needing a Reported-By:

http://marc.info/?l=linux-kernel&m=121037403515110&w=4

I must say the Bisected-by thing did feel a bit childish ("ah, I'm not 
in it to get the bug fixed but to get my name mentioned!") but hey, I 
find things to be offended over easily. From the example it does seem 
you probably want both if you have one; if reporter, nailer-downer and 
fixer are three seperate people.

Rene.

Re: Tracking and crediting bug reporters

[Stefan Richter]

Rene Herman wrote:
> I must say the Bisected-by thing did feel a bit childish ("ah, I'm not 
> in it to get the bug fixed but to get my name mentioned!") but hey, I 
> find things to be offended over easily. From the example it does seem 
> you probably want both if you have one; if reporter, nailer-downer and 
> fixer are three seperate people.

Which tag do I use for the postman who delivered the device which I
needed for debugging?
-- 
Stefan Richter
-=====-==--- -=-= -===-
http://arcgraph.de/sr/

Re: Tracking and crediting bug reporters

[Pavel Machek]

Hi!

> So here's what we would like to try instead.  Whenever somebody sends up a
> patch fixing a reported bug, the name of the person who reported the bug
> would be immortalized with this tag:
> 
> 	Reported-by: A. Bug Reporter <email@goes.here>
> 
> In particular, reporters who work with the developers toward the resolution
> of the bug should be thanked in this way.  If we wanted to take things

I believe we have enough tags already. Plus, if the reporter really
works with the developer till the end, there's already accepter
Tested-by: flag, right?

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Tracking and crediting bug reporters

[Johannes Weiner]

Hi,

Pavel Machek <pavel@ucw.cz> writes:

> Hi!
>
>> So here's what we would like to try instead.  Whenever somebody sends up a
>> patch fixing a reported bug, the name of the person who reported the bug
>> would be immortalized with this tag:
>> 
>> 	Reported-by: A. Bug Reporter <email@goes.here>
>> 
>> In particular, reporters who work with the developers toward the resolution
>> of the bug should be thanked in this way.  If we wanted to take things
>
> I believe we have enough tags already. Plus, if the reporter really
> works with the developer till the end, there's already accepter
> Tested-by: flag, right?

I would more argue to remove Tested-by completely because it does not
tell much.  Code is not bugfree just because someone compiled and ran
it.  And if it breaks on some other systems later on, what does it help
if you know someone tested it?  It still breaks.  And you can not even
blame the tester because of his luck of a working configuration.

While a Reported-by in this case credits a person reporting a bug.  Just
that.  And perhaps that the report was good enough to make a fix.

	Hannes

Re: Tracking and crediting bug reporters

[Pekka Enberg]

On Wed, May 21, 2008 at 4:34 PM, Johannes Weiner <hannes@saeurebad.de> wrote:
> I would more argue to remove Tested-by completely because it does not
> tell much.  Code is not bugfree just because someone compiled and ran
> it.  And if it breaks on some other systems later on, what does it help
> if you know someone tested it?  It still breaks.  And you can not even
> blame the tester because of his luck of a working configuration.

Tested-by is really helpful for identifying potential testers when you
change some part of the kernel. I use git history to identify people
who are able/willing to test NUMA changes to the slab allocators that
require a big ass NUMA box to trigger a bug, for example. Also, in
some cases, testing a bug fix is not trivial and might require a lot
of work that we should give credit for.

Re: Tracking and crediting bug reporters

[Roland]

On Monday, 12 of May 2008, Jonathan Corbet wrote:
> Several members of the Linux Foundation's Technical Advisory Board 
> recently
> got together with Andrew Morton to talk about kernel quality issues.  One
> of the things which came out of that meeting was a desire to improve
> incentives for people who report bugs.  Clearly, actually fixing those 
> bugs
> would qualify; nobody has lost sight of that.  But it was suggested that
> the creation and publication of statistics on bug reporting would also
> help.
>
> One way to do this might be for Andrew (being the only one who actually
> reads every message posted on the list) to keep a spreadsheet along with
> everything else he does.  That idea did not go over very well.
>
> So here's what we would like to try instead.  Whenever somebody sends up a
> patch fixing a reported bug, the name of the person who reported the bug
> would be immortalized with this tag:
>
> 	Reported-by: A. Bug Reporter <email@goes.here>
>
> In particular, reporters who work with the developers toward the 
> resolution
> of the bug should be thanked in this way.  If we wanted to take things
> further, perhaps we could add a Bisected-by: tag for really hard-core
> helpers.
>
> If these tags go into the commit messages in any sort of consistent way, 
> it
> should be possible generate the usual sort of statistics from them.  I'll
> then happily publicize them next to the traditional lists of people who 
> are
> adding new bugs.  The result will certainly be fame, fortune, and job
> offers for the people at the top of the list.  Or something like that.
>
> If the rest of the community is agreeable, it would be nice to make an
> immediate start on this; it's not yet too late to get reasonable data for
> the 2.6.26 kernel, and to have the habits well ingrained for 2.6.27.
>
> Thoughts?

very good idea !

anyhow, i`m browsing trough bugzilla for some time and try helping
where i can and there is one thing which is really noticeable:

there are bugreports not being worked on systematically enough and
reporters often don`t feel welcome there.
i saw bug reports which didn`t get a reply for a year or so.
somebody excused for "posting so much" - all he did was posting a
proper description of his problem.

improving of bugzilla handling would be another incentive for bug reporter,
imho.

someone who reports a bug there and never gets a response (not even a formal
"thanks for your report....") will probably never report again.

i have worked for support (software product) some time ago and i
learned one thing there:

there needs to be someone to keep track of the users` input and to keep
track of the developers action/response.

bugreporters forget about their bug if they found a workaround (buying
new hardware or whatever) and developers mostly prefer working on
new code rather than fixing issues. that`s not true for ever user or 
developer,
but if you really want bugs get fixed, someone acting as a mediator is 
essential.
.
regards
roland