Crediting test authors

Crediting test authors

[Jakub Kicinski]

Hi!

Does anyone have ideas about crediting test authors or tests for bugs
discovered?  We increasingly see situations where someone adds a test
then our subsystem CI uncovers a (1 in a 100 runs) bug using that test.

Using reported-by doesn't feel right. But credit should go to the
person who wrote the test. Is anyone else having this dilemma?

Re: Crediting test authors

[Theodore Ts'o]

On Fri, Jul 25, 2025 at 08:00:23AM -0700, Jakub Kicinski wrote:
> 
> Does anyone have ideas about crediting test authors or tests for bugs
> discovered?  We increasingly see situations where someone adds a test
> then our subsystem CI uncovers a (1 in a 100 runs) bug using that test.

I just list the test that discovered the bug and which was cleared by
the commit, and assume that the test author will get credited in the
upstream test repository (e.g., such as generic/750 from fstests or
inotify02 from ltp).  So it's not something that I've worried about.

I suppose if the test repository isn't as well known, or if the test
hasn't been checked anywhere at all, your concern that the test author
should be credited is something I can understand.  But it hasn't come
up for me.

In other cases, if the commit hasn't been stable yet (say for kunit or
kselftests coming from some other tree), I'll just throw in a Link:
tag pointing at lore.kernel.org.

						- Ted

Re: Crediting test authors

[dan.j.williams]

Jakub Kicinski wrote:
> Hi!
> 
> Does anyone have ideas about crediting test authors or tests for bugs
> discovered?  We increasingly see situations where someone adds a test
> then our subsystem CI uncovers a (1 in a 100 runs) bug using that test.
> 
> Using reported-by doesn't feel right. But credit should go to the
> person who wrote the test. Is anyone else having this dilemma?

Is that not a "credit in the changelog" situation?

"Big thanks to DeveloperX for their recent TestY added with CommitZ for
 catching this case."

Reported-by: Some CI Bot

Re: Crediting test authors

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 928 bytes --]

On Fri, Jul 25, 2025 at 08:00:23AM -0700, Jakub Kicinski wrote:

> Does anyone have ideas about crediting test authors or tests for bugs
> discovered?  We increasingly see situations where someone adds a test
> then our subsystem CI uncovers a (1 in a 100 runs) bug using that test.

> Using reported-by doesn't feel right. But credit should go to the
> person who wrote the test. Is anyone else having this dilemma?

Usually I'd do a reported-by for whoever actually looked at the test
system, triaged the issue and reported it.  Trying to credit test
authorship separately to the testsuite gets cumbersome over time, tests
get updated over time for a range of reasons (toolchain updates, adding
more coverage, improvements in the testsuite's frameworks...) so it's
often not just a single person.  Hopefully the testsuite is keeping
track of things well enough so mentioning the test will point people in
the right direction.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Crediting test authors

[Jakub Kicinski]

On Fri, 25 Jul 2025 17:39:12 +0100 Mark Brown wrote:
> On Fri, Jul 25, 2025 at 08:00:23AM -0700, Jakub Kicinski wrote:
> 
> > Does anyone have ideas about crediting test authors or tests for bugs
> > discovered?  We increasingly see situations where someone adds a test
> > then our subsystem CI uncovers a (1 in a 100 runs) bug using that test.  
> 
> > Using reported-by doesn't feel right. But credit should go to the
> > person who wrote the test. Is anyone else having this dilemma?  
> 
> Usually I'd do a reported-by for whoever actually looked at the test
> system, triaged the issue and reported it.  Trying to credit test
> authorship separately to the testsuite gets cumbersome over time, tests
> get updated over time for a range of reasons (toolchain updates, adding
> more coverage, improvements in the testsuite's frameworks...) so it's
> often not just a single person.  Hopefully the testsuite is keeping
> track of things well enough so mentioning the test will point people in
> the right direction.

Ack, it does get murky overtime. Also with pre-commit testing there
usually wouldn't even be a bug in the tree to credit fixing. 

I guess we just had a lucky(?) string of very clear cut cases where
a good selftest led the maintainer noticing a crash in the CI, and 
fixing something.

Re: Crediting test authors

[Jakub Kicinski]

On Fri, 25 Jul 2025 09:38:28 -0700 dan.j.williams@intel.com wrote:
> Jakub Kicinski wrote:
> > Hi!
> > 
> > Does anyone have ideas about crediting test authors or tests for bugs
> > discovered?  We increasingly see situations where someone adds a test
> > then our subsystem CI uncovers a (1 in a 100 runs) bug using that test.
> > 
> > Using reported-by doesn't feel right. But credit should go to the
> > person who wrote the test. Is anyone else having this dilemma?  
> 
> Is that not a "credit in the changelog" situation?
> 
> "Big thanks to DeveloperX for their recent TestY added with CommitZ for
>  catching this case."

That's what we do usually, but I'm a strong believer in (LWN) statistics
to help people justify the work they do upstream. Feels even more
important for testing than feature development in a way.

So a tag would be ideal. But it's a hard nut to crack. Best I can come
up with would be:

Reproducer: test.case.path # 001122aabb (optimal) commit of the test case

? Could potentially be useful for backporters?

> Reported-by: Some CI Bot

I guess we'd need something like:

Reported-by: subsystem CI # Person Who <developed@the.test> ?

Re: Crediting test authors

[Jakub Kicinski]

On Fri, 25 Jul 2025 10:20:19 -0700 Jakub Kicinski wrote:
> Reproducer: test.case.path # 001122aabb (optimal) commit of the test case

s/optimal/optional/

Re: Crediting test authors

[dan.j.williams]

Jakub Kicinski wrote:
> On Fri, 25 Jul 2025 09:38:28 -0700 dan.j.williams@intel.com wrote:
> > Jakub Kicinski wrote:
> > > Hi!
> > > 
> > > Does anyone have ideas about crediting test authors or tests for bugs
> > > discovered?  We increasingly see situations where someone adds a test
> > > then our subsystem CI uncovers a (1 in a 100 runs) bug using that test.
> > > 
> > > Using reported-by doesn't feel right. But credit should go to the
> > > person who wrote the test. Is anyone else having this dilemma?  
> > 
> > Is that not a "credit in the changelog" situation?
> > 
> > "Big thanks to DeveloperX for their recent TestY added with CommitZ for
> >  catching this case."
> 
> That's what we do usually, but I'm a strong believer in (LWN) statistics
> to help people justify the work they do upstream. Feels even more
> important for testing than feature development in a way.
> 
> So a tag would be ideal. But it's a hard nut to crack. Best I can come
> up with would be:
> 
> Reproducer: test.case.path # 001122aabb (optimal) commit of the test case
> 
> ? Could potentially be useful for backporters?

That's true, more than a few times I have had distro folks reach out to
ask "how do I verify this backport" and end up manually pointing to the
new unit test that backstops a fix.

Although, from that tag I would not know where to get the commit. Maybe:

Test: <git url>

...as a new Link: type?

> > Reported-by: Some CI Bot
> 
> I guess we'd need something like:
> 
> Reported-by: subsystem CI # Person Who <developed@the.test> ?

That looks like a useful convention so that the statistics gathering
script does not need to walk URLs to get author data.

Also:

Tested-by: validation person # test author

...might be another convention, because the validator likely has
interest in getting Cc'd on backports, while the tool author likely
wants the credit but not all the notifications on what happens with
fixes.

Re: Crediting test authors

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1219 bytes --]

On Fri, Jul 25, 2025 at 11:37:13AM -0700, dan.j.williams@intel.com wrote:
> Jakub Kicinski wrote:

> > So a tag would be ideal. But it's a hard nut to crack. Best I can come
> > up with would be:

> > Reproducer: test.case.path # 001122aabb (optimal) commit of the test case

> That's true, more than a few times I have had distro folks reach out to
> ask "how do I verify this backport" and end up manually pointing to the
> new unit test that backstops a fix.

> Although, from that tag I would not know where to get the commit. Maybe:
> 
> Test: <git url>
> 
> ...as a new Link: type?

It seems like there's some overlap here with the work that people have
been intermittently trying to do on test cataloging, eg:

   https://lore.kernel.org/workflows/CAK18DXYitS7hL1mA3QsPLmW9-R0q6Kin0C5Uv9fj=uS90WSnxA@mail.gmail.com/

That's been approached more from the "what tests should I run?" end of
things since it's been driven by people interested in testing and CI,
but it feels like there's a lot of overlap with the describing the
suites part of things.  It'd be a lot easier to write and read tags like
the above if we could define some more compact names than git URLs for
suites/tests.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Crediting test authors

[Gustavo Padovan]

Hello,


---- On Sat, 26 Jul 2025 18:44:29 -0300 Mark Brown <broonie@kernel.org> wrote ---

 > On Fri, Jul 25, 2025 at 11:37:13AM -0700, dan.j.williams@intel.com wrote: 
 > > Jakub Kicinski wrote: 
 >  
 > > > So a tag would be ideal. But it's a hard nut to crack. Best I can come 
 > > > up with would be: 
 >  
 > > > Reproducer: test.case.path # 001122aabb (optimal) commit of the test case 
 >  
 > > That's true, more than a few times I have had distro folks reach out to 
 > > ask "how do I verify this backport" and end up manually pointing to the 
 > > new unit test that backstops a fix. 
 >  
 > > Although, from that tag I would not know where to get the commit. Maybe: 
 > > 
 > > Test: <git url> 
 > > 
 > > ...as a new Link: type? 
 >  
 > It seems like there's some overlap here with the work that people have 
 > been intermittently trying to do on test cataloging, eg: 
 >  
 >  https://lore.kernel.org/workflows/CAK18DXYitS7hL1mA3QsPLmW9-R0q6Kin0C5Uv9fj=uS90WSnxA@mail.gmail.com/ 
 >  
 > That's been approached more from the "what tests should I run?" end of 
 > things since it's been driven by people interested in testing and CI, 
 > but it feels like there's a lot of overlap with the describing the 
 > suites part of things.  It'd be a lot easier to write and read tags like 
 > the above if we could define some more compact names than git URLs for 
 > suites/tests. 


I see the overlap too.  The catalog discussion envisions a mapping of which tests
should be executed for this folder/file or function. This approach is being used,
for example, in the Mesa project for its CI testing. When a new PR comes in,
the system will trigger tests based on the files being modified.

Our discussions on the catalog side are quite basic right now and happening
through the .kernelci.yml file[1]. I believe there is a possible future, built
in a step by step manner, to indentify for given patchset:
 * configs to test
 * arch/hw to run tests on
 * tests must be executed
 * expectation of pass/fail for each test

KernelCI wants to work with maintainers to figure out how to make progress
on that.

[1]  https://www.youtube.com/watch?v=-LtK1fScFww


Best,

- Gus