Re: KernelCI report for stable-rc: v5.15.164

Re: KernelCI report for stable-rc: v5.15.164

[Nikolai Kondrashov]

On 8/6/24 1:03 PM, Nikolai Kondrashov via groups.io wrote:
> On 8/5/24 5:40 PM, Gustavo Padovan via groups.io wrote:
>>  > > Why does this have kernelci, syzbot and tuxsuite. Let's remove it for now.
>>  >
>>  > It lists all the CI systems the report has been aggregated with. Although
> only maestro and broonie reported failures, we included
>>  > builds and test stats from all the 5 CI systems.
>>
>> I see.  Although, we can't really guarantee the quality of data coming from
>> the other origins, hence my request to use only maestro and broonie for the
>> time being. We need a high quality report, not a report with as much results
>> as possible.
>
> This is alright for a PoC. However, I'd like you guys to mention to Greg that
> we have more data from other origins, when you engage him with this, and that
> they're available on the dashboards. Or perhaps better just put that into the
> notification message itself.
>
> Finally, as we've discussed before, if a maintainer has requirements for
> particular result quality, it's best to directly apply them to data, where
> possible (e.g. check that log_excerpt is there), instead of simply selecting
> origins. That could be the next step, once we define what "good quality"
> means exactly.

This is what all the CI systems signed up for, after all: reaching out to
maintainers. And we would be breaking our promise, if we make these decisions
for them, or discriminate by origin.

Nick

Re: KernelCI report for stable-rc: v5.15.164

[Gustavo Padovan]

---- On Tue, 06 Aug 2024 07:06:00 -0300 Nikolai Kondrashov  wrote ---

 > On 8/6/24 1:03 PM, Nikolai Kondrashov via groups.io wrote: 
 > > On 8/5/24 5:40 PM, Gustavo Padovan via groups.io wrote: 
 > >>  > > Why does this have kernelci, syzbot and tuxsuite. Let's remove it for now. 
 > >>  > 
 > >>  > It lists all the CI systems the report has been aggregated with. Although 
 > > only maestro and broonie reported failures, we included 
 > >>  > builds and test stats from all the 5 CI systems. 
 > >> 
 > >> I see.  Although, we can't really guarantee the quality of data coming from 
 > >> the other origins, hence my request to use only maestro and broonie for the 
 > >> time being. We need a high quality report, not a report with as much results 
 > >> as possible. 
 > > 
 > > This is alright for a PoC. However, I'd like you guys to mention to Greg that 
 > > we have more data from other origins, when you engage him with this, and that 
 > > they're available on the dashboards. Or perhaps better just put that into the 
 > > notification message itself. 
 > > 
 > > Finally, as we've discussed before, if a maintainer has requirements for 
 > > particular result quality, it's best to directly apply them to data, where 
 > > possible (e.g. check that log_excerpt is there), instead of simply selecting 
 > > origins. That could be the next step, once we define what "good quality" 
 > > means exactly. 
 >  
 > This is what all the CI systems signed up for, after all: reaching out to 
 > maintainers. And we would be breaking our promise, if we make these decisions 
 > for them, or discriminate by origin. 

Not exactly. That would be the perfectionist take.  We are investing a lot in the test
and results data quality so we can be sure we can be sure we are showing good,
high quality data to maintainers we are engaging actively. 

That's exactly the case of the stable-rc report. Greg has said to us already that just
throwing data at him wouldn't help much. We have to make sure that data can actually 
help him. This has been our goal since we started the current stable-rc report process
since LPC. The Collabora team has invested dozens and dozens of ours into evaluating
and improving the quality of the data.

In summary, we are not breaking any promises. We can really throw any data we have into
maintainers and expect that would be helping them.

Best,

- Gus

Re: KernelCI report for stable-rc: v5.15.164

[Nikolai Kondrashov]

On 8/6/24 7:03 PM, Gustavo Padovan wrote:
> 
> 
> ---- On Tue, 06 Aug 2024 07:06:00 -0300 Nikolai Kondrashov  wrote ---
> 
>  > On 8/6/24 1:03 PM, Nikolai Kondrashov via groups.io wrote: 
>  > > On 8/5/24 5:40 PM, Gustavo Padovan via groups.io wrote: 
>  > >>  > > Why does this have kernelci, syzbot and tuxsuite. Let's remove it for now. 
>  > >>  > 
>  > >>  > It lists all the CI systems the report has been aggregated with. Although 
>  > > only maestro and broonie reported failures, we included 
>  > >>  > builds and test stats from all the 5 CI systems. 
>  > >> 
>  > >> I see.  Although, we can't really guarantee the quality of data coming from 
>  > >> the other origins, hence my request to use only maestro and broonie for the 
>  > >> time being. We need a high quality report, not a report with as much results 
>  > >> as possible. 
>  > > 
>  > > This is alright for a PoC. However, I'd like you guys to mention to Greg that 
>  > > we have more data from other origins, when you engage him with this, and that 
>  > > they're available on the dashboards. Or perhaps better just put that into the 
>  > > notification message itself. 
>  > > 
>  > > Finally, as we've discussed before, if a maintainer has requirements for 
>  > > particular result quality, it's best to directly apply them to data, where 
>  > > possible (e.g. check that log_excerpt is there), instead of simply selecting 
>  > > origins. That could be the next step, once we define what "good quality" 
>  > > means exactly. 
>  >  
>  > This is what all the CI systems signed up for, after all: reaching out to 
>  > maintainers. And we would be breaking our promise, if we make these decisions 
>  > for them, or discriminate by origin. 
> 
> Not exactly. That would be the perfectionist take.  We are investing a lot in the test
> and results data quality so we can be sure we can be sure we are showing good,
> high quality data to maintainers we are engaging actively. 
> 
> That's exactly the case of the stable-rc report. Greg has said to us already that just
> throwing data at him wouldn't help much. We have to make sure that data can actually 
> help him. This has been our goal since we started the current stable-rc report process
> since LPC. The Collabora team has invested dozens and dozens of ours into evaluating
> and improving the quality of the data.
> 
> In summary, we are not breaking any promises. We can really throw any data we have into
> maintainers and expect that would be helping them.

Sure, but our (KCIDB) job as the "middle man" is to advertise results of all
submitters *equally*, let the maintainers *themselves* decide what's good and
bad for them, implement their requirements, and *also* pass their feedback to
the submitters. *This* is what they expect of us.

We should not be deciding *ourselves* alone what's good and bad for
maintainers (without even specifying actual requirements we're applying), and
not telling the submitters we're ignoring their data, while sending our *own*
to maintainers.

> The Collabora team has invested dozens and dozens of ours into evaluating
> and improving the quality of the data

You can be sure that other CI systems worked hard on this too. Microsoft
results were top-notch when they decided to join. Syzbot results are almost
*perfect*, they're doing a *very* good job. CKI has an *army* of QE looking
after tests. And so on.

I understand the desire and the need to hit the mark first try, and sure, this
is fine for the start, as long as we're clear about what we're actually doing,
and what *other* data is available and being omitted from notifications.

Finally, once again, we should be judging data based on its merits, not on who
sent it. Let's specify what kind of quality we're looking for exactly, and
implement filtering according to these requirements as one of the next steps.

Nick