* Re: KernelCI report for stable-rc: v5.15.164 [not found] ` <17E91B8D1664855A.28314@groups.io> @ 2024-08-06 10:06 ` Nikolai Kondrashov 2024-08-06 16:03 ` Gustavo Padovan 0 siblings, 1 reply; 3+ messages in thread From: Nikolai Kondrashov @ 2024-08-06 10:06 UTC (permalink / raw) To: kernelci-results-staging, Jeny Sheth Cc: bot, shreeya.patel, kernelci@lists.linux.dev On 8/6/24 1:03 PM, Nikolai Kondrashov via groups.io wrote: > On 8/5/24 5:40 PM, Gustavo Padovan via groups.io wrote: >> > > Why does this have kernelci, syzbot and tuxsuite. Let's remove it for now. >> > >> > It lists all the CI systems the report has been aggregated with. Although > only maestro and broonie reported failures, we included >> > builds and test stats from all the 5 CI systems. >> >> I see. Although, we can't really guarantee the quality of data coming from >> the other origins, hence my request to use only maestro and broonie for the >> time being. We need a high quality report, not a report with as much results >> as possible. > > This is alright for a PoC. However, I'd like you guys to mention to Greg that > we have more data from other origins, when you engage him with this, and that > they're available on the dashboards. Or perhaps better just put that into the > notification message itself. > > Finally, as we've discussed before, if a maintainer has requirements for > particular result quality, it's best to directly apply them to data, where > possible (e.g. check that log_excerpt is there), instead of simply selecting > origins. That could be the next step, once we define what "good quality" > means exactly. This is what all the CI systems signed up for, after all: reaching out to maintainers. And we would be breaking our promise, if we make these decisions for them, or discriminate by origin. Nick ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: KernelCI report for stable-rc: v5.15.164 2024-08-06 10:06 ` KernelCI report for stable-rc: v5.15.164 Nikolai Kondrashov @ 2024-08-06 16:03 ` Gustavo Padovan 2024-08-06 16:32 ` Nikolai Kondrashov 0 siblings, 1 reply; 3+ messages in thread From: Gustavo Padovan @ 2024-08-06 16:03 UTC (permalink / raw) To: Nikolai Kondrashov Cc: kernelci-results-staging, Jeny Sheth, bot, shreeya.patel, kernelci@lists.linux.dev ---- On Tue, 06 Aug 2024 07:06:00 -0300 Nikolai Kondrashov wrote --- > On 8/6/24 1:03 PM, Nikolai Kondrashov via groups.io wrote: > > On 8/5/24 5:40 PM, Gustavo Padovan via groups.io wrote: > >> > > Why does this have kernelci, syzbot and tuxsuite. Let's remove it for now. > >> > > >> > It lists all the CI systems the report has been aggregated with. Although > > only maestro and broonie reported failures, we included > >> > builds and test stats from all the 5 CI systems. > >> > >> I see. Although, we can't really guarantee the quality of data coming from > >> the other origins, hence my request to use only maestro and broonie for the > >> time being. We need a high quality report, not a report with as much results > >> as possible. > > > > This is alright for a PoC. However, I'd like you guys to mention to Greg that > > we have more data from other origins, when you engage him with this, and that > > they're available on the dashboards. Or perhaps better just put that into the > > notification message itself. > > > > Finally, as we've discussed before, if a maintainer has requirements for > > particular result quality, it's best to directly apply them to data, where > > possible (e.g. check that log_excerpt is there), instead of simply selecting > > origins. That could be the next step, once we define what "good quality" > > means exactly. > > This is what all the CI systems signed up for, after all: reaching out to > maintainers. And we would be breaking our promise, if we make these decisions > for them, or discriminate by origin. Not exactly. That would be the perfectionist take. We are investing a lot in the test and results data quality so we can be sure we can be sure we are showing good, high quality data to maintainers we are engaging actively. That's exactly the case of the stable-rc report. Greg has said to us already that just throwing data at him wouldn't help much. We have to make sure that data can actually help him. This has been our goal since we started the current stable-rc report process since LPC. The Collabora team has invested dozens and dozens of ours into evaluating and improving the quality of the data. In summary, we are not breaking any promises. We can really throw any data we have into maintainers and expect that would be helping them. Best, - Gus ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: KernelCI report for stable-rc: v5.15.164 2024-08-06 16:03 ` Gustavo Padovan @ 2024-08-06 16:32 ` Nikolai Kondrashov 0 siblings, 0 replies; 3+ messages in thread From: Nikolai Kondrashov @ 2024-08-06 16:32 UTC (permalink / raw) To: Gustavo Padovan Cc: kernelci-results-staging, Jeny Sheth, bot, shreeya.patel, kernelci@lists.linux.dev On 8/6/24 7:03 PM, Gustavo Padovan wrote: > > > ---- On Tue, 06 Aug 2024 07:06:00 -0300 Nikolai Kondrashov wrote --- > > > On 8/6/24 1:03 PM, Nikolai Kondrashov via groups.io wrote: > > > On 8/5/24 5:40 PM, Gustavo Padovan via groups.io wrote: > > >> > > Why does this have kernelci, syzbot and tuxsuite. Let's remove it for now. > > >> > > > >> > It lists all the CI systems the report has been aggregated with. Although > > > only maestro and broonie reported failures, we included > > >> > builds and test stats from all the 5 CI systems. > > >> > > >> I see. Although, we can't really guarantee the quality of data coming from > > >> the other origins, hence my request to use only maestro and broonie for the > > >> time being. We need a high quality report, not a report with as much results > > >> as possible. > > > > > > This is alright for a PoC. However, I'd like you guys to mention to Greg that > > > we have more data from other origins, when you engage him with this, and that > > > they're available on the dashboards. Or perhaps better just put that into the > > > notification message itself. > > > > > > Finally, as we've discussed before, if a maintainer has requirements for > > > particular result quality, it's best to directly apply them to data, where > > > possible (e.g. check that log_excerpt is there), instead of simply selecting > > > origins. That could be the next step, once we define what "good quality" > > > means exactly. > > > > This is what all the CI systems signed up for, after all: reaching out to > > maintainers. And we would be breaking our promise, if we make these decisions > > for them, or discriminate by origin. > > Not exactly. That would be the perfectionist take. We are investing a lot in the test > and results data quality so we can be sure we can be sure we are showing good, > high quality data to maintainers we are engaging actively. > > That's exactly the case of the stable-rc report. Greg has said to us already that just > throwing data at him wouldn't help much. We have to make sure that data can actually > help him. This has been our goal since we started the current stable-rc report process > since LPC. The Collabora team has invested dozens and dozens of ours into evaluating > and improving the quality of the data. > > In summary, we are not breaking any promises. We can really throw any data we have into > maintainers and expect that would be helping them. Sure, but our (KCIDB) job as the "middle man" is to advertise results of all submitters *equally*, let the maintainers *themselves* decide what's good and bad for them, implement their requirements, and *also* pass their feedback to the submitters. *This* is what they expect of us. We should not be deciding *ourselves* alone what's good and bad for maintainers (without even specifying actual requirements we're applying), and not telling the submitters we're ignoring their data, while sending our *own* to maintainers. > The Collabora team has invested dozens and dozens of ours into evaluating > and improving the quality of the data You can be sure that other CI systems worked hard on this too. Microsoft results were top-notch when they decided to join. Syzbot results are almost *perfect*, they're doing a *very* good job. CKI has an *army* of QE looking after tests. And so on. I understand the desire and the need to hit the mark first try, and sure, this is fine for the start, as long as we're clear about what we're actually doing, and what *other* data is available and being omitted from notifications. Finally, once again, we should be judging data based on its merits, not on who sent it. Let's specify what kind of quality we're looking for exactly, and implement filtering according to these requirements as one of the next steps. Nick ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2024-08-06 16:32 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <66ae1e21.050a0220.1c5959.0148@mx.google.com>
[not found] ` <1911838cd2e.ddb71f5356531.3914011082698955929@collabora.com>
[not found] ` <19122d7a43d.dde9a53983019.6953377593724593632@collabora.com>
[not found] ` <19122fc3d66.11f5a856693778.1637642610124539898@collabora.com>
[not found] ` <17E91B8D1664855A.28314@groups.io>
2024-08-06 10:06 ` KernelCI report for stable-rc: v5.15.164 Nikolai Kondrashov
2024-08-06 16:03 ` Gustavo Padovan
2024-08-06 16:32 ` Nikolai Kondrashov
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox