kernelci test results

kernelci test results

[Sebastian Andrzej Siewior]

Hi,

I've been poking at the dashboard to see how rt-tests tests are doing.
I tried to summarize my doing/ process and added questions while doing
so. I might be doing things wrong, not the expected way or simply
missing the obvious.


On the dashboard, by clicking hardware, Tree=mainline I see only
kubernetes as platform. There I see on the bottom various builds. There
is a search bar where I can limit to "preempt_rt" and then builds are
limited to "preempt_rt" configuration and it also includes "next".

There I can click on the build-log (and later I get the build.log.gz)
but it appears not to contain the build log of the kernel. I think there
is the "config merging" part and kselftest build but not the actual
build of the kernel itself.

Clicking on tests, I can also filter for preempt_rt but there is only
kselftest. I can't see the results, there is always "No logs available".
Based on the timestamp the results are from today so I wouldn't expect
them to be deleted because of age.

Going back to "trees" and selecting "stable-rt" and branch v6.6-rt
there is more HW. Clicking on bcm2711-rpi-4-b since it has the most
results here.

Here I clicked on configs "defconfig+preempt_rt" and then one of tree
that appeared is "mainline/master" but I did not see this earlier while
I did select that tree.

I did click on "tests" and then "rt-tests" appeared. Expanding it up, I
found cyclictest results for that tree.

Clicking on details I found a link to
	https://lava.collabora.dev/scheduler/job/22810572

where the complete log is available of what has been done on that
hardware. In the bootlog I noticed
| [    0.000000] WARNING: kernel/trace/trace_events.c:420 at test_double_dereference+0x128/0x130, CPU#0: swapper/0/0

which is a warning that appeared during boot. Is this picked up
somewhere? Is there a notification sent somewhere that this has been
noticed?

Later there is the cyclictest invocation, arguments and the result.
There is no max-value set for the test so it passes. Would there be a
notification if the test failed and if so, where?

I noticed that on the lava page, by clicking on the device I can search
for "rt-tests-cyclictest" and get the recent results for this device for
this test. Is something like this possible on the dashboard? I.e. look
for the recent cyclictest results?
Right now I have no idea how to do this without going device by device.

Sebastian

Re: kernelci test results

[Alan Zanoni Peixinho]

Hi Sebastian, I work with the dashboard team, so I hope I am able to help you.

First of all, thanks for using the dashboard, and for taking time to
write this up. Scenarios like this one are exactly what help us find
the UX gaps for improvement.

The dashboard is mostly organized around tree checkouts first.
If i understood it properly you might be able to achieve what you are
looking for via:
1. Selecting a tree (ie stable-rt/v6.6-rt).
2. Clicking on the filters button on the right. There we can apply
filters for configs (defconfig+preempt_rt, etc).
3. We click on the tests tab (default tab is for builds).
4. On the list of tests you can search for test paths (cyclictest,
rt-tests, etc).

Link for the view:
https://dashboard.kernelci.org/tree/stable-rt/v6.6-rt/84925bb144f49eb0d5f4c5c29f690b4bbf84903b?df|c|defconfig%2Bpreempt_rt=true&p=t

That list is for that checkout across hardware, so you do not have to
go board by board. (Unfortunately there is still no search for “this
test on this device over time” across all trees.)

About the other questions.

All the logs presented are the ones that have been submitted via KCI
submissions, so there might be some missing information when not
reported.

On the topic of notifying on log errors, this is a feature on active
discussion, to automate notification for first for boots and later for
test failures as well. Today we only notify registered issues.

We are actually looking for some feedback of dashboard users to help
us steer on the roadmap for new features, and your help would be much
appreciated.

Att
Alan Peixinho
KernelCI Dashboard Team

On Thu, Sep 3, 2026 at 2:36 PM Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
>
> Hi,
>
> I've been poking at the dashboard to see how rt-tests tests are doing.
> I tried to summarize my doing/ process and added questions while doing
> so. I might be doing things wrong, not the expected way or simply
> missing the obvious.
>
>
> On the dashboard, by clicking hardware, Tree=mainline I see only
> kubernetes as platform. There I see on the bottom various builds. There
> is a search bar where I can limit to "preempt_rt" and then builds are
> limited to "preempt_rt" configuration and it also includes "next".
>
> There I can click on the build-log (and later I get the build.log.gz)
> but it appears not to contain the build log of the kernel. I think there
> is the "config merging" part and kselftest build but not the actual
> build of the kernel itself.
>
> Clicking on tests, I can also filter for preempt_rt but there is only
> kselftest. I can't see the results, there is always "No logs available".
> Based on the timestamp the results are from today so I wouldn't expect
> them to be deleted because of age.
>
> Going back to "trees" and selecting "stable-rt" and branch v6.6-rt
> there is more HW. Clicking on bcm2711-rpi-4-b since it has the most
> results here.
>
> Here I clicked on configs "defconfig+preempt_rt" and then one of tree
> that appeared is "mainline/master" but I did not see this earlier while
> I did select that tree.
>
> I did click on "tests" and then "rt-tests" appeared. Expanding it up, I
> found cyclictest results for that tree.
>
> Clicking on details I found a link to
>         https://lava.collabora.dev/scheduler/job/22810572
>
> where the complete log is available of what has been done on that
> hardware. In the bootlog I noticed
> | [    0.000000] WARNING: kernel/trace/trace_events.c:420 at test_double_dereference+0x128/0x130, CPU#0: swapper/0/0
>
> which is a warning that appeared during boot. Is this picked up
> somewhere? Is there a notification sent somewhere that this has been
> noticed?
>
> Later there is the cyclictest invocation, arguments and the result.
> There is no max-value set for the test so it passes. Would there be a
> notification if the test failed and if so, where?
>
> I noticed that on the lava page, by clicking on the device I can search
> for "rt-tests-cyclictest" and get the recent results for this device for
> this test. Is something like this possible on the dashboard? I.e. look
> for the recent cyclictest results?
> Right now I have no idea how to do this without going device by device.
>
> Sebastian
>

Re: kernelci test results

[Denys Fedoryshchenko]

Hi Sebastian, Alan

The system currently creates a short log extract only when a build or boot test is marked as failed. This job completed successfully, including the boot and cyclictest, so the log was stored but not automatically analyzed for an extract.
The kernel warning is present in the full log and our analyzer can recognize it, but it was missed because the overall job passed. This is a limitation in the current reporting flow.
We initially chose to create log extracts only for failed builds or boot tests because successful boots often contain harmless or known warnings, such as spurious IRQ messages. Reporting every warning as an issue could create substantial noise that kernel developers may not find actionable.
We could review this policy and maintain an ignore list for known, non-actionable messages. That would let us highlight new or unexpected warnings without creating unnecessary noise for kernel developers.

From: Alan Zanoni Peixinho <alan.peixinho@profusion.mobi>
To: "Sebastian Andrzej Siewior"<bigeasy@linutronix.de>
Cc: <kernelci@lists.linux.dev>, <kernelci-webdashboard@groups.io>, "John Ogness"<john.ogness@linutronix.de>
Date: Thu, 03 Sep 2026 21:37:09 +0300
Subject: Re: kernelci test results

 > Hi Sebastian, I work with the dashboard team, so I hope I am able to help you.
 > 
 > First of all, thanks for using the dashboard, and for taking time to
 > write this up. Scenarios like this one are exactly what help us find
 > the UX gaps for improvement.
 > 
 > The dashboard is mostly organized around tree checkouts first.
 > If i understood it properly you might be able to achieve what you are
 > looking for via:
 > 1. Selecting a tree (ie stable-rt/v6.6-rt).
 > 2. Clicking on the filters button on the right. There we can apply
 > filters for configs (defconfig+preempt_rt, etc).
 > 3. We click on the tests tab (default tab is for builds).
 > 4. On the list of tests you can search for test paths (cyclictest,
 > rt-tests, etc).
 > 
 > Link for the view:
 > https://dashboard.kernelci.org/tree/stable-rt/v6.6-rt/84925bb144f49eb0d5f4c5c29f690b4bbf84903b?df|c|defconfig%2Bpreempt_rt=true&p=t
 > 
 > That list is for that checkout across hardware, so you do not have to
 > go board by board. (Unfortunately there is still no search for “this
 > test on this device over time” across all trees.)
 > 
 > About the other questions.
 > 
 > All the logs presented are the ones that have been submitted via KCI
 > submissions, so there might be some missing information when not
 > reported.
 > 
 > On the topic of notifying on log errors, this is a feature on active
 > discussion, to automate notification for first for boots and later for
 > test failures as well. Today we only notify registered issues.
 > 
 > We are actually looking for some feedback of dashboard users to help
 > us steer on the roadmap for new features, and your help would be much
 > appreciated.
 > 
 > Att
 > Alan Peixinho
 > KernelCI Dashboard Team
 > 
 > On Thu, Sep 3, 2026 at 2:36 PM Sebastian Andrzej Siewior
 > <bigeasy@linutronix.de> wrote:
 > >
 > > Hi,
 > >
 > > I've been poking at the dashboard to see how rt-tests tests are doing.
 > > I tried to summarize my doing/ process and added questions while doing
 > > so. I might be doing things wrong, not the expected way or simply
 > > missing the obvious.
 > >
 > >
 > > On the dashboard, by clicking hardware, Tree=mainline I see only
 > > kubernetes as platform. There I see on the bottom various builds. There
 > > is a search bar where I can limit to "preempt_rt" and then builds are
 > > limited to "preempt_rt" configuration and it also includes "next".
 > >
 > > There I can click on the build-log (and later I get the build.log.gz)
 > > but it appears not to contain the build log of the kernel. I think there
 > > is the "config merging" part and kselftest build but not the actual
 > > build of the kernel itself.
 > >
 > > Clicking on tests, I can also filter for preempt_rt but there is only
 > > kselftest. I can't see the results, there is always "No logs available".
 > > Based on the timestamp the results are from today so I wouldn't expect
 > > them to be deleted because of age.
 > >
 > > Going back to "trees" and selecting "stable-rt" and branch v6.6-rt
 > > there is more HW. Clicking on bcm2711-rpi-4-b since it has the most
 > > results here.
 > >
 > > Here I clicked on configs "defconfig+preempt_rt" and then one of tree
 > > that appeared is "mainline/master" but I did not see this earlier while
 > > I did select that tree.
 > >
 > > I did click on "tests" and then "rt-tests" appeared. Expanding it up, I
 > > found cyclictest results for that tree.
 > >
 > > Clicking on details I found a link to
 > >         https://lava.collabora.dev/scheduler/job/22810572
 > >
 > > where the complete log is available of what has been done on that
 > > hardware. In the bootlog I noticed
 > > | [    0.000000] WARNING: kernel/trace/trace_events.c:420 at test_double_dereference+0x128/0x130, CPU#0: swapper/0/0
 > >
 > > which is a warning that appeared during boot. Is this picked up
 > > somewhere? Is there a notification sent somewhere that this has been
 > > noticed?
 > >
 > > Later there is the cyclictest invocation, arguments and the result.
 > > There is no max-value set for the test so it passes. Would there be a
 > > notification if the test failed and if so, where?
 > >
 > > I noticed that on the lava page, by clicking on the device I can search
 > > for "rt-tests-cyclictest" and get the recent results for this device for
 > > this test. Is something like this possible on the dashboard? I.e. look
 > > for the recent cyclictest results?
 > > Right now I have no idea how to do this without going device by device.
 > >
 > > Sebastian
 > >
 > 
 > 

Re: kernelci test results

[Ben Copeland]

Hi Sebastian,

Thanks for the report.

On Fri, 4 Sept 2026 at 06:25, Denys Fedoryshchenko
<denys.f@collabora.com> wrote:
>
> Hi Sebastian, Alan
>
> The system currently creates a short log extract only when a build or boot test is marked as failed. This job completed successfully, including the boot and cyclictest, so the log was stored but not automatically analyzed for an extract.
> The kernel warning is present in the full log and our analyzer can recognize it, but it was missed because the overall job passed. This is a limitation in the current reporting flow.
> We initially chose to create log extracts only for failed builds or boot tests because successful boots often contain harmless or known warnings, such as spurious IRQ messages. Reporting every warning as an issue could create substantial noise that kernel developers may not find actionable.
> We could review this policy and maintain an ignore list for known, non-actionable messages. That would let us highlight new or unexpected warnings without creating unnecessary noise for kernel developers.
>
> From: Alan Zanoni Peixinho <alan.peixinho@profusion.mobi>
> To: "Sebastian Andrzej Siewior"<bigeasy@linutronix.de>
> Cc: <kernelci@lists.linux.dev>, <kernelci-webdashboard@groups.io>, "John Ogness"<john.ogness@linutronix.de>
> Date: Thu, 03 Sep 2026 21:37:09 +0300
> Subject: Re: kernelci test results
>
>  > Hi Sebastian, I work with the dashboard team, so I hope I am able to help you.
>  >
>  > First of all, thanks for using the dashboard, and for taking time to
>  > write this up. Scenarios like this one are exactly what help us find
>  > the UX gaps for improvement.
>  >
>  > The dashboard is mostly organized around tree checkouts first.
>  > If i understood it properly you might be able to achieve what you are
>  > looking for via:
>  > 1. Selecting a tree (ie stable-rt/v6.6-rt).
>  > 2. Clicking on the filters button on the right. There we can apply
>  > filters for configs (defconfig+preempt_rt, etc).
>  > 3. We click on the tests tab (default tab is for builds).
>  > 4. On the list of tests you can search for test paths (cyclictest,
>  > rt-tests, etc).
>  >
>  > Link for the view:
>  > https://dashboard.kernelci.org/tree/stable-rt/v6.6-rt/84925bb144f49eb0d5f4c5c29f690b4bbf84903b?df|c|defconfig%2Bpreempt_rt=true&p=t
>  >
>  > That list is for that checkout across hardware, so you do not have to
>  > go board by board. (Unfortunately there is still no search for “this
>  > test on this device over time” across all trees.)
>  >
>  > About the other questions.
>  >
>  > All the logs presented are the ones that have been submitted via KCI
>  > submissions, so there might be some missing information when not
>  > reported.
>  >
>  > On the topic of notifying on log errors, this is a feature on active
>  > discussion, to automate notification for first for boots and later for
>  > test failures as well. Today we only notify registered issues.
>  >
>  > We are actually looking for some feedback of dashboard users to help
>  > us steer on the roadmap for new features, and your help would be much
>  > appreciated.
>  >
>  > Att
>  > Alan Peixinho
>  > KernelCI Dashboard Team
>  >
>  > On Thu, Sep 3, 2026 at 2:36 PM Sebastian Andrzej Siewior
>  > <bigeasy@linutronix.de> wrote:
>  > >
>  > > Hi,
>  > >
>  > > I've been poking at the dashboard to see how rt-tests tests are doing.
>  > > I tried to summarize my doing/ process and added questions while doing
>  > > so. I might be doing things wrong, not the expected way or simply
>  > > missing the obvious.
>  > >
>  > >
>  > > On the dashboard, by clicking hardware, Tree=mainline I see only
>  > > kubernetes as platform. There I see on the bottom various builds. There
>  > > is a search bar where I can limit to "preempt_rt" and then builds are
>  > > limited to "preempt_rt" configuration and it also includes "next".
>  > >
>  > > There I can click on the build-log (and later I get the build.log.gz)
>  > > but it appears not to contain the build log of the kernel. I think there
>  > > is the "config merging" part and kselftest build but not the actual
>  > > build of the kernel itself.

By design. We build with tuxmake, which builds silently unless you
pass --verbose, so a good kernel build emits little more than echoed
commands. What is left is warnings and errors, and for a failed build
that log is what logspec parses. The noise you see is the kselftest
build, which echoes anyway. There is also a reproducer.sh artefact if
you want to reproduce the build locally.

>  > >
>  > > Clicking on tests, I can also filter for preempt_rt but there is only
>  > > kselftest. I can't see the results, there is always "No logs available".

Those aren't LAVA test runs, which is why it looks broken. On the
kubernetes hardware page you are seeing kselftest *build* results
hanging off the kbuild node, e.g. path

  kbuild-gcc-14-arm-preempt_rt-kselftest.build.kselftest.rseq

Those have no log_url, so the viewer says "No logs available", but the
log is there: look at the output files section of the test page for
build_kselftest.log.gz and build_kselftest_stderr.log.gz. Not expiry.

>  > > Based on the timestamp the results are from today so I wouldn't expect
>  > > them to be deleted because of age.
>  > >
>  > > Going back to "trees" and selecting "stable-rt" and branch v6.6-rt
>  > > there is more HW. Clicking on bcm2711-rpi-4-b since it has the most
>  > > results here.
>  > >
>  > > Here I clicked on configs "defconfig+preempt_rt" and then one of tree
>  > > that appeared is "mainline/master" but I did not see this earlier while
>  > > I did select that tree.
>  > >
>  > > I did click on "tests" and then "rt-tests" appeared. Expanding it up, I
>  > > found cyclictest results for that tree.
>  > >
>  > > Clicking on details I found a link to
>  > >         https://lava.collabora.dev/scheduler/job/22810572
>  > >
>  > > where the complete log is available of what has been done on that
>  > > hardware. In the bootlog I noticed
>  > > | [    0.000000] WARNING: kernel/trace/trace_events.c:420 at test_double_dereference+0x128/0x130, CPU#0: swapper/0/0
>  > >
>  > > which is a warning that appeared during boot. Is this picked up
>  > > somewhere? Is there a notification sent somewhere that this has been
>  > > noticed?

Denys mostly already covered this. One thing to add, since it bears on
the ignore list he suggested: our parser was written for the pre-v6.19
WARNING layout. On the reordered format your log uses it extracts

    WARNING at test_double_dereference+0x128/0x130, CPU#0: swapper/0/0

So the file:line is dropped, and the CPU and comm/pid land in the
string we hash into the issue ID. The same warning on a different CPU
becomes a different issue, and an ignore list keyed on that text would
not match reliably. Only affects 6.19 and later, so the -rt branches
are fine today. Its a simple regex fix

On the notifications:
https://github.com/kernelci/dashboard/blob/main/backend/data/notifications/subscriptions/stable-rt.yaml.
This needs to be updated. It looks like there's work here to build
more branches and update notifications, but also to see if we can
automate this going forward so it doesn't slip.

>  > >
>  > > Later there is the cyclictest invocation, arguments and the result.
>  > > There is no max-value set for the test so it passes. Would there be a
>  > > notification if the test failed and if so, where?

I cannot find any thresholds set, which isn't ideal.
parse_rt_tests_results.py prints the literal string "pass" for every
latency line; only cyclictest's return code decides the overall
result. So a max-value parameter on our side would change nothing
until the parser learns about thresholds.

>  > >
>  > > I noticed that on the lava page, by clicking on the device I can search
>  > > for "rt-tests-cyclictest" and get the recent results for this device for
>  > > this test. Is something like this possible on the dashboard? I.e. look
>  > > for the recent cyclictest results?

Alan already covered this; however, after a quick dive into some of
the rt results, I noticed jobs dying with "Unsupported url protocol
scheme". Seems to be an issue with rt-tests + nfs boot. I will have to
dig deeper, but there is some improvement to be had here. I also
noticed that we are missing trixie-rt armhf rootfs, which will be a
problem for imx6q-sabrelite (and other armhf boards about running rt).

So bcm2711-rpi-4-b, where you ended up, is one of the few places
rt-tests actually produces data. I'd need to dig further to see where
else it is running or failing.

Thanks

Ben

>  > > Right now I have no idea how to do this without going device by device.
>  > >
>  > > Sebastian
>  > >
>  >
>  >
>
>
>