PREEMPT_RT testing on kernelci

PREEMPT_RT testing on kernelci

[Sebastian Andrzej Siewior]

Hi,

I remember Daniel Wagner has been asking to include PREEMPT_RT within
kernelci. I don't really know what the status here is. Thank you for the
work.

Since PREEMPT_RT is now available upstream, would it work to compile and
enable some of its tests for the supported architectures on Linus' tree
and maybe the -next tree? I don't think that happens at the moment, I
saw only a few stable-rt trees.

If so, should I just open an issue at
	https://github.com/kernelci/kernelci-core/issues/new

Is there be anything in particular you need help with?

Sebastian

Re: PREEMPT_RT testing on kernelci

[Gustavo Padovan]

Hi Sebastian,

Thank you for your email. We are happy that KernelCI is helping you with tests.

---- On Tue, 11 Feb 2025 06:05:11 -0300 Sebastian Andrzej Siewior  wrote ---

 > Hi, 
 >  
 > I remember Daniel Wagner has been asking to include PREEMPT_RT within 
 > kernelci. I don't really know what the status here is. Thank you for the 
 > work. 

Current status is that we have them enabled only for the stable-rt branches as
you can see here:

https://dashboard.kernelci.org/tree/b8c892b7816074585ff45397712c6310ba5ae26b?df%7Ctp%7Crt-tests=true&i=20&p=t&ti%7Cc=v6.6.74-rt48&ti%7Cch=b8c892b7816074585ff45397712c6310ba5ae26b&ti%7Cgb=v6.6-rt&ti%7Cgu=https%3A%2F%2Fgit.kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Frt%2Flinux-stable-rt.git&ti%7Ct=stable-rt

 >  
 > Since PREEMPT_RT is now available upstream, would it work to compile and 
 > enable some of its tests for the supported architectures on Linus' tree 
 > and maybe the -next tree? I don't think that happens at the moment, I 
 > saw only a few stable-rt trees. 

https://github.com/kernelci/kernelci-pipeline/pull/1010

This will get it enabled for mainline and next. Allow us a few days for processing and
deploying this in production.

Beyond that, we would love any feedback you have to us. A lot of the infra
we have today is new, so there are still rough edges to work on.

Best,

- Gus

Re: PREEMPT_RT testing on kernelci

[Sebastian Andrzej Siewior]

On 2025-02-11 08:33:56 [-0300], Gustavo Padovan wrote:
> Hi Sebastian,
Hi Gustavo,

> Thank you for your email. We are happy that KernelCI is helping you with tests.
;)
>  > Since PREEMPT_RT is now available upstream, would it work to compile and 
>  > enable some of its tests for the supported architectures on Linus' tree 
>  > and maybe the -next tree? I don't think that happens at the moment, I 
>  > saw only a few stable-rt trees. 
> 
> https://github.com/kernelci/kernelci-pipeline/pull/1010
> 
> This will get it enabled for mainline and next. Allow us a few days for processing and
> deploying this in production.

Sure. Just poke if you have something to inspect or look at because of
$reason.

The architectures, that support RT are what you see in
| git grep ARCH_SUPPORTS_RT arch/

so this does not include arm at current stage for instance. So in
general, if you can't select CONFIG_PREEMPT_RT you can skip the
config/ build/ test.

RiscV on the other hand is supported.

> Beyond that, we would love any feedback you have to us. A lot of the infra
> we have today is new, so there are still rough edges to work on.

Sure. How do I see easily what is tested/ done? I could maybe wait a few
days and check the dashboard. The link from above showed timerlat has
been but I didn't manage to figure what has been. I guess there is a
script somewhere handling the logic.

> Best,
> 
> - Gus

Sebastian

Re: PREEMPT_RT testing on kernelci

[Gustavo Padovan]

Hi Sebastian,


---- On Tue, 11 Feb 2025 12:20:18 -0300 Sebastian Andrzej Siewior  wrote ---

 > On 2025-02-11 08:33:56 [-0300], Gustavo Padovan wrote: 
 > > Hi Sebastian, 
 > Hi Gustavo, 
 >  
 > > Thank you for your email. We are happy that KernelCI is helping you with tests. 
 > ;) 
 > >  > Since PREEMPT_RT is now available upstream, would it work to compile and 
 > >  > enable some of its tests for the supported architectures on Linus' tree 
 > >  > and maybe the -next tree? I don't think that happens at the moment, I 
 > >  > saw only a few stable-rt trees. 
 > > 
 > > https://github.com/kernelci/kernelci-pipeline/pull/1010 
 > > 
 > > This will get it enabled for mainline and next. Allow us a few days for processing and 
 > > deploying this in production. 
 >  
 > Sure. Just poke if you have something to inspect or look at because of 
 > $reason. 
 >  
 > The architectures, that support RT are what you see in 
 > | git grep ARCH_SUPPORTS_RT arch/ 
 >  
 > so this does not include arm at current stage for instance. So in 
 > general, if you can't select CONFIG_PREEMPT_RT you can skip the 
 > config/ build/ test. 
 >  
 > RiscV on the other hand is supported.

Right. Let me look at how we can limit this to x86 only.
Unfortunatelly, we don't have much RISC-V hw support in KernelCI yet.

 >  
 > > Beyond that, we would love any feedback you have to us. A lot of the infra 
 > > we have today is new, so there are still rough edges to work on. 
 >  
 > Sure. How do I see easily what is tested/ done? I could maybe wait a few 
 > days and check the dashboard. The link from above showed timerlat has 
 > been but I didn't manage to figure what has been. I guess there is a 
 > script somewhere handling the logic. 

That is one the current limitations in the dashboard. The CI systems are not sending
a clear signal yet that they completed all tests they wanted to run. They just submit
test result individually as they get finished. 

Imo, it is a priority to fix this as it is second time this week I get this feedback.

Best,

- Gus

Re: PREEMPT_RT testing on kernelci

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 770 bytes --]

On Tue, Feb 11, 2025 at 12:50:47PM -0300, Gustavo Padovan wrote:

>  > The architectures, that support RT are what you see in 
>  > | git grep ARCH_SUPPORTS_RT arch/ 

>  > so this does not include arm at current stage for instance. So in 
>  > general, if you can't select CONFIG_PREEMPT_RT you can skip the 
>  > config/ build/ test. 

>  > RiscV on the other hand is supported.

> Right. Let me look at how we can limit this to x86 only.
> Unfortunatelly, we don't have much RISC-V hw support in KernelCI yet.

We should enable it on arm64 as well, there's plenty of boards for that.
The current list of supported architectures is arm64, loongarch, riscv
and x86.

I'd be happy to have it running in my lab for the boards I've got
multiples of.  

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: PREEMPT_RT testing on kernelci

[Sebastian Andrzej Siewior]

On 2025-02-11 12:50:47 [-0300], Gustavo Padovan wrote:
> Hi Sebastian,
Hi Gustavo,

> That is one the current limitations in the dashboard. The CI systems are not sending
> a clear signal yet that they completed all tests they wanted to run. They just submit
> test result individually as they get finished. 
> 
> Imo, it is a priority to fix this as it is second time this week I get this feedback.

Ah okay. No need to rush. If you have something to look at, let me know
and I will. Other than that I will lean back and wait for the bot to
complain ;)

Thank you.

> Best,
> 
> - Gus

Sebastian

Re: PREEMPT_RT testing on kernelci

[Sebastian Andrzej Siewior]

On 2025-02-13 09:52:56 [+0100], To Gustavo Padovan wrote:
> On 2025-02-11 12:50:47 [-0300], Gustavo Padovan wrote:
> > Hi Sebastian,
Hi Gustavo,

> > That is one the current limitations in the dashboard. The CI systems are not sending
> > a clear signal yet that they completed all tests they wanted to run. They just submit
> > test result individually as they get finished. 
> > 
> > Imo, it is a priority to fix this as it is second time this week I get this feedback.
> 
> Ah okay. No need to rush. If you have something to look at, let me know
> and I will. Other than that I will lean back and wait for the bot to
> complain ;)
> 
> Thank you.

any updates on this?

> > Best,
> > 
> > - Gus

Sebastian

Re: PREEMPT_RT testing on kernelci

[Gustavo Padovan]

Hi Sebastian,

---- On Fri, 11 Apr 2025 16:44:32 -0300 Sebastian Andrzej Siewior  wrote ---

 > On 2025-02-13 09:52:56 [+0100], To Gustavo Padovan wrote: 
 > > On 2025-02-11 12:50:47 [-0300], Gustavo Padovan wrote: 
 > > > Hi Sebastian, 
 > Hi Gustavo, 
 >  
 > > > That is one the current limitations in the dashboard. The CI systems are not sending 
 > > > a clear signal yet that they completed all tests they wanted to run. They just submit 
 > > > test result individually as they get finished. 
 > > > 
 > > > Imo, it is a priority to fix this as it is second time this week I get this feedback. 
 > > 
 > > Ah okay. No need to rush. If you have something to look at, let me know 
 > > and I will. Other than that I will lean back and wait for the bot to 
 > > complain ;) 
 > > 
 > > Thank you. 
 >  
 > any updates on this? 

We had some progress, but not enough yet. KCIDB gained support for CI systems to
send when they completed all test execution for a given branch checkout, but we have
to plumb this across this pipeline yet.

Regardless of that, the data is all available in the dashboard and you look up for the 
rt-tests results there.

kci-dev cli - http://kci.dev/ - can also be used. For example: 

 kci-dev results tests --giturl 'https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git' --branch master --latest --filter=filter.yaml

Where filter can contain:

----
test:
  - rt-tests.cyclictest
  - rt-tests.cyclicdeadline
  ...
----

I recommend using latest git kci-dev for the time being as we keep fixing and evolving.
Eg, I just sent a PR to fix a crash that happens when we use the filter file with no hw filter in it.

I also just created an issue for being able to use wildcards in filters. eg `rt-tests.*`
https://github.com/kernelci/kci-dev/issues/163

Both dashboard and kci-dev are pretty young, so features to help the community more and
more are growing month by month. We appreciate the patience and understanding of the community.

On the the above, I am working on our notifications system in the dashboard. You usecase to be cc'ed
on notification for certains tests is not yet support, but we can look at it if we tell me exactly what will 
help you.

In my understanding you want to know when results for any rt-tests change status (eg PASS -> FAIL or
vice versa) for a selection of trees (eg mainline, next, etc). Can you confirm?

Best,

- Gus

Re: PREEMPT_RT testing on kernelci

[Sebastian Andrzej Siewior]

On 2025-04-14 12:14:44 [-0300], Gustavo Padovan wrote:
> Hi Sebastian,
Hi Gustavo,

> In my understanding you want to know when results for any rt-tests change status (eg PASS -> FAIL or
> vice versa) for a selection of trees (eg mainline, next, etc). Can you confirm?

The results change is definitely interesting. The other thing is what is
tested and how. I remember you added/ changed some architectures for the
RT subset. I also assume that warnings at bootime (as in lockdep or
general warnings) are reported.
I remember Daniel asked for some cyclictests/rt-tests integration so I
don't know what is tested here ;)

From the commandline you sent that there are a few boards running the
cyclictests and everything either passes or is in state "INCONCLUSIVE".
So this looks nice.

> Best,
> 
> - Gus

Sebastian

Re: PREEMPT_RT testing on kernelci

[Gustavo Padovan]

---- On Thu, 17 Apr 2025 07:04:10 -0300 Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote ---

 > On 2025-04-14 12:14:44 [-0300], Gustavo Padovan wrote: 
 > > Hi Sebastian, 
 > Hi Gustavo, 
 >  
 > > In my understanding you want to know when results for any rt-tests change status (eg PASS -> FAIL or 
 > > vice versa) for a selection of trees (eg mainline, next, etc). Can you confirm? 
 >  
 > The results change is definitely interesting. The other thing is what is 
 > tested and how. I remember you added/ changed some architectures for the 
 > RT subset. I also assume that warnings at bootime (as in lockdep or 
 > general warnings) are reported. 

We have some support around that, but this work is still evolving.

 > I remember Daniel asked for some cyclictests/rt-tests integration so I 
 > don't know what is tested here ;) 
 >  
 > From the commandline you sent that there are a few boards running the 
 > cyclictests and everything either passes or is in state "INCONCLUSIVE". 
 > So this looks nice. 

Okay.

As for notifications, for which trees/branches you want to see it?

Re: PREEMPT_RT testing on kernelci

[Sebastian Andrzej Siewior]

On 2025-04-17 10:10:43 [-0300], Gustavo Padovan wrote:
> 
> 
> ---- On Thu, 17 Apr 2025 07:04:10 -0300 Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote ---
> 
>  > On 2025-04-14 12:14:44 [-0300], Gustavo Padovan wrote: 
>  > > Hi Sebastian, 
>  > Hi Gustavo, 
>  >  
>  > > In my understanding you want to know when results for any rt-tests change status (eg PASS -> FAIL or 
>  > > vice versa) for a selection of trees (eg mainline, next, etc). Can you confirm? 
>  >  
>  > The results change is definitely interesting. The other thing is what is 
>  > tested and how. I remember you added/ changed some architectures for the 
>  > RT subset. I also assume that warnings at bootime (as in lockdep or 
>  > general warnings) are reported. 
> 
> We have some support around that, but this work is still evolving.

Okay.

>  > I remember Daniel asked for some cyclictests/rt-tests integration so I 
>  > don't know what is tested here ;) 
>  >  
>  > From the commandline you sent that there are a few boards running the 
>  > cyclictests and everything either passes or is in state "INCONCLUSIVE". 
>  > So this looks nice. 
> 
> Okay.
> 
> As for notifications, for which trees/branches you want to see it?

Linus' master and the -next tree would be good. If you plan add to add
Cc here, it would be nice if you could add linux-rt-devel@lists.linux.dev

Sebastian

Re: PREEMPT_RT testing on kernelci

[Gustavo Padovan]

---- On Thu, 17 Apr 2025 12:11:13 -0300 Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote ---

 > On 2025-04-17 10:10:43 [-0300], Gustavo Padovan wrote: 
 > > 
 > > 
 > > ---- On Thu, 17 Apr 2025 07:04:10 -0300 Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote --- 
 > > 
 > >  > On 2025-04-14 12:14:44 [-0300], Gustavo Padovan wrote: 
 > >  > > Hi Sebastian, 
 > >  > Hi Gustavo, 
 > >  > 
 > >  > > In my understanding you want to know when results for any rt-tests change status (eg PASS -> FAIL or 
 > >  > > vice versa) for a selection of trees (eg mainline, next, etc). Can you confirm? 
 > >  > 
 > >  > The results change is definitely interesting. The other thing is what is 
 > >  > tested and how. I remember you added/ changed some architectures for the 
 > >  > RT subset. I also assume that warnings at bootime (as in lockdep or 
 > >  > general warnings) are reported. 
 > > 
 > > We have some support around that, but this work is still evolving. 
 >  
 > Okay. 
 >  
 > >  > I remember Daniel asked for some cyclictests/rt-tests integration so I 
 > >  > don't know what is tested here ;) 
 > >  > 
 > >  > From the commandline you sent that there are a few boards running the 
 > >  > cyclictests and everything either passes or is in state "INCONCLUSIVE". 
 > >  > So this looks nice. 
 > > 
 > > Okay. 
 > > 
 > > As for notifications, for which trees/branches you want to see it? 
 >  
 > Linus' master and the -next tree would be good. If you plan add to add 
 > Cc here, it would be nice if you could add linux-rt-devel@lists.linux.dev 

Sounds good. Let me look at how to enable that in the coming weeks, as 
today we don't support report "profiles"  for a given tree.. Eg, the mainline
report comes with all the tests[1], but that would be noise to you.

[0]  https://groups.io/g/kernelci-results/message/58797


Best,

- Gus

Re: PREEMPT_RT testing on kernelci

[Guenter Roeck]

On Tue, Feb 11, 2025 at 1:05 AM Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
>
> Hi,
>
> I remember Daniel Wagner has been asking to include PREEMPT_RT within
> kernelci. I don't really know what the status here is. Thank you for the
> work.
>
> Since PREEMPT_RT is now available upstream, would it work to compile and
> enable some of its tests for the supported architectures on Linus' tree
> and maybe the -next tree? I don't think that happens at the moment, I
> saw only a few stable-rt trees.
>
> If so, should I just open an issue at
>         https://github.com/kernelci/kernelci-core/issues/new
>
> Is there be anything in particular you need help with?
>

Unrelated to KernelCI - how stable is PREEMPT_RT in practice ? Reason
for asking is that I enabled it for x86_64 together with lock
debugging and got an almost immediate "sleeping function called"
backtrace.

Thanks,
Guenter

Re: PREEMPT_RT testing on kernelci

[Sebastian Andrzej Siewior]

On 2025-02-11 07:33:57 [-0800], Guenter Roeck wrote:
> Unrelated to KernelCI - how stable is PREEMPT_RT in practice ? Reason
> for asking is that I enabled it for x86_64 together with lock
> debugging and got an almost immediate "sleeping function called"
> backtrace.

If I would have seen it I would have acted. My guess is that this is
something not a lot of people observed.

> Thanks,
> Guenter

Sebastian

Re: PREEMPT_RT testing on kernelci

[Guenter Roeck]

On Tue, Feb 11, 2025 at 7:38 AM Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
>
> On 2025-02-11 07:33:57 [-0800], Guenter Roeck wrote:
> > Unrelated to KernelCI - how stable is PREEMPT_RT in practice ? Reason
> > for asking is that I enabled it for x86_64 together with lock
> > debugging and got an almost immediate "sleeping function called"
> > backtrace.
>
> If I would have seen it I would have acted. My guess is that this is
> something not a lot of people observed.
>

I suspect that not many run the code with lock debugging enabled. I'll
send you the backtrace separately.

Guenter

> > Thanks,
> > Guenter
>
> Sebastian

Re: PREEMPT_RT testing on kernelci

[Donald Zickus]

On Tue, Feb 11, 2025 at 10:43 AM Guenter Roeck <groeck@google.com> wrote:
>
> On Tue, Feb 11, 2025 at 1:05 AM Sebastian Andrzej Siewior
> <bigeasy@linutronix.de> wrote:
> >
> > Hi,
> >
> > I remember Daniel Wagner has been asking to include PREEMPT_RT within
> > kernelci. I don't really know what the status here is. Thank you for the
> > work.
> >
> > Since PREEMPT_RT is now available upstream, would it work to compile and
> > enable some of its tests for the supported architectures on Linus' tree
> > and maybe the -next tree? I don't think that happens at the moment, I
> > saw only a few stable-rt trees.
> >
> > If so, should I just open an issue at
> >         https://github.com/kernelci/kernelci-core/issues/new
> >
> > Is there be anything in particular you need help with?
> >
>
> Unrelated to KernelCI - how stable is PREEMPT_RT in practice ? Reason
> for asking is that I enabled it for x86_64 together with lock
> debugging and got an almost immediate "sleeping function called"
> backtrace.

Red Hat has been selling a kernel-rt product for almost a decade now.
It should be fairly stable.  We also enable lock debugging on our
debug version of kernel-rt and I don't recall anyone seeing anything
(though those debug kernels are slow).  We may not have the same
config options set and haven't seen it yet.  Stacktrace?

Cheers,
Don

>
> Thanks,
> Guenter
>

Re: PREEMPT_RT testing on kernelci

[Guenter Roeck]

On Wed, Feb 12, 2025 at 9:18 AM Donald Zickus <dzickus@redhat.com> wrote:
>
> On Tue, Feb 11, 2025 at 10:43 AM Guenter Roeck <groeck@google.com> wrote:
> >
> > On Tue, Feb 11, 2025 at 1:05 AM Sebastian Andrzej Siewior
> > <bigeasy@linutronix.de> wrote:
> > >
> > > Hi,
> > >
> > > I remember Daniel Wagner has been asking to include PREEMPT_RT within
> > > kernelci. I don't really know what the status here is. Thank you for the
> > > work.
> > >
> > > Since PREEMPT_RT is now available upstream, would it work to compile and
> > > enable some of its tests for the supported architectures on Linus' tree
> > > and maybe the -next tree? I don't think that happens at the moment, I
> > > saw only a few stable-rt trees.
> > >
> > > If so, should I just open an issue at
> > >         https://github.com/kernelci/kernelci-core/issues/new
> > >
> > > Is there be anything in particular you need help with?
> > >
> >
> > Unrelated to KernelCI - how stable is PREEMPT_RT in practice ? Reason
> > for asking is that I enabled it for x86_64 together with lock
> > debugging and got an almost immediate "sleeping function called"
> > backtrace.
>
> Red Hat has been selling a kernel-rt product for almost a decade now.
> It should be fairly stable.  We also enable lock debugging on our
> debug version of kernel-rt and I don't recall anyone seeing anything
> (though those debug kernels are slow).  We may not have the same
> config options set and haven't seen it yet.  Stacktrace?
>

Two so far.

https://lore.kernel.org/linux-rt-devel/760e34f9-6034-40e0-82a5-ee9becd24438@roeck-us.net/
https://lore.kernel.org/linux-rt-devel/d9480845-7e81-46a2-8bbd-bd0ebdbc6a5f@roeck-us.net/

Guenter