Two kernel oopses

Two kernel oopses

[Ismo 1. Puustinen (Nokia)]

Hi list!

I'm not sure if the kernel bug tracker is used for tracking the RT-related issues, so I thought I would also send a message here. This is regarding these two issues: https://bugzilla.kernel.org/show_bug.cgi?id=219919 and https://bugzilla.kernel.org/show_bug.cgi?id=219920. Briefly, on a HPE DL110 Gen11 system running 6.12.13 kernel (Talos Linux), I see daily crashes which seem to be related to the PREEMPT_RT kernel.

The first issue seems to be triggering an assert on line kernel/sched/rt.c:1035 . Based on my understanding, the nr_rq->rt_running value drops to 0, which should not be possible. The second issue relates to "not-present page" issue. I'm guessing it might be related to RT because I don't see it with non-RT kernel, and rt spin lock seems to be part of the backtrace. The crash occurs with various backtraces but seems to relate to address 0000000000000051 each time it happens. The system has a Mellanox and an Intel NIC using mlx5 and ice drivers.

Any help or debugging tips would be appreciated! Thanks!

Ismo

Re: Two kernel oopses

[Sebastian Andrzej Siewior]

On 2025-03-27 08:56:16 [+0000], Ismo 1. Puustinen (Nokia) wrote:
> Hi list!
Hi,

> I'm not sure if the kernel bug tracker is used for tracking the
> RT-related issues, so I thought I would also send a message here. This
> is regarding these two issues:
> https://bugzilla.kernel.org/show_bug.cgi?id=219919 and
> https://bugzilla.kernel.org/show_bug.cgi?id=219920. Briefly, on a HPE
> DL110 Gen11 system running 6.12.13 kernel (Talos Linux), I see daily
> crashes which seem to be related to the PREEMPT_RT kernel.

I don't know what Talos Linux is. Do you have a vanilla v6.12.13 or do
you have any patches (like the PREEMPT_RT patch for v6.12) on top?

> The first issue seems to be triggering an assert on line
> kernel/sched/rt.c:1035 . Based on my understanding, the
> nr_rq->rt_running value drops to 0, which should not be possible. The
> second issue relates to "not-present page" issue. I'm guessing it
> might be related to RT because I don't see it with non-RT kernel, and
> rt spin lock seems to be part of the backtrace. The crash occurs with
> various backtraces but seems to relate to address 0000000000000051
> each time it happens. The system has a Mellanox and an Intel NIC using
> mlx5 and ice drivers.

Both things are scheduler related and mess up its internal state/
tracking. I haven't see anything like it so far. It would be good to
know if you have any external patches or if you do anything else usual
to track this down.

> Any help or debugging tips would be appreciated! Thanks!
> 
> Ismo

Sebastian

RE: Two kernel oopses

[Ismo 1. Puustinen (Nokia)]

On 2025-03-31 09:32:11 [+0300], Sebastian Andrzej Siewior wrote:

> I don't know what Talos Linux is. Do you have a vanilla v6.12.13 or do you
> have any patches (like the PREEMPT_RT patch for v6.12) on top?

Talos Linux is a minimal Linux distribution for running a Kubernetes node. The system has vanilla 6.12.13. We build it ourselves because Talos doesn't provide a RT kernel. I'm also not sure why the separate PREEMPT_RT patch would be needed on 6.12 -- should I use it?

> Both things are scheduler related and mess up its internal state/ tracking. I
> haven't see anything like it so far. It would be good to know if you have any
> external patches or if you do anything else usual to track this down.

There isn't anything special that I can think of. The crashes seem to happen on a "mostly idle" system even without any special kernel parameters or run the real-time workloads. There are lots of processes running on the system though, but the cpu load is small. The server itself is a large one with 64 cores, so I guess any race conditions are easy to trigger there. The BIOS version is recent, using an older version doesn't seem to affect the issues. Also, for what it's worth, we see the same crashes also on 6.12.6 kernel. The rt.c:1035 crash seems to happen in both IRQ and TASK context. The crashes happen on several (identical) servers.

I'll add here the first lines of a couple of oopses I have in store, I can attach the full backtraces if it helps. I can also add full kernel configuration or other data points if you think that it may help, and if you think there is something I can try out to help find the issue I'm happy to do so.

[24208.761392] kernel BUG at kernel/sched/rt.c:1035!
[24208.761399] Oops: invalid opcode: 0000 [#1] PREEMPT_RT SMP PTI
[24208.761402] CPU: 37 UID: 0 PID: 19 Comm: rcu_preempt Not tainted 6.12.6-talos #1
[24208.761405] Hardware name: HPE ProLiant DL110 Gen11/ProLiant DL110 Gen11, BIOS 2.44 01/17/2025
[24208.761406] RIP: 0010:dequeue_rt_stack+0x359/0x3b0

[37718.448733] kernel BUG at kernel/sched/rt.c:1035!
[37718.448747] Oops: invalid opcode: 0000 [#1] PREEMPT_RT SMP PTI
[37718.448756] CPU: 11 UID: 0 PID: 0 Comm: swapper/11 Not tainted 6.12.6-talos #1
[37718.448763] Hardware name: HPE ProLiant DL110 Gen11/ProLiant DL110 Gen11, BIOS 2.16 03/01/2024
[37718.448766] RIP: 0010:dequeue_rt_stack+0x359/0x3b0

[20045.137840] kernel BUG at kernel/sched/rt.c:1035!
[20045.137848] Oops: invalid opcode: 0000 [#1] PREEMPT_RT SMP PTI
[20045.137851] CPU: 5 UID: 0 PID: 423 Comm: rcuog/48 Not tainted 6.12.6-talos #1
[20045.137853] Hardware name: HPE ProLiant DL110 Gen11/ProLiant DL110 Gen11, BIOS 2.48 03/11/2025
[20045.137854] RIP: 0010:dequeue_rt_stack+0x359/0x3b0

Thanks!

Ismo

Re: RE: Two kernel oopses

[Sebastian Andrzej Siewior]

On 2025-04-01 11:22:24 [+0000], Ismo 1. Puustinen (Nokia) wrote:
> On 2025-03-31 09:32:11 [+0300], Sebastian Andrzej Siewior wrote:
> 
> > I don't know what Talos Linux is. Do you have a vanilla v6.12.13 or do you
> > have any patches (like the PREEMPT_RT patch for v6.12) on top?
> 
> Talos Linux is a minimal Linux distribution for running a Kubernetes node. The system has vanilla 6.12.13. We build it ourselves because Talos doesn't provide a RT kernel. I'm also not sure why the separate PREEMPT_RT patch would be needed on 6.12 -- should I use it?

There are a few extensions which are not yet part of v6.12. Nothing
that would explain this kind of crashes.

> > Both things are scheduler related and mess up its internal state/ tracking. I
> > haven't see anything like it so far. It would be good to know if you have any
> > external patches or if you do anything else usual to track this down.
> 
> There isn't anything special that I can think of. The crashes seem to
> happen on a "mostly idle" system even without any special kernel
> parameters or run the real-time workloads. There are lots of processes
> running on the system though, but the cpu load is small. The server
> itself is a large one with 64 cores, so I guess any race conditions
> are easy to trigger there. The BIOS version is recent, using an older
> version doesn't seem to affect the issues. Also, for what it's worth,
> we see the same crashes also on 6.12.6 kernel. The rt.c:1035 crash
> seems to happen in both IRQ and TASK context. The crashes happen on
> several (identical) servers.

So no external patches.
Could you build one kernel with CONFIG_DEBUG_ATOMIC_SLEEP and check if
something pops up in the kernel buffer?
If the crash happens without additional data could you try to enable
CONFIG_PROVE_LOCKING and try if this option creates additional output?

> I'll add here the first lines of a couple of oopses I have in store, I
> can attach the full backtraces if it helps. I can also add full kernel
> configuration or other data points if you think that it may help, and
> if you think there is something I can try out to help find the issue
> I'm happy to do so.

I've seen the backtraces in bugzilla, that is okay. If you could try the
steps above if this creates an additional source of information what
might be going on, that would be nice.

> Thanks!
> 
> Ismo

Sebastian

RE: Two kernel oopses

[Ismo 1. Puustinen (Nokia)]

>> There isn't anything special that I can think of. The crashes seem to 
>> happen on a "mostly idle" system even without any special kernel 
>> parameters or run the real-time workloads. There are lots of processes 
>> running on the system though, but the cpu load is small. The server 
>> itself is a large one with 64 cores, so I guess any race conditions 
>> are easy to trigger there. The BIOS version is recent, using an older 
>> version doesn't seem to affect the issues. Also, for what it's worth, 
>> we see the same crashes also on 6.12.6 kernel. The rt.c:1035 crash 
>> seems to happen in both IRQ and TASK context. The crashes happen on 
>> several (identical) servers.
>
> So no external patches.
> Could you build one kernel with CONFIG_DEBUG_ATOMIC_SLEEP and check if something pops up in the kernel buffer?
> If the crash happens without additional data could you try to enable CONFIG_PROVE_LOCKING and try if this option creates additional output?

Ok, I tested on two kernels with the debug options turned on, and there is just one extra log line (psi: inconsistent task state) that appears right before the crash for both bugs. I updated also from 6.12.13 to 6.12.18 so the added log line might be a side effect of that. I'm not sure how to continue debugging this -- if you have any suggestions, please let me know.

I was running the same kernel on the system with almost no processes running, and did not see the crash happening in a week. However when I add more processes there, the crash frequency seems to around once per day. I was thinking that I could maybe run stress-ng or similar software to simulate more load, in case I could be able to reproduce this faster.

This is the log line:

[27811.789630] psi: inconsistent task state! task=16461:operator[27811.789630] psi: inconsistent task state! task=16461:operator cpu=6 psi_flags=10 clear=14 set=0
[27811.808097] ------------[ cut here ]------------
[27811.808099] kernel BUG at kernel/sched/rt.c:1035!
[27811.808106] Oops: invalid opcode: 0000 [#1] PREEMPT_RT SMP PTI
[27811.808109] CPU: 6 UID: 0 PID: 159 Comm: rcuog/16 Not tainted 6.12.18-mxie #1
[27811.808111] Hardware name: HPE ProLiant DL110 Gen11/ProLiant DL110 Gen11, BIOS 2.16 03/01/2024
[27811.808113] RIP: 0010:dequeue_rt_stack+0x359/0x3b0

Ismo