[Question] Detecting Sleep-in-Atomic Context in PREEMPT_RT via RV (Runtime Verification) monitor rtapp:sleep

[Question] Detecting Sleep-in-Atomic Context in PREEMPT_RT via RV (Runtime Verification) monitor rtapp:sleep

[Yunseong Kim]

Hi Nam,

I've been very interested in RV (Runtime Verification) to proactively detect
"sleep in atomic" scenarios on PREEMPT_RT kernels. Specifically, I'm looking
for ways to find cases where sleeping spinlocks or memory allocations are used
within preemption-disabled or irq-disabled contexts. While searching for
solutions, I discovered the RV subsystem.

I've tested with it as follows, and I have a few questions.

# cat /sys/kernel/tracing/rv/available_monitors
wwnr
rtapp
rtapp:sleep

# cat /sys/kernel/tracing/rv/available_reactors
nop
printk
panic

# echo printk > /sys/kernel/tracing/rv/monitors/rtapp/sleep/reactors

# cat /sys/kernel/tracing/rv/monitors/rtapp/sleep/enable
1

# echo rtapp:sleep > /sys/kernel/tracing/rv/enabled_monitors

> [192735.309072] [   T6957] rv: sleep: multipathd[6957]: violation detected

# echo panic > /sys/kernel/tracing/rv/monitors/rtapp/sleep/reactors

> [ T6957] Kernel panic - not syncing: rv: sleep: multipathd[6957]: violation detected
> [193521.768666][ T6957] CPU: 4 UID: 0 PID: 6957 Comm: multipathd Not tainted 6.17.0-rc3-g39f90c196721 #1 PREEMPT_{RT,(full)}
> [193521.771727][ T6957] Hardware name: QEMU KVM Virtual Machine, BIOS 2025.02-8ubuntu3 10/08/2025
> [193521.774126][ T6957] Call trace:
> [193521.774998][ T6957]  show_stack+0x2c/0x3c (C)
> [193521.776281][ T6957]  __dump_stack+0x30/0x40
> [193521.777523][ T6957]  dump_stack_lvl+0x34/0x2bc
> [193521.778797][ T6957]  dump_stack+0x1c/0x48
> [193521.779984][ T6957]  vpanic+0x220/0x618
> [193521.781211][ T6957]  oom_killer_enable+0x0/0x30
> [193521.782512][ T6957]  ltl_validate+0x7ac/0xb1c
> [193521.783870][ T6957]  ltl_atom_update+0xd0/0x32c
> [193521.785198][ T6957]  handle_sched_set_state+0xb8/0x12c
> [193521.786773][ T6957]  __trace_set_current_state+0x128/0x174
> [193521.788450][ T6957]  do_nanosleep+0x128/0x2a4
> [193521.789731][ T6957]  hrtimer_nanosleep+0xb4/0x160
> [193521.791167][ T6957]  common_nsleep+0x6c/0x84
> [193521.792404][ T6957]  __arm64_sys_clock_nanosleep+0x1a8/0x1f0
> [193521.794031][ T6957]  invoke_syscall+0x64/0x168
> [193521.795353][ T6957]  el0_svc_common+0x134/0x164
> [193521.796707][ T6957]  do_el0_svc+0x2c/0x3c
> [193521.797897][ T6957]  el0_svc+0x58/0x184
> [193521.799048][ T6957]  el0t_64_sync_handler+0x84/0x12c
> [193521.800514][ T6957]  el0t_64_sync+0x1b8/0x1bc
> [193521.801818][ T6957] SMP: stopping secondary CPUs
> [193521.803320][ T6957] Dumping ftrace buffer:
> [193521.804510][ T6957]    (ftrace buffer empty)
> [193521.805848][ T6957] Kernel Offset: disabled
> [193521.807084][ T6957] CPU features: 0xc0000,00007800,149a3161,357ff667
> [193521.808941][ T6957] Memory Limit: none
> [193522.655297][ T6957] Rebooting in 86400 seconds..

Here are my questions:

1. Does the rtapp:sleep monitor proactively detect scenarios that
   could lead to sleeping in atomic context, perhaps before
   CONFIG_DEBUG_ATOMIC_SLEEP (enabled) would trigger at the actual point of
   sleeping?

2. Is there a way to enable this monitor (e.g., rtapp:sleep)
   immediately as soon as the RV subsystem is loaded during boot time?
   (How to make this "default turn on"?)

3. When a "violation detected" message occurs at runtime, is it
   possible to get a call stack of the location that triggered the
   violation? The panic reactor provides a full stack, but I'm
   wondering if this is also possible with the printk reactor.


Here is some background on why I'm so interested in this topic:

Recently, I was fuzzing the PREEMPT_RT kernel with syzkaller but ran into
issues where fuzzing wouldn't proceed smoothly. It turned out to be a problem
in the kcov USB API. This issue was fixed after I reported it, together
with Sebastian’s patch.

[PATCH] kcov, usb: Don't disable interrupts in kcov_remote_start_usb_softirq()
 - https://lore.kernel.org/all/20250811082745.ycJqBXMs@linutronix.de/

After this fix, syzkaller fuzzing ran well and was able to detect several
runtime "sleep in atomic context" bugs:

[PATCH] USB: gadget: dummy-hcd: Fix locking bug in RT-enabled kernels
 - https://lore.kernel.org/all/bb192ae2-4eee-48ee-981f-3efdbbd0d8f0@rowland.harvard.edu/

[BUG] usbip: vhci: Sleeping function called from invalid context in
vhci_urb_enqueue on PREEMPT_RT
 - https://lore.kernel.org/all/c6c17f0d-b71d-4a44-bcef-2b65e4d634f7@kzalloc.com/

This led me to research ways to find these issues proactively at a
static analysis level, and I created some regex and coccinelle scripts
to detect them.

[BUG] gfs2: sleeping lock in gfs2_quota_init() with preempt disabled
on PREEMPT_RT
 - https://lore.kernel.org/all/20250812103808.3mIVpgs9@linutronix.de/t/#u

[PATCH] md/raid5-ppl: Fix invalid context sleep in
ppl_io_unit_finished() on PREEMPT_RT
 - https://lore.kernel.org/all/f2dbf110-e2a7-4101-b24c-0444f708fd4e@kernel.org/t/#u

Tomas, the author of the rtlockscope project, also gave me some deep
insights into this static analysis approach.

Re: [WIP] coccinelle: rt: Add coccicheck on sleep in atomic context on
PREEMPT_RT
 - https://lore.kernel.org/all/CAP4=nvTOE9W+6UtVZ5-5gAoYeEQE8g4cgG602FJDPesNko-Bgw@mail.gmail.com/


Thank you!

Best regards,
Yunseong Kim

Re: [Question] Detecting Sleep-in-Atomic Context in PREEMPT_RT via RV (Runtime Verification) monitor rtapp:sleep

[Gabriele Monaco]

On Mon, 2025-10-27 at 15:54 +0900, Yunseong Kim wrote:
> Hi Nam,
> 
> I've been very interested in RV (Runtime Verification) to proactively detect
> "sleep in atomic" scenarios on PREEMPT_RT kernels. Specifically, I'm looking
> for ways to find cases where sleeping spinlocks or memory allocations are used
> within preemption-disabled or irq-disabled contexts. While searching for
> solutions, I discovered the RV subsystem.
> 

Hi Yunseong,

I'm sure Nam can be more specific on this, but let me add my 2 cents here.

The sleep monitor doesn't really do what you want, its violations are real time
tasks (typically userspace tasks with RR/FIFO policies) sleeping in a way that
might incur latencies. For instance using non PI locks or imprecise sleep.

What you need here is to validate kernel code, RV was actually designed for
that, but there's currently no monitor that does what you want.

The closest thing I can think of is monitors like scpd and snep in the sched
collection [1]. Those however won't catch what you need because they focus on
the preemption tracepoints and schedule, which works fine also in your scenario.

We could add similar monitors to catch what you want though:

                     |
                     |
                     v
                   +-----------------+
                   |   cant_sleep    | <+
                   +-----------------+  |
                     |                  |
                     | preempt_enable   | preempt_disable
                     v                  |
    kmalloc                             |
    lock_acquire                        |
  +---------------      can_sleep       |
  |                                     |
  +-------------->                     -+

which would become slightly more complicated if considering irq enable/disable
too. This is a deterministic automaton representation (see [1] for examples),
you could use an LTL like sleep as well, I assume (needs a per-CPU monitor which
is not merged yet for LTL).

This is simplified but you can of course put conditions on what kind of
allocations and locks you're interested in.

Now this specific case would require lockdep for the definition of lock_acquire
tracepoints. So I'm not sure how useful this monitor would be since lockdep is
going to complain too. You could use contention tracepoints to catch exactly
when sleep is going to occur and not /potential/ failures.

I only gave a quick thought on this, there may be better models/event fitting
your usecase, but I hope you get the idea.

[1] - https://docs.kernel.org/trace/rv/monitor_sched.html#monitor-scpd

> Here are my questions:
> 
> 1. Does the rtapp:sleep monitor proactively detect scenarios that
>    could lead to sleeping in atomic context, perhaps before
>    CONFIG_DEBUG_ATOMIC_SLEEP (enabled) would trigger at the actual point of
>    sleeping?

I guess I answered this already, but TL;DR no, you'd need a dedicated monitor.

> 2. Is there a way to enable this monitor (e.g., rtapp:sleep)
>    immediately as soon as the RV subsystem is loaded during boot time?
>    (How to make this "default turn on"?)

Currently not, but you could probably use any sort of startup script to turn it
on soon enough.

> 3. When a "violation detected" message occurs at runtime, is it
>    possible to get a call stack of the location that triggered the
>    violation? The panic reactor provides a full stack, but I'm
>    wondering if this is also possible with the printk reactor.

You can use ftrace and rely on error tracepoints instead of reactors. Each RV
violation triggers a tracepoint (e.g. error_sleep) and you can print a call
stack there. E.g.:

  echo stacktrace > /sys/kernel/tracing/events/rv/error_sleep/trigger

Here I use sleep as an example, but all monitors have their own error events
(e.g. error_wwnr, error_snep, etc.).

Does this all look useful in your scenario?

Gabriele

> 
> Here is some background on why I'm so interested in this topic:
> 
> Recently, I was fuzzing the PREEMPT_RT kernel with syzkaller but ran into
> issues where fuzzing wouldn't proceed smoothly. It turned out to be a problem
> in the kcov USB API. This issue was fixed after I reported it, together
> with Sebastian’s patch.
> 
> [PATCH] kcov, usb: Don't disable interrupts in kcov_remote_start_usb_softirq()
>  - https://lore.kernel.org/all/20250811082745.ycJqBXMs@linutronix.de/
> 
> After this fix, syzkaller fuzzing ran well and was able to detect several
> runtime "sleep in atomic context" bugs:
> 
> [PATCH] USB: gadget: dummy-hcd: Fix locking bug in RT-enabled kernels
>  -
> https://lore.kernel.org/all/bb192ae2-4eee-48ee-981f-3efdbbd0d8f0@rowland.harvard.edu/
> 
> [BUG] usbip: vhci: Sleeping function called from invalid context in
> vhci_urb_enqueue on PREEMPT_RT
>  -
> https://lore.kernel.org/all/c6c17f0d-b71d-4a44-bcef-2b65e4d634f7@kzalloc.com/
> 
> This led me to research ways to find these issues proactively at a
> static analysis level, and I created some regex and coccinelle scripts
> to detect them.
> 
> [BUG] gfs2: sleeping lock in gfs2_quota_init() with preempt disabled
> on PREEMPT_RT
>  - https://lore.kernel.org/all/20250812103808.3mIVpgs9@linutronix.de/t/#u
> 
> [PATCH] md/raid5-ppl: Fix invalid context sleep in
> ppl_io_unit_finished() on PREEMPT_RT
>  -
> https://lore.kernel.org/all/f2dbf110-e2a7-4101-b24c-0444f708fd4e@kernel.org/t/#u
> 
> Tomas, the author of the rtlockscope project, also gave me some deep
> insights into this static analysis approach.
> 
> Re: [WIP] coccinelle: rt: Add coccicheck on sleep in atomic context on
> PREEMPT_RT
>  -
> https://lore.kernel.org/all/CAP4=nvTOE9W+6UtVZ5-5gAoYeEQE8g4cgG602FJDPesNko-Bgw@mail.gmail.com/
> 
> 
> Thank you!
> 
> Best regards,
> Yunseong Kim

Re: [Question] Detecting Sleep-in-Atomic Context in PREEMPT_RT via RV (Runtime Verification) monitor rtapp:sleep

[Yunseong Kim]

Hi Gabriele,

On 10/27/25 9:20 PM, Gabriele Monaco wrote:
> On Mon, 2025-10-27 at 15:54 +0900, Yunseong Kim wrote:
>> Hi Nam,
>>
>> I've been very interested in RV (Runtime Verification) to proactively detect
>> "sleep in atomic" scenarios on PREEMPT_RT kernels. Specifically, I'm looking
>> for ways to find cases where sleeping spinlocks or memory allocations are used
>> within preemption-disabled or irq-disabled contexts. While searching for
>> solutions, I discovered the RV subsystem.
>>
> 
> Hi Yunseong,
> 
> I'm sure Nam can be more specific on this, but let me add my 2 cents here.

Thank you so much for your detailed response! It cleared up many of the
questions I had.

> The sleep monitor doesn't really do what you want, its violations are real time
> tasks (typically userspace tasks with RR/FIFO policies) sleeping in a way that
> might incur latencies. For instance using non PI locks or imprecise sleep.

So that’s the role of rtapp:sleep you mentioned. Thank you again for
clarifying it.

> What you need here is to validate kernel code, RV was actually designed for
> that, but there's currently no monitor that does what you want.

It’s a valuable chance to make a contribution to RV!

> The closest thing I can think of is monitors like scpd and snep in the sched
> collection [1]. Those however won't catch what you need because they focus on
> the preemption tracepoints and schedule, which works fine also in your scenario.
> 
> We could add similar monitors to catch what you want though:
> 
>                      |
>                      |
>                      v
>                    +-----------------+
>                    |   cant_sleep    | <+
>                    +-----------------+  |
>                      |                  |
>                      | preempt_enable   | preempt_disable
>                      v                  |
>     kmalloc                             |
>     lock_acquire                        |
>   +---------------      can_sleep       |
>   |                                     |
>   +-------------->                     -+
> 
> which would become slightly more complicated if considering irq enable/disable
> too. This is a deterministic automaton representation (see [1] for examples),
> you could use an LTL like sleep as well, I assume (needs a per-CPU monitor which
> is not merged yet for LTL).
> 
> This is simplified but you can of course put conditions on what kind of
> allocations and locks you're interested in.

If the goal is to detect this state before the output from __might_resched()
under CONFIG_DEBUG_ATOMIC_SLEEP (i.e., before an actual context switch occurs),
I am considering whether Deterministic Automata (.dot/DA) or Linear Temporal
Logic (.ltl/LTL) would be more appropriate for modeling this check. I'm also
thinking about whether I need to create a comprehensive table of all sleepable
functions for this purpose on the PREEMPT_RT kernel.

If this check is necessary, I’m planning to try the following verification:

RULE = always ((IN_ATOMIC or IRQS_DISABLED) imply not CALLS_RT_SLEEPER)

I’m also planning to add sleepable functions, including sleepable spinlocks
and memory allocations callable under PREEMPT_RT preempt/IRQ-disabled states,
to the RV monitor kernel module.

I’m considering adding the following functions as a result:

 // Mutex & Semaphore (or Lockdep's 'lock_acquire' for lock cases)
 "mutex_lock",
 "mutex_lock_interruptible",
 "mutex_lock_killable",
 "down_interruptible",
 "down_killable",
 "rwsem_down_read_failed",
 "rwsem_down_write_failed",
 "ww_mutex_lock",
 "rt_spin_lock",
 "rt_read_lock",
 "rt_write_lock",
 // or just "lock_acquire" for LOCKDEP enabled kernel.

 // sleep & schedule
 "msleep",
 "ssleep",
 "usleep_range",
 "wait_for_completion",
 "schedule",
 "cond_resched",

 // User-space memory access
 "copy_from_user",
 "copy_to_user",
 "__get_user_asm",
 "__put_user_asm",

 // memory allocation
 "__vmalloc",
 "__kmalloc"

> Now this specific case would require lockdep for the definition of lock_acquire
> tracepoints. So I'm not sure how useful this monitor would be since lockdep is
> going to complain too. You could use contention tracepoints to catch exactly
> when sleep is going to occur and not /potential/ failures.

I’ll look into this lockdep realated part further as well.

> I only gave a quick thought on this, there may be better models/event fitting
> your usecase, but I hope you get the idea.
> 
> [1] - https://docs.kernel.org/trace/rv/monitor_sched.html#monitor-scpd

Thank you for providing a diagram and references that make it easier to
understand!

>> Here are my questions:
>>
>> 1. Does the rtapp:sleep monitor proactively detect scenarios that
>>    could lead to sleeping in atomic context, perhaps before
>>    CONFIG_DEBUG_ATOMIC_SLEEP (enabled) would trigger at the actual point of
>>    sleeping?
> 
> I guess I answered this already, but TL;DR no, you'd need a dedicated monitor.
> 
>> 2. Is there a way to enable this monitor (e.g., rtapp:sleep)
>>    immediately as soon as the RV subsystem is loaded during boot time?
>>    (How to make this "default turn on"?)
> 
> Currently not, but you could probably use any sort of startup script to turn it
> on soon enough.
> 
>> 3. When a "violation detected" message occurs at runtime, is it
>>    possible to get a call stack of the location that triggered the
>>    violation? The panic reactor provides a full stack, but I'm
>>    wondering if this is also possible with the printk reactor.
> 
> You can use ftrace and rely on error tracepoints instead of reactors. Each RV
> violation triggers a tracepoint (e.g. error_sleep) and you can print a call
> stack there. E.g.:
> 
>   echo stacktrace > /sys/kernel/tracing/events/rv/error_sleep/trigger
> 
> Here I use sleep as an example, but all monitors have their own error events
> (e.g. error_wwnr, error_snep, etc.).
> 
> Does this all look useful in your scenario?

Thank you once again for your thorough explanation. Many of the questions
I initially had have now been resolved!

> Gabriele

Best regards,
Yunseong Kim

Re: [Question] Detecting Sleep-in-Atomic Context in PREEMPT_RT via RV (Runtime Verification) monitor rtapp:sleep

[Gabriele Monaco]

On Wed, 2025-10-29 at 07:53 +0900, Yunseong Kim wrote:
> > What you need here is to validate kernel code, RV was actually designed for
> > that, but there's currently no monitor that does what you want.
> 
> It’s a valuable chance to make a contribution to RV!

And could be quite a useful model!

> If the goal is to detect this state before the output from __might_resched()
> under CONFIG_DEBUG_ATOMIC_SLEEP (i.e., before an actual context switch
> occurs),
> I am considering whether Deterministic Automata (.dot/DA) or Linear Temporal
> Logic (.ltl/LTL) would be more appropriate for modeling this check. I'm also
> thinking about whether I need to create a comprehensive table of all sleepable
> functions for this purpose on the PREEMPT_RT kernel.
> 
> If this check is necessary, I’m planning to try the following verification:
> 
> RULE = always ((IN_ATOMIC or IRQS_DISABLED) imply not CALLS_RT_SLEEPER)

Yes, in this case DA or LTL is mostly down to preference, one thing to keep in
mind is that this is going to be a per-cpu monitor (i.e. the rule stands for
each CPU, as the irq/preemption state is per-cpu).

LTL support for per-cpu is added in [1] (not merged), so you will need to pull
that in if you want to play with LTL.

[1] -
https://lore.kernel.org/lkml/e7fb580ca898c707573fe1dcf6312f0c2d7682c5.1754900299.git.namcao@linutronix.de

> I’m also planning to add sleepable functions, including sleepable spinlocks
> and memory allocations callable under PREEMPT_RT preempt/IRQ-disabled states,
> to the RV monitor kernel module.
> 
> I’m considering adding the following functions as a result:
> 
>  // Mutex & Semaphore (or Lockdep's 'lock_acquire' for lock cases)
>  "mutex_lock",
>  "mutex_lock_interruptible",
>  "mutex_lock_killable",
>  "down_interruptible",
>  "down_killable",
>  "rwsem_down_read_failed",
>  "rwsem_down_write_failed",
>  "ww_mutex_lock",
>  "rt_spin_lock",
>  "rt_read_lock",
>  "rt_write_lock",
>  // or just "lock_acquire" for LOCKDEP enabled kernel.
> 
>  // sleep & schedule
>  "msleep",
>  "ssleep",
>  "usleep_range",
>  "wait_for_completion",
>  "schedule",
>  "cond_resched",
> 
>  // User-space memory access
>  "copy_from_user",
>  "copy_to_user",
>  "__get_user_asm",
>  "__put_user_asm",
> 
>  // memory allocation
>  "__vmalloc",
>  "__kmalloc"
> 

Here you're talking about direct kernel functions, currently RV relies on
tracepoints (that's why I mentioned those earlier). You have two routes:

1. use existing tracepoints and/or add new ones in strategical points
2. use kprobes and attach wherever you want

1. is very easy in RV and you may use tracepoints arguments to narrow down the
search (e.g. just transition state on certain locks, certain allocations), you
may need to discuss with various maintainers to add new ones, but that's usually
alright, have a look at the V2 of the linked thread for an example [2].

2. is a bit more involved, you'd be able to access precisely the functions you
want (usually), but I'm not sure about the overhead of plugging 15 kprobes.
Also RV doesn't support kprobes, although extending it is rather trivial.

You can mix both, of course. But yes, you'd need to identify all the "events"
you care about. I'd start simple with some of those (e.g. malloc and lock
contention tracepoints) and see if it satisfies your needs.

You may also be counting things twice (isn't malloc calling locks, which may end
up calling schedule?), just an idea, but you may find common paths in the above
list.

Gabriele

[2] -
https://lore.kernel.org/lkml/f87ce0cb979daa3e8221c496de16883ca53f3950.1754466623.git.namcao@linutronix.de

> > Now this specific case would require lockdep for the definition of
> > lock_acquire
> > tracepoints. So I'm not sure how useful this monitor would be since lockdep
> > is
> > going to complain too. You could use contention tracepoints to catch exactly
> > when sleep is going to occur and not /potential/ failures.
> 
> I’ll look into this lockdep realated part further as well.
> 
> > I only gave a quick thought on this, there may be better models/event
> > fitting
> > your usecase, but I hope you get the idea.
> > 
> > [1] - https://docs.kernel.org/trace/rv/monitor_sched.html#monitor-scpd
> 
> Thank you for providing a diagram and references that make it easier to
> understand!
> 
> > > Here are my questions:
> > > 
> > > 1. Does the rtapp:sleep monitor proactively detect scenarios that
> > >    could lead to sleeping in atomic context, perhaps before
> > >    CONFIG_DEBUG_ATOMIC_SLEEP (enabled) would trigger at the actual point
> > > of
> > >    sleeping?
> > 
> > I guess I answered this already, but TL;DR no, you'd need a dedicated
> > monitor.
> > 
> > > 2. Is there a way to enable this monitor (e.g., rtapp:sleep)
> > >    immediately as soon as the RV subsystem is loaded during boot time?
> > >    (How to make this "default turn on"?)
> > 
> > Currently not, but you could probably use any sort of startup script to turn
> > it
> > on soon enough.
> > 
> > > 3. When a "violation detected" message occurs at runtime, is it
> > >    possible to get a call stack of the location that triggered the
> > >    violation? The panic reactor provides a full stack, but I'm
> > >    wondering if this is also possible with the printk reactor.
> > 
> > You can use ftrace and rely on error tracepoints instead of reactors. Each
> > RV
> > violation triggers a tracepoint (e.g. error_sleep) and you can print a call
> > stack there. E.g.:
> > 
> >   echo stacktrace > /sys/kernel/tracing/events/rv/error_sleep/trigger
> > 
> > Here I use sleep as an example, but all monitors have their own error events
> > (e.g. error_wwnr, error_snep, etc.).
> > 
> > Does this all look useful in your scenario?
> 
> Thank you once again for your thorough explanation. Many of the questions
> I initially had have now been resolved!
> 
> > Gabriele
> 
> Best regards,
> Yunseong Kim

Re: [Question] Detecting Sleep-in-Atomic Context in PREEMPT_RT via RV (Runtime Verification) monitor rtapp:sleep

[Nam Cao]

Yunseong Kim <ysk@kzalloc.com> writes:
> Hi Nam,
>
> I've been very interested in RV (Runtime Verification)

Cool, happy to learn you find it interesting.

> to proactively detect
> "sleep in atomic" scenarios on PREEMPT_RT kernels. Specifically, I'm looking
> for ways to find cases where sleeping spinlocks or memory allocations are used
> within preemption-disabled or irq-disabled contexts. While searching for
> solutions, I discovered the RV subsystem.
...
> Here are my questions:
>
> 1. Does the rtapp:sleep monitor proactively detect scenarios that
>    could lead to sleeping in atomic context, perhaps before
>    CONFIG_DEBUG_ATOMIC_SLEEP (enabled) would trigger at the actual point of
>    sleeping?

No it does not, as explained by Gabriele.

I am a bit confused, because CONFIG_DEBUG_ATOMIC_SLEEP seems to already
do what you need. CONFIG_DEBUG_ATOMIC_SLEEP does warn before the actual
point of sleeping. Sleeping locks and memory allocations are marked with
might_sleep(). When they are called in atomic context, we have a warning
regardless of whether actual sleep happens. See the comment above
might_sleep():

"This is a useful debugging help to be able to catch problems early and
not be bitten later when the calling function happens to sleep when it
is not supposed to"

For sure you can implement this functionality in RV, but I don't think
RV can do more. An advantage of doing it in RV is the ability to toggle
at run-time, but that's a different discussion.

> 2. Is there a way to enable this monitor (e.g., rtapp:sleep)
>    immediately as soon as the RV subsystem is loaded during boot time?
>    (How to make this "default turn on"?)

At the moment, no. But if you need this, we could look into implementing it.

> 3. When a "violation detected" message occurs at runtime, is it
>    possible to get a call stack of the location that triggered the
>    violation? The panic reactor provides a full stack, but I'm
>    wondering if this is also possible with the printk reactor.

You can use monitor's tracepoint to get the stacktrace, as mentioned by Gabriele.

> This led me to research ways to find these issues proactively at a
> static analysis level, and I created some regex and coccinelle scripts
> to detect them.
...
> Tomas, the author of the rtlockscope project, also gave me some deep
> insights into this static analysis approach.

RV is not a static checker, it is a run-time checker.

Just in case you are not aware yet, there is also Smatch:
https://github.com/error27/smatch. But I can't offer much help there.

Nam

Re: [Question] Detecting Sleep-in-Atomic Context in PREEMPT_RT via RV (Runtime Verification) monitor rtapp:sleep

[Nam Cao]

> RV is not a static checker, it is a run-time checker.
>
> Just in case you are not aware yet, there is also Smatch:
> https://github.com/error27/smatch. But I can't offer much help there.

I was looking up something unrelated, and I found that Smatch does
detect this sleep-in-atomic problem:
https://staticthinking.wordpress.com/2024/05/24/sleeping-in-atomic-warnings/

I'm not sure if its design takes PREEMPT_RT into consideration. But
seems worth having a look.

Nam

Re: [Question] Detecting Sleep-in-Atomic Context in PREEMPT_RT via RV (Runtime Verification) monitor rtapp:sleep

[Sebastian Andrzej Siewior]

On 2025-12-02 12:14:09 [+0100], Nam Cao wrote:
> > RV is not a static checker, it is a run-time checker.
> >
> > Just in case you are not aware yet, there is also Smatch:
> > https://github.com/error27/smatch. But I can't offer much help there.
> 
> I was looking up something unrelated, and I found that Smatch does
> detect this sleep-in-atomic problem:
> https://staticthinking.wordpress.com/2024/05/24/sleeping-in-atomic-warnings/
> 
> I'm not sure if its design takes PREEMPT_RT into consideration. But
> seems worth having a look.

It does not look like it does. Judging from
	https://github.com/error27/smatch/blob/master/check_preempt_info.c

it increments the preempt-counter on preempt_disable() and spin_lock().
So it won't yell at
	preempt_disable();
	spin_lock();

while CONFIG_DEBUG_ATOMIC_SLEEP would.

> Nam

Sebastian

Re: [Question] Detecting Sleep-in-Atomic Context in PREEMPT_RT via RV (Runtime Verification) monitor rtapp:sleep

[Yunseong Kim]

Hi Nam and Gabriele,

Thanks for the great presentation at LPC 2025.

I have a follow-up question during the Gabriele's Session RV and 
real-time properties, regarding spinlock behavior in atomic
contexts (IRQ/preempt disabled) on the PREEMPT_RT kernel.

On 12/2/25 20:26, Sebastian Andrzej Siewior wrote:
> On 2025-12-02 12:14:09 [+0100], Nam Cao wrote:
>>> RV is not a static checker, it is a run-time checker.
>>>
>>> Just in case you are not aware yet, there is also Smatch:
>>> https://github.com/error27/smatch. But I can't offer much help there.
>>
>> I was looking up something unrelated, and I found that Smatch does
>> detect this sleep-in-atomic problem:
>> https://staticthinking.wordpress.com/2024/05/24/sleeping-in-atomic-warnings/
>>
>> I'm not sure if its design takes PREEMPT_RT into consideration. But
>> seems worth having a look.
> 
> It does not look like it does. Judging from
> 	https://github.com/error27/smatch/blob/master/check_preempt_info.c
> 
> it increments the preempt-counter on preempt_disable() and spin_lock().
> So it won't yell at
> 	preempt_disable();
> 	spin_lock();
> 
> while CONFIG_DEBUG_ATOMIC_SLEEP would.

I specifically believe that RV can encompass the role of
CONFIG_DEBUG_ATOMIC_SLEEP and even go beyond it.

My reasoning is that even if a sleepable (PREEMPT_RT) spinlock is used 
within an IRQ/preemption disabled section, CONFIG_DEBUG_ATOMIC_SLEEP 
might not trigger a warning if scheduling does not actually occur (i.e., 
if there is no contention for that spinlock). This is because the actual 
debugging check happens in __might_resched().

Therefore, I think RV could catch these potential bugs that might
otherwise be missed. Is my understanding in the right direction?

>> Nam
> 
> Sebastian

I really appreciate Nam and Sebastian for sharing valuable insights on
static analysis tools again.

I recently discussed this with the Kernel CVE team, and Greg confirmed
that such usage (sleepable spinlocks in IRQ/preempt disabled sections)
is handled as a CVE.

I would appreciate your insights on this matter.

Best regards,
Yunseong Kim

Re: [Question] Detecting Sleep-in-Atomic Context in PREEMPT_RT via RV (Runtime Verification) monitor rtapp:sleep

[Nam Cao]

Yunseong Kim <ysk@kzalloc.com> writes:
> I specifically believe that RV can encompass the role of
> CONFIG_DEBUG_ATOMIC_SLEEP and even go beyond it.
>
> My reasoning is that even if a sleepable (PREEMPT_RT) spinlock is used 
> within an IRQ/preemption disabled section, CONFIG_DEBUG_ATOMIC_SLEEP 
> might not trigger a warning if scheduling does not actually occur (i.e., 
> if there is no contention for that spinlock). This is because the actual 
> debugging check happens in __might_resched().

That's not how it works. See the description of CONFIG_DEBUG_ATOMIC_SLEEP:

"If you say Y here, various routines which may sleep will become very
noisy if they are called inside atomic sections: when a spinlock is
held, inside an rcu read side critical section, inside preempt disabled
sections, inside an interrupt, etc..."

Nam

Re: [Question] Detecting Sleep-in-Atomic Context in PREEMPT_RT via RV (Runtime Verification) monitor rtapp:sleep

[Yunseong Kim]

Thanks you Nam, for pointing that out!

On 12/11/25 14:42, Nam Cao wrote:
> Yunseong Kim <ysk@kzalloc.com> writes:
>> I specifically believe that RV can encompass the role of
>> CONFIG_DEBUG_ATOMIC_SLEEP and even go beyond it.
>>
>> My reasoning is that even if a sleepable (PREEMPT_RT) spinlock is used
>> within an IRQ/preemption disabled section, CONFIG_DEBUG_ATOMIC_SLEEP
>> might not trigger a warning if scheduling does not actually occur (i.e.,
>> if there is no contention for that spinlock). This is because the actual
>> debugging check happens in __might_resched().
> 
> That's not how it works. See the description of CONFIG_DEBUG_ATOMIC_SLEEP:
> 
> "If you say Y here, various routines which may sleep will become very
> noisy if they are called inside atomic sections: when a spinlock is
> held, inside an rcu read side critical section, inside preempt disabled
> sections, inside an interrupt, etc..."

I am currently considering how to model this to cover cases that go 
beyond what CONFIG_DEBUG_ATOMIC_SLEEP covers.

My specific concern is about custom functions where might_sleep() might 
be missing. In such cases, if the code hits a fast path (no scheduling),
CONFIG_DEBUG_ATOMIC_SLEEP won't trigger. I'm wondering if RV can detect
these potential bugs.

> Nam


Thanks a lot for the quick reply!

Best regards,
Yunseong Kim

Re: [Question] Detecting Sleep-in-Atomic Context in PREEMPT_RT via RV (Runtime Verification) monitor rtapp:sleep

[Gabriele Monaco]

On Thu, 2025-12-11 at 16:58 +0900, Yunseong Kim wrote:
> I am currently considering how to model this to cover cases that go 
> beyond what CONFIG_DEBUG_ATOMIC_SLEEP covers.
> 
> My specific concern is about custom functions where might_sleep() might 
> be missing. In such cases, if the code hits a fast path (no scheduling),
> CONFIG_DEBUG_ATOMIC_SLEEP won't trigger. I'm wondering if RV can detect
> these potential bugs.

Hi Yunseong,

I finally got to reflect a bit more on this.
As we discussed at LPC, RV wouldn't help with not annotated functions because
you'd need to add tracepoints there, so probably the best is to just add a
might_sleep and be done with it.

Other than that, I double checked and CONFIG_DEBUG_ATOMIC_SLEEP is in fact
disabled on non-debug kernels in Fedora/RHEL (and I presume also in Debian and
friends), so the other concern you raised is indeed valid.

Although perhaps rather unlikely, I'm not sure we can completely rule out cases
in which non-debug kernel (where several other debugging features besides
CONFIG_DEBUG_ATOMIC_SLEEP are disabled) behave differently enough for some bugs,
otherwise invisible on debug kernels, to pop up.

Another use could be debugging kernel modules on non-debug kernels, perhaps to
avoid some other debug features to be present while still relying on the
packaged kernel.

For this to work you'd need to add a tracepoint on might_sleep when
CONFIG_DEBUG_ATOMIC_SLEEP is disabled (and it's currently a NOP), hopefully that
wouldn't make people too angry.

Again, none of this would be ground-breaking but it doesn't look useless either.

Any thoughts?

Thanks,
Gabriele

Re: [Question] Detecting Sleep-in-Atomic Context in PREEMPT_RT via RV (Runtime Verification) monitor rtapp:sleep

[Yunseong Kim]

Hi Gabriele,

Thanks for the follow-up and for taking the time to double-check the
configuration details.

On 12/22/25 4:40 PM, Gabriele Monaco wrote:
> On Thu, 2025-12-11 at 16:58 +0900, Yunseong Kim wrote:
>> I am currently considering how to model this to cover cases that go 
>> beyond what CONFIG_DEBUG_ATOMIC_SLEEP covers.
>>
>> My specific concern is about custom functions where might_sleep() might 
>> be missing. In such cases, if the code hits a fast path (no scheduling),
>> CONFIG_DEBUG_ATOMIC_SLEEP won't trigger. I'm wondering if RV can detect
>> these potential bugs.
> 
> Hi Yunseong,
> 
> I finally got to reflect a bit more on this.
> As we discussed at LPC, RV wouldn't help with not annotated functions because
> you'd need to add tracepoints there, so probably the best is to just add a
> might_sleep and be done with it.

Thanks for your insight! I agree that for unannotated functions, adding
might_sleep() directly is the most straightforward approach.

I will also double-check sleepable memory allocations in the context of
PREEMPT_RT kernels to see if there are other patterns being missed.

> Other than that, I double checked and CONFIG_DEBUG_ATOMIC_SLEEP is in fact
> disabled on non-debug kernels in Fedora/RHEL (and I presume also in Debian and
> friends), so the other concern you raised is indeed valid.

I agree that CONFIG_DEBUG_ATOMIC_SLEEP covers most cases, but as you
pointed out, the fact that it's disabled in production kernels (Fedora,
RHEL, Debian, Ubuntu etc.) gives this approach real-world value. I was
hesitant about whether this was worth a contribution, so I truly appreciate
your encouraging feedback.

> Although perhaps rather unlikely, I'm not sure we can completely rule out cases
> in which non-debug kernel (where several other debugging features besides
> CONFIG_DEBUG_ATOMIC_SLEEP are disabled) behave differently enough for some bugs,
> otherwise invisible on debug kernels, to pop up.
> 
> Another use could be debugging kernel modules on non-debug kernels, perhaps to
> avoid some other debug features to be present while still relying on the
> packaged kernel.
> 
> For this to work you'd need to add a tracepoint on might_sleep when
> CONFIG_DEBUG_ATOMIC_SLEEP is disabled (and it's currently a NOP), hopefully that
> wouldn't make people too angry.

My interest in this arose because I've been seeing these legacy patterns
consistently reported as bugs in the Real-Time (PREEMPT_RT) kernel. It
seems many kernel developers are still unaware that these patterns are
problematic, partly because some older Linux learning materials actually
recommended them. This makes it quite likely that such code will continue
to find its way into the upstream.

> Again, none of this would be ground-breaking but it doesn't look useless either.
> 
> Any thoughts?

I will keep looking into more meaningful contributions for RV. For instance,
I’ve been considering a monitor to detect excessive memory compaction issues
under specific workloads, which I’ve encountered during kernel development.

> Thanks,
> Gabriele

It was a great pleasure discussing these ideas with you and Nam. I sorry
for any lack of clarity in my previous explanations during LPC 25.

Thanks again for your insights! :)

Best regards,
Yunseong

Re: [Question] Detecting Sleep-in-Atomic Context in PREEMPT_RT via RV (Runtime Verification) monitor rtapp:sleep

[Gabriele Monaco]

Hi Yunseong,

> I will also double-check sleepable memory allocations in the context of
> PREEMPT_RT kernels to see if there are other patterns being missed.

You may want to play with lockdep as well, it's much heavier but can catch a lot
of those issues. I believe it still relies on annotations though, so it might
have blind spots too (though it's battle-tested on locks).

> My interest in this arose because I've been seeing these legacy patterns
> consistently reported as bugs in the Real-Time (PREEMPT_RT) kernel. It
> seems many kernel developers are still unaware that these patterns are
> problematic, partly because some older Linux learning materials actually
> recommended them. This makes it quite likely that such code will continue
> to find its way into the upstream.

Yeah that's quite common indeed, I see kernel functions claiming to be atomic
while in fact using (sleeping) spinlocks.

Reading the material Daniel left (RV's original author), he thought of a similar
model, so at least we know there was interest in this kind of thing.


> I will keep looking into more meaningful contributions for RV. For instance,
> I’ve been considering a monitor to detect excessive memory compaction issues
> under specific workloads, which I’ve encountered during kernel development.

Looking forward to knowing more!

Cheers,
Gabriele

Re: [Question] Detecting Sleep-in-Atomic Context in PREEMPT_RT via RV (Runtime Verification) monitor rtapp:sleep

[Dan Carpenter]

On Tue, Dec 02, 2025 at 12:26:44PM +0100, Sebastian Andrzej Siewior wrote:
> On 2025-12-02 12:14:09 [+0100], Nam Cao wrote:
> > > RV is not a static checker, it is a run-time checker.
> > >
> > > Just in case you are not aware yet, there is also Smatch:
> > > https://github.com/error27/smatch. But I can't offer much help there.
> > 
> > I was looking up something unrelated, and I found that Smatch does
> > detect this sleep-in-atomic problem:
> > https://staticthinking.wordpress.com/2024/05/24/sleeping-in-atomic-warnings/
> > 
> > I'm not sure if its design takes PREEMPT_RT into consideration. But
> > seems worth having a look.
> 
> It does not look like it does. Judging from
> 	https://github.com/error27/smatch/blob/master/check_preempt_info.c
> 
> it increments the preempt-counter on preempt_disable() and spin_lock().
> So it won't yell at
> 	preempt_disable();
> 	spin_lock();
> 
> while CONFIG_DEBUG_ATOMIC_SLEEP would.

Yeah.  Sorry, I haven't updated this check for PREEMPT_RT.

There is a bug to do with cleanup.h that affects it as well.  If you
have a goto from inside a scoped guard that isn't handled correctly.
I need to fix that as well.

regards,
dan carpenter