Re: [ANNOUNCE] v6.3-rc2-rt3

Re: [ANNOUNCE] v6.3-rc2-rt3

[Sebastian Andrzej Siewior]

On 2023-03-14 18:05:04 [+0100], To Thomas Gleixner wrote:
> Dear RT folks!
> 
> I'm pleased to announce the v6.3-rc2-rt3 patch set. 
> 
> Changes since v6.3-rc2-rt2:
> 
>   - The i915 and other driver using io_mapping_map_atomic_wc() function
>     could deadlock. Reported by Richard Weinberger.
> 
>   - A larger printk rework by John Ogness. This printk series is based
>     on the latest work by John and most of the patches have been
>     submitted for upstream review.
>     As in the previous version the concept of an atomic console
>     remained. That means a crash (BUG(), panic(), …) without a atomic
>     console driver remains invisible on PREEMPT_RT. The only available
>     atomic console driver is the 8250 UART and has been tested on X86.
> 
> Known issues
>      - Crystal Wood reported that a filesystem (XFS) may deadlock while
>        flushing during schedule.
> 
> The delta patch against v6.3-rc2-rt2 is appended below and can be found here:
>  
>      https://cdn.kernel.org/pub/linux/kernel/projects/rt/6.3/incr/patch-6.3-rc2-rt2-rt3.patch.xz
> 
> You can get this release via the git tree at:
> 
>     git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git v6.3-rc2-rt3
> 
> The RT patch against v6.3-rc2 can be found here:
> 
>     https://cdn.kernel.org/pub/linux/kernel/projects/rt/6.3/older/patch-6.3-rc2-rt3.patch.xz
> 
> The split quilt queue is available at:
> 
>     https://cdn.kernel.org/pub/linux/kernel/projects/rt/6.3/older/patches-6.3-rc2-rt3.tar.xz

This email didn't pass the size restriction on linux-rt-users@ so this
is just a small ping that this release happened. The whole email is in
the archive:
	https://lore.kernel.org/20230314170502.OHw1_FK3@linutronix.de
 
Sebastian

Re: [ANNOUNCE] v6.3-rc2-rt3

[Pierre Gondois]

Hello Sebastian,
The following appears between v6.3-rc2-rt2 and v6.3-rc2-rt3,
it is probably due to b111e33ae076 ("printk: Update John's printk series.")

=============================
[ BUG: Invalid wait context ]
6.3.0-rc5-rt8-gacb52bd349a2 #417 Not tainted
-----------------------------
swapper/0/1 is trying to lock:
ffff00097eea1180 (cpu){....}-{3:3}, at: __printk_safe_enter (kernel/printk/printk_safe.c:28 (discriminator 3))
other info that might help us debug this:
context-{5:5}
1 lock held by swapper/0/1:
#0: ffff80000ba77028 (rcu_tasks_rude.cbs_gbl_lock){....}-{2:2}, at: cblist_init_generic (kernel/rcu/tasks.h:233)
stack backtrace:
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.3.0-rc5-rt8-gacb52bd349a2 #417
Hardware name: ARM Juno development board (r2) (DT)
Call trace:
dump_backtrace (arch/arm64/kernel/stacktrace.c:160)
show_stack (arch/arm64/kernel/stacktrace.c:167)
dump_stack_lvl (lib/dump_stack.c:107)
dump_stack (lib/dump_stack.c:114)
__lock_acquire (kernel/locking/lockdep.c:4710)
lock_acquire (./arch/arm64/include/asm/percpu.h:40)
__printk_safe_enter (./include/linux/local_lock_internal.h:30 (discriminator 3))
__down_trylock_console_sem (kernel/printk/printk.c:327)
console_trylock (kernel/printk/printk.c:2710 (discriminator 4))
vprintk_emit (kernel/printk/printk.c:1960)
vprintk_default (kernel/printk/printk.c:2367)
vprintk (kernel/printk/printk_safe.c:69)
_printk (kernel/printk/printk.c:2380)
cblist_init_generic (kernel/rcu/tasks.h:236)
rcu_init_tasks_generic (kernel/rcu/tasks.h:1142)
kernel_init_freeable (./arch/arm64/include/asm/jump_label.h:21)
kernel_init (init/main.c:1530)
ret_from_fork (arch/arm64/kernel/entry.S:871)

Regards,
Pierre

On 3/14/23 18:12, Sebastian Andrzej Siewior wrote:
> On 2023-03-14 18:05:04 [+0100], To Thomas Gleixner wrote:
>> Dear RT folks!
>>
>> I'm pleased to announce the v6.3-rc2-rt3 patch set.
>>
>> Changes since v6.3-rc2-rt2:
>>
>>    - The i915 and other driver using io_mapping_map_atomic_wc() function
>>      could deadlock. Reported by Richard Weinberger.
>>
>>    - A larger printk rework by John Ogness. This printk series is based
>>      on the latest work by John and most of the patches have been
>>      submitted for upstream review.
>>      As in the previous version the concept of an atomic console
>>      remained. That means a crash (BUG(), panic(), …) without a atomic
>>      console driver remains invisible on PREEMPT_RT. The only available
>>      atomic console driver is the 8250 UART and has been tested on X86.
>>
>> Known issues
>>       - Crystal Wood reported that a filesystem (XFS) may deadlock while
>>         flushing during schedule.
>>
>> The delta patch against v6.3-rc2-rt2 is appended below and can be found here:
>>   
>>       https://cdn.kernel.org/pub/linux/kernel/projects/rt/6.3/incr/patch-6.3-rc2-rt2-rt3.patch.xz
>>
>> You can get this release via the git tree at:
>>
>>      git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git v6.3-rc2-rt3
>>
>> The RT patch against v6.3-rc2 can be found here:
>>
>>      https://cdn.kernel.org/pub/linux/kernel/projects/rt/6.3/older/patch-6.3-rc2-rt3.patch.xz
>>
>> The split quilt queue is available at:
>>
>>      https://cdn.kernel.org/pub/linux/kernel/projects/rt/6.3/older/patches-6.3-rc2-rt3.tar.xz
> 
> This email didn't pass the size restriction on linux-rt-users@ so this
> is just a small ping that this release happened. The whole email is in
> the archive:
> 	https://lore.kernel.org/20230314170502.OHw1_FK3@linutronix.de
>   
> Sebastian
> 

Re: [ANNOUNCE] v6.3-rc2-rt3

[John Ogness]

Hi Pierre,

On 2023-04-14, Pierre Gondois <pierre.gondois@arm.com> wrote:
> =============================
> [ BUG: Invalid wait context ]
> 6.3.0-rc5-rt8-gacb52bd349a2 #417 Not tainted
> -----------------------------
> swapper/0/1 is trying to lock:
> ffff00097eea1180 (cpu){....}-{3:3}, at: __printk_safe_enter (kernel/printk/printk_safe.c:28 (discriminator 3))
> other info that might help us debug this:
> context-{5:5}
> 1 lock held by swapper/0/1:
> #0: ffff80000ba77028 (rcu_tasks_rude.cbs_gbl_lock){....}-{2:2}, at: cblist_init_generic (kernel/rcu/tasks.h:233)

This is actually a mainline issue relating to legacy consoles. In
mainline you see it as a warning when CONFIG_PROVE_RAW_LOCK_NESTING is
enabled.

If you enable CONFIG_PREEMPT_RT you will not have the problem.

I will look at how the issue can be gracefully ignored in
!CONFIG_PREEMPT_RT.

Thanks for reporting.

John Ogness

Re: [ANNOUNCE] v6.3-rc2-rt3

[Vlastimil Babka]

On 4/14/23 15:49, John Ogness wrote:
> Hi Pierre,
> 
> On 2023-04-14, Pierre Gondois <pierre.gondois@arm.com> wrote:
>> =============================
>> [ BUG: Invalid wait context ]
>> 6.3.0-rc5-rt8-gacb52bd349a2 #417 Not tainted
>> -----------------------------
>> swapper/0/1 is trying to lock:
>> ffff00097eea1180 (cpu){....}-{3:3}, at: __printk_safe_enter (kernel/printk/printk_safe.c:28 (discriminator 3))
>> other info that might help us debug this:
>> context-{5:5}
>> 1 lock held by swapper/0/1:
>> #0: ffff80000ba77028 (rcu_tasks_rude.cbs_gbl_lock){....}-{2:2}, at: cblist_init_generic (kernel/rcu/tasks.h:233)
> 
> This is actually a mainline issue relating to legacy consoles. In
> mainline you see it as a warning when CONFIG_PROVE_RAW_LOCK_NESTING is
> enabled.
> 
> If you enable CONFIG_PREEMPT_RT you will not have the problem.
> 
> I will look at how the issue can be gracefully ignored in
> !CONFIG_PREEMPT_RT.

Is it perhaps a similar situation to this thread?

https://lore.kernel.org/all/20230412124735.GE628377@hirez.programming.kicks-ass.net/

> Thanks for reporting.
> 
> John Ogness

Re: [ANNOUNCE] v6.3-rc2-rt3

[John Ogness]

On 2023-04-14, Vlastimil Babka <vbabka@suse.cz> wrote:
> Is it perhaps a similar situation to this thread?
>
> https://lore.kernel.org/all/20230412124735.GE628377@hirez.programming.kicks-ass.net/

We are dealing with a spin_lock() inside a raw_spin_lock() section. The
legacy console drivers do this. The fix is the new atomic/threaded
consoles. For CONFIG_PREEMPT_RT a workaround is implemented so that the
legacy consoles avoid this. For !CONFIG_PREEMPT_RT you can expect
CONFIG_PROVE_RAW_LOCK_NESTING to cause a splat on that printk().

Or perhaps that particular printk() in cblist_init_generic() should be
changed to printk_deferred() as a temporary whack-a-mole
workaround.

John Ogness

Re: [ANNOUNCE] v6.3-rc2-rt3

[Vlastimil Babka]

On 4/14/23 17:01, John Ogness wrote:
> On 2023-04-14, Vlastimil Babka <vbabka@suse.cz> wrote:
>> Is it perhaps a similar situation to this thread?
>>
>> https://lore.kernel.org/all/20230412124735.GE628377@hirez.programming.kicks-ass.net/
> 
> We are dealing with a spin_lock() inside a raw_spin_lock() section. The
> legacy console drivers do this. The fix is the new atomic/threaded
> consoles. For CONFIG_PREEMPT_RT a workaround is implemented so that the
> legacy consoles avoid this. For !CONFIG_PREEMPT_RT you can expect
> CONFIG_PROVE_RAW_LOCK_NESTING to cause a splat on that printk().
> 
> Or perhaps that particular printk() in cblist_init_generic() should be
> changed to printk_deferred() as a temporary whack-a-mole
> workaround.

What I meant that in the linked thread a solution seems to be forming in the
form of annotation for lockdep/CONFIG_PROVE_RAW_LOCK_NESTING to make it
aware that on CONFIG_PREEMPT_RT the problem it sees is side-stepped so it
shouldn't warn about it on !PREEMPT_RT, and maybe that solution could be
used for the printk issue as well (I admit I didn't check the code, just by
reading your mail it sounded very similar).

> John Ogness

Re: [ANNOUNCE] v6.3-rc2-rt3

[Sebastian Andrzej Siewior]

On 2023-04-14 17:38:56 [+0200], Vlastimil Babka wrote:
> What I meant that in the linked thread a solution seems to be forming in the
> form of annotation for lockdep/CONFIG_PROVE_RAW_LOCK_NESTING to make it
> aware that on CONFIG_PREEMPT_RT the problem it sees is side-stepped so it
> shouldn't warn about it on !PREEMPT_RT, and maybe that solution could be
> used for the printk issue as well (I admit I didn't check the code, just by
> reading your mail it sounded very similar).

Yes, you right and it would work and I would include it. Once the
DEBUG_OBJECT part is merged (the LD_LOCK_WAIT logic is missing) I would
add something here.

Sebastian