* Re: Backport submission - rcu: Avoid tracing a few functions executed in stop machine
2023-11-24 16:10 ` Greg KH
@ 2023-11-29 14:08 ` Ronald Monthero
2023-11-30 12:07 ` Ronald Monthero
2023-12-02 6:56 ` [PATCH] " Ronald Monthero
2023-12-02 8:11 ` Ronald Monthero
2 siblings, 1 reply; 9+ messages in thread
From: Ronald Monthero @ 2023-11-29 14:08 UTC (permalink / raw)
To: Greg KH; +Cc: stable
On Sat, Nov 25, 2023 at 2:10 AM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> On Tue, Nov 21, 2023 at 12:09:38AM +1000, Ronald Monthero wrote:
> > Dear stable maintainers,
> > I like to indicate the oops encountered and request the below patch to
> > be backported to v 5.15. The fix is important to avoid recurring oops
> > in context of rcu detected stalls.
> >
> > subject: rcu: Avoid tracing a few functions executed in stop machine
> > commit 48f8070f5dd8
> > Target kernel version v 5.15
> > Reason for Application: To avoid oops due to rcu_prempt detect stalls
> > on cpus/tasks
> >
> > Environment and oops context: Issue was observed in my environment on
> > 5.15.193 kernel (arm platform). The patch is helpful to avoid the
> > below oops indicated in [1] and [2]
>
> As the patch does not apply cleanly, we need a working and tested
> backport so we know to apply the correct version.
>
> Can you please provide that as you've obviously already done this?
Hi Greg,
Sorry I notice my typo error 193 instead of 93. I have tested on the
5.15.93-rt58 kernel.
BR,
Ronald
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Backport submission - rcu: Avoid tracing a few functions executed in stop machine
2023-11-29 14:08 ` Ronald Monthero
@ 2023-11-30 12:07 ` Ronald Monthero
2023-11-30 12:22 ` Greg KH
0 siblings, 1 reply; 9+ messages in thread
From: Ronald Monthero @ 2023-11-30 12:07 UTC (permalink / raw)
To: Greg KH; +Cc: stable
On Thu, Nov 30, 2023 at 12:08 AM Ronald Monthero
<debug.penguin32@gmail.com> wrote:
>
> On Sat, Nov 25, 2023 at 2:10 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> >
> > On Tue, Nov 21, 2023 at 12:09:38AM +1000, Ronald Monthero wrote:
> > > Dear stable maintainers,
> > > I like to indicate the oops encountered and request the below patch to
> > > be backported to v 5.15. The fix is important to avoid recurring oops
> > > in context of rcu detected stalls.
> > >
> > > subject: rcu: Avoid tracing a few functions executed in stop machine
> > > commit 48f8070f5dd8
> > > Target kernel version v 5.15
> > > Reason for Application: To avoid oops due to rcu_prempt detect stalls
> > > on cpus/tasks
> > >
> > > Environment and oops context: Issue was observed in my environment on
> > > 5.15.193 kernel (arm platform). The patch is helpful to avoid the
> > > below oops indicated in [1] and [2]
> >
> > As the patch does not apply cleanly, we need a working and tested
> > backport so we know to apply the correct version.
> >
> > Can you please provide that as you've obviously already done this?
>
> Hi Greg,
> Sorry I notice my typo error 193 instead of 93. I have tested on the
> 5.15.93-rt58 kernel.
Hi Greg,
I used a 5.15.93 kernel
- on arm32 bit platform I tested with 5.15.93-rt58 (rt kernel) , on
real hardware - Freescale LS1021A, 32 bit Cortex A7 processor
- on x86_64 platform I tested non rt kernel 5.15.93 - virtual
machine - qemu platform
Below is the build log after patch to kernel/rcu/tree.h on x86_64
linux-5.15.93$ make
CALL scripts/checksyscalls.sh
CALL scripts/atomic/check-atomics.sh
DESCEND objtool
DESCEND bpf/resolve_btfids
CHK include/generated/compile.h
CC kernel/rcu/tree.o <<<
AR kernel/rcu/built-in.a <<<
AR kernel/built-in.a
CHK kernel/kheaders_data.tar.xz
GEN .version
CHK include/generated/compile.h
UPD include/generated/compile.h
CC init/version.o
AR init/built-in.a
LD vmlinux.o
MODPOST vmlinux.symvers
MODINFO modules.builtin.modinfo
GEN modules.builtin
LD .tmp_vmlinux.btf
BTF .btf.vmlinux.bin.o
LD .tmp_vmlinux.kallsyms1
< snipped >
BTF [M] sound/usb/usx2y/snd-usb-usx2y.ko
BTF [M] sound/virtio/virtio_snd.ko
BTF [M] sound/x86/snd-hdmi-lpe-audio.ko
BTF [M] sound/xen/snd_xen_front.ko
BTF [M] virt/lib/irqbypass.ko
linux-5.15.93$
BR,
ronald
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Backport submission - rcu: Avoid tracing a few functions executed in stop machine
2023-11-30 12:07 ` Ronald Monthero
@ 2023-11-30 12:22 ` Greg KH
0 siblings, 0 replies; 9+ messages in thread
From: Greg KH @ 2023-11-30 12:22 UTC (permalink / raw)
To: Ronald Monthero; +Cc: stable
On Thu, Nov 30, 2023 at 10:07:03PM +1000, Ronald Monthero wrote:
> On Thu, Nov 30, 2023 at 12:08 AM Ronald Monthero
> <debug.penguin32@gmail.com> wrote:
> >
> > On Sat, Nov 25, 2023 at 2:10 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> > >
> > > On Tue, Nov 21, 2023 at 12:09:38AM +1000, Ronald Monthero wrote:
> > > > Dear stable maintainers,
> > > > I like to indicate the oops encountered and request the below patch to
> > > > be backported to v 5.15. The fix is important to avoid recurring oops
> > > > in context of rcu detected stalls.
> > > >
> > > > subject: rcu: Avoid tracing a few functions executed in stop machine
> > > > commit 48f8070f5dd8
> > > > Target kernel version v 5.15
> > > > Reason for Application: To avoid oops due to rcu_prempt detect stalls
> > > > on cpus/tasks
> > > >
> > > > Environment and oops context: Issue was observed in my environment on
> > > > 5.15.193 kernel (arm platform). The patch is helpful to avoid the
> > > > below oops indicated in [1] and [2]
> > >
> > > As the patch does not apply cleanly, we need a working and tested
> > > backport so we know to apply the correct version.
> > >
> > > Can you please provide that as you've obviously already done this?
> >
> > Hi Greg,
> > Sorry I notice my typo error 193 instead of 93. I have tested on the
> > 5.15.93-rt58 kernel.
>
> Hi Greg,
> I used a 5.15.93 kernel
> - on arm32 bit platform I tested with 5.15.93-rt58 (rt kernel) , on
> real hardware - Freescale LS1021A, 32 bit Cortex A7 processor
> - on x86_64 platform I tested non rt kernel 5.15.93 - virtual
> machine - qemu platform
>
> Below is the build log after patch to kernel/rcu/tree.h on x86_64
>
> linux-5.15.93$ make
> CALL scripts/checksyscalls.sh
> CALL scripts/atomic/check-atomics.sh
> DESCEND objtool
> DESCEND bpf/resolve_btfids
> CHK include/generated/compile.h
> CC kernel/rcu/tree.o <<<
> AR kernel/rcu/built-in.a <<<
> AR kernel/built-in.a
> CHK kernel/kheaders_data.tar.xz
> GEN .version
> CHK include/generated/compile.h
> UPD include/generated/compile.h
> CC init/version.o
> AR init/built-in.a
> LD vmlinux.o
> MODPOST vmlinux.symvers
> MODINFO modules.builtin.modinfo
> GEN modules.builtin
> LD .tmp_vmlinux.btf
> BTF .btf.vmlinux.bin.o
> LD .tmp_vmlinux.kallsyms1
>
> < snipped >
>
> BTF [M] sound/usb/usx2y/snd-usb-usx2y.ko
> BTF [M] sound/virtio/virtio_snd.ko
> BTF [M] sound/x86/snd-hdmi-lpe-audio.ko
> BTF [M] sound/xen/snd_xen_front.ko
> BTF [M] virt/lib/irqbypass.ko
> linux-5.15.93$
I don't understand what you are showing here, sorry.
I do not have a working backport anywhere that I can see, that is what
we need. As you seem to have one, can you please submit it?
Also note, if you are using the -rt kernel, that changes lots of stuff
that we know nothing about, please work with the -rt kernel developers
about that.
thanks,
greg k-h
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH] rcu: Avoid tracing a few functions executed in stop machine
2023-11-24 16:10 ` Greg KH
2023-11-29 14:08 ` Ronald Monthero
@ 2023-12-02 6:56 ` Ronald Monthero
2023-12-02 7:46 ` Greg KH
2023-12-02 8:11 ` Ronald Monthero
2 siblings, 1 reply; 9+ messages in thread
From: Ronald Monthero @ 2023-12-02 6:56 UTC (permalink / raw)
To: stable; +Cc: gregkh, Ronald Monthero
[upstream commit 48f8070f5dd8]
This backport patch for kernel 5.15v is derived from upstream
48f8070f5dd8. On 5.15 kernel it fixes recurring oops in context of rcu
detected stalls, indicated below.
log :
root@ls1021atwr:~# uname -r
5.15.93-rt58+ge0f69a158d5b
oops dump stack
** ID_531 main/smp_fsm.c:1884 <inrcu: INFO: rcu_preempt detected
stalls on CPUs/tasks: <<< [1]
rcu: Tasks blocked on level-0 rcu_node (CPUs 0-1): P116/2:b..l
(detected by 1, t=2102 jiffies, g=12741, q=1154)
task:irq/31-arm-irq1 state:D stack: 0 pid:116 ppid:2 flags:0x00000000
[<8064b97f>] (__schedule) from [<8064bb01>] (schedule+0x8d/0xc2)
[<8064bb01>] (schedule) from [<8064fa65>] (schedule_timeout+0x6d/0xa0)
[<8064fa65>] (schedule_timeout) from [<804ba353>]
(fsl_ifc_run_command+0x6f/0x178)
[<804ba353>] (fsl_ifc_run_command) from [<804ba72f>]
(fsl_ifc_cmdfunc+0x203/0x2b8)
[<804ba72f>] (fsl_ifc_cmdfunc) from [<804b135f>]
....
< snipped >
rcu: rcu_preempt kthread timer wakeup didn't happen for 764 jiffies!
g12741 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x1000
rcu: Possible timer handling issue on cpu=0 timer-softirq=1095
rcu: rcu_preempt kthread starved for 765 jiffies! g12741 f0x0
RCU_GP_WAIT_FQS(5) ->state=0x1000 ->cpu=0 <<< [2]
rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is
now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt state:D stack: 0 pid: 13 ppid: 2 flags:0x00000000
[<8064b97f>] (__schedule) from [<8064ba03>] (schedule_rtlock+0x1b/0x2e)
[<8064ba03>] (schedule_rtlock) from [<8064ea6f>]
(rtlock_slowlock_locked+0x93/0x108)
[<8064ea6f>] (rtlock_slowlock_locked) from [<8064eb1b>]
[<8064eb1b>] (rt_spin_lock) from [<8021b723>] (__local_bh_disable_ip+0x6b/0x110)
[<8021b723>] (__local_bh_disable_ip) from [<8025a90f>]
(del_timer_sync+0x7f/0xe0)
[<8025a90f>] (del_timer_sync) from [<8064fa6b>] (schedule_timeout+0x73/0xa0)
Exception stack(0x820fffb0 to 0x820ffff8)
rcu: Stack dump where RCU GP kthread last ran:
...
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
< .. >
Signed-off-by: Ronald Monthero <debug.penguin32@gmail.com>
---
kernel/rcu/tree_plugin.h | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index d070059163d7..36ca6bacd430 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -458,7 +458,7 @@ static bool rcu_preempt_has_tasks(struct rcu_node *rnp)
* be quite short, for example, in the case of the call from
* rcu_read_unlock_special().
*/
-static void
+static notrace void
rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags)
{
bool empty_exp;
@@ -578,7 +578,7 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags)
* is disabled. This function cannot be expected to understand these
* nuances, so the caller must handle them.
*/
-static bool rcu_preempt_need_deferred_qs(struct task_struct *t)
+static notrace bool rcu_preempt_need_deferred_qs(struct task_struct *t)
{
return (__this_cpu_read(rcu_data.exp_deferred_qs) ||
READ_ONCE(t->rcu_read_unlock_special.s)) &&
@@ -592,7 +592,7 @@ static bool rcu_preempt_need_deferred_qs(struct task_struct *t)
* evaluate safety in terms of interrupt, softirq, and preemption
* disabling.
*/
-static void rcu_preempt_deferred_qs(struct task_struct *t)
+static notrace void rcu_preempt_deferred_qs(struct task_struct *t)
{
unsigned long flags;
@@ -922,7 +922,7 @@ static bool rcu_preempt_has_tasks(struct rcu_node *rnp)
* Because there is no preemptible RCU, there can be no deferred quiescent
* states.
*/
-static bool rcu_preempt_need_deferred_qs(struct task_struct *t)
+static notrace bool rcu_preempt_need_deferred_qs(struct task_struct *t)
{
return false;
}
--
2.34.1
^ permalink raw reply related [flat|nested] 9+ messages in thread* Re: [PATCH] rcu: Avoid tracing a few functions executed in stop machine
2023-12-02 6:56 ` [PATCH] " Ronald Monthero
@ 2023-12-02 7:46 ` Greg KH
2023-12-02 8:05 ` Ronald Monthero
0 siblings, 1 reply; 9+ messages in thread
From: Greg KH @ 2023-12-02 7:46 UTC (permalink / raw)
To: Ronald Monthero; +Cc: stable
On Sat, Dec 02, 2023 at 04:56:16PM +1000, Ronald Monthero wrote:
>
> Signed-off-by: Ronald Monthero <debug.penguin32@gmail.com>
You lost the signed-off-by and didn't cc: everyone on the original
commit, how come?
thanks,
greg k-h
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] rcu: Avoid tracing a few functions executed in stop machine
2023-12-02 7:46 ` Greg KH
@ 2023-12-02 8:05 ` Ronald Monthero
0 siblings, 0 replies; 9+ messages in thread
From: Ronald Monthero @ 2023-12-02 8:05 UTC (permalink / raw)
To: Greg KH; +Cc: stable
Greg ack sorry missed that block, will update and send it now. thanks
BR,
ronald
On Sat, Dec 2, 2023 at 5:46 PM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> On Sat, Dec 02, 2023 at 04:56:16PM +1000, Ronald Monthero wrote:
> >
> > Signed-off-by: Ronald Monthero <debug.penguin32@gmail.com>
>
> You lost the signed-off-by and didn't cc: everyone on the original
> commit, how come?
>
> thanks,
>
> greg k-h
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH] rcu: Avoid tracing a few functions executed in stop machine
2023-11-24 16:10 ` Greg KH
2023-11-29 14:08 ` Ronald Monthero
2023-12-02 6:56 ` [PATCH] " Ronald Monthero
@ 2023-12-02 8:11 ` Ronald Monthero
2 siblings, 0 replies; 9+ messages in thread
From: Ronald Monthero @ 2023-12-02 8:11 UTC (permalink / raw)
To: stable
Cc: patrick.wang.shcn, rostedt, quic_neeraju, paulmck, gregkh,
Ronald Monthero
[upstream commit 48f8070f5dd8e13148ae4647780a452d53c457a2]
This backport patch for kernel 5.15v is derived from upstream
48f8070f5dd8. On 5.15 kernel it fixes recurring oops in context of rcu
detected stalls, indicated below.
log :
root@ls1021atwr:~# uname -r
5.15.93-rt58+ge0f69a158d5b
oops dump stack
** ID_531 main/smp_fsm.c:1884 <inrcu: INFO: rcu_preempt detected
stalls on CPUs/tasks: <<< [1]
rcu: Tasks blocked on level-0 rcu_node (CPUs 0-1): P116/2:b..l
(detected by 1, t=2102 jiffies, g=12741, q=1154)
task:irq/31-arm-irq1 state:D stack: 0 pid:116 ppid:2 flags:0x00000000
[<8064b97f>] (__schedule) from [<8064bb01>] (schedule+0x8d/0xc2)
[<8064bb01>] (schedule) from [<8064fa65>] (schedule_timeout+0x6d/0xa0)
[<8064fa65>] (schedule_timeout) from [<804ba353>]
(fsl_ifc_run_command+0x6f/0x178)
[<804ba353>] (fsl_ifc_run_command) from [<804ba72f>]
(fsl_ifc_cmdfunc+0x203/0x2b8)
[<804ba72f>] (fsl_ifc_cmdfunc) from [<804b135f>]
....
< snipped >
rcu: rcu_preempt kthread timer wakeup didn't happen for 764 jiffies!
g12741 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x1000
rcu: Possible timer handling issue on cpu=0 timer-softirq=1095
rcu: rcu_preempt kthread starved for 765 jiffies! g12741 f0x0
RCU_GP_WAIT_FQS(5) ->state=0x1000 ->cpu=0 <<< [2]
rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is
now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt state:D stack: 0 pid: 13 ppid: 2 flags:0x00000000
[<8064b97f>] (__schedule) from [<8064ba03>] (schedule_rtlock+0x1b/0x2e)
[<8064ba03>] (schedule_rtlock) from [<8064ea6f>]
(rtlock_slowlock_locked+0x93/0x108)
[<8064ea6f>] (rtlock_slowlock_locked) from [<8064eb1b>]
[<8064eb1b>] (rt_spin_lock) from [<8021b723>] (__local_bh_disable_ip+0x6b/0x110)
[<8021b723>] (__local_bh_disable_ip) from [<8025a90f>]
(del_timer_sync+0x7f/0xe0)
[<8025a90f>] (del_timer_sync) from [<8064fa6b>] (schedule_timeout+0x73/0xa0)
Exception stack(0x820fffb0 to 0x820ffff8)
rcu: Stack dump where RCU GP kthread last ran:
...
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
< .. >
upstream commit:
Signed-off-by: Patrick Wang <patrick.wang.shcn@gmail.com>
Acked-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Neeraj Upadhyay <quic_neeraju@quicinc.com>
Signed-off-by: Ronald Monthero <debug.penguin32@gmail.com>
---
kernel/rcu/tree_plugin.h | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index d070059163d7..36ca6bacd430 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -458,7 +458,7 @@ static bool rcu_preempt_has_tasks(struct rcu_node *rnp)
* be quite short, for example, in the case of the call from
* rcu_read_unlock_special().
*/
-static void
+static notrace void
rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags)
{
bool empty_exp;
@@ -578,7 +578,7 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags)
* is disabled. This function cannot be expected to understand these
* nuances, so the caller must handle them.
*/
-static bool rcu_preempt_need_deferred_qs(struct task_struct *t)
+static notrace bool rcu_preempt_need_deferred_qs(struct task_struct *t)
{
return (__this_cpu_read(rcu_data.exp_deferred_qs) ||
READ_ONCE(t->rcu_read_unlock_special.s)) &&
@@ -592,7 +592,7 @@ static bool rcu_preempt_need_deferred_qs(struct task_struct *t)
* evaluate safety in terms of interrupt, softirq, and preemption
* disabling.
*/
-static void rcu_preempt_deferred_qs(struct task_struct *t)
+static notrace void rcu_preempt_deferred_qs(struct task_struct *t)
{
unsigned long flags;
@@ -922,7 +922,7 @@ static bool rcu_preempt_has_tasks(struct rcu_node *rnp)
* Because there is no preemptible RCU, there can be no deferred quiescent
* states.
*/
-static bool rcu_preempt_need_deferred_qs(struct task_struct *t)
+static notrace bool rcu_preempt_need_deferred_qs(struct task_struct *t)
{
return false;
}
--
2.34.1
^ permalink raw reply related [flat|nested] 9+ messages in thread