From: Shrikanth Hegde <sshegde@linux.ibm.com>
To: "Michal Suchánek" <msuchanek@suse.de>
Cc: "Christophe Leroy (CS GROUP)" <chleroy@kernel.org>,
Michael Ellerman <mpe@ellerman.id.au>,
harshpb@linux.ibm.com, Gautam Menghani <gautam@linux.ibm.com>,
amachhiw@linux.ibm.com,
Christian Zigotzky <chzigotzky@xenosoft.de>,
linuxppc-dev <linuxppc-dev@lists.ozlabs.org>,
"kvm-ppc@vger.kernel.org" <kvm-ppc@vger.kernel.org>,
Madhavan Srinivasan <maddy@linux.ibm.com>,
Peter Zijlstra <peterz@infradead.org>,
"R.T.Dickinson" <rtd2@xtra.co.nz>,
hypexed@yahoo.com.au, mad skateman <madskateman@gmail.com>,
Christian Zigotzky <info@xenosoft.de>,
Samir M <samir@linux.ibm.com>,
Venkat Rao Bagalkote <venkat88@linux.ibm.com>,
Narayana Murty N <nnmlinux@linux.ibm.com>,
Hari Bathini <hbathini@linux.ibm.com>
Subject: Re: [Linux PPC] Disable PREEMPT
Date: Thu, 17 Sep 2026 15:58:42 +0530 [thread overview]
Message-ID: <b2a5d53f-05e8-48ae-adb7-12fc74523b4e@linux.ibm.com> (raw)
In-Reply-To: <aqu9hA1Bq58GFrVw@kunlun.suse.cz>
Hi Michal,
On 9/17/26 3:44 PM, Michal Suchánek wrote:
> Hello,
>
> On Wed, Sep 16, 2026 at 11:41:38PM +0530, Shrikanth Hegde wrote:
>> Hi Michal,
>>
>>>
>>> It seems easy to re-create.
>>> The key is running bcc/bpf program in parallel to kernel build.
>>>
>>> For example, these two in parallal, leads to stack corruption panic.
>>> Stacktrace does differ sometimes.
>>>
>>> ./funccount rcu* -d 100
>>> make -j 32
>>>
>>> Boom. Will try to debug this further.
>>
>> A bit more debugging.
>>
>> 1. A few functions seems to crash, specially ones that call schedule.
>> So picked schedule itself for tracing
>> 2. Used plain ftrace. No crash.
>> 3. Used kprobes. Boom. Crash appears. But this helped to avoid BPF JIT, or BCC part of it.
>> 4. Did some preempt count validation with annotations. Seems to be holding fine.
>> 5. Since unlock can call schedule while tracing schedule, there is possible looping.
>> Need to see further why it happens to be crashing there. Maybe regs or paca
>> handling needs a closer look.
>>
>> For now, below patch helps to keep the system stable under kprobes, bcc tools.
>> I will look further into see why exactly it is crashing and below one is going to
>> be the proper fix or not.
>>
>> Feel free to give it a try.
>
> thanks for the patch.
>
> While it is mangled by the e-mail client in different way compared to
> the previous patch it does seem to mitigate the issue.
Thanks for trying.
>
> As it is unlikely the board where KVM lockup was originally observed is
> running kpbobes it will probably not address the original problem.
>
Give it a try.
I am suspecting both lockup, stack corruptions are secondary symptoms here.
While recreate, I have different type of traces.
> Thanks
>
> Michal
>
>>
>> ---
>> arch/powerpc/kernel/kprobes-ftrace.c | 2 ++
>> 1 file changed, 2 insertions(+)
>>
>> diff --git a/arch/powerpc/kernel/kprobes-ftrace.c b/arch/powerpc/kernel/kprobes-ftrace.c
>> index f8208c027148..bcb1e8a8995d 100644
>> --- a/arch/powerpc/kernel/kprobes-ftrace.c
>> +++ b/arch/powerpc/kernel/kprobes-ftrace.c
>> @@ -63,7 +63,9 @@ void kprobe_ftrace_handler(unsigned long nip, unsigned long parent_nip,
>> __this_cpu_write(current_kprobe, NULL);
>> }
>> out:
>> + preempt_disable_notrace();
>> ftrace_test_recursion_unlock(bit);
>> + preempt_enable_no_resched_notrace();
>> }
>> NOKPROBE_SYMBOL(kprobe_ftrace_handler);
>> --
>> 2.52.0
next prev parent reply other threads:[~2026-09-17 10:29 UTC|newest]
Thread overview: 78+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-02-14 14:02 [Linux PPC] KVM PR and KVM HV do not work if the kernel was compiled with PREEMPT Christian Zigotzky
2026-02-17 5:02 ` Shrikanth Hegde
2026-02-17 8:52 ` Christian Zigotzky
2026-02-17 9:03 ` Shrikanth Hegde
2026-02-17 9:44 ` Christian Zigotzky
2026-02-17 10:17 ` Madhavan Srinivasan
2026-02-17 10:26 ` Christian Zigotzky
2026-02-17 11:34 ` Shrikanth Hegde
2026-02-21 5:30 ` Christian Zigotzky
2026-02-23 3:45 ` Shrikanth Hegde
2026-02-23 16:48 ` [Linux PPC] 7.0-rc1: e500_mmu.c:923:26: error: assignment to 'struct tlbe_priv *' from incompatible pointer type 'struct tlbe_ref *' Christian Zigotzky
2026-02-23 18:07 ` Christophe Leroy (CS GROUP)
2026-02-23 19:38 ` Christophe Leroy (CS GROUP)
2026-02-23 22:40 ` Kees Cook
2026-02-24 6:30 ` Christian Zigotzky
2026-02-24 7:18 ` Christophe Leroy (CS GROUP)
2026-02-25 5:54 ` [Linux PPC] Disable PREEMPT Christian Zigotzky
2026-02-25 6:23 ` Shrikanth Hegde
2026-02-25 7:16 ` Christophe Leroy (CS GROUP)
2026-02-25 11:36 ` Peter Zijlstra
2026-02-26 5:37 ` Christian Zigotzky
2026-02-26 13:24 ` Christophe Leroy (CS GROUP)
2026-09-03 9:53 ` Michal Suchánek
2026-09-03 10:56 ` Shrikanth Hegde
2026-09-03 11:25 ` Michal Suchánek
2026-09-03 11:31 ` Shrikanth Hegde
2026-09-03 11:40 ` Michal Suchánek
2026-09-03 12:04 ` Shrikanth Hegde
2026-09-03 12:25 ` Michal Suchánek
2026-09-03 12:36 ` Harsh Prateek Bora
2026-09-03 12:55 ` Michal Suchánek
2026-09-03 15:17 ` Amit Machhiwal
2026-09-03 13:28 ` Michal Suchánek
2026-09-04 7:39 ` Michal Suchánek
2026-09-21 8:06 ` Michal Suchánek
2026-09-21 16:46 ` Michal Suchánek
2026-09-21 17:20 ` Shrikanth Hegde
2026-09-22 8:22 ` Michal Suchánek
2026-09-03 17:22 ` Shrikanth Hegde
2026-09-04 7:35 ` Michal Suchánek
2026-09-04 9:04 ` Shrikanth Hegde
2026-09-10 8:59 ` Michal Suchánek
2026-09-10 10:41 ` Shrikanth Hegde
2026-09-10 11:29 ` Michal Suchánek
2026-09-10 12:16 ` Michal Suchánek
2026-09-11 17:09 ` Shrikanth Hegde
2026-09-14 10:58 ` Michal Suchánek
2026-09-15 5:47 ` Narayana Murty N
2026-09-15 7:35 ` Michal Suchánek
2026-09-15 9:08 ` Shrikanth Hegde
2026-09-16 18:11 ` Shrikanth Hegde
2026-09-17 10:14 ` Michal Suchánek
2026-09-17 10:28 ` Shrikanth Hegde [this message]
2026-09-17 13:12 ` Michal Suchánek
2026-09-17 14:15 ` Shrikanth Hegde
2026-09-18 2:39 ` Christian Zigotzky
2026-09-18 7:10 ` Michal Suchánek
2026-09-20 11:49 ` Christian Zigotzky
2026-09-20 13:04 ` Christian Zigotzky
2026-09-21 5:12 ` Shrikanth Hegde
2026-09-23 5:09 ` Christian Zigotzky
2026-09-23 5:13 ` Shrikanth Hegde
2026-09-23 16:01 ` Christian Zigotzky
2026-09-23 18:42 ` Shrikanth Hegde
2026-09-24 17:47 ` Christian Zigotzky
2026-09-25 6:40 ` Shrikanth Hegde
2026-09-18 9:26 ` Shrikanth Hegde
2026-09-18 11:52 ` Michal Suchánek
2026-09-04 8:55 ` Christian Zigotzky
2026-09-04 9:20 ` Christian Zigotzky
2026-09-04 9:24 ` Shrikanth Hegde
2026-09-04 9:34 ` Christian Zigotzky
2026-09-05 13:07 ` Christian Zigotzky
2026-09-07 8:50 ` Michal Suchánek
2026-09-07 11:32 ` Christian Zigotzky
2026-09-10 8:54 ` Michal Suchánek
2026-09-10 10:31 ` Michal Suchánek
2026-09-10 11:31 ` Michal Suchánek
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=b2a5d53f-05e8-48ae-adb7-12fc74523b4e@linux.ibm.com \
--to=sshegde@linux.ibm.com \
--cc=amachhiw@linux.ibm.com \
--cc=chleroy@kernel.org \
--cc=chzigotzky@xenosoft.de \
--cc=gautam@linux.ibm.com \
--cc=harshpb@linux.ibm.com \
--cc=hbathini@linux.ibm.com \
--cc=hypexed@yahoo.com.au \
--cc=info@xenosoft.de \
--cc=kvm-ppc@vger.kernel.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=maddy@linux.ibm.com \
--cc=madskateman@gmail.com \
--cc=mpe@ellerman.id.au \
--cc=msuchanek@suse.de \
--cc=nnmlinux@linux.ibm.com \
--cc=peterz@infradead.org \
--cc=rtd2@xtra.co.nz \
--cc=samir@linux.ibm.com \
--cc=venkat88@linux.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox