* selftests/ftrace kprobe_eventname test fails on s390x QEMU (KVM/Linux)
@ 2024-06-27 5:03 Yunseong Kim
2024-06-27 14:05 ` Steven Rostedt
` (2 more replies)
0 siblings, 3 replies; 5+ messages in thread
From: Yunseong Kim @ 2024-06-27 5:03 UTC (permalink / raw)
To: Akanksha J N, Naveen N. Rao, Sven Schnelle, Vasily Gorbik,
Martin Schwidefsky
Cc: MichelleJin, linux-s390, linux-kselftest, linux-trace-kernel
Hi all,
In my s390x archtecture, kprobe_eventname selftest have always failed
because of rcu_sched stalls.
My environment is QEMU Ubuntu 24.04 KVM Machine Linux version
6.8.0-36-generic (buildd@bos01-s390x-012) (s390x-linux-gnu-gcc-13
(Ubuntu 13.2.0-23ubuntu4) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #36
1 configured CPUs, Linux is running under KVM in 64-bit mode
qemu-system-s390x -no-reboot -name auto-inst-test -nographic -m 4096 \
-drive file=disk-image.qcow2,format=qcow2,cache=none,if=virtio \
-netdev user,id=enc0,hostfwd=tcp::10000-:22 \
-device virtio-net-ccw,netdev=enc0 \
-qmp tcp:localhost:4444,server,nowait
Currently, This failure can be always reproduced by this kselftests script:
# tools/testing/selftests/ftrace/ftracetest
tools/testing/selftests/ftrace/test.d/kprobe/kprobe_eventname.tc
I've investigating cause of line, then I find this line.
for i in `seq 0 255`; do
echo p $FUNCTION_FORK+${i} >> kprobe_events || continue
done
cat kprobe_events >> $testlog
echo 1 > events/kprobes/enable # <<<
This line makes "rcu_sched detected stalls" log and stall the system.
[ 7825.578940] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
After this line, the test doesn't go any further.
This test was added in the patch below.
("selftests/ftrace: Add new test case which adds multiple consecutive
probes in a function")
Link:
https://lore.kernel.org/linux-trace-kernel/20230428163842.95118-2-akanksha@linux.ibm.com/
I've attached a link to a compressed version of vmcore, vmlinux and
config files that I dumped from my environment.
https://drive.google.com/file/d/1O2bCKrRbyJ-yP4zTz_sAd_qM80nHnCGr/view?usp=sharing
I used QEMU QMP to dump the vmcore.
$ telnet localhost 4444
{"execute": "qmp_capabilities"}
{"execute":"dump-guest-memory","arguments":
{"paging":false,"protocol":"file:/home/paran/vmcore1.img"}}
rcu: INFO: rcu_sched detected stalls on CPUs/tasks:s:
rcu:
(detected by 0, t=6002 jiffies, g=24353, q=1 ncpus=1)1)
rcu: All QSes seen, last rcu_sched kthread activity 6002
(4294978930-4294972928), jiffies_till_next_fqs=1, root ->qsmask 0x0x0
rcu: rcu_sched kthread starved for 6002 jiffies! g24353 f0x2
RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0=0
rcu:
Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected
behavior.r.
rcu: RCU grace-period kthread stack dump:p:
task:rcu_sched state:R running task stack:0 pid:16
tgid:16 ppid:2 flags:0x0000000000
Call Trace:e:
__schedule+0x346/0x8b8 8
schedule+0x36/0x148 8
schedule_timeout+0x8e/0x148 8
rcu_gp_fqs_loop+0x444/0x548 8
rcu_gp_kthread+0x146/0x198 8
kthread+0x124/0x128 8
__ret_from_fork+0x40/0x58 8
ret_from_fork+0xa/0x30 0
rcu: Stack dump where RCU GP kthread last ran:n:
CPU: 0 PID: 1077 Comm: ftracetest Not tainted 6.8.0-36-generic #36-Ubuntu
Hardware name: QEMU 8561 QEMU (KVM/Linux)
Krnl PSW : 0704f00180000000 0000000000121d32 kprobe_exceptions_notify
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/kprobes.c:519
(discriminator 1))
R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:3 PM:0 RI:0 EA:3:3
Krnl GPRS: 0000000000000000 0000000000000000 0000000000008001
0000000000000000
0000000000000000 0000000000000000 0000000000000000
0000000001956720
ffffffffffffffff 0000000000121c98 0000000001958d40
00000380000cfbd8
000003ff938dbc78 00000380000cfab8 0000000000121d1c
00000380000cf980
Krnl Code: 0000000000121d26: 9103b008
Code starting with the faulting instruction
===========================================
8(%r11),3
0000000000121d2a: a7840004
8,0000000000121d32
#0000000000121d2e: ad03f0a0
stosm
160(%r15),3
>0000000000121d32: b9140022
lgfr
%r2,%r2
0000000000121d36: ebbff0a80004
%r11,%r15,168(%r15)
0000000000121d3c: a7190000
lghi
%r1,0
0000000000121d40: a7390000
lghi
%r3,0
0000000000121d44: a7490000
lghi
%r4,0
Call Trace:
kprobe_exceptions_notify
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/kprobes.c:519
(discriminator 1))
kprobe_exceptions_notify
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/kprobes.c:504
(discriminator 1))
notifier_call_chain (/build/linux-3nCxw2/linux-6.8.0/kernel/notifier.c:93)
notify_die (/build/linux-3nCxw2/linux-6.8.0/kernel/notifier.c:597)
do_per_trap
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/traps.c:75
(discriminator 1))
__do_pgm_check
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/include/asm/irqflags.h:47
(discriminator 1)
/build/linux-3nCxw2/linux-6.8.0/arch/s390/include/asm/irqflags.h:52
(discriminator 1)
/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/traps.c:356
(discriminator 1))
pgm_check_handler
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/entry.S:383)
kernel_clone
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/include/asm/uaccess.h:119
(discriminator 1) /build/linux-3nCxw2/linux-6.8.0/kernel/fork.c:2927
(discriminator 1))
__do_sys_clone (/build/linux-3nCxw2/linux-6.8.0/kernel/fork.c:3055)
__s390x_sys_clone (/build/linux-3nCxw2/linux-6.8.0/kernel/fork.c:3027)
__do_syscall
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/include/asm/ptrace.h:195
(discriminator 3)
/build/linux-3nCxw2/linux-6.8.0/arch/s390/include/asm/ptrace.h:200
(discriminator 3)
/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/syscall.c:145
(discriminator 3)
/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/syscall.c:168
(discriminator 3))
system_call (/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/entry.S:309)
Last Breaking-Event-Address:
0xfdf5045050
?rcu: INFO: rcu_sched detected stalls on CPUs/tasks:s:
@rcu:
(detected by 0, t=24007 jiffies, g=24353, q=1 ncpus=1)
rcu: All QSes seen, last rcu_sched kthread activity 24007
(4294996935-4294972928), jiffies_till_next_fqs=1, root ->qsmask 0x0x0
rcu: rcu_sched kthread starved for 24007 jiffies! g24353 f0x2
RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0=0
rcu:
Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected
behavior.r.
rcu: RCU grace-period kthread stack dump:p:
task:rcu_sched state:R running task stack:0 pid:16
tgid:16 ppid:2 flags:0x0000000000
Call Trace:e:
__schedule+0x346/0x8b8 8
schedule+0x36/0x148 8
schedule_timeout+0x8e/0x148 8
rcu_gp_fqs_loop+0x444/0x548 8
rcu_gp_kthread+0x146/0x198 8
kthread+0x124/0x128 8
__ret_from_fork+0x40/0x58 8
ret_from_fork+0xa/0x30 0
rcu: Stack dump where RCU GP kthread last ran:n:
CPU: 0 PID: 1077 Comm: ftracetest Not tainted 6.8.0-36-generic #36-Ubuntu
Hardware name: QEMU 8561 QEMU (KVM/Linux)
Krnl PSW : 0704d00180000000 0000000000ebe0b2 __do_pgm_check
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/traps.c:353)
R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3:3
Krnl GPRS: 0704c00180000000 0000000000000000 00000380000cfb97
0000000000000000
0000000000000000 0704c00180000000 0000000000000000
0000000000000000
0704c00180000000 00000000001a8388 0000000000000000
00000380000cfbd8
000003ff938dbc78 0000000000ed1c6c 0000000000ebe024
00000380000cfaf0
Krnl Code: 0000000000ebe0a4: a504bfff
Code starting with the faulting instruction
===========================================
nihh
%r0,49151
0000000000ebe0a8: e300f0a80024
%r0,168(%r15)
#0000000000ebe0ae: 8000f0a8
168(%r15)
>0000000000ebe0b2: 5850b0a0
%r5,160(%r11)
0000000000ebe0b6: c05b0000007f
nilf
%r5,127
0000000000ebe0bc: a7840012
8,0000000000ebe0e0
0000000000ebe0c0: b91600e5
llgfr
%r14,%r5
0000000000ebe0c4: c0400051121e
larl
%r4,00000000018e0500
Call Trace:
__do_pgm_check
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/traps.c:353)
__do_pgm_check
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/traps.c:318)
pgm_check_handler
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/entry.S:383)
kernel_clone
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/include/asm/uaccess.h:119
(discriminator 1) /build/linux-3nCxw2/linux-6.8.0/kernel/fork.c:2927
(discriminator 1))
__do_sys_clone (/build/linux-3nCxw2/linux-6.8.0/kernel/fork.c:3055)
__s390x_sys_clone (/build/linux-3nCxw2/linux-6.8.0/kernel/fork.c:3027)
__do_syscall
(/build/linux-3nCxw2/linux-6.8.0/arch/s390/include/asm/ptrace.h:195
(discriminator 3)
/build/linux-3nCxw2/linux-6.8.0/arch/s390/include/asm/ptrace.h:200
(discriminator 3)
/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/syscall.c:145
(discriminator 3)
/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/syscall.c:168
(discriminator 3))
system_call (/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/entry.S:309)
Last Breaking-Event-Address:
0x4404c0018000000000
It's not easy for me to resolve this issue. If advice or guidance can be
provided on how to resolve this issue, I'll try sending a patch!
Warm regards,
Yunseong Kim
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: selftests/ftrace kprobe_eventname test fails on s390x QEMU (KVM/Linux)
2024-06-27 5:03 selftests/ftrace kprobe_eventname test fails on s390x QEMU (KVM/Linux) Yunseong Kim
@ 2024-06-27 14:05 ` Steven Rostedt
2024-06-27 14:34 ` Masami Hiramatsu
2024-06-27 16:15 ` Sven Schnelle
2 siblings, 0 replies; 5+ messages in thread
From: Steven Rostedt @ 2024-06-27 14:05 UTC (permalink / raw)
To: Yunseong Kim
Cc: Akanksha J N, Naveen N. Rao, Sven Schnelle, Vasily Gorbik,
Martin Schwidefsky, MichelleJin, linux-s390, linux-kselftest,
linux-trace-kernel
On Thu, 27 Jun 2024 14:03:14 +0900
Yunseong Kim <yskelg@gmail.com> wrote:
> My environment is QEMU Ubuntu 24.04 KVM Machine Linux version
> 6.8.0-36-generic (buildd@bos01-s390x-012) (s390x-linux-gnu-gcc-13
> (Ubuntu 13.2.0-23ubuntu4) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #36
Could you see if you have the same issue on the most recent kernel?
-- Steve
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: selftests/ftrace kprobe_eventname test fails on s390x QEMU (KVM/Linux)
2024-06-27 5:03 selftests/ftrace kprobe_eventname test fails on s390x QEMU (KVM/Linux) Yunseong Kim
2024-06-27 14:05 ` Steven Rostedt
@ 2024-06-27 14:34 ` Masami Hiramatsu
2024-06-27 16:15 ` Sven Schnelle
2 siblings, 0 replies; 5+ messages in thread
From: Masami Hiramatsu @ 2024-06-27 14:34 UTC (permalink / raw)
To: Yunseong Kim
Cc: Akanksha J N, Naveen N. Rao, Sven Schnelle, Vasily Gorbik,
Martin Schwidefsky, MichelleJin, linux-s390, linux-kselftest,
linux-trace-kernel
Hi,
Thanks for reporting!
On Thu, 27 Jun 2024 14:03:14 +0900
Yunseong Kim <yskelg@gmail.com> wrote:
> Hi all,
>
>
> In my s390x archtecture, kprobe_eventname selftest have always failed
> because of rcu_sched stalls.
>
> My environment is QEMU Ubuntu 24.04 KVM Machine Linux version
> 6.8.0-36-generic (buildd@bos01-s390x-012) (s390x-linux-gnu-gcc-13
> (Ubuntu 13.2.0-23ubuntu4) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #36
>
> 1 configured CPUs, Linux is running under KVM in 64-bit mode
>
>
> qemu-system-s390x -no-reboot -name auto-inst-test -nographic -m 4096 \
> -drive file=disk-image.qcow2,format=qcow2,cache=none,if=virtio \
> -netdev user,id=enc0,hostfwd=tcp::10000-:22 \
> -device virtio-net-ccw,netdev=enc0 \
> -qmp tcp:localhost:4444,server,nowait
>
>
> Currently, This failure can be always reproduced by this kselftests script:
>
> # tools/testing/selftests/ftrace/ftracetest
> tools/testing/selftests/ftrace/test.d/kprobe/kprobe_eventname.tc
>
> I've investigating cause of line, then I find this line.
>
> for i in `seq 0 255`; do
> echo p $FUNCTION_FORK+${i} >> kprobe_events || continue
> done
>
> cat kprobe_events >> $testlog
>
> echo 1 > events/kprobes/enable # <<<
Isn't it kprobe_insn_boundary.tc?
Also, can you modify it as
for i in `seq 0 255`; do
echo p $FUNCTION_FORK+${i} >> kprobe_events || continue
echo 1 > events/kprobes/enable
done
And run it with --console option? It may tell us when the stall happens
with what offset.
Also, please try it with the latest kernel instead of 6.8.x
Thank you,
>
>
> This line makes "rcu_sched detected stalls" log and stall the system.
>
> [ 7825.578940] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
>
> After this line, the test doesn't go any further.
>
>
> This test was added in the patch below.
>
> ("selftests/ftrace: Add new test case which adds multiple consecutive
> probes in a function")
> Link:
> https://lore.kernel.org/linux-trace-kernel/20230428163842.95118-2-akanksha@linux.ibm.com/
>
>
> I've attached a link to a compressed version of vmcore, vmlinux and
> config files that I dumped from my environment.
>
> https://drive.google.com/file/d/1O2bCKrRbyJ-yP4zTz_sAd_qM80nHnCGr/view?usp=sharing
>
>
> I used QEMU QMP to dump the vmcore.
>
> $ telnet localhost 4444
>
> {"execute": "qmp_capabilities"}
>
> {"execute":"dump-guest-memory","arguments":
> {"paging":false,"protocol":"file:/home/paran/vmcore1.img"}}
>
>
> rcu: INFO: rcu_sched detected stalls on CPUs/tasks:s:
> rcu:
> (detected by 0, t=6002 jiffies, g=24353, q=1 ncpus=1)1)
> rcu: All QSes seen, last rcu_sched kthread activity 6002
> (4294978930-4294972928), jiffies_till_next_fqs=1, root ->qsmask 0x0x0
> rcu: rcu_sched kthread starved for 6002 jiffies! g24353 f0x2
> RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0=0
> rcu:
> Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected
> behavior.r.
> rcu: RCU grace-period kthread stack dump:p:
> task:rcu_sched state:R running task stack:0 pid:16
> tgid:16 ppid:2 flags:0x0000000000
> Call Trace:e:
> __schedule+0x346/0x8b8 8
> schedule+0x36/0x148 8
> schedule_timeout+0x8e/0x148 8
> rcu_gp_fqs_loop+0x444/0x548 8
> rcu_gp_kthread+0x146/0x198 8
> kthread+0x124/0x128 8
> __ret_from_fork+0x40/0x58 8
> ret_from_fork+0xa/0x30 0
> rcu: Stack dump where RCU GP kthread last ran:n:
> CPU: 0 PID: 1077 Comm: ftracetest Not tainted 6.8.0-36-generic #36-Ubuntu
> Hardware name: QEMU 8561 QEMU (KVM/Linux)
> Krnl PSW : 0704f00180000000 0000000000121d32 kprobe_exceptions_notify
> (/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/kprobes.c:519
> (discriminator 1))
> R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:3 PM:0 RI:0 EA:3:3
> Krnl GPRS: 0000000000000000 0000000000000000 0000000000008001
> 0000000000000000
> 0000000000000000 0000000000000000 0000000000000000
> 0000000001956720
> ffffffffffffffff 0000000000121c98 0000000001958d40
> 00000380000cfbd8
> 000003ff938dbc78 00000380000cfab8 0000000000121d1c
> 00000380000cf980
> Krnl Code: 0000000000121d26: 9103b008
>
> Code starting with the faulting instruction
> ===========================================
> 8(%r11),3
> 0000000000121d2a: a7840004
> 8,0000000000121d32
> #0000000000121d2e: ad03f0a0
> stosm
> 160(%r15),3
> >0000000000121d32: b9140022
> lgfr
> %r2,%r2
> 0000000000121d36: ebbff0a80004
> %r11,%r15,168(%r15)
> 0000000000121d3c: a7190000
> lghi
> %r1,0
> 0000000000121d40: a7390000
> lghi
> %r3,0
> 0000000000121d44: a7490000
> lghi
> %r4,0
> Call Trace:
> kprobe_exceptions_notify
> (/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/kprobes.c:519
> (discriminator 1))
> kprobe_exceptions_notify
> (/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/kprobes.c:504
> (discriminator 1))
> notifier_call_chain (/build/linux-3nCxw2/linux-6.8.0/kernel/notifier.c:93)
> notify_die (/build/linux-3nCxw2/linux-6.8.0/kernel/notifier.c:597)
> do_per_trap
> (/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/traps.c:75
> (discriminator 1))
> __do_pgm_check
> (/build/linux-3nCxw2/linux-6.8.0/arch/s390/include/asm/irqflags.h:47
> (discriminator 1)
> /build/linux-3nCxw2/linux-6.8.0/arch/s390/include/asm/irqflags.h:52
> (discriminator 1)
> /build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/traps.c:356
> (discriminator 1))
> pgm_check_handler
> (/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/entry.S:383)
> kernel_clone
> (/build/linux-3nCxw2/linux-6.8.0/arch/s390/include/asm/uaccess.h:119
> (discriminator 1) /build/linux-3nCxw2/linux-6.8.0/kernel/fork.c:2927
> (discriminator 1))
> __do_sys_clone (/build/linux-3nCxw2/linux-6.8.0/kernel/fork.c:3055)
> __s390x_sys_clone (/build/linux-3nCxw2/linux-6.8.0/kernel/fork.c:3027)
> __do_syscall
> (/build/linux-3nCxw2/linux-6.8.0/arch/s390/include/asm/ptrace.h:195
> (discriminator 3)
> /build/linux-3nCxw2/linux-6.8.0/arch/s390/include/asm/ptrace.h:200
> (discriminator 3)
> /build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/syscall.c:145
> (discriminator 3)
> /build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/syscall.c:168
> (discriminator 3))
> system_call (/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/entry.S:309)
> Last Breaking-Event-Address:
> 0xfdf5045050
> ?rcu: INFO: rcu_sched detected stalls on CPUs/tasks:s:
> @rcu:
> (detected by 0, t=24007 jiffies, g=24353, q=1 ncpus=1)
> rcu: All QSes seen, last rcu_sched kthread activity 24007
> (4294996935-4294972928), jiffies_till_next_fqs=1, root ->qsmask 0x0x0
> rcu: rcu_sched kthread starved for 24007 jiffies! g24353 f0x2
> RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0=0
> rcu:
> Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected
> behavior.r.
> rcu: RCU grace-period kthread stack dump:p:
> task:rcu_sched state:R running task stack:0 pid:16
> tgid:16 ppid:2 flags:0x0000000000
> Call Trace:e:
> __schedule+0x346/0x8b8 8
> schedule+0x36/0x148 8
> schedule_timeout+0x8e/0x148 8
> rcu_gp_fqs_loop+0x444/0x548 8
> rcu_gp_kthread+0x146/0x198 8
> kthread+0x124/0x128 8
> __ret_from_fork+0x40/0x58 8
> ret_from_fork+0xa/0x30 0
> rcu: Stack dump where RCU GP kthread last ran:n:
> CPU: 0 PID: 1077 Comm: ftracetest Not tainted 6.8.0-36-generic #36-Ubuntu
> Hardware name: QEMU 8561 QEMU (KVM/Linux)
> Krnl PSW : 0704d00180000000 0000000000ebe0b2 __do_pgm_check
> (/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/traps.c:353)
> R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3:3
> Krnl GPRS: 0704c00180000000 0000000000000000 00000380000cfb97
> 0000000000000000
> 0000000000000000 0704c00180000000 0000000000000000
> 0000000000000000
> 0704c00180000000 00000000001a8388 0000000000000000
> 00000380000cfbd8
> 000003ff938dbc78 0000000000ed1c6c 0000000000ebe024
> 00000380000cfaf0
> Krnl Code: 0000000000ebe0a4: a504bfff
>
> Code starting with the faulting instruction
> ===========================================
> nihh
> %r0,49151
> 0000000000ebe0a8: e300f0a80024
> %r0,168(%r15)
> #0000000000ebe0ae: 8000f0a8
> 168(%r15)
> >0000000000ebe0b2: 5850b0a0
> %r5,160(%r11)
> 0000000000ebe0b6: c05b0000007f
> nilf
> %r5,127
> 0000000000ebe0bc: a7840012
> 8,0000000000ebe0e0
> 0000000000ebe0c0: b91600e5
> llgfr
> %r14,%r5
> 0000000000ebe0c4: c0400051121e
> larl
> %r4,00000000018e0500
> Call Trace:
> __do_pgm_check
> (/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/traps.c:353)
> __do_pgm_check
> (/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/traps.c:318)
> pgm_check_handler
> (/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/entry.S:383)
> kernel_clone
> (/build/linux-3nCxw2/linux-6.8.0/arch/s390/include/asm/uaccess.h:119
> (discriminator 1) /build/linux-3nCxw2/linux-6.8.0/kernel/fork.c:2927
> (discriminator 1))
> __do_sys_clone (/build/linux-3nCxw2/linux-6.8.0/kernel/fork.c:3055)
> __s390x_sys_clone (/build/linux-3nCxw2/linux-6.8.0/kernel/fork.c:3027)
> __do_syscall
> (/build/linux-3nCxw2/linux-6.8.0/arch/s390/include/asm/ptrace.h:195
> (discriminator 3)
> /build/linux-3nCxw2/linux-6.8.0/arch/s390/include/asm/ptrace.h:200
> (discriminator 3)
> /build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/syscall.c:145
> (discriminator 3)
> /build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/syscall.c:168
> (discriminator 3))
> system_call (/build/linux-3nCxw2/linux-6.8.0/arch/s390/kernel/entry.S:309)
> Last Breaking-Event-Address:
> 0x4404c0018000000000
>
>
> It's not easy for me to resolve this issue. If advice or guidance can be
> provided on how to resolve this issue, I'll try sending a patch!
>
>
> Warm regards,
> Yunseong Kim
>
--
Masami Hiramatsu (Google) <mhiramat@kernel.org>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: selftests/ftrace kprobe_eventname test fails on s390x QEMU (KVM/Linux)
2024-06-27 5:03 selftests/ftrace kprobe_eventname test fails on s390x QEMU (KVM/Linux) Yunseong Kim
2024-06-27 14:05 ` Steven Rostedt
2024-06-27 14:34 ` Masami Hiramatsu
@ 2024-06-27 16:15 ` Sven Schnelle
2024-06-27 21:44 ` Yunseong Kim
2 siblings, 1 reply; 5+ messages in thread
From: Sven Schnelle @ 2024-06-27 16:15 UTC (permalink / raw)
To: Yunseong Kim
Cc: Akanksha J N, Naveen N. Rao, Vasily Gorbik, Martin Schwidefsky,
MichelleJin, linux-s390, linux-kselftest, linux-trace-kernel
Yunseong Kim <yskelg@gmail.com> writes:
> In my s390x archtecture, kprobe_eventname selftest have always failed
> because of rcu_sched stalls.
>
> My environment is QEMU Ubuntu 24.04 KVM Machine Linux version
> 6.8.0-36-generic (buildd@bos01-s390x-012) (s390x-linux-gnu-gcc-13
> (Ubuntu 13.2.0-23ubuntu4) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #36
>
> 1 configured CPUs, Linux is running under KVM in 64-bit mode
>
>
> qemu-system-s390x -no-reboot -name auto-inst-test -nographic -m 4096 \
> -drive file=disk-image.qcow2,format=qcow2,cache=none,if=virtio \
> -netdev user,id=enc0,hostfwd=tcp::10000-:22 \
> -device virtio-net-ccw,netdev=enc0 \
> -qmp tcp:localhost:4444,server,nowait
>
>
> Currently, This failure can be always reproduced by this kselftests script:
>
> # tools/testing/selftests/ftrace/ftracetest
> tools/testing/selftests/ftrace/test.d/kprobe/kprobe_eventname.tc
>
> I've investigating cause of line, then I find this line.
>
> for i in `seq 0 255`; do
> echo p $FUNCTION_FORK+${i} >> kprobe_events || continue
> done
I think this is fixed with:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v6.10-rc5&id=c239c83ed5c558be3b5926c7f11639f02c8acd00
Regards,
Sven
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: selftests/ftrace kprobe_eventname test fails on s390x QEMU (KVM/Linux)
2024-06-27 16:15 ` Sven Schnelle
@ 2024-06-27 21:44 ` Yunseong Kim
0 siblings, 0 replies; 5+ messages in thread
From: Yunseong Kim @ 2024-06-27 21:44 UTC (permalink / raw)
To: Steven Rostedt, Masami Hiramatsu (Google), Sven Schnelle
Cc: Akanksha J N, Naveen N. Rao, Vasily Gorbik, Martin Schwidefsky,
MichelleJin, linux-s390, linux-kselftest, linux-trace-kernel
Hi all,
On 6/28/24 1:15 오전, Sven Schnelle wrote:
> Yunseong Kim <yskelg@gmail.com> writes:
>
>> In my s390x archtecture, kprobe_eventname selftest have always failed
>> because of rcu_sched stalls.
>>
>> My environment is QEMU Ubuntu 24.04 KVM Machine Linux version
>> 6.8.0-36-generic (buildd@bos01-s390x-012) (s390x-linux-gnu-gcc-13
>> (Ubuntu 13.2.0-23ubuntu4) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #36
>>
>> 1 configured CPUs, Linux is running under KVM in 64-bit mode
>>
>>
>> qemu-system-s390x -no-reboot -name auto-inst-test -nographic -m 4096 \
>> -drive file=disk-image.qcow2,format=qcow2,cache=none,if=virtio \
>> -netdev user,id=enc0,hostfwd=tcp::10000-:22 \
>> -device virtio-net-ccw,netdev=enc0 \
>> -qmp tcp:localhost:4444,server,nowait
>>
>>
>> Currently, This failure can be always reproduced by this kselftests script:
>>
>> # tools/testing/selftests/ftrace/ftracetest
>> tools/testing/selftests/ftrace/test.d/kprobe/kprobe_eventname.tc
> Could you see if you have the same issue on the most recent kernel?
>
> -- Steve
Thanks you Steve,
As soon as I saw your mail, I pulled the latest kernel.
Linux version 6.10.0-rc5+ (paran@fedora) (s390x-linux-gnu-gcc (GCC)
14.1.1 20240507 (Red Hat Cross 14.1.1-1), GNU ld version 2.41-1.fc40)
#30 SMP Fri Jun 28 04:32:20 KST 2024
My Linux is running under QEMU KVM in 64-bit mode, cpu: 1 configured
CPUs, 0 standby CPUs.
[ 726.069032] Kernel panic - not syncing: Corrupt kernel stack, can't
continue.
[ 726.069297] CPU: 0 PID: 1023 Comm: ftracetest Not tainted 6.10.0-rc5+ #30
[ 726.069335] Hardware name: QEMU 8561 QEMU (KVM/Linux)
[ 726.069361] Call Trace:
[ 726.069383] [<000003ffe0dc545e>] dump_stack_lvl+0x76/0xa8
[ 726.069446] [<000003ffe0db7170>] panic+0x170/0x358
[ 726.069485] [<000003ffe0000f16>] kernel_stack_overflow+0x56/0x60
[ 726.069522] [<0000000000000200>] 0x200
[ 726.069987] Dumping ftrace buffer:
[ 726.070144] ---------------------------------
[ 726.071356] <...>-1023 0..... 1829038us : p_kernel_clone_0:
(kernel_clone+0x0/0x390)
[ 726.071452] <...>-1023 0dN... 1829717us : p_kernel_clone_6:
(kernel_clone+0x6/0x390)
[ 726.071506] <...>-1023 0dN... 1830009us : p_kernel_clone_12:
(kernel_clone+0xc/0x390)
[ 726.071558] <...>-1023 0dN... 1830064us : p_kernel_clone_16:
...
I have changed the QEMU setting a bit to run latest kernel.
qemu-system-s390x -no-reboot -name auto-inst-test -nographic -m 4096 \
-kernel bzImage-6.10.0-rc5-00035-gafcd48134c58 \
-initrd 6.8.0-36-generic/initrd.img-6.8.0-36-generic \
-append
"root=/dev/disk/by-id/dm-uuid-LVM-7JeH1Hjp3H1U60GH8m82Od6YXP47ItB01EcvyJ33Ya80VPeRinI1vfMjI9Ndzcc3
crashkernel=512M-:1024M oops=panic panic_on_warn=1
ftrace_dump_on_oops=orig_cpu slub_debug=UZ nokaslr gcov_persist=0 " \
-drive file=disk-image.qcow2,format=qcow2,cache=none,if=virtio \
-netdev user,id=enc0,hostfwd=tcp::10000-:22 \
-device virtio-net-ccw,netdev=enc0 \
-qmp tcp:localhost:4444,server,nowait
> Hi,
>
> Thanks for reporting!
Hi, Masami, It's my pleasure.
>> # tools/testing/selftests/ftrace/ftracetest
>> tools/testing/selftests/ftrace/test.d/kprobe/kprobe_eventname.tc
>>
>> I've investigating cause of line, then I find this line.
>>
>> for i in `seq 0 255`; do
>> echo p $FUNCTION_FORK+${i} >> kprobe_events || continue
>> done
>>
>> cat kprobe_events >> $testlog
>>
>> echo 1 > events/kprobes/enable # <<<
>
> Isn't it kprobe_insn_boundary.tc?
That's exactly right!
I'm sorry, I slipped up again. :(
This is my exactly tested command.
$ cd tools/testing/selftests/ftrace/
$ sudo ./ftracetest --console test.d/kprobe/kprobe_insn_boundary.tc
> Also, can you modify it as
>
> for i in `seq 0 255`; do
> echo p $FUNCTION_FORK+${i} >> kprobe_events || continue
> echo 1 > events/kprobes/enable
> done
>
> And run it with --console option? It may tell us when the stall happens
> with what offset.
>
> Also, please try it with the latest kernel instead of 6.8.x
>
> Thank you,
>
I took what you said and applied it in test script.
--- a/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_insn_boundary.tc
+++ b/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_insn_boundary.tc
@@ -6,11 +6,11 @@
for i in `seq 0 255`; do
echo p $FUNCTION_FORK+${i} >> kprobe_events || continue
+ echo 1 > events/kprobes/enable
done
cat kprobe_events >> $testlog
-echo 1 > events/kprobes/enable
( echo "forked" )
echo 0 > events/kprobes/enable
echo > kprobe_events
This is the results from console.
=== Ftrace unit tests ===
[1] Register multiple kprobe events in a function+ checkreq
/home/paran/linux-source/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_insn_boundary.tc
+ + grep ^#[ \t]*requires:
/home/paran/linux-source-6.8.0/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_insn_boundary.tc
cut -f2- -d:
+ requires= kprobe_events
+ eval check_requires kprobe_events
+ check_requires kprobe_events
+ p=kprobe_events
+ r=kprobe_events
+ t=kprobe_events
+ [ kprobe_events != kprobe_events ]
+ [ kprobe_events != kprobe_events ]
+ [ kprobe_events != kprobe_events ]
+ [ ! -e kprobe_events ]
+ initialize_ftrace
+ disable_tracing
+ echo 0
+ reset_tracer
+ echo nop
+ reset_trigger
+ [ -d events/synthetic ]
+ reset_trigger_file events/alarmtimer/alarmtimer_cancel/trigger
events/alarmtimer/alarmtimer_fired/trigger events/alarmtimer+
....
clear_trace
+ echo
+ enable_tracing
+ echo 1
+ .
/home/paran/linux-source/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_insn_boundary.tc
+ seq 0 255
+ echo p kernel_clone+0
+ echo 1
+ echo p kernel_clone+1
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+2
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+3
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+4
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+5
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+6
+ echo 1
+ echo p kernel_clone+7
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+8
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+9
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+10
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+11
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+12
+ echo 1
+ echo p kernel_clone+13
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+14
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+15
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+16
+ echo 1
+ echo p kernel_clone+17
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+18
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+19
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+20
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+21
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+22
+ echo 1
+ echo p kernel_clone+23
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+24
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+25
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+26
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+27
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+28
+ echo 1
+ echo p kernel_clone+29
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+30
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+31
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+32
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+33
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+34
+ echo 1
+ echo p kernel_clone+35
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+36
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+37
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+38
+ echo 1
+ echo p kernel_clone+39
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+40
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+41
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+42
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+43
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+44
+ echo 1
+ echo p kernel_clone+45
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+46
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+47
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+48
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+49
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+50
+ echo 1
+ echo p kernel_clone+51
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+52
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+53
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+54
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+55
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+56
+ echo 1
+ echo p kernel_clone+57
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+58
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+59
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+60
+ echo 1
+ echo p kernel_clone+61
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+62
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+63
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+64
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+65
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+66
+ echo 1
+ echo p kernel_clone+67
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+68
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+69
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+70
+ echo 1
+ echo p kernel_clone+71
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+72
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+73
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+74
+ echo 1
+ echo p kernel_clone+75
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+76
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+77
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+78
+ echo 1
+ echo p kernel_clone+79
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+80
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+81
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+82
+ echo 1
+ echo p kernel_clone+83
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+84
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+85
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+86
+ echo 1
+ echo p kernel_clone+87
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+88
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+89
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+90
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+91
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+92
+ echo 1
+ echo p kernel_clone+93
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+94
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+95
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+96
+ echo 1
+ echo p kernel_clone+97
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+98
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+99
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+100
+ echo 1
+ echo p kernel_clone+101
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+102
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+103
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+104
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+105
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+106
+ echo 1
+ echo p kernel_clone+107
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+108
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+109
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+110
+ echo 1
+ echo p kernel_clone+111
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+112
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+113
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+114
+ echo 1
+ echo p kernel_clone+115
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+116
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+117
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+118
+ echo 1
+ echo p kernel_clone+119
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+120
+ echo 1
+ echo p kernel_clone+121
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+122
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+123
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+124
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+125
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+126
+ echo 1
+ echo p kernel_clone+127
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+128
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+129
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+130
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+131
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+132
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+133
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+134
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+135
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+136
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+137
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+138
+ echo 1
+ echo p kernel_clone+139
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+140
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+141
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+142
+ echo 1
+ echo p kernel_clone+143
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+144
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+145
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+146
+ echo 1
+ echo p kernel_clone+147
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+148
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+149
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+150
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+151
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+152
+ echo 1
+ echo p kernel_clone+153
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+154
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+155
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+156
+ echo 1
+ echo p kernel_clone+157
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+158
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+159
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+160
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+161
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+162
+ echo 1
+ echo p kernel_clone+163
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+164
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+165
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+166
+ echo 1
+ echo p kernel_clone+167
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+168
+ echo 1
+ echo p kernel_clone+169
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+170
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+171
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+172
+ echo 1
+ echo p kernel_clone+173
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+174
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+175
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+176
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+177
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+178
+ echo 1
+ echo p kernel_clone+179
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+180
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+181
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+182
+ echo 1
+ echo p kernel_clone+183
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+184
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+185
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+186
+ echo 1
+ echo p kernel_clone+187
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+188
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+189
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+190
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+191
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+192
+ echo 1
+ echo p kernel_clone+193
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+194
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+195
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+196
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+197
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+198
+ echo 1
+ echo p kernel_clone+199
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+200
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+201
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+202
+ echo 1
+ echo p kernel_clone+203
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+204
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+205
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+206
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+207
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+208
+ echo 1
+ echo p kernel_clone+209
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+210
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+211
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+212
+ echo 1
+ echo p kernel_clone+213
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+214
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+215
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+216
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+217
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+218
+ echo 1
+ echo p kernel_clone+219
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+220
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+221
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+222
+ echo 1
+ echo p kernel_clone+223
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+224
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+225
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+226
+ echo 1
+ echo p kernel_clone+227
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+228
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+229
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+230
+ echo 1
+ echo p kernel_clone+231
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+232
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+233
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+234
+ echo 1
+ echo p kernel_clone+235
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+236
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+237
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+238
+ echo 1
+ echo p kernel_clone+239
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+240
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+241
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+242
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+243
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+244
+ echo 1
+ echo p kernel_clone+245
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+246
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+247
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+248
+ echo 1
+ echo p kernel_clone+249
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+250
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+251
./ftracetest: 8: echo: echo: I/O error
+ continue
+ echo p kernel_clone+252
>> I've investigating cause of line, then I find this line.
>>
>> for i in `seq 0 255`; do
>> echo p $FUNCTION_FORK+${i} >> kprobe_events || continue
>> done
>
> I think this is fixed with:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v6.10-rc5&id=c239c83ed5c558be3b5926c7f11639f02c8acd00
>
> Regards,
> Sven
Hi, Sven, Oh thank you so much for the link.
I have also checked that the patch(c239c83ed5c55) you mentioned has been
applied to the code.
It looks like have another problem.
This is the whole call trace.
[ 726.069032] Kernel panic - not syncing: Corrupt kernel stack, can't
continue.
[ 726.069297] CPU: 0 PID: 1023 Comm: ftracetest Not tainted 6.10.0-rc5+ #30
[ 726.069335] Hardware name: QEMU 8561 QEMU (KVM/Linux)
[ 726.069361] Call Trace:
[ 726.069383] [<000003ffe0dc545e>] dump_stack_lvl+0x76/0xa8
[ 726.069446] [<000003ffe0db7170>] panic+0x170/0x358
[ 726.069485] [<000003ffe0000f16>] kernel_stack_overflow+0x56/0x60
[ 726.069522] [<0000000000000200>] 0x200
[ 726.069987] Dumping ftrace buffer:
[ 726.070144] ---------------------------------
[ 726.071356] <...>-1023 0..... 1829038us : p_kernel_clone_0:
(kernel_clone+0x0/0x390)
[ 726.071452] <...>-1023 0dN... 1829717us : p_kernel_clone_6:
(kernel_clone+0x6/0x390)
[ 726.071506] <...>-1023 0dN... 1830009us : p_kernel_clone_12:
(kernel_clone+0xc/0x390)
[ 726.071558] <...>-1023 0dN... 1830064us : p_kernel_clone_16:
(kernel_clone+0x10/0x390)
[ 726.071610] <...>-1023 0dN... 1830116us : p_kernel_clone_22:
(kernel_clone+0x16/0x390)
[ 726.071661] <...>-1023 0dN... 1830161us : p_kernel_clone_28:
(kernel_clone+0x1c/0x390)
[ 726.071711] <...>-1023 0dN... 1830204us : p_kernel_clone_34:
(kernel_clone+0x22/0x390)
[ 726.071762] <...>-1023 0dN... 1830252us : p_kernel_clone_38:
(kernel_clone+0x26/0x390)
[ 726.071815] <...>-1023 0dN... 1830301us : p_kernel_clone_44:
(kernel_clone+0x2c/0x390)
[ 726.071866] <...>-1023 0dN... 1830355us : p_kernel_clone_50:
(kernel_clone+0x32/0x390)
[ 726.071917] <...>-1023 0dN... 1830399us : p_kernel_clone_56:
(kernel_clone+0x38/0x390)
[ 726.071967] <...>-1023 0dN... 1830441us : p_kernel_clone_60:
(kernel_clone+0x3c/0x390)
[ 726.072018] <...>-1023 0dN... 1830495us : p_kernel_clone_66:
(kernel_clone+0x42/0x390)
[ 726.072068] <...>-1023 0dN... 1830553us : p_kernel_clone_70:
(kernel_clone+0x46/0x390)
[ 726.072118] <...>-1023 0dN... 1830597us : p_kernel_clone_74:
(kernel_clone+0x4a/0x390)
[ 726.072168] <...>-1023 0dN... 1830639us : p_kernel_clone_78:
(kernel_clone+0x4e/0x390)
[ 726.072217] <...>-1023 0dN... 1830681us : p_kernel_clone_82:
(kernel_clone+0x52/0x390)
[ 726.072267] <...>-1023 0dN... 1830722us : p_kernel_clone_86:
(kernel_clone+0x56/0x390)
[ 726.072318] <...>-1023 0dN... 1830765us : p_kernel_clone_92:
(kernel_clone+0x5c/0x390)
[ 726.072368] <...>-1023 0dN... 1830806us : p_kernel_clone_96:
(kernel_clone+0x60/0x390)
[ 726.072418] <...>-1023 0dN... 1830847us : p_kernel_clone_100:
(kernel_clone+0x64/0x390)
[ 726.072468] <...>-1023 0dN... 1830911us : p_kernel_clone_106:
(kernel_clone+0x6a/0x390)
[ 726.072518] <...>-1023 0dN... 1830953us : p_kernel_clone_110:
(kernel_clone+0x6e/0x390)
[ 726.072568] <...>-1023 0dN... 1830994us : p_kernel_clone_114:
(kernel_clone+0x72/0x390)
[ 726.072619] <...>-1023 0dN... 1831035us : p_kernel_clone_118:
(kernel_clone+0x76/0x390)
[ 726.072669] <...>-1023 0dN... 1831092us : p_kernel_clone_120:
(kernel_clone+0x78/0x390)
[ 726.072718] <...>-1023 0dN... 1831133us : p_kernel_clone_126:
(kernel_clone+0x7e/0x390)
[ 726.072769] <...>-1023 0dN... 1831175us : p_kernel_clone_138:
(kernel_clone+0x8a/0x390)
[ 726.072819] <...>-1023 0dN... 1831221us : p_kernel_clone_142:
(kernel_clone+0x8e/0x390)
[ 726.072869] <...>-1023 0dN... 1831262us : p_kernel_clone_146:
(kernel_clone+0x92/0x390)
[ 726.072919] <...>-1023 0dN... 1831303us : p_kernel_clone_152:
(kernel_clone+0x98/0x390)
[ 726.072970] <...>-1023 0dN... 1831344us : p_kernel_clone_156:
(kernel_clone+0x9c/0x390)
[ 726.073020] <...>-1023 0dN... 1831385us : p_kernel_clone_162:
(kernel_clone+0xa2/0x390)
[ 726.073070] <...>-1023 0dN... 1831427us : p_kernel_clone_166:
(kernel_clone+0xa6/0x390)
[ 726.073120] <...>-1023 0dN... 1831469us : p_kernel_clone_168:
(kernel_clone+0xa8/0x390)
[ 726.073910] <...>-1023 0dN... 1831511us : p_kernel_clone_172:
(kernel_clone+0xac/0x390)
[ 726.073983] <...>-1023 0dN... 1831554us : p_kernel_clone_178:
(kernel_clone+0xb2/0x390)
[ 726.074047] <...>-1023 0dN... 1831601us : p_kernel_clone_182:
(kernel_clone+0xb6/0x390)
[ 726.074109] <...>-1023 0dN... 1831642us : p_kernel_clone_186:
(kernel_clone+0xba/0x390)
[ 726.074172] <...>-1023 0dN... 1831683us : p_kernel_clone_192:
(kernel_clone+0xc0/0x390)
[ 726.074234] <...>-1023 0dN... 1831725us : p_kernel_clone_198:
(kernel_clone+0xc6/0x390)
[ 726.074296] <...>-1023 0dN... 1831766us : p_kernel_clone_202:
(kernel_clone+0xca/0x390)
[ 726.074358] <...>-1023 0dN... 1831807us : p_kernel_clone_208:
(kernel_clone+0xd0/0x390)
[ 726.074419] <...>-1023 0dN... 1831848us : p_kernel_clone_212:
(kernel_clone+0xd4/0x390)
[ 726.074481] <...>-1023 0dN... 1831895us : p_kernel_clone_218:
(kernel_clone+0xda/0x390)
[ 726.074542] <...>-1023 0dN... 1831941us : p_kernel_clone_222:
(kernel_clone+0xde/0x390)
[ 726.074603] <...>-1023 0dN... 1831982us : p_kernel_clone_226:
(kernel_clone+0xe2/0x390)
[ 726.074665] <...>-1023 0dN... 1832023us : p_kernel_clone_230:
(kernel_clone+0xe6/0x390)
[ 726.074727] <...>-1023 0dN... 1832064us : p_kernel_clone_234:
(kernel_clone+0xea/0x390)
[ 726.074787] <...>-1023 0dN... 1832105us : p_kernel_clone_238:
(kernel_clone+0xee/0x390)
[ 726.074846] <...>-1023 0dN... 1832146us : p_kernel_clone_244:
(kernel_clone+0xf4/0x390)
[ 726.074906] <...>-1023 0dN... 1832188us : p_kernel_clone_248:
(kernel_clone+0xf8/0x390)
[ 726.074965] <...>-1023 0dN... 1832229us : p_kernel_clone_252:
(kernel_clone+0xfc/0x390)
[ 726.075024] <...>-1023 0dN... 1832286us : p_kernel_clone_74:
(kernel_clone+0x4a/0x390)
[ 726.075083] <...>-1023 0dN... 1832324us : p_kernel_clone_78:
(kernel_clone+0x4e/0x390)
[ 726.075142] <...>-1023 0dN... 1832361us : p_kernel_clone_82:
(kernel_clone+0x52/0x390)
[ 726.075200] <...>-1023 0dN... 1832398us : p_kernel_clone_86:
(kernel_clone+0x56/0x390)
[ 726.075259] <...>-1023 0dN... 1832436us : p_kernel_clone_92:
(kernel_clone+0x5c/0x390)
[ 726.075317] <...>-1023 0dN... 1832472us : p_kernel_clone_96:
(kernel_clone+0x60/0x390)
[ 726.075376] <...>-1023 0dN... 1832510us : p_kernel_clone_100:
(kernel_clone+0x64/0x390)
[ 726.075434] <...>-1023 0dN... 1832547us : p_kernel_clone_106:
(kernel_clone+0x6a/0x390)
[ 726.075493] <...>-1023 0dN... 1832584us : p_kernel_clone_110:
(kernel_clone+0x6e/0x390)
[ 726.075552] <...>-1023 0dN... 1832620us : p_kernel_clone_114:
(kernel_clone+0x72/0x390)
[ 726.075611] <...>-1023 0dN... 1832657us : p_kernel_clone_118:
(kernel_clone+0x76/0x390)
[ 726.075669] <...>-1023 0dN... 1832694us : p_kernel_clone_120:
(kernel_clone+0x78/0x390)
[ 726.075734] <...>-1023 0dN... 1832731us : p_kernel_clone_126:
(kernel_clone+0x7e/0x390)
[ 726.075793] <...>-1023 0dN... 1832768us : p_kernel_clone_138:
(kernel_clone+0x8a/0x390)
[ 726.075860] <...>-1023 0dN... 1832805us : p_kernel_clone_142:
(kernel_clone+0x8e/0x390)
[ 726.075922] <...>-1023 0dN... 1832848us : p_kernel_clone_146:
(kernel_clone+0x92/0x390)
[ 726.075980] <...>-1023 0dN... 1832885us : p_kernel_clone_152:
(kernel_clone+0x98/0x390)
[ 726.076046] <...>-1023 0dN... 1832922us : p_kernel_clone_156:
(kernel_clone+0x9c/0x390)
[ 726.076107] <...>-1023 0dN... 1832959us : p_kernel_clone_162:
(kernel_clone+0xa2/0x390)
[ 726.076168] <...>-1023 0dN... 1832996us : p_kernel_clone_166:
(kernel_clone+0xa6/0x390)
[ 726.076229] <...>-1023 0dN... 1833038us : p_kernel_clone_168:
(kernel_clone+0xa8/0x390)
[ 726.076290] <...>-1023 0dN... 1833078us : p_kernel_clone_172:
(kernel_clone+0xac/0x390)
[ 726.076376] <...>-1023 0dN... 1833117us : p_kernel_clone_178:
(kernel_clone+0xb2/0x390)
[ 726.076438] <...>-1023 0dN... 1833154us : p_kernel_clone_182:
(kernel_clone+0xb6/0x390)
[ 726.076505] <...>-1023 0dN... 1833190us : p_kernel_clone_186:
(kernel_clone+0xba/0x390)
[ 726.076565] <...>-1023 0dN... 1833229us : p_kernel_clone_192:
(kernel_clone+0xc0/0x390)
[ 726.076634] <...>-1023 0dN... 1833266us : p_kernel_clone_198:
(kernel_clone+0xc6/0x390)
[ 726.076695] <...>-1023 0dN... 1833302us : p_kernel_clone_202:
(kernel_clone+0xca/0x390)
[ 726.076779] <...>-1023 0dN... 1833339us : p_kernel_clone_208:
(kernel_clone+0xd0/0x390)
[ 726.076842] <...>-1023 0dN... 1833376us : p_kernel_clone_212:
(kernel_clone+0xd4/0x390)
[ 726.076886] ---------------------------------
Thank you Steve, Masami and Sven for the great advice on what I reported.
If there are any additional aspects I should check or any advice,
please let me know and I will look into it.
Warm regards,
Yunseong Kim
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2024-06-27 21:44 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-06-27 5:03 selftests/ftrace kprobe_eventname test fails on s390x QEMU (KVM/Linux) Yunseong Kim
2024-06-27 14:05 ` Steven Rostedt
2024-06-27 14:34 ` Masami Hiramatsu
2024-06-27 16:15 ` Sven Schnelle
2024-06-27 21:44 ` Yunseong Kim
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).