linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [BUG][6.15][perf] Kernel panic not syncing: Fatal exception in interrupt
@ 2025-06-01 17:36 David Wang
  0 siblings, 0 replies; 7+ messages in thread
From: David Wang @ 2025-06-01 17:36 UTC (permalink / raw)
  To: peterz, mingo, acme, namhyung, mingo, yeoreum.yun, leo.yan
  Cc: linux-perf-users, linux-kernel, David Wang

Hi,

Caught a kernel panic when rebooting, system stuck until pressing power button.
I have only a screenshot when it happens, following logs were extracted from a
captured picture.

863.881960] sysved_call_function_sing le+0x4c/0xc0
863.881301] asm_sysvec_call_function_single+0x16/0x20
869.881344] RIP: 0633:0x7f9alcea3367
663.681373] Code: 00 66 99 b8 ff ff ff ff c3 66 ....
863.881524] RSP: 002b:00007fffa526fcf8 EFLAGS: 00000246
869.881567] RAX: 0000562060c962d0 RBX: 0000000000000002 RCX: 00007f9a1cff1c60
863.881625] RDX: 00007f9a0c000030 RSI: 00007f9alcff1c60 RDI: 00007f9a1ca91c20
863.081682] RBP: 0000000000000001 R08: 0000000000000000 R09: 00007f9a1d6217a0
869.881740] R10: 00007f9alca91c10 R11: 0000000000000246 R12: 00007f9a1d70c020
869.881798] R13: 00007fffa5270030 R14: 00007fffa526fd00 R15: 0000000000000000
863.881860] </TASK>
863.881876) Modules linked in: snd_seq_dummy (E) snd_hrtimer (E)...
...
863.887142] button (E)
863.912127] CR2: ffffe4afcc079650
863.914593] --- [ end trace 0000000000000000 1--
864.042750] RIP: 0010:ctx_sched_out+0x1ce/0x210
864.045214] Code: 89 c6 4c 8b b9 de 00 00 00 48 ...
864.050343] RSP: 0000:ffffaa4ec0f3fe60 EFLAGS: 00010086
864.052929] RAX: 0000000000000002 RBX: ffff8e8eeed2a580 RCX: ffff8e8bded9bf00
864.055518] RDX: 000000c92340b051 RSI: 000000c92340b051 RDI: ffff
864.058093] RBP: 0000000000000000 R08: 0000000000000002 R09: 00
864.060654] R10: 0000000000000000 R11: 0000000000000000 R12: 000
864.063183] R13: ffff8e8eeed2a580 R14: 0000000000000007 R15: ffffe4afcc079650
864.065729] FS: 00007f9a1ca91940 (0000) GS:ffff8e8f6b1c3000(0000) knIGS:0000000000000000
864.068312] CS: 0010 DS: 0000 ES: 0000 CRO: 0000000080050033
864.070898] CR2: ffffe4afcc079650 CR3: 00000001136d8000 CR4: 0000000000350ef0
864.673523] Kernel panic - not syncing: Fatal exception in interrupt
864.076410] Kernel Offset: 0xc00000 from 0xffffffff81000000 (relocation range: 0xff
864.205401] --- [ end Kernel panic - not syncing: Fatal exception in interrupt ]---

This happens ever since 6.15-rc1, from time to time, I would get kernel panic when
reboot; it is only recently that I figured out a precedure reproducing
this with *high* probability:

1. create a cgroup.
2. perf_event_open(PERF_FLAG_FD_CLOEXEC|PERF_FLAG_PID_CGROUP) for each cpu with following attrs:
	attr.type = PERF_TYPE_SOFTWARE;
	attr.size = sizeof(attr);
	attr.config = PERF_COUNT_SW_CPU_CLOCK;
	attr.sample_freq = 9999;
	attr.freq = 1;
	attr.wakeup_events = 16;
	attr.sample_type = PERF_SAMPLE_CALLCHAIN;
	attr.sample_max_stack = 32;
	attr.exclude_callchain_user = 1;
3. close all perf_event_open after several minutes
4. reboot

And after an exhausting bisect on events/core.c, (I need 5 rounds to conclude a good bisect)
I think I reach the conclusion, with very high probability, that this is caused by

commit a3c3c66670cee11eb13aa43905904bf29cb92d32
Author: Yeoreum Yun <yeoreum.yun@arm.com>
Date:   Wed Mar 26 08:20:03 2025 +0000

    perf/core: Fix child_total_time_enabled accounting bug at task exit

Reverting this can fix it: I run the test 10 rounds, no kernel panic observed.

The changes made to __perf_remove_from_context by commit a3c3c6667("perf/core:
Fix child_total_time_enabled accounting bug at task exit") has wider effect
than the callchain mentioned in commit message, and I think an esay fix would
be just restricting the effect to that callchain only, and restore other changes back.

I have test the patch below several rounds, and so far so good, and I will have
more tests on it.

Signed-off-by: David Wang <00107082@163.com>
---
 kernel/events/core.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/kernel/events/core.c b/kernel/events/core.c
index 95e703891b24..95d15105e0e7 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -2482,12 +2482,15 @@ __perf_remove_from_context(struct perf_event *event,
 		state = PERF_EVENT_STATE_DEAD;
 	}
 	event_sched_out(event, ctx);
-	perf_event_set_state(event, min(event->state, state));
+	if (flags & DETACH_EXIT)
+		perf_event_set_state(event, min(event->state, state));
 	if (flags & DETACH_GROUP)
 		perf_group_detach(event);
 	if (flags & DETACH_CHILD)
 		perf_child_detach(event);
 	list_del_event(event, ctx);
+	if (flags & DETACH_DEAD)
+		event->state = PERF_EVENT_STATE_DEAD;
 
 	if (!pmu_ctx->nr_events) {
 		pmu_ctx->rotate_necessary = 0;
-- 
2.39.2




Thanks
David


^ permalink raw reply related	[flat|nested] 7+ messages in thread

* Re: [BUG][6.15][perf] Kernel panic not syncing: Fatal exception in interrupt
@ 2025-06-02 12:06 Yeoreum Yun
  2025-06-02 12:33 ` David Wang
  2025-06-02 13:19 ` Yeoreum Yun
  0 siblings, 2 replies; 7+ messages in thread
From: Yeoreum Yun @ 2025-06-02 12:06 UTC (permalink / raw)
  To: 00107082
  Cc: peterz, mingo, acme, namhyung, mingo, yeoreum.yun, leo.yan,
	linux-perf-users, linux-kernel

Sorry to make noise all.
I've forgotten to cc mailing list.
If you receive duplicate mail, Sorry again...

====================================================
Hi David,

> Hi,
>
> Caught a kernel panic when rebooting, system stuck until pressing power button.
> I have only a screenshot when it happens, following logs were extracted from a
> captured picture.
>
> 863.881960] sysved_call_function_sing le+0x4c/0xc0
> 863.881301] asm_sysvec_call_function_single+0x16/0x20
> 869.881344] RIP: 0633:0x7f9alcea3367
> 663.681373] Code: 00 66 99 b8 ff ff ff ff c3 66 ....
> 863.881524] RSP: 002b:00007fffa526fcf8 EFLAGS: 00000246
> 869.881567] RAX: 0000562060c962d0 RBX: 0000000000000002 RCX: 00007f9a1cff1c60
> 863.881625] RDX: 00007f9a0c000030 RSI: 00007f9alcff1c60 RDI: 00007f9a1ca91c20
> 863.081682] RBP: 0000000000000001 R08: 0000000000000000 R09: 00007f9a1d6217a0
> 869.881740] R10: 00007f9alca91c10 R11: 0000000000000246 R12: 00007f9a1d70c020
> 869.881798] R13: 00007fffa5270030 R14: 00007fffa526fd00 R15: 0000000000000000
> 863.881860] </TASK>
> 863.881876) Modules linked in: snd_seq_dummy (E) snd_hrtimer (E)...
> ...
> 863.887142] button (E)
> 863.912127] CR2: ffffe4afcc079650
> 863.914593] --- [ end trace 0000000000000000 1--
> 864.042750] RIP: 0010:ctx_sched_out+0x1ce/0x210
> 864.045214] Code: 89 c6 4c 8b b9 de 00 00 00 48 ...
> 864.050343] RSP: 0000:ffffaa4ec0f3fe60 EFLAGS: 00010086
> 864.052929] RAX: 0000000000000002 RBX: ffff8e8eeed2a580 RCX: ffff8e8bded9bf00
> 864.055518] RDX: 000000c92340b051 RSI: 000000c92340b051 RDI: ffff
> 864.058093] RBP: 0000000000000000 R08: 0000000000000002 R09: 00
> 864.060654] R10: 0000000000000000 R11: 0000000000000000 R12: 000
> 864.063183] R13: ffff8e8eeed2a580 R14: 0000000000000007 R15: ffffe4afcc079650
> 864.065729] FS: 00007f9a1ca91940 (0000) GS:ffff8e8f6b1c3000(0000) knIGS:0000000000000000
> 864.068312] CS: 0010 DS: 0000 ES: 0000 CRO: 0000000080050033
> 864.070898] CR2: ffffe4afcc079650 CR3: 00000001136d8000 CR4: 0000000000350ef0
> 864.673523] Kernel panic - not syncing: Fatal exception in interrupt
> 864.076410] Kernel Offset: 0xc00000 from 0xffffffff81000000 (relocation range: 0xff
> 864.205401] --- [ end Kernel panic - not syncing: Fatal exception in interrupt ]---
>
> This happens ever since 6.15-rc1, from time to time, I would get kernel panic when
> reboot; it is only recently that I figured out a precedure reproducing
> this with *high* probability:
>
> 1. create a cgroup.
> 2. perf_event_open(PERF_FLAG_FD_CLOEXEC|PERF_FLAG_PID_CGROUP) for each cpu with following attrs:
> 	attr.type = PERF_TYPE_SOFTWARE;
> 	attr.size = sizeof(attr);
> 	attr.config = PERF_COUNT_SW_CPU_CLOCK;
> 	attr.sample_freq = 9999;
> 	attr.freq = 1;
> 	attr.wakeup_events = 16;
> 	attr.sample_type = PERF_SAMPLE_CALLCHAIN;
> 	attr.sample_max_stack = 32;
> 	attr.exclude_callchain_user = 1;
> 3. close all perf_event_open after several minutes
> 4. reboot
>
> And after an exhausting bisect on events/core.c, (I need 5 rounds to conclude a good bisect)
> I think I reach the conclusion, with very high probability, that this is caused by
>
> commit a3c3c66670cee11eb13aa43905904bf29cb92d32
> Author: Yeoreum Yun <yeoreum.yun@arm.com>
> Date:   Wed Mar 26 08:20:03 2025 +0000
>
>    perf/core: Fix child_total_time_enabled accounting bug at task exit
>
> Reverting this can fix it: I run the test 10 rounds, no kernel panic observed.
>
> The changes made to __perf_remove_from_context by commit a3c3c6667("perf/core:
> Fix child_total_time_enabled accounting bug at task exit") has wider effect
> than the callchain mentioned in commit message, and I think an esay fix would
> be just restricting the effect to that callchain only, and restore other changes back.
>
> I have test the patch below several rounds, and so far so good, and I will have
> more tests on it.
>
> Signed-off-by: David Wang <00107082@163.com>

Thanks for your reporting and Sorry for my bad.
By my change, the tracking nr_cgorups is broken which could make a dangling
pointer for cpuctx->cgrp.

Could you test with below change please?

diff --git a/kernel/events/core.c b/kernel/events/core.c
index 95e703891b24..d0a9096735b9 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -2116,18 +2116,6 @@ list_del_event(struct perf_event *event, struct perf_event_context *ctx)
        if (event->group_leader == event)
                del_event_from_groups(event, ctx);

-       /*
-        * If event was in error state, then keep it
-        * that way, otherwise bogus counts will be
-        * returned on read(). The only way to get out
-        * of error state is by explicit re-enabling
-        * of the event
-        */
-       if (event->state > PERF_EVENT_STATE_OFF) {
-               perf_cgroup_event_disable(event, ctx);
-               perf_event_set_state(event, PERF_EVENT_STATE_OFF);
-       }
-
        ctx->generation++;
        event->pmu_ctx->nr_events--;
 }
@@ -2471,6 +2459,16 @@ __perf_remove_from_context(struct perf_event *event,

        ctx_time_update(cpuctx, ctx);

+       /*
+        * If event was in error state, then keep it
+        * that way, otherwise bogus counts will be
+        * returned on read(). The only way to get out
+        * of error state is by explicit re-enabling
+        * of the event
+        */
+       if (event->state > PERF_EVENT_STATE_OFF)
+               perf_cgroup_event_disable(event, ctx);
+
        /*
         * Ensure event_sched_out() switches to OFF, at the very least
         * this avoids raising perf_pending_task() at this time.

Thanks


--
Sincerely,
Yeoreum Yun

^ permalink raw reply related	[flat|nested] 7+ messages in thread

* Re: [BUG][6.15][perf] Kernel panic not syncing: Fatal exception in interrupt
  2025-06-02 12:06 [BUG][6.15][perf] Kernel panic not syncing: Fatal exception in interrupt Yeoreum Yun
@ 2025-06-02 12:33 ` David Wang
  2025-06-02 13:19 ` Yeoreum Yun
  1 sibling, 0 replies; 7+ messages in thread
From: David Wang @ 2025-06-02 12:33 UTC (permalink / raw)
  To: yeoreum.yun
  Cc: peterz, mingo, acme, namhyung, mingo, yeoreum.yun, leo.yan,
	linux-perf-users, linux-kernel



At 2025-06-02 20:06:10, "Yeoreum Yun" <> wrote:
>Sorry to make noise all.
>I've forgotten to cc mailing list.
>If you receive duplicate mail, Sorry again...
>
>====================================================
>Hi David,
>
>> Hi,
>>
>> Caught a kernel panic when rebooting, system stuck until pressing power button.
>> I have only a screenshot when it happens, following logs were extracted from a
>> captured picture.
>>
>> 863.881960] sysved_call_function_sing le+0x4c/0xc0
>> 863.881301] asm_sysvec_call_function_single+0x16/0x20
>> 869.881344] RIP: 0633:0x7f9alcea3367
>> 663.681373] Code: 00 66 99 b8 ff ff ff ff c3 66 ....
>> 863.881524] RSP: 002b:00007fffa526fcf8 EFLAGS: 00000246
>> 869.881567] RAX: 0000562060c962d0 RBX: 0000000000000002 RCX: 00007f9a1cff1c60
>> 863.881625] RDX: 00007f9a0c000030 RSI: 00007f9alcff1c60 RDI: 00007f9a1ca91c20
>> 863.081682] RBP: 0000000000000001 R08: 0000000000000000 R09: 00007f9a1d6217a0
>> 869.881740] R10: 00007f9alca91c10 R11: 0000000000000246 R12: 00007f9a1d70c020
>> 869.881798] R13: 00007fffa5270030 R14: 00007fffa526fd00 R15: 0000000000000000
>> 863.881860] </TASK>
>> 863.881876) Modules linked in: snd_seq_dummy (E) snd_hrtimer (E)...
>> ...
>> 863.887142] button (E)
>> 863.912127] CR2: ffffe4afcc079650
>> 863.914593] --- [ end trace 0000000000000000 1--
>> 864.042750] RIP: 0010:ctx_sched_out+0x1ce/0x210
>> 864.045214] Code: 89 c6 4c 8b b9 de 00 00 00 48 ...
>> 864.050343] RSP: 0000:ffffaa4ec0f3fe60 EFLAGS: 00010086
>> 864.052929] RAX: 0000000000000002 RBX: ffff8e8eeed2a580 RCX: ffff8e8bded9bf00
>> 864.055518] RDX: 000000c92340b051 RSI: 000000c92340b051 RDI: ffff
>> 864.058093] RBP: 0000000000000000 R08: 0000000000000002 R09: 00
>> 864.060654] R10: 0000000000000000 R11: 0000000000000000 R12: 000
>> 864.063183] R13: ffff8e8eeed2a580 R14: 0000000000000007 R15: ffffe4afcc079650
>> 864.065729] FS: 00007f9a1ca91940 (0000) GS:ffff8e8f6b1c3000(0000) knIGS:0000000000000000
>> 864.068312] CS: 0010 DS: 0000 ES: 0000 CRO: 0000000080050033
>> 864.070898] CR2: ffffe4afcc079650 CR3: 00000001136d8000 CR4: 0000000000350ef0
>> 864.673523] Kernel panic - not syncing: Fatal exception in interrupt
>> 864.076410] Kernel Offset: 0xc00000 from 0xffffffff81000000 (relocation range: 0xff
>> 864.205401] --- [ end Kernel panic - not syncing: Fatal exception in interrupt ]---
>>
>> This happens ever since 6.15-rc1, from time to time, I would get kernel panic when
>> reboot; it is only recently that I figured out a precedure reproducing
>> this with *high* probability:
>>
>> 1. create a cgroup.
>> 2. perf_event_open(PERF_FLAG_FD_CLOEXEC|PERF_FLAG_PID_CGROUP) for each cpu with following attrs:
>> 	attr.type = PERF_TYPE_SOFTWARE;
>> 	attr.size = sizeof(attr);
>> 	attr.config = PERF_COUNT_SW_CPU_CLOCK;
>> 	attr.sample_freq = 9999;
>> 	attr.freq = 1;
>> 	attr.wakeup_events = 16;
>> 	attr.sample_type = PERF_SAMPLE_CALLCHAIN;
>> 	attr.sample_max_stack = 32;
>> 	attr.exclude_callchain_user = 1;
>> 3. close all perf_event_open after several minutes
>> 4. reboot
>>
>> And after an exhausting bisect on events/core.c, (I need 5 rounds to conclude a good bisect)
>> I think I reach the conclusion, with very high probability, that this is caused by
>>
>> commit a3c3c66670cee11eb13aa43905904bf29cb92d32
>> Author: Yeoreum Yun <yeoreum.yun@arm.com>
>> Date:   Wed Mar 26 08:20:03 2025 +0000
>>
>>    perf/core: Fix child_total_time_enabled accounting bug at task exit
>>
>> Reverting this can fix it: I run the test 10 rounds, no kernel panic observed.
>>
>> The changes made to __perf_remove_from_context by commit a3c3c6667("perf/core:
>> Fix child_total_time_enabled accounting bug at task exit") has wider effect
>> than the callchain mentioned in commit message, and I think an esay fix would
>> be just restricting the effect to that callchain only, and restore other changes back.
>>
>> I have test the patch below several rounds, and so far so good, and I will have
>> more tests on it.
>>
>> Signed-off-by: David Wang <00107082@163.com>
>
>Thanks for your reporting and Sorry for my bad.
>By my change, the tracking nr_cgorups is broken which could make a dangling
>pointer for cpuctx->cgrp.
>
>Could you test with below change please?
>
>diff --git a/kernel/events/core.c b/kernel/events/core.c
>index 95e703891b24..d0a9096735b9 100644
>--- a/kernel/events/core.c
>+++ b/kernel/events/core.c
>@@ -2116,18 +2116,6 @@ list_del_event(struct perf_event *event, struct perf_event_context *ctx)
>        if (event->group_leader == event)
>                del_event_from_groups(event, ctx);
>
>-       /*
>-        * If event was in error state, then keep it
>-        * that way, otherwise bogus counts will be
>-        * returned on read(). The only way to get out
>-        * of error state is by explicit re-enabling
>-        * of the event
>-        */
>-       if (event->state > PERF_EVENT_STATE_OFF) {
>-               perf_cgroup_event_disable(event, ctx);
>-               perf_event_set_state(event, PERF_EVENT_STATE_OFF);
>-       }
>-
>        ctx->generation++;
>        event->pmu_ctx->nr_events--;
> }
>@@ -2471,6 +2459,16 @@ __perf_remove_from_context(struct perf_event *event,
>
>        ctx_time_update(cpuctx, ctx);
>
>+       /*
>+        * If event was in error state, then keep it
>+        * that way, otherwise bogus counts will be
>+        * returned on read(). The only way to get out
>+        * of error state is by explicit re-enabling
>+        * of the event
>+        */
>+       if (event->state > PERF_EVENT_STATE_OFF)
>+               perf_cgroup_event_disable(event, ctx);
>+
>        /*
>         * Ensure event_sched_out() switches to OFF, at the very least
>         * this avoids raising perf_pending_task() at this time.
>
>Thanks
>
>
>--
>Sincerely,
>Yeoreum Yun

Before I start testing, I feel concerned about following chain:

./kernel/fork.c:
bad_fork_cleanup_perf:
    perf_event_free_task() 
        perf_free_event()
            list_del_event()

This patch seems changes the behavior in this callchain.
Would this have other side-effect?


David

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [BUG][6.15][perf] Kernel panic not syncing: Fatal exception in interrupt
  2025-06-02 12:06 [BUG][6.15][perf] Kernel panic not syncing: Fatal exception in interrupt Yeoreum Yun
  2025-06-02 12:33 ` David Wang
@ 2025-06-02 13:19 ` Yeoreum Yun
  2025-06-02 13:31   ` David Wang
  2025-06-02 15:13   ` Yeoreum Yun
  1 sibling, 2 replies; 7+ messages in thread
From: Yeoreum Yun @ 2025-06-02 13:19 UTC (permalink / raw)
  To: 83b4d26.3362.19730a21115.Coremail.00107082
  Cc: 00107082, peterz, mingo, acme, namhyung, mingo, leo.yan,
	linux-perf-users, linux-kernel

Hi David,

> Before I start testing, I feel concerned about following chain:
>
> ./kernel/fork.c:
> bad_fork_cleanup_perf:
>     perf_event_free_task()
>         perf_free_event()
>             list_del_event()
>
> This patch seems changes the behavior in this callchain.
> Would this have other side-effect?

What behavior is changed you're worry about?
both error patch is handled by __perf_remove_from_context(),
There wouldn't be no problem since this patch just move the
time of disabling cgroup before changing event state.

also, the cgroup event is for only cpuctx not added in taskctx.
So, there's no effect for event attached in taskctx.

Thanks.

--
Sincerely,
Yeoreum Yun

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [BUG][6.15][perf] Kernel panic not syncing: Fatal exception in interrupt
  2025-06-02 13:19 ` Yeoreum Yun
@ 2025-06-02 13:31   ` David Wang
  2025-06-02 15:13   ` Yeoreum Yun
  1 sibling, 0 replies; 7+ messages in thread
From: David Wang @ 2025-06-02 13:31 UTC (permalink / raw)
  To: Yeoreum Yun
  Cc: peterz, mingo, acme, namhyung, mingo, leo.yan, linux-perf-users,
	linux-kernel


At 2025-06-02 21:19:11, "Yeoreum Yun" <yeoreum.yun@arm.com> wrote:
>Hi David,
>
>> Before I start testing, I feel concerned about following chain:
>>
>> ./kernel/fork.c:
>> bad_fork_cleanup_perf:
>>     perf_event_free_task()
>>         perf_free_event()
>>             list_del_event()
>>
>> This patch seems changes the behavior in this callchain.
>> Would this have other side-effect?
>
>What behavior is changed you're worry about?
>both error patch is handled by __perf_remove_from_context(),
>There wouldn't be no problem since this patch just move the
>time of disabling cgroup before changing event state.
>
>also, the cgroup event is for only cpuctx not added in taskctx.
>So, there's no effect for event attached in taskctx.
>
>Thanks.
>
>--
>Sincerely,
>Yeoreum Yun

Am I reading it wrong? 
The call chain I mentioned above dose not walk through __perf_remove_from_context,
It is a fail path in fork, which happens rarely, but still possible. I guess...

David


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [BUG][6.15][perf] Kernel panic not syncing: Fatal exception in interrupt
  2025-06-02 13:19 ` Yeoreum Yun
  2025-06-02 13:31   ` David Wang
@ 2025-06-02 15:13   ` Yeoreum Yun
  2025-06-02 15:32     ` David Wang
  1 sibling, 1 reply; 7+ messages in thread
From: Yeoreum Yun @ 2025-06-02 15:13 UTC (permalink / raw)
  To: 1f4a84e5.32ca.19730d781aa.Coremail.00107082
  Cc: 00107082, peterz, mingo, acme, namhyung, mingo, leo.yan,
	linux-perf-users, linux-kernel

Hi David,

> > > Before I start testing, I feel concerned about following chain:
> > >
> > > ./kernel/fork.c:
> > > bad_fork_cleanup_perf:
> > >     perf_event_free_task()
> > >         perf_free_event()
> > >             list_del_event()
> > >
> > > This patch seems changes the behavior in this callchain.
> > > Would this have other side-effect?
> >
> > What behavior is changed you're worry about?
> > both error patch is handled by __perf_remove_from_context(),
> > There wouldn't be no problem since this patch just move the
> > time of disabling cgroup before changing event state.
> >
> > also, the cgroup event is for only cpuctx not added in taskctx.
> > So, there's no effect for event attached in taskctx.
> >
> > Thanks.
>
> Am I reading it wrong?
> The call chain I mentioned above dose not walk through __perf_remove_from_context,
> It is a fail path in fork, which happens rarely, but still possible. I guess...

Since commit 90661365021a
("perf Unify perf_event_free_task() / perf_evenet_exit_task_context()")

perf_event_free_task() is integrated with perf_event_exit_task_context()
So, it calls __perf_remove_from_context().

In v6.15, I think you can test with below change only:
@@ -2471,6 +2459,16 @@ __perf_remove_from_context(struct perf_event *event,

        ctx_time_update(cpuctx, ctx);

+       /*
+        * If event was in error state, then keep it
+        * that way, otherwise bogus counts will be
+        * returned on read(). The only way to get out
+        * of error state is by explicit re-enabling
+        * of the event
+        */
+       if (event->state > PERF_EVENT_STATE_OFF)
+               perf_cgroup_event_disable(event, ctx);
+
        /*
         * Ensure event_sched_out() switches to OFF, at the very least
         * this avoids raising perf_pending_task() at this time.

not with modification with "list_del_event()".

Thanks
--
Sincerely,
Yeoreum Yun

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [BUG][6.15][perf] Kernel panic not syncing: Fatal exception in interrupt
  2025-06-02 15:13   ` Yeoreum Yun
@ 2025-06-02 15:32     ` David Wang
  0 siblings, 0 replies; 7+ messages in thread
From: David Wang @ 2025-06-02 15:32 UTC (permalink / raw)
  To: Yeoreum Yun
  Cc: peterz, mingo, acme, namhyung, mingo, leo.yan, linux-perf-users,
	linux-kernel


At 2025-06-02 23:13:10, "Yeoreum Yun" <yeoreum.yun@arm.com> wrote:
>Hi David,
>
>> > > Before I start testing, I feel concerned about following chain:
>> > >
>> > > ./kernel/fork.c:
>> > > bad_fork_cleanup_perf:
>> > >     perf_event_free_task()
>> > >         perf_free_event()
>> > >             list_del_event()
>> > >
>> > > This patch seems changes the behavior in this callchain.
>> > > Would this have other side-effect?
>> >
>> > What behavior is changed you're worry about?
>> > both error patch is handled by __perf_remove_from_context(),
>> > There wouldn't be no problem since this patch just move the
>> > time of disabling cgroup before changing event state.
>> >
>> > also, the cgroup event is for only cpuctx not added in taskctx.
>> > So, there's no effect for event attached in taskctx.
>> >
>> > Thanks.
>>
>> Am I reading it wrong?
>> The call chain I mentioned above dose not walk through __perf_remove_from_context,
>> It is a fail path in fork, which happens rarely, but still possible. I guess...
>
>Since commit 90661365021a
>("perf Unify perf_event_free_task() / perf_evenet_exit_task_context()")
>
>perf_event_free_task() is integrated with perf_event_exit_task_context()
>So, it calls __perf_remove_from_context().


Good to know~

>
>In v6.15, I think you can test with below change only:
>@@ -2471,6 +2459,16 @@ __perf_remove_from_context(struct perf_event *event,
>
>        ctx_time_update(cpuctx, ctx);
>
>+       /*
>+        * If event was in error state, then keep it
>+        * that way, otherwise bogus counts will be
>+        * returned on read(). The only way to get out
>+        * of error state is by explicit re-enabling
>+        * of the event
>+        */
>+       if (event->state > PERF_EVENT_STATE_OFF)
>+               perf_cgroup_event_disable(event, ctx);
>+
>        /*
>         * Ensure event_sched_out() switches to OFF, at the very least
>         * this avoids raising perf_pending_task() at this time.
>
>not with modification with "list_del_event()".

... I apply your original patch on 6.15, up to now, 7 rounds of test show no sign of kernel panic.
I think, the patch does fix it.

Tested-by: David Wang <00107082@163.com>


Thanks
David


^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2025-06-02 15:33 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-06-02 12:06 [BUG][6.15][perf] Kernel panic not syncing: Fatal exception in interrupt Yeoreum Yun
2025-06-02 12:33 ` David Wang
2025-06-02 13:19 ` Yeoreum Yun
2025-06-02 13:31   ` David Wang
2025-06-02 15:13   ` Yeoreum Yun
2025-06-02 15:32     ` David Wang
  -- strict thread matches above, loose matches on Subject: below --
2025-06-01 17:36 David Wang

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).