Re: [PATCH v4 0/4] arm64: cross-CPU NMI via SDEI

[Kiryl Shutsemau <kirill@shutemov.name>]

On Tue, Jun 30, 2026 at 11:04:14AM +0100, Kiryl Shutsemau wrote:
> On Mon, Jun 29, 2026 at 05:53:57PM +0100, Kiryl Shutsemau wrote:
> > On Mon, Jun 29, 2026 at 04:54:18PM +0100, Catalin Marinas wrote:
> > > Have you tried SDEI_EVENT_COMPLETE_AND_RESUME instead? Just COMPLETE
> > > won't return to the kernel. We have sdei_handler_abort() to complete the
> > > event and, hopefully, you can continue with the CPU_OFF. It's a work
> > > around the TF-A non-compliance but I think this is useful even if you
> > > don't issue the CPU_OFF (e.g. no CPU hotplug, just the park loop).
> > 
> > Tried it. The result is the opposite of what I expected, and it argues
> > against doing the complete at all under QEMU's TF-A.
> 
> I have to walk back on this. My test setup was broken. I will be back to
> you with proper data.

Back with proper data. Short version: CPU_OFF from the SDEI stop does not
work on Grace, whether or not the event is completed first, so the series
stays with the park. Details below.

What was broken before
======================
My QEMU kdump harness had three faults that together made CPU_OFF look
like it broke the capture kernel:

  - a race between the sysrq-c crash and the buddy detector -- the script
    that wedges the CPU wasn't pinned off it, so which crash path ran (and
    thus the outcome) was random;
  - the capture kernel's console went quiet at the boot-console handover,
    so "nothing after Bye!" was mostly lost output, not a hang (keep_bootcon
    shows it booting on);
  - the one genuine early stall was the capture kernel's CFI probe of the
    secure NOR flash -- it reproduces with the whole series compiled out
    (CONFIG_ARM_SDEI_NMI=n) and disappears with
    initcall_blacklist=physmap_init, i.e. nothing to do with SDEI.

With those fixed, QEMU/TF-A boots the capture kernel to userspace for both
park and raw CPU_OFF, and CPU_OFF really powers the PE off. But QEMU can't
decide the question that matters: its post-kexec hold-pen can't revive any
CPU_OFF'd core (even a cleanly hotplug-offlined one), so re-onlining is
undecidable there. Hence Grace.

Grace (Neoverse V2, EL3 TF-A)
=============================
  - Series as posted (park): the SDEI rung stops the wedged CPU, its
    context lands in the vmcore, the capture kernel boots. Works.
    Bringing up secondary CPUs complains that it cannot get the CPU up,
    but proceeds.
  - raw CPU_OFF from the uncompleted SDEI event: the capture kernel boots
    but hangs at "smp: Bringing up secondary CPUs ..." -- it cannot
    re-online the CPU that was CPU_OFF'd. This is exactly the TF-A point
    you raised: the SDEI dispatch is left dangling on CPU_OFF and the PE
    won't come back cleanly.
  - complete-then-CPU_OFF: same hang at secondary bring-up. Completing the
    event first does not help.

For that last one I did not use sdei_handler_abort(): it can't express
"complete, then CPU_OFF" from the handler, because its COMPLETE_AND_RESUME
lands on the "1: ret" trampoline, which resumes the *interrupted* (wedged)
context -- so nothing after it runs. I used a COMPLETE_AND_RESUME whose
resume PC is a PSCI CPU_OFF stub, so the event is genuinely completed and
then the PE powers off. It still can't be re-onlined. (I suspect the
"complete-then-CPU_OFF wedged EL3" I mentioned earlier was in fact the
sdei_handler_abort() path resuming the wedged loop, i.e. never doing the
CPU_OFF at all.)

Conclusion
==========
CPU_OFF from the SDEI stop is unusable on current TF-A, with or without
completing the event first -- a concrete real-HW consequence of the
non-compliance you flagged (TF-A subscribes SDEI to PSCI CPU_ON and
suspend-wakeup, but not CPU_OFF). So the series keeps the park: the dump
is complete either way, and only re-onlining the SDEI-stopped CPU in an
SMP capture kernel is lost -- the same fallback the shared stop path
already takes when CPU_OFF is unavailable. Powering the PE off for reuse
is a firmware follow-up (TF-A completing or tearing down the SDEI dispatch
on PSCI CPU_OFF), not something the kernel can paper over.

Please consider applying v5.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov