From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>
To: Vladimir Murzin <vladimir.murzin@arm.com>
Cc: mark.rutland@arm.com, catalin.marinas@arm.com,
Jinjie Ruan <ruanjinjie@huawei.com>,
liaochang1@huawei.com, ashok.raj@oss.qualcomm.com,
maz@kernel.org, will@kernel.org, lpieralisi@kernel.org,
linux-arm-kernel@lists.infradead.org
Subject: Re: [PATCH v2 00/39] arm64: Add support for FEAT_NMI
Date: Thu, 19 Nov 2026 16:07:19 +0000 [thread overview]
Message-ID: <av8bByU93_nCbPqE@shutemov.name> (raw)
In-Reply-To: <d4bb2979-4bc2-436c-8f4a-c9a5decea900@arm.com>
On Thu, Nov 19, 2026 at 02:50:50PM +0000, Vladimir Murzin wrote:
> Any chance you can attach debugger or collect traces so we know what those
> CPUs are doing?
I think they are idle with SDEI masked when the broadcast lands, and
the firmware leaves them unable to take interrupts.
A CPU entering a power-down idle state masks SDEI from the cpu_pm
notifier. If event 0 lands on it while it is masked, TF-A's
handle_masked_trigger() returns for event 0 without EOI:
CPU A CPU B (power-down idle)
cpu_pm_enter()
sdei_mask_local_cpu()
sdei_nmi_raise_backtrace()
SDEI_EVENT_SIGNAL(0, CPU B)
EL3: sdei_intr_handler()
acknowledge the SGI
handle_masked_trigger()
// event 0: return, no EOI
cpu_pm_exit()
sdei_unmask_local_cpu()
The SGI stays active, so CPU B runs at the SDEI priority from then on:
no Non-secure interrupts and no more event 0. Other CPUs then hang in
csd_lock_wait() on their next synchronous cross-call to it.
I reproduced it on QEMU with TF-A v2.15 minus the check described below.
After event 0 lands on a masked CPU, ICC_RPR_EL1 there reads 0 and stays
0 after SDEI_PE_UNMASK, and RCU reports "1-...!: ... idle=c078/0/0x0 ...
(false positive?)" as in Jinjie's rc3 log. With an EOI added on that
path, the same test passes.
That path has been in TF-A since the SDEI dispatcher was added and is
still in master. TF-A v2.13 made SDEI_EVENT_SIGNAL refuse a masked
target, which leaves only a race with a target that masks after the
check.
A fix is under review:
https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/55414
Jinjie, what is your firmware based on?
Two checks would confirm it:
- does dmesg show "SDEI_EVENT_SIGNAL to CPU N failed" after the
sysrq-l? None means the firmware accepted the signals to masked
CPUs;
- does the broadcast work with cpuidle.off=1? Without power-down idle
nothing masks SDEI, so all CPUs should answer.
Ways out I see:
- Firmware: end the interrupt on that path, or keep event 0 pending
until unmask as 55414 does. That is the real fix, but deployed
firmware will not have it for a while.
- Kernel: never signal a CPU that has SDEI masked. The signalling CPU
and the cpu_pm notifier can handshake: the sender marks the target
and IPIs it instead if it is already masked, and the notifier
refuses the power-down state while a signal is in flight. SDEI keeps
serving every running CPU, and the race goes away whatever the
firmware does. It needs a hook in arm_sdei.c.
- Kernel: send the IPI first and SDEI only to the CPUs that do not
answer, as smp_send_stop() does. Simple, but then SDEI runs only
when a CPU is really stuck, and a problem like this one stays hidden
until the day it is needed.
- Kernel: use the NMI IPI when there is one, as you suggested. That
takes SDEI out of the picture where FEAT_NMI or pseudo-NMI is in use,
but not on machines without them, which are the ones SDEI is for.
- A switch to turn SDEI NMI off, like Jinjie's patch, as an escape
hatch whichever way this goes.
Any comments?
--
Kiryl Shutsemau / Kirill A. Shutemov
next prev parent reply other threads:[~2026-11-19 16:07 UTC|newest]
Thread overview: 56+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 01/39] arm64: irqflags: Introduce arm64-specific irqflags type Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 02/39] arm64: irqflags: save and use both DAIF and PMR Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 03/39] arm64: interrupts: Add common exception state helpers Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 04/39] arm64: process: Lift check for exception state to generic infrastructure Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 05/39] arm64: entry: Introduce entry specific exception masking helpers Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 06/39] arm64: entry: replace DAIF helpers with entry helpers Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 07/39] arm64: interrupts: Introduce exception masking save/restore helpers Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 08/39] arm64: interrupts: introduce a helper for GIC priority initialization Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 09/39] arm64: replace local_daif helpers Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 10/39] arm64: cpuidle: use new helpers to bypass interrupt priority masking Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 11/39] arm64: remove daifflags.h Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 12/39] arm64: gicv3: remove GIC_PRIO_PSR_I_SET Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 13/39] arm64: cpufeature: Remove system_has_prio_mask_debugging() Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 14/39] arm64: irqflags: Switch to CONFIG_DEBUG_IRQFLAGS Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 15/39] arm64: Kconfig: Remove CONFIG_ARM64_DEBUG_PRIORITY_MASKING Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 16/39] efi/runtime-wrappers: Permit architectures to override IRQ flags checks Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 17/39] arm64/efi: Implement override for " Vladimir Murzin
2026-11-13 13:22 ` Mark Brown
2026-11-12 13:20 ` [PATCH v2 18/39] arm64: booting: Document boot requirements for FEAT_NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 19/39] arm64: sysreg: Add definitions for immediate versions of MSR ALLINT Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 20/39] arm64: probes: Reject stepping over ALLINT Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 21/39] arm64: ptrace: Add PSR_ALLINT_BIT Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 22/39] arm64: idreg: Add an override for FEAT_NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 23/39] arm64: cpufeature: Detect PE support " Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 24/39] arm64: nmi: Manage masking for superpriority interrupts Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 25/39] arm64: irq: Report FEAT_NMI masking local IRQs Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 26/39] arm64: nmi: Add handling of superpriority interrupts as NMIs Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 27/39] arm64: suspend: Always initialise PSTATE.ALLINT Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 28/39] arm64/efi: Add ALLINT to IRQ flags checks Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 29/39] arm64: kprobes: Disable NMIs Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 30/39] arm64: smp: Abstract SGI and LPI operations Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 31/39] arm64: smp: Fall back to IRQ when IPI NMI request fails Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 32/39] arm64: nmi: Add Kconfig for NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 33/39] arm64: nmi: Select HAVE_PERF_EVENTS_NMI for FEAT_NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 34/39] irqchip/gic-v3: Rename nmi to pseudo_nmi Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 35/39] irqchip/gic-v3: Implement FEAT_GICv3_NMI support Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 36/39] arm64: smp: Add NMI support for LPI-backed IPIs Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 37/39] genirq: Trust local irqchip when considering IRQ for NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 38/39] genirq: Do not call add_interrupt_randomness() in NMI context Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 39/39] irqchip/gic-v5: Add NMI support Vladimir Murzin
2026-11-14 5:03 ` [PATCH v2 00/39] arm64: Add support for FEAT_NMI Ashok Raj
2026-11-19 11:18 ` Vladimir Murzin
2026-11-16 6:58 ` Jinjie Ruan
2026-11-16 10:02 ` Vladimir Murzin
2026-11-16 11:01 ` Jinjie Ruan
2026-11-18 13:22 ` Vladimir Murzin
2026-11-19 2:33 ` Jinjie Ruan
2026-11-18 12:17 ` Jinjie Ruan
2026-11-19 1:54 ` Jinjie Ruan
2026-11-19 10:14 ` Vladimir Murzin
2026-11-19 12:45 ` Jinjie Ruan
2026-11-19 14:50 ` Vladimir Murzin
2026-11-19 16:07 ` Kiryl Shutsemau (Meta) [this message]
2026-11-19 11:41 ` Jinjie Ruan
2026-11-19 14:46 ` Vladimir Murzin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=av8bByU93_nCbPqE@shutemov.name \
--to=kas@kernel.org \
--cc=ashok.raj@oss.qualcomm.com \
--cc=catalin.marinas@arm.com \
--cc=liaochang1@huawei.com \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=lpieralisi@kernel.org \
--cc=mark.rutland@arm.com \
--cc=maz@kernel.org \
--cc=ruanjinjie@huawei.com \
--cc=vladimir.murzin@arm.com \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox