Linux-ARM-Kernel Archive on lore.kernel.org
 help / color / mirror / Atom feed
From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>
To: Vladimir Murzin <vladimir.murzin@arm.com>
Cc: mark.rutland@arm.com, catalin.marinas@arm.com,
	Jinjie Ruan <ruanjinjie@huawei.com>,
	liaochang1@huawei.com, ashok.raj@oss.qualcomm.com,
	maz@kernel.org, will@kernel.org, lpieralisi@kernel.org,
	linux-arm-kernel@lists.infradead.org
Subject: Re: [PATCH v2 00/39] arm64: Add support for FEAT_NMI
Date: Thu, 19 Nov 2026 16:07:19 +0000	[thread overview]
Message-ID: <av8bByU93_nCbPqE@shutemov.name> (raw)
In-Reply-To: <d4bb2979-4bc2-436c-8f4a-c9a5decea900@arm.com>

On Thu, Nov 19, 2026 at 02:50:50PM +0000, Vladimir Murzin wrote:
> Any chance you can attach debugger or collect traces so we know what those
> CPUs are doing?

I think they are idle with SDEI masked when the broadcast lands, and
the firmware leaves them unable to take interrupts.

A CPU entering a power-down idle state masks SDEI from the cpu_pm
notifier. If event 0 lands on it while it is masked, TF-A's
handle_masked_trigger() returns for event 0 without EOI:

    CPU A                           CPU B (power-down idle)
                                    cpu_pm_enter()
                                      sdei_mask_local_cpu()
    sdei_nmi_raise_backtrace()
      SDEI_EVENT_SIGNAL(0, CPU B)
                                    EL3: sdei_intr_handler()
                                      acknowledge the SGI
                                      handle_masked_trigger()
                                        // event 0: return, no EOI
                                    cpu_pm_exit()
                                      sdei_unmask_local_cpu()

The SGI stays active, so CPU B runs at the SDEI priority from then on:
no Non-secure interrupts and no more event 0. Other CPUs then hang in
csd_lock_wait() on their next synchronous cross-call to it.

I reproduced it on QEMU with TF-A v2.15 minus the check described below.

After event 0 lands on a masked CPU, ICC_RPR_EL1 there reads 0 and stays
0 after SDEI_PE_UNMASK, and RCU reports "1-...!: ... idle=c078/0/0x0 ...
(false positive?)" as in Jinjie's rc3 log. With an EOI added on that
path, the same test passes.

That path has been in TF-A since the SDEI dispatcher was added and is
still in master. TF-A v2.13 made SDEI_EVENT_SIGNAL refuse a masked
target, which leaves only a race with a target that masks after the
check.

A fix is under review:

  https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/55414

Jinjie, what is your firmware based on?

Two checks would confirm it:

  - does dmesg show "SDEI_EVENT_SIGNAL to CPU N failed" after the
    sysrq-l? None means the firmware accepted the signals to masked
    CPUs;

  - does the broadcast work with cpuidle.off=1? Without power-down idle
    nothing masks SDEI, so all CPUs should answer.

Ways out I see:

  - Firmware: end the interrupt on that path, or keep event 0 pending
    until unmask as 55414 does. That is the real fix, but deployed
    firmware will not have it for a while.

  - Kernel: never signal a CPU that has SDEI masked. The signalling CPU
    and the cpu_pm notifier can handshake: the sender marks the target
    and IPIs it instead if it is already masked, and the notifier
    refuses the power-down state while a signal is in flight. SDEI keeps
    serving every running CPU, and the race goes away whatever the
    firmware does. It needs a hook in arm_sdei.c.

  - Kernel: send the IPI first and SDEI only to the CPUs that do not
    answer, as smp_send_stop() does. Simple, but then SDEI runs only
    when a CPU is really stuck, and a problem like this one stays hidden
    until the day it is needed.

  - Kernel: use the NMI IPI when there is one, as you suggested. That
    takes SDEI out of the picture where FEAT_NMI or pseudo-NMI is in use,
    but not on machines without them, which are the ones SDEI is for.

  - A switch to turn SDEI NMI off, like Jinjie's patch, as an escape
    hatch whichever way this goes.

Any comments?

-- 
  Kiryl Shutsemau / Kirill A. Shutemov


  reply	other threads:[~2026-11-19 16:07 UTC|newest]

Thread overview: 56+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 01/39] arm64: irqflags: Introduce arm64-specific irqflags type Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 02/39] arm64: irqflags: save and use both DAIF and PMR Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 03/39] arm64: interrupts: Add common exception state helpers Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 04/39] arm64: process: Lift check for exception state to generic infrastructure Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 05/39] arm64: entry: Introduce entry specific exception masking helpers Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 06/39] arm64: entry: replace DAIF helpers with entry helpers Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 07/39] arm64: interrupts: Introduce exception masking save/restore helpers Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 08/39] arm64: interrupts: introduce a helper for GIC priority initialization Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 09/39] arm64: replace local_daif helpers Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 10/39] arm64: cpuidle: use new helpers to bypass interrupt priority masking Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 11/39] arm64: remove daifflags.h Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 12/39] arm64: gicv3: remove GIC_PRIO_PSR_I_SET Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 13/39] arm64: cpufeature: Remove system_has_prio_mask_debugging() Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 14/39] arm64: irqflags: Switch to CONFIG_DEBUG_IRQFLAGS Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 15/39] arm64: Kconfig: Remove CONFIG_ARM64_DEBUG_PRIORITY_MASKING Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 16/39] efi/runtime-wrappers: Permit architectures to override IRQ flags checks Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 17/39] arm64/efi: Implement override for " Vladimir Murzin
2026-11-13 13:22   ` Mark Brown
2026-11-12 13:20 ` [PATCH v2 18/39] arm64: booting: Document boot requirements for FEAT_NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 19/39] arm64: sysreg: Add definitions for immediate versions of MSR ALLINT Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 20/39] arm64: probes: Reject stepping over ALLINT Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 21/39] arm64: ptrace: Add PSR_ALLINT_BIT Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 22/39] arm64: idreg: Add an override for FEAT_NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 23/39] arm64: cpufeature: Detect PE support " Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 24/39] arm64: nmi: Manage masking for superpriority interrupts Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 25/39] arm64: irq: Report FEAT_NMI masking local IRQs Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 26/39] arm64: nmi: Add handling of superpriority interrupts as NMIs Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 27/39] arm64: suspend: Always initialise PSTATE.ALLINT Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 28/39] arm64/efi: Add ALLINT to IRQ flags checks Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 29/39] arm64: kprobes: Disable NMIs Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 30/39] arm64: smp: Abstract SGI and LPI operations Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 31/39] arm64: smp: Fall back to IRQ when IPI NMI request fails Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 32/39] arm64: nmi: Add Kconfig for NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 33/39] arm64: nmi: Select HAVE_PERF_EVENTS_NMI for FEAT_NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 34/39] irqchip/gic-v3: Rename nmi to pseudo_nmi Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 35/39] irqchip/gic-v3: Implement FEAT_GICv3_NMI support Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 36/39] arm64: smp: Add NMI support for LPI-backed IPIs Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 37/39] genirq: Trust local irqchip when considering IRQ for NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 38/39] genirq: Do not call add_interrupt_randomness() in NMI context Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 39/39] irqchip/gic-v5: Add NMI support Vladimir Murzin
2026-11-14  5:03 ` [PATCH v2 00/39] arm64: Add support for FEAT_NMI Ashok Raj
2026-11-19 11:18   ` Vladimir Murzin
2026-11-16  6:58 ` Jinjie Ruan
2026-11-16 10:02   ` Vladimir Murzin
2026-11-16 11:01     ` Jinjie Ruan
2026-11-18 13:22       ` Vladimir Murzin
2026-11-19  2:33         ` Jinjie Ruan
2026-11-18 12:17 ` Jinjie Ruan
2026-11-19  1:54   ` Jinjie Ruan
2026-11-19 10:14     ` Vladimir Murzin
2026-11-19 12:45       ` Jinjie Ruan
2026-11-19 14:50         ` Vladimir Murzin
2026-11-19 16:07           ` Kiryl Shutsemau (Meta) [this message]
2026-11-19 11:41   ` Jinjie Ruan
2026-11-19 14:46     ` Vladimir Murzin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=av8bByU93_nCbPqE@shutemov.name \
    --to=kas@kernel.org \
    --cc=ashok.raj@oss.qualcomm.com \
    --cc=catalin.marinas@arm.com \
    --cc=liaochang1@huawei.com \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=lpieralisi@kernel.org \
    --cc=mark.rutland@arm.com \
    --cc=maz@kernel.org \
    --cc=ruanjinjie@huawei.com \
    --cc=vladimir.murzin@arm.com \
    --cc=will@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox