All of lore.kernel.org
 help / color / mirror / Atom feed
From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>
To: Vladimir Murzin <vladimir.murzin@arm.com>
Cc: mark.rutland@arm.com, catalin.marinas@arm.com,
	Jinjie Ruan <ruanjinjie@huawei.com>,
	liaochang1@huawei.com, ashok.raj@oss.qualcomm.com,
	maz@kernel.org, will@kernel.org, lpieralisi@kernel.org,
	linux-arm-kernel@lists.infradead.org
Subject: Re: [PATCH v2 00/39] arm64: Add support for FEAT_NMI
Date: Thu, 19 Nov 2026 16:07:19 +0000	[thread overview]
Message-ID: <av8bByU93_nCbPqE@shutemov.name> (raw)
In-Reply-To: <d4bb2979-4bc2-436c-8f4a-c9a5decea900@arm.com>

On Thu, Nov 19, 2026 at 02:50:50PM +0000, Vladimir Murzin wrote:
> Any chance you can attach debugger or collect traces so we know what those
> CPUs are doing?

I think they are idle with SDEI masked when the broadcast lands, and
the firmware leaves them unable to take interrupts.

A CPU entering a power-down idle state masks SDEI from the cpu_pm
notifier. If event 0 lands on it while it is masked, TF-A's
handle_masked_trigger() returns for event 0 without EOI:

    CPU A                           CPU B (power-down idle)
                                    cpu_pm_enter()
                                      sdei_mask_local_cpu()
    sdei_nmi_raise_backtrace()
      SDEI_EVENT_SIGNAL(0, CPU B)
                                    EL3: sdei_intr_handler()
                                      acknowledge the SGI
                                      handle_masked_trigger()
                                        // event 0: return, no EOI
                                    cpu_pm_exit()
                                      sdei_unmask_local_cpu()

The SGI stays active, so CPU B runs at the SDEI priority from then on:
no Non-secure interrupts and no more event 0. Other CPUs then hang in
csd_lock_wait() on their next synchronous cross-call to it.

I reproduced it on QEMU with TF-A v2.15 minus the check described below.

After event 0 lands on a masked CPU, ICC_RPR_EL1 there reads 0 and stays
0 after SDEI_PE_UNMASK, and RCU reports "1-...!: ... idle=c078/0/0x0 ...
(false positive?)" as in Jinjie's rc3 log. With an EOI added on that
path, the same test passes.

That path has been in TF-A since the SDEI dispatcher was added and is
still in master. TF-A v2.13 made SDEI_EVENT_SIGNAL refuse a masked
target, which leaves only a race with a target that masks after the
check.

A fix is under review:

  https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/55414

Jinjie, what is your firmware based on?

Two checks would confirm it:

  - does dmesg show "SDEI_EVENT_SIGNAL to CPU N failed" after the
    sysrq-l? None means the firmware accepted the signals to masked
    CPUs;

  - does the broadcast work with cpuidle.off=1? Without power-down idle
    nothing masks SDEI, so all CPUs should answer.

Ways out I see:

  - Firmware: end the interrupt on that path, or keep event 0 pending
    until unmask as 55414 does. That is the real fix, but deployed
    firmware will not have it for a while.

  - Kernel: never signal a CPU that has SDEI masked. The signalling CPU
    and the cpu_pm notifier can handshake: the sender marks the target
    and IPIs it instead if it is already masked, and the notifier
    refuses the power-down state while a signal is in flight. SDEI keeps
    serving every running CPU, and the race goes away whatever the
    firmware does. It needs a hook in arm_sdei.c.

  - Kernel: send the IPI first and SDEI only to the CPUs that do not
    answer, as smp_send_stop() does. Simple, but then SDEI runs only
    when a CPU is really stuck, and a problem like this one stays hidden
    until the day it is needed.

  - Kernel: use the NMI IPI when there is one, as you suggested. That
    takes SDEI out of the picture where FEAT_NMI or pseudo-NMI is in use,
    but not on machines without them, which are the ones SDEI is for.

  - A switch to turn SDEI NMI off, like Jinjie's patch, as an escape
    hatch whichever way this goes.

Any comments?

-- 
  Kiryl Shutsemau / Kirill A. Shutemov


  reply	other threads:[~2026-11-19 16:07 UTC|newest]

Thread overview: 59+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 01/39] arm64: irqflags: Introduce arm64-specific irqflags type Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 02/39] arm64: irqflags: save and use both DAIF and PMR Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 03/39] arm64: interrupts: Add common exception state helpers Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 04/39] arm64: process: Lift check for exception state to generic infrastructure Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 05/39] arm64: entry: Introduce entry specific exception masking helpers Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 06/39] arm64: entry: replace DAIF helpers with entry helpers Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 07/39] arm64: interrupts: Introduce exception masking save/restore helpers Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 08/39] arm64: interrupts: introduce a helper for GIC priority initialization Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 09/39] arm64: replace local_daif helpers Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 10/39] arm64: cpuidle: use new helpers to bypass interrupt priority masking Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 11/39] arm64: remove daifflags.h Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 12/39] arm64: gicv3: remove GIC_PRIO_PSR_I_SET Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 13/39] arm64: cpufeature: Remove system_has_prio_mask_debugging() Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 14/39] arm64: irqflags: Switch to CONFIG_DEBUG_IRQFLAGS Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 15/39] arm64: Kconfig: Remove CONFIG_ARM64_DEBUG_PRIORITY_MASKING Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 16/39] efi/runtime-wrappers: Permit architectures to override IRQ flags checks Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 17/39] arm64/efi: Implement override for " Vladimir Murzin
2026-11-13 13:22   ` Mark Brown
2026-11-12 13:20 ` [PATCH v2 18/39] arm64: booting: Document boot requirements for FEAT_NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 19/39] arm64: sysreg: Add definitions for immediate versions of MSR ALLINT Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 20/39] arm64: probes: Reject stepping over ALLINT Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 21/39] arm64: ptrace: Add PSR_ALLINT_BIT Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 22/39] arm64: idreg: Add an override for FEAT_NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 23/39] arm64: cpufeature: Detect PE support " Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 24/39] arm64: nmi: Manage masking for superpriority interrupts Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 25/39] arm64: irq: Report FEAT_NMI masking local IRQs Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 26/39] arm64: nmi: Add handling of superpriority interrupts as NMIs Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 27/39] arm64: suspend: Always initialise PSTATE.ALLINT Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 28/39] arm64/efi: Add ALLINT to IRQ flags checks Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 29/39] arm64: kprobes: Disable NMIs Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 30/39] arm64: smp: Abstract SGI and LPI operations Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 31/39] arm64: smp: Fall back to IRQ when IPI NMI request fails Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 32/39] arm64: nmi: Add Kconfig for NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 33/39] arm64: nmi: Select HAVE_PERF_EVENTS_NMI for FEAT_NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 34/39] irqchip/gic-v3: Rename nmi to pseudo_nmi Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 35/39] irqchip/gic-v3: Implement FEAT_GICv3_NMI support Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 36/39] arm64: smp: Add NMI support for LPI-backed IPIs Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 37/39] genirq: Trust local irqchip when considering IRQ for NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 38/39] genirq: Do not call add_interrupt_randomness() in NMI context Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 39/39] irqchip/gic-v5: Add NMI support Vladimir Murzin
2026-11-14  5:03 ` [PATCH v2 00/39] arm64: Add support for FEAT_NMI Ashok Raj
2026-11-19 11:18   ` Vladimir Murzin
2026-11-16  6:58 ` Jinjie Ruan
2026-11-16 10:02   ` Vladimir Murzin
2026-11-16 11:01     ` Jinjie Ruan
2026-11-18 13:22       ` Vladimir Murzin
2026-11-19  2:33         ` Jinjie Ruan
2026-11-18 12:17 ` Jinjie Ruan
2026-11-19  1:54   ` Jinjie Ruan
2026-11-19 10:14     ` Vladimir Murzin
2026-11-19 12:45       ` Jinjie Ruan
2026-11-19 14:50         ` Vladimir Murzin
2026-11-19 16:07           ` Kiryl Shutsemau (Meta) [this message]
2026-11-19 16:38             ` Kiryl Shutsemau (Meta)
2026-11-20  2:11             ` Jinjie Ruan
2026-11-20 15:07               ` Kiryl Shutsemau (Meta)
2026-11-19 11:41   ` Jinjie Ruan
2026-11-19 14:46     ` Vladimir Murzin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=av8bByU93_nCbPqE@shutemov.name \
    --to=kas@kernel.org \
    --cc=ashok.raj@oss.qualcomm.com \
    --cc=catalin.marinas@arm.com \
    --cc=liaochang1@huawei.com \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=lpieralisi@kernel.org \
    --cc=mark.rutland@arm.com \
    --cc=maz@kernel.org \
    --cc=ruanjinjie@huawei.com \
    --cc=vladimir.murzin@arm.com \
    --cc=will@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.