From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from devianza.investici.org (devianza.investici.org [198.167.222.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 02319371885 for ; Sun, 26 Jul 2026 19:04:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.167.222.108 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785092664; cv=none; b=a5P+i39yowIqwMAP33DcOHcqnLQ40ybJUxTe07vsxikRSosEZ4BvZDE7JFQgsn+oELIgq3tl5x33TyvCw9VjTqVPiS5D12CltZcyVerzKvOogPAKhceVQ4ygIFe1BJoVnG2RDbVn2QpGSQG240xm6kORE5iRzon10WoSJN7o1Tg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785092664; c=relaxed/simple; bh=6gI5AMFUKYS1PCvL1M8ounjshXKXppMYKVtaRtFp594=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=kSQti+Vu7TVD7r1yplvoVICNrMBK4fA7YDBNH9urhkzaKdCS40tyxUDlpwhC8fUKSRt9V34aNE4B+oTfZYejlecxz07O0HTVDKOHP1ASOeMZgFsNj5nxUKWbnwO4z4mQ4qwwTFmFJRxJ9ykq3Vz9cp7MRHUUzrjogvttI/l8xrQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=grrlz.net; spf=pass smtp.mailfrom=grrlz.net; dkim=pass (1024-bit key) header.d=grrlz.net header.i=@grrlz.net header.b=p9pIpyfz; arc=none smtp.client-ip=198.167.222.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=grrlz.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=grrlz.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=grrlz.net header.i=@grrlz.net header.b="p9pIpyfz" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=grrlz.net; s=stigmate; t=1785092652; bh=z/Og1KnH0E37cK3ByoZrJ4noX0hMO3ih33xmvOqF0ww=; h=From:To:Cc:Subject:Date:From; b=p9pIpyfzv8hkk2nwnBKSBYTH3Nlt7WrB8BuJkfkjDm1Tth3YOTV127FBo3fX6OqQn WcAphUI0ODo/4Ufo2PuIBhEHjXZdrhhjRzLubES8QDr9+Ll6Mae+pA0SVMw9W23lAF V3NEsRzI/GEs2ctRh8JGWx610vifowJeG3HZQ4aA= Received: from mx2.investici.org (unknown [127.0.0.1]) by devianza.investici.org (Postfix) with ESMTP id 4h7WNr6KhTz6vQ0; Sun, 26 Jul 2026 19:04:12 +0000 (UTC) Received: by mx2.investici.org (Postfix) id 4h7WNr0ylCz4y2Q; Sun, 26 Jul 2026 19:04:12 +0000 (UTC) From: Bradley Morgan To: Andrew Morton Cc: Petr Mladek , Jinchao Wang , Feng Tang , Rio , Pnina Feder , Petr Pavlu , Sergey Senozhatsky , linux-kernel@vger.kernel.org, Bradley Morgan Subject: [PATCH v5 0/4] panic: fix panic_force_cpu= redirect races and NMI bypass Date: Sun, 26 Jul 2026 19:04:08 +0000 Message-ID: <20260726190412.10891-1-include@grrlz.net> X-Mailer: git-send-email 2.47.3 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit The panic_force_cpu= parameter redirects a panic to a specific CPU so the crash kernel runs there. The redirect code in panic_try_force_cpu() had two races, a va_list reuse bug and an ordering bug that bypassed the redirect for NMI panics, all found by Sashiko. This series closes them. This is v5, addressing Petr's review of v4. The earlier standalone submissions of these fixes were dropped from mm, so the series is now self contained: the va_list fix rejoins it as patch 3. Patch 1: fix the redirect CPU race. The redirect is gated by an atomic cmpxchg on panic_redirect_cpu, so only one CPU sends the redirect IPI. The cmpxchg loser used to return false and fall through into vpanic(), where it could win panic_try_start() and run crash_kexec on the wrong CPU before the target ever received the IPI. The loser has to stop. It cannot just return true, though, because panic_try_force_cpu() can be called twice on the same CPU (nested NMI during the message formatting, before the IPI is sent), and a blind stop on that second call would abandon the panic with no IPI sent. The loser now returns true to stop, unless it is reentering on the same CPU (old_cpu == this_cpu), in which case it returns false and falls through. Patch 1 also fixes the panic_in_progress() guard. We must never redirect when panic_cpu is already taken, so the guard stays. But it now returns true (stop) when the panic is on another CPU and false (proceed) when it is on this CPU, instead of returning false either way. The two races side by side, two non target CPUs A and B (target is C), then a reentry on the redirect winner: cpu A cpu B ---------- ---------- panic_try_force_cpu() panic_try_force_cpu() cmpxchg wins cmpxchg fails IPI -> C return false <- old BUG return true panic_try_start() wins panic_smp_self_stop() __crash_kexec() on B (A stops) (target C bypassed) cpu A (1st) cpu A (nested NMI) ---------- ---------- panic_try_force_cpu() cmpxchg wins (redirect = A) vsnprintf(msg) ... <-- NMI --> panic_try_force_cpu() cmpxchg fails old_cpu == A == this_cpu return true <- would abandon self_stop (IPI never sent) Patch 2: flatten nmi_panic control flow. A behavior preserving cleanup. panic() is noreturn, so the else after it is dropped and the body flattened, ready for patch 4 to add the redirect step without piling more onto the if else chain. Patch 3: fix va_list reuse in panic_try_force_cpu(). vsnprintf() consumes the caller's va_list. When the redirect fails, vpanic() reuses it for the panic message, which is undefined behavior. Fixed with va_copy(). This is the earlier standalone fix, already reviewed by Petr, rejoining the series unchanged so patch 4 can build on it. Patch 4: allow force_cpu redirect from an NMI. A panic from an NMI used to bypass the redirect entirely. nmi_panic() called panic_try_start() first, which claims panic_cpu, so by the time panic() reached panic_try_force_cpu() the panic_in_progress() check saw panic_cpu set, returned false, and never sent the redirect IPI. The crash kernel ran on the CPU that took the NMI instead of the requested one. The buggy call order, on a CPU X that is not the target (target is C): nmi_panic() panic_try_start() wins, panic_cpu = X panic("%s", msg) vpanic() panic_try_force_cpu() panic_in_progress() true, panic_cpu is X return false redirect bypassed panic_try_start() already won __crash_kexec() on X, not C The fix tries the redirect before claiming panic_cpu. nmi_panic() calls panic_try_force_cpu() first and only calls panic_try_start() when no redirect happens. The requested CPU then claims panic_cpu itself when its panic() runs, so panic_cpu is never handed off. nmi_panic() receives the final message as a plain string and has no va_list, and only a variadic function can create one. Instead of a wrapper, panic_try_force_cpu() now takes a va_list pointer, where NULL means that the format string already is the final message. vpanic() hands over a disposable copy of its arguments because the address of a va_list function parameter cannot be taken portably (on x86_64 va_list is an array type). The redirect IPI still goes out via smp_call_function_single_async(). Per Petr's v4 review this is not guaranteed to be safe from NMI context. It is best effort, and worth the risk because the redirect is only used when the crash kernel would not work on the panicking CPU anyway. Note: checkpatch complains about "spacing around '*'" on the new panic_try_force_cpu() prototype. It does not recognize va_list as a type name; the code is a regular pointer parameter. Changes since v4: - Patch 1: use panic_on_other_cpu() instead of the open coded negation, per Petr. Added Petr's Reviewed-by. - Patch 2: unchanged. Added Petr's Reviewed-by. - Patch 3: the earlier standalone va_copy() fix rejoins the series unchanged, per Petr, with his earlier Reviewed-by. - Patch 4: dropped panic_try_force_cpu_fmt(). panic_try_force_cpu() takes a va_list pointer instead and nmi_panic() calls it directly, per Petr. - Patch 4: fixed a v4 bug where the unconditional self_stop dropped the "return when already panicking on this CPU" behavior. A nested NMI during panic(), for example with unknown_nmi_panic, would have parked the CPU and hung the interrupted panic. - Patch 4: the redirecting CPU now marks itself offline before stopping, like vpanic() does, so panic_other_cpus_shutdown() on the target does not wait for it. - Patch 4: the IPI-from-NMI justification is reworded as best effort, per Petr. - Added Fixes: tags, and Cc: stable on patches 1, 3 and 4. Changes since v3 (v4 numbering, see the v4 cover letter): - Patch 1 now also fixes the panic_in_progress() guard to return true or false depending on which CPU owns panic_cpu, and drops the recursion framing in the comment per Petr's review. - Patch 3 no longer changes the panic_try_force_cpu() signature or formats the message before the redirect cmpxchg. Petr pointed out the static buf is only safe under panic_cpu ownership, so the formatting stays inside the cmpxchg guarded path. - Patch 3 adds the NMI safety justification for smp_call_function_single_async(), answering Petr's v1 question. - The nmi_panic() control flow cleanup is split into its own patch (patch 2), per Petr's request to split changes. Bradley Morgan (4): panic: fix redirect CPU race in panic_try_force_cpu() panic: flatten nmi_panic control flow panic: fix va_list reuse in panic_try_force_cpu() panic: allow force_cpu redirect from an NMI kernel/panic.c | 77 +++++++++++++++++++++++++++++++++++--------------- 1 file changed, 55 insertions(+), 22 deletions(-) -- 2.47.3