From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from confino.investici.org (confino.investici.org [93.190.126.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5FC3170836 for ; Sat, 25 Jul 2026 19:19:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=93.190.126.19 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785007190; cv=none; b=tR0LZq9sxvubZljgOHgbKJY/apF+WpYb5DlbIMo7aIeNqN6QVVfKSoiWQNuW7VuujhYusQnJEd1SJl2MSffWrvoKnEdsU12Vyg5pOwIp1IKkLgAtJ2BVenvU+Lrc8P7GB+Kz/We7ONHVFY7nYdR5tb7yMrA9O6tl7x8SxSqVCcE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785007190; c=relaxed/simple; bh=MWmHmAwTqRYYYvfKW8vCxliMxqSGIxugHvWtBqrtX7Y=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=A78GaH4cKJcnmObjTCSKKEwKuDgE9rw4pOuVSIDWGesG9rEg+X8tw4LUiJJhaizR8t5f6KMpxI8HoIWFgIw4fdtuXSjH21S+oiD/w6omwxjVbdHPOegXZBMNnao5c8NAOH3w982CJRROdXi2H6ZPIbyA4XfjiMR6uQLS5o8KUzA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=grrlz.net; spf=pass smtp.mailfrom=grrlz.net; dkim=pass (1024-bit key) header.d=grrlz.net header.i=@grrlz.net header.b=C+6ZdaOX; arc=none smtp.client-ip=93.190.126.19 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=grrlz.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=grrlz.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=grrlz.net header.i=@grrlz.net header.b="C+6ZdaOX" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=grrlz.net; s=stigmate; t=1785007178; bh=KnYEmxmXlkEBCXT4emWsU8xxWzDddpcuvx5tm78zQXg=; h=From:To:Cc:Subject:Date:From; b=C+6ZdaOXW5rmZUvUY5ml5IZJ/kdJ+x0cQs+j51C/7e6vlh86gUvdqYjacAx9qQ14S rHUksrPYOozl5m5M2U1q+0BaedoCJQ4ldd5aZe5L74oLeXOD7s3hsu6u/pJMpSE2bd U3wm7kvAh6rR0Y2oZy5TxFBETGTSx9SnT6FU0oGU= Received: from mx1.investici.org (unknown [127.0.0.1]) by confino.investici.org (Postfix) with ESMTP id 4h6vn664gFz1144; Sat, 25 Jul 2026 19:19:38 +0000 (UTC) Received: by mx1.investici.org (Postfix) id 4h6vn56ZWcz111C; Sat, 25 Jul 2026 19:19:37 +0000 (UTC) From: Bradley Morgan To: Andrew Morton Cc: linux-kernel@vger.kernel.org, Peter Zijlstra , Tejun Heo , Bradley Morgan Subject: [PATCH v2] stop_machine: Warn when a stop state stalls Date: Sat, 25 Jul 2026 19:19:29 +0000 Message-ID: <20260725191930.23668-1-include@grrlz.net> X-Mailer: git-send-email 2.47.3 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit multi_cpu_stop() feeds the NMI watchdog and rcu_momentary_eqs() on every spin. so when a CPU never acks a state transition the machine hangs, and the two detectors that wouldve caught it are the ones being fed. no tick either, irqs are off everywhere, hung task is blind too. the hang logs exactly nothing. Stamp every state transition in set_state() and let waiters yell once after ten seconds stuck in the same state. every waiter reports itself, the culprit is the CPU missing from the output. its local_clock() not jiffies, jiffies is dead once everyone has irqs off. One branch in a path that already pokes two watchdogs per spin. nothing changes while stop_machine() behaves. Signed-off-by: Bradley Morgan --- Changes since V1: - Oh no! forgot to CC loads of people.. - No code was changed.. kernel/stop_machine.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c index 773d8e9ae30c..1e1d1d46c24d 100644 --- a/kernel/stop_machine.c +++ b/kernel/stop_machine.c @@ -15,12 +15,14 @@ #include #include #include +#include #include #include #include #include #include #include +#include #include /* @@ -174,6 +176,7 @@ struct multi_stop_data { enum multi_stop_state state; atomic_t thread_ack; + u64 tstamp; }; static void set_state(struct multi_stop_data *msdata, @@ -181,6 +184,7 @@ static void set_state(struct multi_stop_data *msdata, { /* Reset ack counter. */ atomic_set(&msdata->thread_ack, msdata->num_threads); + WRITE_ONCE(msdata->tstamp, local_clock()); smp_wmb(); WRITE_ONCE(msdata->state, newstate); } @@ -197,6 +201,9 @@ notrace void __weak stop_machine_yield(const struct cpumask *cpumask) cpu_relax(); } +/* how long a state may sit unacked before waiting CPUs complain */ +#define MULTI_STOP_STUCK_NS (10ULL * NSEC_PER_SEC) + /* This is the cpu_stop function which stops the CPU. */ static int multi_cpu_stop(void *data) { @@ -205,7 +212,7 @@ static int multi_cpu_stop(void *data) int cpu = smp_processor_id(), err = 0; const struct cpumask *cpumask; unsigned long flags; - bool is_active; + bool is_active, stuck = false; /* * When called from stop_machine_from_inactive_cpu(), irq might @@ -250,6 +257,18 @@ static int multi_cpu_stop(void *data) touch_nmi_watchdog(); /* Also suppress RCU CPU stall warnings. */ rcu_momentary_eqs(); + /* + * the watchdogs above eat the evidence, a CPU that + * never acks hangs us silently. every waiter yells + * once, the culprit is the one missing from the log. + */ + if (!stuck && + local_clock() - READ_ONCE(msdata->tstamp) > MULTI_STOP_STUCK_NS) { + stuck = true; + pr_warn("%s: CPU %d stuck in state %d (%ps), %d acks missing\n", + __func__, cpu, curstate, msdata->fn, + atomic_read(&msdata->thread_ack)); + } } } while (curstate != MULTI_STOP_EXIT); -- 2.47.3