From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8B7CA389109 for ; Fri, 13 Mar 2026 12:58:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773406691; cv=none; b=jNu1D5aUvt1IKuD9Qt/W0o+syRE4oSkBAS6lEBW/GLunAnr93YjD7uNAnno23Z6/7HN5ALTBm5wp+tBCcXjNTsk109SE5GZW5crK1zyOHqN+EHpEPxuK5X7SV9BRycEXxsbVVyogB/8KfsrkWH91VAHPSz/XM79f5hdnxwXjj0k= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773406691; c=relaxed/simple; bh=0QJQr4a2WCbI2SjI6vKEAMlNwuEFS4n9yZgWDcZ5wjI=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=Lyz3HwRD4SK9qUL+AJydMNOmeGLlZvlj9/tr/VRzy1GdKDdXYk5BSBTLUJX3sF6JENASb/uiPLTvE4LxStXYKwCTcjlfOM/HIr+DEa/419mAOKtnehAnjzGZBBT3j3NeOfg5tRZWttZRIHtBq+3Tpgc/8fz+J7GgngxTvMQ2Rkc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org; spf=none smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=bkSsSzCL; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="bkSsSzCL" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:In-Reply-To:Content-Transfer-Encoding: Content-Type:MIME-Version:References:Message-ID:Subject:Cc:To:From:Date: Reply-To:Content-ID:Content-Description; bh=ukn6uYw3SohJPHIIRgkaDrzYkw4OpSpqgGFILMoNVZI=; b=bkSsSzCLqcnGe5ppnz3RSEwTsN 4XGlIaHmmCQCK6hKCexAY3BKiFmIUoxmUKqX7g/yBsT+6VE8wFrCQyRgdegs9qjbnjbpwrVWAasjZ r5v+PJr0XNFTOX738r2XmHbQtE8akfIvBN3+vqn66eL+OoNbrzqVtqLTwRKSHU6iME5J09kGdgo4I 4RbxXlpIAgy1AUOpPJiL9YNKT+viW7n10k1osHp3jDQ0udoExnW5uVWt7xB7xs4RiT8/aIR6bQXlW 9fiTORorg2NEvKTv0P00tC3a747VvK5Cdvmo8Gr2P0CkmwzGctTMdd3Buz8uemEMGyIHia/5BAfD+ osL15wwQ==; Received: from authenticated user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.94.2) (envelope-from ) id 1w125c-005KwN-Bj; Fri, 13 Mar 2026 12:58:04 +0000 Date: Fri, 13 Mar 2026 05:57:59 -0700 From: Breno Leitao To: Petr Mladek Cc: Tejun Heo , Lai Jiangshan , Andrew Morton , linux-kernel@vger.kernel.org, Omar Sandoval , Song Liu , Danielle Costantino , kasan-dev@googlegroups.com, kernel-team@meta.com Subject: Re: [PATCH v2 4/5] workqueue: Show all busy workers in stall diagnostics Message-ID: References: <20260305-wqstall_start-at-v2-0-b60863ee0899@debian.org> <20260305-wqstall_start-at-v2-4-b60863ee0899@debian.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Debian-User: leitao On Thu, Mar 12, 2026 at 06:03:03PM +0100, Petr Mladek wrote: > On Thu 2026-03-05 08:15:40, Breno Leitao wrote: > > show_cpu_pool_hog() only prints workers whose task is currently running > > on the CPU (task_is_running()). This misses workers that are busy > > processing a work item but are sleeping or blocked — for example, a > > worker that clears PF_WQ_WORKER and enters wait_event_idle(). > > IMHO, it is misleading. AFAIK, workers clear PF_WQ_WORKER flag only > when they are going to die. They never do so when going to sleep. > > > Such a > > worker still occupies a pool slot and prevents progress, yet produces > > an empty backtrace section in the watchdog output. > > > > This is happening on real arm64 systems, where > > toggle_allocation_gate() IPIs every single CPU in the machine (which > > lacks NMI), causing workqueue stalls that show empty backtraces because > > toggle_allocation_gate() is sleeping in wait_event_idle(). > > The wait_event_idle() called in toggle_allocation_gate() should not > cause a stall. The scheduler should call wq_worker_sleeping(tsk) > and wake up another idle worker. It should guarantee the progress. > > > Remove the task_is_running() filter so every in-flight worker in the > > pool's busy_hash is dumped. The busy_hash is protected by pool->lock, > > which is already held. > > As I explained in reply to the cover letter, sleeping workers should > not block forward progress. It seems that in this case, the system was > not able to wake up the other idle worker or it was the last idle > worker and was not able to fork a new one. > > IMHO, we should warn about this when there is no running worker. > It might be more useful than printing backtraces of the sleeping > workers because they likely did not cause the problem. > > I believe that the problem, in this particular situation, is that > the system can't schedule or fork new processes. It might help > to warn about it and maybe show backtrace of the currently > running process on the stalled CPU. Do you mean checking if pool->busy_hash is empty, and then warning? Commit fc36ad49ce7160907bcbe4f05c226595611ac293 Author: Breno Leitao Date: Fri Mar 13 05:35:02 2026 -0700 workqueue: warn when stalled pool has no running workers When the workqueue watchdog detects a pool stall and the pool's busy_hash is empty (no workers executing any work item), print a diagnostic warning with the pool state and trigger a backtrace of the currently running task on the stalled CPU. Signed-off-by: Breno Leitao Suggested-by: Petr Mladek diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 6ee52ba9b14f7..d538067754123 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -7655,6 +7655,17 @@ static void show_cpu_pool_busy_workers(struct worker_pool *pool) raw_spin_lock_irqsave(&pool->lock, irq_flags); + if (hash_empty(pool->busy_hash)) { + raw_spin_unlock_irqrestore(&pool->lock, irq_flags); + + pr_info("pool %d: no running workers, cpu=%d is %s (nr_workers=%d nr_idle=%d)\n", + pool->id, pool->cpu, + idle_cpu(pool->cpu) ? "idle" : "busy", + pool->nr_workers, pool->nr_idle); + trigger_single_cpu_backtrace(pool->cpu); + return; + } + hash_for_each(pool->busy_hash, bkt, worker, hentry) { if (task_is_running(worker->task)) { /*