From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D38B730595B for ; Tue, 16 Jun 2026 13:34:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781616867; cv=none; b=YiJXTv2Y8h1Ry8g94suV/BeOdAVPHn0X/fIpJhyJSwfOCDFOl/4kgq+tfS8jOC9o2O0wYBFpwGOXmRnBU1YygNZGL3fzs4zbXsmVfxY/NIJbAjd1r1pysDCZcbC2bbrFltwEEhCABiF1z4rLm+idFt53u5+As4Rs4E+5VVdZLtw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781616867; c=relaxed/simple; bh=T1pYdejgTvmkfWhEvOrtwPKTKPG76pSgLubIPGhMhEA=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=qqS57rolSEobhQieDAjMB8zS76lG82nsPHoyPtGjKhjp1gTWNqGXSZvzohRTjavu7i9nTmUsXJF+IxYFQQrld4oP9mhjpckA+APfs3jViKc4SAW7GDBXdn56/ztspxcrvTp/QKnbPlmtLY0QU0eP79DgmE12OnWTXySo4SpsHZg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org; spf=pass smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=i0uEL4kL; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="i0uEL4kL" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:In-Reply-To:References: Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description; bh=+UE5RJJ6/r0jdZCVwMgidgI40TgUBq+lPP0mK3G0/84=; b=i0uEL4kLZ+a4m+jL3wUGeN02uT 6pl86MsGGWWBDQ2EAueg0l6cYz6SLuLMoS0SVmJw2lidnXocZtl/b5ArDb/pchfnUKWMSWbf0+0vQ JwwoQmNGO1ErpCm/releez4kT5SQhVMCAIY3Tn33IfhYz3utsL6q7XwfyCbizGimCMRuabQgSc3e5 HduYEOa5ueGp2gqtbqCnPhT/SIoquH4vgUYfjjQ71d2A9EMvlGhmqzbA7ZmcY7DGDilvNR5cbj8Am 2gAti3sShI3fHZnV8U0zAHDtKo6JXhDcfLZjpDn1G7xCBh5TEawQVjL74OSenxjCFAAPOg357kde3 PuivOwWA==; Received: from authenticated-user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.96) (envelope-from ) id 1wZTvo-00DsxL-2y; Tue, 16 Jun 2026 13:34:21 +0000 From: Breno Leitao Date: Tue, 16 Jun 2026 06:33:32 -0700 Subject: [PATCH v3 2/3] workqueue: defer the worker wakeup outside pool->lock in __queue_work() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20260616-fastwake-v3-2-79da19fcd08f@debian.org> References: <20260616-fastwake-v3-0-79da19fcd08f@debian.org> In-Reply-To: <20260616-fastwake-v3-0-79da19fcd08f@debian.org> To: Tejun Heo , Lai Jiangshan Cc: linux-kernel@vger.kernel.org, marco.crivellari@suse.com, frederic@kernel.org, bigeasy@linutronix.de, Hillf Danton , Breno Leitao , kernel-team@meta.com, kmagar@redhat.com, psuriset@redhat.com X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=openpgp-sha256; l=1679; i=leitao@debian.org; h=from:subject:message-id; bh=T1pYdejgTvmkfWhEvOrtwPKTKPG76pSgLubIPGhMhEA=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBqMVDQFy6++1Izu+GvyZRj2SlR+JK5vYKB55zt+ DJD0XMZVPOJAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCajFQ0AAKCRA1o5Of/Hh3 bdN9EACkufXCeSO8ABeGby8xKoJUhGBJHhi3OiBCgxrXjGkWEgkVL8YhpDEAejyeRYcdB61V/3S aa8lQTUE5uEXT0AwE6GhZZq/5ErIdVQfY/0cxlYTWv3IeTwK1WovfDmsS62oWBDgLgBax6TXjOn ufO5Jk7/OJAzWBIbNdlPdd0qvxto/rWu9KUPBlxLltGGYd8NEftzOosphVas2Nh08ga9jC5amgo 30rpT6fZOirGMl98kzbqFujYJiApbLtBG+KIVLGOE4Vxgam8ebDnvnCn4Yqlxz5rjMD9GDTMilz 4k/z4lnRXVNBN/9V3tu1xSWSzbbM9/2wvgO8GUMZtD2jbqfK8PyWj1cgq7jrQinBlGYwx8J4b0Y PRswJLxgzQHC4bjot3nwGHQ1n+W83taJPACsd6JjZZkQQyWRRTp76ZtJSDlEr/C721Hs2EAaHnh x7vnomg6QcyTAKkQIlS0Dt1jt7BYFHUBLtfLVHEMtEO6kNEgEZDtWpZiJNxeyQ/rDEa35BlD3LU 3ECOIbHyaJC4+IAKO+CmannWbMh/51qzUouGp6svZtuJDGFY1Zo88nXxX2sSHfQbqknu1E3h0zz VzyHLI5gXHEulNTASJoIEYC41yTK328ObgtQpisz7/B4/As3KsYcnTIyXkiI6qiV9rhl6+QmEIO IhyFnqWqO90wFWw== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao __queue_work() is the enqueue hot path: it inserts the work item and calls kick_pool() while holding pool->lock. kick_pool() ends in a wakeup, which takes the target task's rq->lock, so rq->lock nests under pool->lock on every enqueue that wakes a worker on a contended unbound pool. Use kick_pool_pick() to select and claim the worker under pool->lock, queue it on an on-stack wake_q, and issue the wakeup with wake_up_q() right after dropping the lock via raw_spin_unlock_wake(). Worker selection, wake_cpu setup and claiming the worker off pool->idle_list still happen under the lock; only the rq->lock acquisition moves out. Signed-off-by: Breno Leitao --- kernel/workqueue.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index fd3b5bc78df9e..44ad3450ff77f 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -2299,6 +2299,7 @@ static void __queue_work(int cpu, struct workqueue_struct *wq, { struct pool_workqueue *pwq; struct worker_pool *last_pool, *pool; + DEFINE_WAKE_Q(wakeq); unsigned int work_flags; unsigned int req_cpu = cpu; @@ -2421,14 +2422,14 @@ static void __queue_work(int cpu, struct workqueue_struct *wq, trace_workqueue_activate_work(work); insert_work(pwq, work, &pool->worklist, work_flags); - kick_pool(pool); + kick_pool_pick(pool, &wakeq); } else { work_flags |= WORK_STRUCT_INACTIVE; insert_work(pwq, work, &pwq->inactive_works, work_flags); } out: - raw_spin_unlock(&pool->lock); + raw_spin_unlock_wake(&pool->lock, &wakeq); rcu_read_unlock(); } -- 2.53.0-Meta