From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EB1942FF17A for ; Tue, 16 Jun 2026 13:34:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781616865; cv=none; b=WwEfJzU7zNYYpSpir5rJcPaQPublTMJocETcQ86+aPneGbMBeyATkW8EX5dGR9buNlKo+rDWwcxZax5zrZJJxgShjxawvczjtzdS9mlNPZHG/8S39ziGeFmtoXmZ853A8mPlWz/KHtGTUXYSd0UbDiNvqk7xkd549U2zjpycpdU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781616865; c=relaxed/simple; bh=HcTOTCPjwKo567vH/x6XYbJ6gB4ev/DgMY+eGKFNlmk=; h=From:Subject:Date:Message-Id:MIME-Version:Content-Type:To:Cc; b=JWDvP81ogCzPVRq+GqTAJCwYsn8quLmP5se8PGGsRISio5crsy+LQrdWu/aX+pBDdBhLDtd+Q+T/swwb54Us/vxL1XIjyhzd76InuyKiFGLPQsv5vsIfSxdaSFiJ9PIPu8GMkgYjkOQ/TBHiOERl0sBgDtuAMMtOXlVoUAkMJn8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org; spf=pass smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=NMvbBoB4; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="NMvbBoB4" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:Content-Transfer-Encoding: Content-Type:MIME-Version:Message-Id:Date:Subject:From:Reply-To:Content-ID: Content-Description:In-Reply-To:References; bh=4xpYPvOLLDiOl0oU8om3UijpgMC1OYPJqF7mbfncHCU=; b=NMvbBoB4sr8fckyj6uiGv2Mp2b N42m3d71cMv+FAFutYLhE2bWd6ht7kTzKgpJDRE3lYO2SKPSlGH1TpD2b1rpRFAGWFz0ib4lyik9A qnRD52UldfneRt1RYAXdTU6T+vf9mGUpq2AKtwuGN1qVHqAgbVtbI6ZLe/OH3uXaLvCJbTW3OgwJT zkzmwCD8Z9qlg3emUjAS4e/P/dv4e89cRkjAOafLAiqsZ2AObTXGe2wwVTGH6lQaSuHhs8qMJlpqW m82QhhJSxuLV90dL3S+P43AFbkcSsrpDExFdFoN35xWAvFb+HpYZvMGO4l6MKacWmgR/8tqKC3FeB XQPRzUdg==; Received: from authenticated-user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.96) (envelope-from ) id 1wZTvg-00Dsx1-1w; Tue, 16 Jun 2026 13:34:13 +0000 From: Breno Leitao Subject: [PATCH v3 0/3] workqueue: Shrink the lock time Date: Tue, 16 Jun 2026 06:33:30 -0700 Message-Id: <20260616-fastwake-v3-0-79da19fcd08f@debian.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-B4-Tracking: v=1; b=H4sIAKpQMWoC/0WNQQ6CMBBFr9LMmpoyhCKsvIdhUWAKowmQtlYN4 e6msHD5k/ff28CTY/LQiA0cRfa8zNCIIhPQT2YeSfIAjQBUqFWJWlrjw9s8SSqsr2gHrYscIRO wOrL8OVT39tz+1T2oD+mfiIl9WNz3aEVM3KnVqvhrI0olsa6qMkdLlTW3gTo282VxI7T7vv8Av yYn57MAAAA= X-Change-ID: 20260526-fastwake-02982fd66312 To: Tejun Heo , Lai Jiangshan Cc: linux-kernel@vger.kernel.org, marco.crivellari@suse.com, frederic@kernel.org, bigeasy@linutronix.de, Hillf Danton , Breno Leitao , kernel-team@meta.com, kmagar@redhat.com, psuriset@redhat.com X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=openpgp-sha256; l=3700; i=leitao@debian.org; h=from:subject:message-id; bh=HcTOTCPjwKo567vH/x6XYbJ6gB4ev/DgMY+eGKFNlmk=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBqMVDPyjRoejv/1wp2mxLZEFBJCdar/+sew/ahe fuYTytLCJiJAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCajFQzwAKCRA1o5Of/Hh3 bXpqD/9sCuTIH7R+bWYc706CkA3JwUlfsFQcfn1rg6B0+CDe9F8vVL8KL2DvnMInzeQG7E4kBJT W1xgVi0WnBvDK+tOjOA+3FIYVowz4YWGTiLxmI3A5t7psbig41iNEmJz5rlfIT2JwhoRA/niPXm p3r/lHJfi+vs0pwsPx1pii/nQ3ORKj6xcfZc43HNzFiz1JAwakdcDVm0+RANhtBH10szY5TtH0b VzgKuj8RCuplPYVP5PxmsvUDhQno89uwe8voV/AuxeBJWnPCJA2DCixF9PBU81EMErnEsUmotzA dyIvEIiLhYdMMCZjPFKQAnkH7FfFsrLOQExtZDwGLOcAZkA3QZtrRtRpHWIBg5ejRmOr4sD3QDC VuEfNJ/24Mmkp2fC8LzudDWcGcp/4rZcMdi/iKZ9X+767Krla6xNY7GPAml9/f1aZAO+tgWVv3z 1yFN9I6fgfMf9Mrth8gz3TP1n18OPd8BHmCS4NgdZpnEoBqlvTtrwYokK/inzuViU1aVY3Xp2Ft 9HIlzgFv5ZEuLwHQeyyu8hb0QE4RksU+Q5XRlApIeqz5kPw4OOlpVHk+k8S9YtGudgKjmknmpcb aK3qGO7AUw4RuRh3mpdOR83zZGpbwA+cibL0yCyIKrJnd2Zi5yFWy+8ni+PcwpqWORmUD8i4zAh B3qPB10j01UFBOA== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao The goal of this patchset is to decrease the time spent under the workqueue pool->lock. Currently the worker process is woken up inside pool->lock. The wakeup ends in wake_up_process(), which takes the target task's rq->lock, so rq->lock nests under pool->lock on the two hottest paths of a contended unbound workqueue (__queue_work() enqueue and process_one_work() chain kick). On some architectures the wakeup is even more expensive: on arm64 waking a CPU that is idle (in wfi) issues an IPI. Doing all of that while holding pool->lock lengthens the locked region and hurts throughput on contended unbound pools. This series shortens the locked region by selecting and claiming the worker to wake under pool->lock, but issuing the actual wakeup after the lock is dropped, using the wake_q machinery (wake_q_add() under the lock, wake_up_q() after). Because the win is a shorter pool->lock hold time, it shows up most clearly as lower enqueue latency under contention. Performance numbers (based on in-kernel workqueue microbenchmark) VMs and arm64 (Grace) is where this series is meant to pay off -- waking an idle CPU sitting in wfi costs an IPI (on arm; similar type of operation on VMs), so doing it under pool->lock lengthens the critical section. The arm64 bare-metal numbers match what the x86-or-arm64 VM showed: affinity_scope baseline patched tput p95 (items/s) (items/s) gain drop -------------- --------- --------- ------ ------ cpu 2,569,880 3,029,740 +17.9% -13.6% smt 2,586,485 3,044,788 +17.7% -14.0% cache_shard 572,055 797,621 +39.4% -37.1% cache 538,132 724,997 +34.7% -30.1% numa 528,673 658,215 +24.5% -20.5% system 524,287 614,486 +17.2% -21.1% (p95 drop = change in p95 enqueue latency; negative is better.) (tput gain = number of requests enqueued per sec; bigger is better.) Patch 1 is a pure refactor introducing kick_pool_pick(). Patch 2 defers the wakeup on the enqueue path (__queue_work()). Patch 3 defers the wakeup on the per-work chain-kick path (process_one_work()). Changes in v3: - Drop the "park kicked worker on pool->kicked_list" patch (v2 1/4). * That is a fix that is independent of this patch, in case we want to revamp it, it can be sent separately. - Link to v2: https://lore.kernel.org/r/20260603-fastwake-v2-0-2977512fe7fa@debian.org Changes in v2: - Close the idle_cull_fn() vs kicked-worker race by parking the kicked worker on a new pool->kicked_list under pool->lock (new patch 1). Reported by Hillf Danton. - Use the wake_q machinery (wake_q_add() / wake_up_q() via raw_spin_unlock_wake()) instead of plumbing a task_struct out of the helper by hand. Suggested by Sebastian Andrzej Siewior. - Link to v1: https://lore.kernel.org/r/20260526-fastwake-v1-0-e69ad86923e6@debian.org Signed-off-by: Breno Leitao --- To: Tejun Heo To: Lai Jiangshan Cc: linux-kernel@vger.kernel.org --- Breno Leitao (3): workqueue: split kick_pool() into kick_pool_pick() + wake_up_q() workqueue: defer the worker wakeup outside pool->lock in __queue_work() workqueue: defer the worker wakeup outside pool->lock in process_one_work() kernel/workqueue.c | 42 +++++++++++++++++++++++++++++++++--------- 1 file changed, 33 insertions(+), 9 deletions(-) --- base-commit: 8d6dbbbe3ba62de0a63e962ee004afb848c8e3ac change-id: 20260526-fastwake-02982fd66312 Best regards, -- Breno Leitao