From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 18BCD3EB0E6 for ; Fri, 26 Jun 2026 09:58:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782467897; cv=none; b=fS4wVVo8jCOrUTK8JCfw/sdt7up3uwfnNKJdKpzC4xbe+hNdUxqsVqUYNsJkIglmaYUnDFe50PVYUCnz38E1WZSw37TeLjDqlW9k1Z+uS+NFXn26a5Oon657r0FrYUNzZNJgbs9VjrGQBOScUC3HdeFEFWUeaB/7+9susYsttZc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782467897; c=relaxed/simple; bh=h+zFXio8vesvVp2GdePl59IS/03IfMdabWk1v18Av5Y=; h=From:Subject:Date:Message-Id:MIME-Version:Content-Type:To:Cc; b=kzyhbLtCcoV99DVZ9i0flLJozzL4zyYxjN5taWFXikMo/RAHxIyIBxrkK2H4DkW4INifTuCGVygeBC1T8xcYJ+8XVeZNdFhQ/HdtKo9rU+crH+lCtJj0UHRSFmpIsLzrdImWGyXpHkfkbLWxFJpUpwKs+k2Grby27O2589Y+K5U= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org; spf=pass smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=TwXpwG12; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="TwXpwG12" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:Content-Transfer-Encoding: Content-Type:MIME-Version:Message-Id:Date:Subject:From:Reply-To:Content-ID: Content-Description:In-Reply-To:References; bh=yoESphuiIurZlmcTuzYbMuUmh3t945TEa2gee5s+etE=; b=TwXpwG12jomcfDxHcRYTKI013p sra2qKDK+4BMcR5Er/1y/vYxndUUhOKphYRNin3vllS22VGFoT3xEm1jfDcS1/KkbQ98GVWuTiS3C jUbXZAJurr5rQ+jWBUM+zrWvFA5R+NxY+/+DDTvz2GrRRR9srr3Hlla0mQAexuP5lIT4lAvXRr9bd EPJvfTk/Os9GDM2uWY+aMllMPpBH4A4qTca+W0WzyrX1eZ0FJWwqitBed/Ds9veC3rYNo2DOZwOR6 q+FDVzUhDwTJTWSRkXt9vB5B7VtI+C/7VhFRXXDQsCsVMkNz9tsE736GpGIdxbzw6VJyjGahj1hg+ wSl8DCxw==; Received: from authenticated-user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.96) (envelope-from ) id 1wd3Jy-003t4I-2B; Fri, 26 Jun 2026 09:58:03 +0000 From: Breno Leitao Subject: [PATCH v5 0/3] workqueue: Shrink the lock time Date: Fri, 26 Jun 2026 02:57:52 -0700 Message-Id: <20260626-fastwake-v5-0-9ae2f1867234@debian.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-B4-Tracking: v=1; b=H4sIACFNPmoC/2XOwQ7CIAyA4VchPYthHQPZyfcwHtgoiiabgYkas 3c36MEZj036/e0TEsVACVr2hEg5pDAO0LJmxaA/2uFAPDhoGaBAJRpU3Ns03eyZuECzQe+Uqiu EFYNLJB/u79Ru/5nTtTtRPxVfNo4hTWN8vG9lLHufrBL1N5uRC45G66ZCT9rbraMu2GE9xgOUb q4Xslo8lGsuuDbOVsb3Tmz8n5QLiXIhZZGdcrqTRlopf+Q8zy+JfhyHJwEAAA== X-Change-ID: 20260526-fastwake-02982fd66312 To: Tejun Heo , Lai Jiangshan Cc: linux-kernel@vger.kernel.org, marco.crivellari@suse.com, frederic@kernel.org, bigeasy@linutronix.de, Hillf Danton , Breno Leitao , kernel-team@meta.com, kmagar@redhat.com, psuriset@redhat.com, david.dai@linux.dev X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=openpgp-sha256; l=3709; i=leitao@debian.org; h=from:subject:message-id; bh=h+zFXio8vesvVp2GdePl59IS/03IfMdabWk1v18Av5Y=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBqPk0mMh7uP4dGiwlQK9h7ZakJpXArL9W21A/cR muCmmm9O2yJAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCaj5NJgAKCRA1o5Of/Hh3 bdr5EACSbhy9LcITotIcD2AT1lGHBD2/TeMGjuRYMk2UPme6uL74ZAfyUbbL+C4oCcYz9oR1gPh fzFLMW6G5D8hRmK4MC2fknAu6/TJGYxxVycHrX8YiKjvaH2Q13aE5V2yI4s9+ns+uDFAmQbcIkF /k7KrMF/NkAbUfab/iQqOna/bJLIlxE92ZxjvWWHx5zkgI4nakEiKXzMBaDlycTfF9eoSTQd2bd eNB1JrBtY3VgyQxbJh7MHzMGlZXQp6/cCUXTELCugmbad+rV+/nmH2++yw4vN9s64957pFPoW0k UqMTbozCR+ukbegi+UraZSBQNh1DX8dlewlPnhee0NNbb27wCZDv4t0jOHXM7GJwaTRrgKbhEGE SbXaitbIVILkd5fu4Oj/BsDrO3+ITATTU6qHZax5AgW+9X3Qj+6ILbMvCnA8SJVLDKFzUayIF7U Ty0lOtFEdFQYBKvxamdO2Sun7xs/Ybv+J8yIXzsjlFcB9V523HLgpfo4GDc27tuzkY7qL4Df1+R kEqV28vFZg9ZCBcjAN1dr8is/r9XzP+ctnxgmlaxLmZmblzflUopD3SGXeiADdW/Z2Q2+Cdx73t rqLlHIx7I5azFVFzj+1Dt5dWS/dleEXYLk7UvCnBLX4cxamiOLti/g5hEatQs9H7pzoLzDJ7hMl vYd9JKAg4ypHMwQ== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao The goal of this patchset is to decrease the time spent under the workqueue pool->lock. Currently the worker process is woken up inside pool->lock. The wakeup ends in wake_up_process(), which takes the target task's rq->lock, so rq->lock nests under pool->lock on the two hottest paths of a contended unbound workqueue (__queue_work() enqueue and process_one_work() chain kick). On some architectures the wakeup is even more expensive: on arm64 waking a CPU that is idle (in wfi) issues an IPI. Doing all of that while holding pool->lock lengthens the locked region and hurts throughput on contended unbound pools. This series shortens the locked region by selecting and claiming the worker to wake under pool->lock, but issuing the actual wakeup after the lock is dropped, using the wake_q machinery (wake_q_add() under the lock, wake_up_q() after). Because the win is a shorter pool->lock hold time, it shows up most clearly as lower enqueue latency under contention. Performance numbers (based on in-kernel workqueue microbenchmark) VMs and arm64 (Grace) is where this series is meant to pay off -- waking an idle CPU sitting in wfi costs an IPI (on arm; similar type of operation on VMs), so doing it under pool->lock lengthens the critical section. Latested number (from v5) on a Grace arm64 host: affinity_scope baseline patched tput p95 (items/s) (items/s) gain drop -------------- --------- --------- ------ ------ cpu 3,580,440 3,486,014 -2.6% +3.5% smt 3,545,763 3,512,633 -0.9% +2.8% cache_shard 3,397,678 3,651,063 +7.5% -4.2% cache 720,368 797,914 +10.8% -9.8% numa 719,794 794,049 +10.3% -10.3% system 721,058 798,010 +10.7% -10.0% Signed-off-by: Breno Leitao Changes in v5: - Use wake_up_process() instead of the fancy wake_q_add() as raised by tejun. - Dropped the review-by from Sebastian, given the code changed. - Link to v4: https://lore.kernel.org/r/20260624-fastwake-v4-0-7b6d7b494a44@debian.org Changes in v4: - replace raw_spin_unlock_wake() with a standard raw_spin_unlock() + wake_up_q() (Sebastian Andrzej Siewior) - Link to v3: https://lore.kernel.org/r/20260616-fastwake-v3-0-79da19fcd08f@debian.org Changes in v3: - Drop the "park kicked worker on pool->kicked_list" patch (v2 1/4). * That is a fix that is independent of this patch, in case we want to revamp it, it can be sent separately. - Link to v2: https://lore.kernel.org/r/20260603-fastwake-v2-0-2977512fe7fa@debian.org Changes in v2: - Close the idle_cull_fn() vs kicked-worker race by parking the kicked worker on a new pool->kicked_list under pool->lock (new patch 1). Reported by Hillf Danton. - Use the wake_q machinery (wake_q_add() / wake_up_q() via raw_spin_unlock_wake()) instead of plumbing a task_struct out of the helper by hand. Suggested by Sebastian Andrzej Siewior. - Link to v1: https://lore.kernel.org/r/20260526-fastwake-v1-0-e69ad86923e6@debian.org --- Breno Leitao (3): workqueue: split kick_pool() into kick_pool_pick() workqueue: defer the worker wakeup outside pool->lock in __queue_work() workqueue: defer the worker wakeup outside pool->lock in process_one_work() kernel/workqueue.c | 51 ++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 44 insertions(+), 7 deletions(-) --- base-commit: 8d6dbbbe3ba62de0a63e962ee004afb848c8e3ac change-id: 20260526-fastwake-02982fd66312 Best regards, -- Breno Leitao