From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mta1.migadu.com (out-13.mta1.migadu.com [95.215.58.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5B6E71A01BE for ; Mon, 24 Aug 2026 12:20:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.13 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787574031; cv=none; b=BDbWzsA+X7YnWaHobbuJOQi1kz1qJwa0JvbQstjlbMOKYvlQFdg+VUfUDrc5ezAsAnMyQiVMUVthXEw6SHLB7aPjubBJeOKo2yfstpoRCaJgimYSgokSnkOCz0heccTKyPioGqVNvXVCgB3WMvEmn5LvWndFR6ZyjqjZy4kUjpc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787574031; c=relaxed/simple; bh=GT1jLiel5M2UG4rmfoDZ+9MbixFcM3ZroPSpTa3Q+Uk=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=TwyJU7Yl3DuHC1+aNxeFcIvUEKCnL9YroOjIoUmp7bDCyPFVhP048a07uR+kzznyUHTJCLM5ZouWnKDQocb9/eZY+grmFjhmiy6M024VW3Q2+iG+/ri8H4cg6hPT/fvv6ppbKzyEOeVfWH8wF2AJAigkJheOTKjBBPyNSRm9xSk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=GL8u/+fA; arc=none smtp.client-ip=95.215.58.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="GL8u/+fA" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=GT1jLiel5M2UG4rmfoDZ+9MbixFcM3ZroPSpTa3Q+Uk=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1787574027; v=1; x=1788178827; b=GL8u/+fAV4bUEAkYQnYZifh5BVpxSa5ps63Cn9XtNr6sho2ly39wH0J5f10MXM8gyCDl+jjI IHD5Abq8+KvuZ7pzi1kVm+q8F4UV5oNOHZzNfxJoLQOCdiTqrWw+bmSZAHNRV9InL7JxNuJ/pW8 6nGJR7ZFT9aDxaob/aCXWgvA= X-Envelope-To: linux-kernel@vger.kernel.org Received: from fedora (117.129.78.49) by smtp.migadu.com with ESMTPS id ac3c30d9b503febc; Mon, 24 Aug 2026 12:20:27 +0000 X-Mizu-Trace-ID: ac3c30d9b503febc X-Migadu-Flow: FLOW_OUT From: Hao Li To: vbabka@kernel.org, harry@kernel.org, akpm@linux-foundation.org Cc: cl@gentwo.org, rientjes@google.com, roman.gushchin@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Hao Li Subject: [RFC PATCH 0/2] mm/slub: reduce list_lock contention with slab parking Date: Mon, 24 Aug 2026 20:19:50 +0800 Message-ID: <20260824122004.3652-1-hao.li@linux.dev> X-Mailer: git-send-email 2.54.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit This patch series might sound a bit wild, but the initial numbers don't look too bad so far. I would really appreciate any feedback and discussion :) On a will-it-scale mmap1 run with 192 processes, list_lock is the top contention point: __slab_free() and __refill_objects_node() together spend 44% of cycles in native_queued_spin_lock_slowpath. The free slowpath takes the lock mainly to add slabs that became non-full to the partial list. By adding extra instrumentation to __slab_free(), I collect the following data for the maple_node cache (in counts): partial->partial 843017414 full->partial 550719384 partial->empty 17459564 full->empty 2 We can see that full -> partial transitions account for a significant proportion, and optimizing them can help reduce lock contention to some extent. This series introduces the parking mechanism to address this issue. When the trylock fails during a full -> partial/empty transition, __slab_free() parks the slab on a per-node llist instead of waiting. The paths that consume the partial list (sheaf refill, alloc slowpath, shrink, cache destruction) unpark the slabs after taking the lock, and a delayed work covers the case where none of them runs. Patch 1 cleans up the case handling in __slab_free(), no functional change. Patch 2 introduces the parking mechanism. Tested with will-it-scale mmap1 (192 processes). Summary data ------------ throughput 29237910 -> 35585663 (+21.7%) alloc_slab,free_slab -52% cmpxchg_double_fail -85% perf data without this patchset: - 44.22% [kernel] [k] native_queued_spin_lock_slowpath 43.48% native_queued_spin_lock_slowpath - _raw_spin_lock_irqsave - 23.80% __refill_objects_node - 19.12% __slab_free perf data with this patchset: - 30.39% [kernel] [k] native_queued_spin_lock_slowpath 29.82% native_queued_spin_lock_slowpath - _raw_spin_lock_irqsave - 29.06% __refill_objects_node Additionally, the number of partial slabs and the number of objects show no noticeable change before and after applying this patchset, indicating that this change has a negligible impact on slab fragmentation. Detailed data ------------- metric before after delta change ========================================================================================== alloc_fastpath 155,417 168,534 13,117 +8.44% alloc_slab 55,679,646 26,702,510 -28,977,136 -52.04% alloc_slowpath 0 0 0 +0.00% barn_get 2,715 2,771 56 +2.06% barn_get_fail 2 0 -2 -100.00% barn_put 2,715 2,770 55 +2.03% barn_put_fail 1,370,488,457 1,668,257,974 297,769,517 +21.73% cmpxchg_double_fail 3,827,935 549,427 -3,278,508 -85.65% free_add_partial 1,684,340,964 2,161,921,625 477,580,661 +28.35% free_fastpath 31,766 32,979 1,213 +3.82% free_rcu_sheaf 43,855,689,417 53,384,314,553 9,528,625,136 +21.73% free_rcu_sheaf_fail 0 0 0 +0.00% free_remove_partial 55,678,459 26,685,274 -28,993,185 -52.07% free_slab 55,678,459 26,701,099 -28,977,360 -52.04% free_slowpath 107,771,739 63,197,344 -44,574,395 -41.36% min_partial 5 5 0 +0.00% object_size 256 256 0 +0.00% objects 14,504 14,336 -168 -1.16% objects_partial 14,504 14,208 -296 -2.04% objs_per_slab 64 64 0 +0.00% park_slab - 2,147,963,530 - absent partial 1,398 1,367 -31 -2.22% sheaf_alloc 743,660,288 1,284,618,991 540,958,703 +72.74% sheaf_capacity 32 32 0 +0.00% sheaf_flush 43,855,651,858 53,384,274,983 9,528,623,125 +21.73% sheaf_free 743,660,280 1,284,618,967 540,958,687 +72.74% sheaf_prefill_fast 17,585,335,850 21,378,958,833 3,793,622,983 +21.57% sheaf_prefill_oversize 0 0 0 +0.00% sheaf_prefill_slow 2,060 2,051 -9 -0.44% sheaf_refill 43,963,424,505 53,447,473,167 9,484,048,662 +21.57% sheaf_return_fast 17,585,336,335 21,378,959,334 3,793,622,999 +21.57% sheaf_return_slow 1,402 1,277 -125 -8.92% slabs 1,398 1,369 -29 -2.07% total_objects 89,472 87,616 -1,856 -2.07% unpark_event - 315,397,838 - absent unpark_slab - 2,147,963,530 - absent derived before after change ============================================================================================= PARK_SLAB / FREE_ADD_PARTIAL - 99.35% absent UNPARK_SLAB / UNPARK_EVENT - 6.81 absent PARK_SLAB - UNPARK_SLAB - 0 absent page allocator churn (alloc_slab + free_slab) 111,358,105 53,403,609 -52.04% Note that some metrics have very small absolute values (such as alloc_fastpath, partial, slabs, and total_objects) and are subject to noise. Across multiple test runs, their rate of change fluctuates between positive and negative, which supports the hypothesis that this is measurement noise and demonstrates that this patch has no noticeable impact on these metrics. For metrics with large absolute values, their trends are distinct. The data indicates that the primary benefit of this approach is significantly relieved pressure on the buddy system, with page allocator churn reduced by 52%. Additionally, free_slowpath decreases by 41%, and cmpxchg_double_fail decreases by 85%. The PARK_SLAB / FREE_ADD_PARTIAL ratio reaches 99.35%, which indicates that the vast majority of partial slabs are added back to the partial list via the parking mechanism, reflecting that the lock stayed saturated and nearly all additions avoided waiting for the lock. The ratio of UNPARK_SLAB / UNPARK_EVENT shows that each unpark event processes roughly 6 slabs. PARK_SLAB - UNPARK_SLAB being 0 confirms that no parked slabs are left stranded. I also observe increases in both sheaf_alloc and sheaf_free, which could currently be attributed to faster allocation and free paths resulting from the overall performance improvement. However, I'm not sure about this, which is part of why this is posted as an RFC. Based on slab/for-next. Hao Li (2): mm/slub: make the case handling in __slab_free() easier to follow mm/slub: introduce slab parking to reduce list_lock contention mm/slub.c | 326 +++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 274 insertions(+), 52 deletions(-) base-commit: e7f630142df2afccce90555e4972e60008222311 -- 2.55.0