From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out-174.mta1.migadu.com (out-174.mta1.migadu.com [95.215.58.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B8104481B1 for ; Mon, 25 May 2026 03:23:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.174 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779679396; cv=none; b=Wu8PS6B8fB7kS9gVDs+BFp26J8IoUy9KwzaVjlzTbgvwQm8DLXBxfOub7dpR6Qk7bke/XGy5QdQS8Koyv5L1kA/84yVJsF0fXQb79a2Al1yr53V4gppZAgP8Mw/cYAM/bwfVxAc6/Gb61NDjD3ekz747POzO8EONFG9Vn2txs8E= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779679396; c=relaxed/simple; bh=4NmaLOi2gqawQmKHI9ZA6K96X4IKUDPKlHGUYnTLa2Q=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=aamLZRiVB0vndOruM9g3ZeSEUeIY0M9hwXObREKmgA0qizGKc6EBVxRJhwXWSPZh14DUGmmA44joUpfZ+3R5ADQX13lFa0gYezmfGc8Bu3f+/xW8WYQg9+A96bv9JQ2w0rRq5sHICGJycRUGD+f2pfbm2C2MmcZFOZUHmmWtfXQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=rUMQfxxC; arc=none smtp.client-ip=95.215.58.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="rUMQfxxC" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1779679391; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=oowUe/5TJUbyngfeGH7m6ljYwCauEoDHldpXCalFn1M=; b=rUMQfxxCsJGkN9LPb7hW9/kqAT9WysixDLbH9wBFKq4J9cHfT4gllSv+fzggu3vvWYnAW4 ayoomg0m/J8gQHUqIWr1s/6e6qMCFgCfrRWrBx9JGy5YCGfQu9Hm8TXldl4TN6vANUtpgo WZmQjpKuW8h4G6/DdNGg+LTt2Fxwfhc= From: Hao Li To: vbabka@kernel.org, harry@kernel.org, akpm@linux-foundation.org Cc: cl@gentwo.org, rientjes@google.com, roman.gushchin@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Hao Li Subject: [PATCH] mm/slub: batch-detach node partial slabs Date: Mon, 25 May 2026 11:22:09 +0800 Message-ID: <20260525032233.10847-1-hao.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT get_partial_node_bulk() used to move each selected slab from the node partial list to the local pc->slabs list using a remove_partial() and list_add() pair. In practice, the loop often detaches several adjacent slabs, so this repeatedly manipulates list pointers while holding n->list_lock, which causes unnecessary churn. Instead, track contiguous runs of matching slabs and move each run with list_bulk_move_tail() in one operation. This reduces list pointer churn inside the lock critical section. The mmap2 testcase shows a 5% improvement after applying this patch. Signed-off-by: Hao Li --- mm/slub.c | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 04692a6f9128..180973a4a3d2 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3739,6 +3739,7 @@ static bool get_partial_node_bulk(struct kmem_cache *s, bool allow_spin) { struct slab *slab, *slab2; + struct slab *first = NULL, *last = NULL; unsigned int total_free = 0; unsigned long flags; @@ -3757,8 +3758,15 @@ static bool get_partial_node_bulk(struct kmem_cache *s, struct freelist_counters flc; unsigned int slab_free; - if (!pfmemalloc_match(slab, pc->flags)) + if (!pfmemalloc_match(slab, pc->flags)) { + if (first) { + list_bulk_move_tail(&pc->slabs, + &first->slab_list, + &last->slab_list); + first = NULL; + } continue; + } /* * determine the number of free objects in the slab racily @@ -3775,15 +3783,21 @@ static bool get_partial_node_bulk(struct kmem_cache *s, && total_free + slab_free > pc->max_objects) break; - remove_partial(n, slab); - - list_add(&slab->slab_list, &pc->slabs); + if (!first) + first = slab; + last = slab; + slab_clear_node_partial(slab); + n->nr_partial--; total_free += slab_free; if (total_free >= pc->max_objects) break; } + if (first) + list_bulk_move_tail(&pc->slabs, &first->slab_list, + &last->slab_list); + spin_unlock_irqrestore(&n->list_lock, flags); return total_free > 0; } -- 2.54.0