From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BBEDD38F63E for ; Mon, 2 Mar 2026 09:55:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772445352; cv=none; b=jHN2054ChPQHNf/oSuT70KczPp9DtfQdnAaZmZNZMmM/rgPo+qxHIpTkz0kpbqJnYVytntIHmYvYjUm2+3W7T5pZOirYME3T10M30lxPhxUr5/uReGlnn4YsL7DNpu2lCvKRemvCBIZvwsMyjpwSgvj1f97Iijdkm3sctShuaPU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772445352; c=relaxed/simple; bh=baMHQ+epwV7528eqrCDXzQnEtKpJ0piYIDHEF8wqGvQ=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=TpY3f1NOUKxbi0TaLSBoWXslHpSg5KKd5xZ/WnziVqbeEQnq+ZEkjQ1RwuDx2cFzAnHb3iPIE5Y73v2EMXrYRGoEAqXgzaJO8g63CyXbmGkB+CEa6a9Fh97OS1WbCmm8b3jJBDT9B+QzM7TDaVndSLYi0XBp8fzTvaYGTqw7DYU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=kER6V9GK; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="kER6V9GK" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 53495C19423; Mon, 2 Mar 2026 09:55:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1772445352; bh=baMHQ+epwV7528eqrCDXzQnEtKpJ0piYIDHEF8wqGvQ=; h=From:To:Cc:Subject:Date:From; b=kER6V9GKv2gWtuTlCGqhiEJO6jHnAn2bc761DfOsrxnkjFj0Tbk8h45aeIeCb4SZp 9PEkkeeaLilDKfDqioTLJpBQ1GSU3Bkjg9U5zqgBor992dYrkytXkarmFIApB3oLh/ SCASmQHA8Ja+ZipqZDgNBO8AXhfY9IDDDyWWeVAz84HzcmR26X1wb1aDRv0OLntU4R hWBsiruMkKQz1BJEVPh8evSMxdf8j4XiYQW4Za3HHW/zXZGHU8ao33y6fL+Jt6r94b vKegqlMBioJWx/Sy2XwO+tTogwgRPeAifcU9Vbb/RS9Py4/o7eU9oBnojVFAL2eIk6 O+8VhalSzt6aw== From: "Vlastimil Babka (SUSE)" To: Harry Yoo Cc: Hao Li , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Vlastimil Babka (SUSE)" , Ming Lei Subject: [PATCH slab/for-next-fixes] mm/slab: allow sheaf refill if blocking is not allowed Date: Mon, 2 Mar 2026 10:55:37 +0100 Message-ID: <20260302095536.34062-2-vbabka@kernel.org> X-Mailer: git-send-email 2.53.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Ming Lei reported [1] a regression in the ublk null target benchmark due to sheaves. The profile shows that the alloc_from_pcs() fastpath fails and allocations fall back to ___slab_alloc(). It also shows the allocations happen through mempool_alloc(). The strategy of mempool_alloc() is to call the underlying allocator (here slab) without __GFP_DIRECT_RECLAIM first. This does not play well with __pcs_replace_empty_main() checking for gfpflags_allow_blocking() to decide if it should refill an empty sheaf or fallback to the slowpath, so we end up falling back. We could change the mempool strategy but there might be other paths doing the same ting. So instead allow sheaf refill when blocking is not allowed, changing the condition to gfpflags_allow_spinning(). The original condition was unnecessarily restrictive. Note this doesn't fully resolve the regression [1] as another component of that are memoryless nodes, which is to be addressed separately. Reported-by: Ming Lei Fixes: e47c897a2949 ("slab: add sheaves to most caches") Link: https://lore.kernel.org/all/aZ0SbIqaIkwoW2mB@fedora/ [1] Signed-off-by: Vlastimil Babka (SUSE) --- mm/slub.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index b1e9f16ba435..17b200695e9b 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4567,7 +4567,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, struct slab_sheaf *empty = NULL; struct slab_sheaf *full; struct node_barn *barn; - bool can_alloc; + bool allow_spin; lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock)); @@ -4588,8 +4588,9 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, return NULL; } - full = barn_replace_empty_sheaf(barn, pcs->main, - gfpflags_allow_spinning(gfp)); + allow_spin = gfpflags_allow_spinning(gfp); + + full = barn_replace_empty_sheaf(barn, pcs->main, allow_spin); if (full) { stat(s, BARN_GET); @@ -4599,9 +4600,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, stat(s, BARN_GET_FAIL); - can_alloc = gfpflags_allow_blocking(gfp); - - if (can_alloc) { + if (allow_spin) { if (pcs->spare) { empty = pcs->spare; pcs->spare = NULL; @@ -4612,7 +4611,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, local_unlock(&s->cpu_sheaves->lock); - if (!can_alloc) + if (!allow_spin) return NULL; if (empty) { @@ -4632,11 +4631,8 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, if (!full) return NULL; - /* - * we can reach here only when gfpflags_allow_blocking - * so this must not be an irq - */ - local_lock(&s->cpu_sheaves->lock); + if (!local_trylock(&s->cpu_sheaves->lock)) + goto barn_put; pcs = this_cpu_ptr(s->cpu_sheaves); /* @@ -4667,6 +4663,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, return pcs; } +barn_put: barn_put_full_sheaf(barn, full); stat(s, BARN_PUT); -- 2.53.0