From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A4EE73FF1D8 for ; Fri, 15 May 2026 16:24:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778862286; cv=none; b=VImflEfJ2NT/WcfzTSGFdp/AWFd0WgnTjwXS7KZBhBiVPnO1hqkeCt8F6jk+6gUoC38hd98AeA4ESfVRD28uXWQhlhyvjoD7+Cf1Yi9iEjEsHyMyuyYcYvmyAq4tfOPReY+9vECffD2pk4iXliKlYGUlw9X+NyP5aW/y0oNTzLc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778862286; c=relaxed/simple; bh=n8roWf7IiSB3Cp7Yv9UEETQTWbilYSL2KXuyP3eF33Y=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=rWZqRHWO0rUe/R2J6KTHDrbwcT/8XHBZrd0xPWZ2CPAPOkUeGRDFG0FsCeWUwmJI90IbSrV5Mwwz8iTosWutQKpdRH1Lomkl5dgVlZn6nlZumnwzKTlWjB9G6VK5NZl1je3dzsK2vsu/7H30/eC7zcNR6s8fTbfaf8xR+K1m9y0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=eJWAZeyo; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="eJWAZeyo" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8423DC2BCB0; Fri, 15 May 2026 16:24:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1778862286; bh=n8roWf7IiSB3Cp7Yv9UEETQTWbilYSL2KXuyP3eF33Y=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=eJWAZeyo/z3+nAssoq1IiCClb1Yh4ZM4vmqc0mW1Gep3eYOUhL8PFm77TbOUlgeG1 BG8DVRvBDu0hnBjONljT94jzp9iWd2R+y7pTPYdR3NoDAck08rwJ6SxvSbk6I0YKaf fiX7WKYgD3dVkOhyWqiCvnr9MbialyZydP2x67YjY+YX5bhJYGSCf726SKLN6AudvR lDE8VIS5iPd/YQOnHjtHhVy88qsQXjBw2WoGcsiqYuVMkX7HSHmtL4HdeK0IVfbNOW Ud+MfKejBmlxfwrGudKM+HAyLlgtqEgt/A7FQfOHePI+xnHbeDCkNP8XwlfBsIGCFX rM5Snaz55NGJw== From: "Harry Yoo (Oracle)" Date: Sat, 16 May 2026 01:24:27 +0900 Subject: [PATCH RFC 3/8] mm/slab: track capacity per sheaf Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20260516-sheaves-tuning-v1-3-221aa3e1d829@kernel.org> References: <20260516-sheaves-tuning-v1-0-221aa3e1d829@kernel.org> In-Reply-To: <20260516-sheaves-tuning-v1-0-221aa3e1d829@kernel.org> To: Vlastimil Babka , Andrew Morton , Hao Li , Christoph Lameter , David Rientjes , Roman Gushchin Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Suren Baghdasaryan , "Liam R. Howlett" X-Mailer: b4 0.16-dev Currently, only prefilled sheaves have a capacity field, used to record the requested (possibly oversized) capacity. To allow changing sheaf capacity at runtime, track the capacity for each sheaf so that checking if a sheaf is full would work even when changing cache capacity concurrently. Signed-off-by: Harry Yoo (Oracle) --- mm/slub.c | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index a1974523bba9..44f36ae32570 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -417,11 +417,9 @@ struct slab_sheaf { struct rcu_head rcu_head; struct list_head barn_list; /* only used for prefilled sheafs */ - struct { - unsigned short capacity; - bool pfmemalloc; - }; + bool pfmemalloc; }; + unsigned short capacity; unsigned short size; int node; /* only used for rcu_sheaf */ void *objects[]; @@ -2780,6 +2778,8 @@ static struct slab_sheaf *__alloc_empty_sheaf(struct kmem_cache *s, gfp_t gfp, if (unlikely(!sheaf)) return NULL; + sheaf->capacity = capacity; + stat(s, SHEAF_ALLOC); return sheaf; @@ -2816,7 +2816,7 @@ refill_objects(struct kmem_cache *s, void **p, gfp_t gfp, unsigned int min, static int refill_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf, gfp_t gfp) { - int to_fill = s->sheaf_capacity - sheaf->size; + int to_fill = sheaf->capacity - sheaf->size; int filled; if (!to_fill) @@ -5063,7 +5063,6 @@ kmem_cache_prefill_sheaf(struct kmem_cache *s, gfp_t gfp, unsigned short size) sheaf = alloc_empty_sheaf(s, gfp); if (sheaf) { - sheaf->capacity = s->sheaf_capacity; sheaf->pfmemalloc = false; if (sheaf->size < size && @@ -5688,13 +5687,13 @@ static void __pcs_install_empty_sheaf(struct kmem_cache *s, * Unlikely because if the main sheaf had space, we would have just * freed to it. Get rid of our empty sheaf. */ - if (pcs->main->size < s->sheaf_capacity) { + if (pcs->main->size < pcs->main->capacity) { barn_put_empty_sheaf(barn, empty); return; } /* Also unlikely for the same reason */ - if (pcs->spare->size < s->sheaf_capacity) { + if (pcs->spare->size < pcs->spare->capacity) { swap(pcs->main, pcs->spare); barn_put_empty_sheaf(barn, empty); return; @@ -5752,7 +5751,7 @@ __pcs_replace_full_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, goto alloc_empty; } - if (pcs->spare->size < s->sheaf_capacity) { + if (pcs->spare->size < pcs->spare->capacity) { swap(pcs->main, pcs->spare); return pcs; } @@ -5819,7 +5818,7 @@ __pcs_replace_full_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, * but in case we got preempted or migrated, we need to * check again */ - if (pcs->main->size == s->sheaf_capacity) + if (pcs->main->size == pcs->main->capacity) goto restart; return pcs; @@ -5850,7 +5849,7 @@ bool free_to_pcs(struct kmem_cache *s, void *object, bool allow_spin) pcs = this_cpu_ptr(s->cpu_sheaves); - if (unlikely(pcs->main->size == s->sheaf_capacity)) { + if (unlikely(pcs->main->size == pcs->main->capacity)) { pcs = __pcs_replace_full_main(s, pcs, allow_spin); if (unlikely(!pcs)) @@ -6015,7 +6014,7 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj) */ rcu_sheaf->objects[rcu_sheaf->size++] = obj; - if (likely(rcu_sheaf->size < s->sheaf_capacity)) { + if (likely(rcu_sheaf->size < rcu_sheaf->capacity)) { rcu_sheaf = NULL; } else { pcs->rcu_free = NULL; @@ -6139,7 +6138,7 @@ static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p) pcs = this_cpu_ptr(s->cpu_sheaves); - if (likely(pcs->main->size < s->sheaf_capacity)) + if (likely(pcs->main->size < pcs->main->capacity)) goto do_free; barn = get_barn(s); @@ -6156,7 +6155,7 @@ static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p) goto do_free; } - if (pcs->spare->size < s->sheaf_capacity) { + if (pcs->spare->size < pcs->spare->capacity) { swap(pcs->main, pcs->spare); goto do_free; } @@ -6172,7 +6171,7 @@ static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p) do_free: main = pcs->main; - batch = min_t(size_t, size, s->sheaf_capacity - main->size); + batch = min_t(size_t, size, main->capacity - main->size); memcpy(main->objects + main->size, p, batch * sizeof(void *)); main->size += batch; @@ -7613,7 +7612,7 @@ static int init_percpu_sheaves(struct kmem_cache *s) /* * Bootstrap sheaf has zero size so fast-path allocation fails. - * It has also size == s->sheaf_capacity, so fast-path free + * It has also size == sheaf->capacity, so fast-path free * fails. In the slow paths we recognize the situation by * checking s->sheaf_capacity. This allows fast paths to assume * s->cpu_sheaves and pcs->main always exists and are valid. -- 2.43.0