From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 053BDC4167B for ; Thu, 2 Nov 2023 03:24:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 798A68E0018; Wed, 1 Nov 2023 23:24:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7223F8E0009; Wed, 1 Nov 2023 23:24:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 599CB8E0018; Wed, 1 Nov 2023 23:24:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 417638E0009 for ; Wed, 1 Nov 2023 23:24:51 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id F27B31CB59F for ; Thu, 2 Nov 2023 03:24:50 +0000 (UTC) X-FDA: 81411572340.15.E249B8A Received: from out-186.mta1.migadu.com (out-186.mta1.migadu.com [95.215.58.186]) by imf10.hostedemail.com (Postfix) with ESMTP id 40118C0004 for ; Thu, 2 Nov 2023 03:24:49 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="v1GKoBc/"; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf10.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.186 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698895489; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IPS3iWC0ordY9Z/gFrasLVfoOU0ee6NMzPFkJtmjEdA=; b=y0FSEBn4zmEm+t9ZLRqPbSnajqqH1ocfqhccFNY/uKvOndBRIktv0sEbCr+o+TN0NR0DxU N+8IRJVCKjHvPuE5RS4Iyv8YmgBrpBADAFvkTiPsTuqaa6D+tlJgUjeOnc2ysS9Uv2AXMZ gLTjdPNf0As67x3Wly00nuO0P3PwxZE= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="v1GKoBc/"; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf10.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.186 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698895489; a=rsa-sha256; cv=none; b=GBQmvEnQLS855tIVI6k4lT2KbBssgQKAgeYSCKNdWPyHKaGxHMW6w2naB8jTikEy/QP3bQ xnlRjBt/pA2l5uDrQp9AYH0JF+s+wMSbfprLZHyp+a9cNOO+NS+KOvNVCneS8YFiLwl6nf vUgaeZh9ZX4LJdLpqx+tZCRrlK62TIU= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1698895487; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IPS3iWC0ordY9Z/gFrasLVfoOU0ee6NMzPFkJtmjEdA=; b=v1GKoBc/GWi/6wA+3MMyQ6L15tftj7e2VTfNMKRmvlXo85TA0YUcUSnVVJ8Kga11asCboO btJ3b5nOE+cya1I08GWukxT+lICauUTNrau2aealf4F9CdmRScWkmlPsoHeV6UcabxAXZx LYjHaUg928T27sxO5LCHMigYm2gu+Xw= From: chengming.zhou@linux.dev To: vbabka@suse.cz, cl@linux.com, penberg@kernel.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou , Matthew Wilcox Subject: [PATCH v5 3/9] slub: Keep track of whether slub is on the per-node partial list Date: Thu, 2 Nov 2023 03:23:24 +0000 Message-Id: <20231102032330.1036151-4-chengming.zhou@linux.dev> In-Reply-To: <20231102032330.1036151-1-chengming.zhou@linux.dev> References: <20231102032330.1036151-1-chengming.zhou@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 40118C0004 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: cn8zwqgx98buif8nomugsg3z1fhwkzsf X-HE-Tag: 1698895489-574151 X-HE-Meta: U2FsdGVkX19PJBZNEXIlYfAT8injr3nxCEwyiloXNdc5I2/1kZwa3zHicBokbyixgvYysTPfVxIC8qTbSZ0XGH7HcJ8l7wVAfqukHw22vATUFPIXHanm3UoAJSXES1XS9yvc5oPhmYj/AuTrvZotJu0oYGoILuGkEopHtI3N92yE1Rj7wKAIdeZ0tgWPoJCcv4NaYuTuZ8lvZfB4tK7mmIpRxuo1F/dVrGF2tJ+4djmaOBSr9AgyVRRF+QAnLkuQVx2BuN91X/RnRs8pJMxGzEsxVOOep4sSSnBdhPUT/7ajCdmCe743CRf15kAkk944rP1pb6srkyI9w5DkrrAv3c+ULNgWN0Fr+vB/yScoOqItctm93uAwp4QXheu0U5Q7J+FyDtr+B1efw7+/F9htpLzkcHi1weZ01W3zxVlw5SIG9kITYnJQ584Ii8N/wTgxmTYm0nc20ivcMJm2tC/jh8kQQZwrsEK3Ur7WwwVXnxpckrpmYludMHFXreu38g6RWJ4t0icqdLYBfduq5Netyaj+D9M5MBev1hmdl57SsxmSYzIbrjB3lcqLRH3qxBPxVcW7esX+d9PgmVoBPki3MSbphxyHZflG7xE+VOeycI60IcCfC0iffFEdbYUatj8E6PTS+Ecr8ev2UqV3HrUEx4qyeosqbdmEQGVsMTwfmKciCZWBE3IMWFuczu2EYpk8p1pzjSKAyjzaVHr2yntXJefiBeVWLyvuhq1mac/RCx/bfyTTH/9FVBAlZMXXRTn2NEois76W8u6KK2GFKO2WuwikdnhCQLIlSOQwM5idT4SK2SJ4tS+j5S9W/pgvzjo4k0khD91b+vJy2H3Z+Hvgvx+SqnzyfM2Ehuc8DcqAW+2ooltZPTG9LHbKFPgmUFsvCaZBIECjjUziuP6HHhqNXK9BVvKVV+mwgyguptkbzkZi89/qZ+pkmJZEGCQlb1mVxrClOi8IvzNM8U/wkcL 2JwzGSKm RpdUT/mJPfIjwfjqotvTv/FydGQIemXo+3NU3xurJhT6pOhpZCOpC3/KgijWKzA8we7bQyLCXuXztTNlzlPtE90QFwpAk7p/wOBSLJ0nzvtVY+fUpi0pf708ZIozvj0phUDZBRlW567+quGDG+fQMIlSHoHDncNtG0xEC8pjSe2NYLYydNUXwND74osGdHliLJH+gZTY4+5n6SbHKNylziJYOWLyfbPe1KrFHjvbK1zn82Xa9D8ztCeakfaTEt/jb0e+eF0mrYBE+skz9CuQ08GGPrcsx8WCw7CEt X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chengming Zhou Now we rely on the "frozen" bit to see if we should manipulate the slab->slab_list, which will be changed in the following patch. Instead we introduce another way to keep track of whether slub is on the per-node partial list, here we reuse the PG_workingset bit. We use __set_bit and __clear_bit directly instead of the atomic version for better performance and it's safe since it's protected by the slub node list_lock. Suggested-by: Matthew Wilcox Signed-off-by: Chengming Zhou Reviewed-by: Vlastimil Babka Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/slub.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/mm/slub.c b/mm/slub.c index 03384cd965c5..eed8ae0dbaf9 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2116,6 +2116,25 @@ static void discard_slab(struct kmem_cache *s, struct slab *slab) free_slab(s, slab); } +/* + * SLUB reuses PG_workingset bit to keep track of whether it's on + * the per-node partial list. + */ +static inline bool slab_test_node_partial(const struct slab *slab) +{ + return folio_test_workingset((struct folio *)slab_folio(slab)); +} + +static inline void slab_set_node_partial(struct slab *slab) +{ + __set_bit(PG_workingset, folio_flags(slab_folio(slab), 0)); +} + +static inline void slab_clear_node_partial(struct slab *slab) +{ + __clear_bit(PG_workingset, folio_flags(slab_folio(slab), 0)); +} + /* * Management of partially allocated slabs. */ @@ -2127,6 +2146,7 @@ __add_partial(struct kmem_cache_node *n, struct slab *slab, int tail) list_add_tail(&slab->slab_list, &n->partial); else list_add(&slab->slab_list, &n->partial); + slab_set_node_partial(slab); } static inline void add_partial(struct kmem_cache_node *n, @@ -2141,6 +2161,7 @@ static inline void remove_partial(struct kmem_cache_node *n, { lockdep_assert_held(&n->list_lock); list_del(&slab->slab_list); + slab_clear_node_partial(slab); n->nr_partial--; } @@ -4833,6 +4854,7 @@ static int __kmem_cache_do_shrink(struct kmem_cache *s) if (free == slab->objects) { list_move(&slab->slab_list, &discard); + slab_clear_node_partial(slab); n->nr_partial--; dec_slabs_node(s, node, slab->objects); } else if (free <= SHRINK_PROMOTE_MAX) -- 2.20.1