From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3E95C3DA7F for ; Wed, 31 Jul 2024 06:49:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9381B6B0088; Wed, 31 Jul 2024 02:49:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 71DE86B0092; Wed, 31 Jul 2024 02:49:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 28E5F6B0088; Wed, 31 Jul 2024 02:49:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id C88DF6B0093 for ; Wed, 31 Jul 2024 02:49:26 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6B401A215F for ; Wed, 31 Jul 2024 06:49:26 +0000 (UTC) X-FDA: 82399121532.21.0AB7589 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf02.hostedemail.com (Postfix) with ESMTP id B61D080014 for ; Wed, 31 Jul 2024 06:49:24 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=aGUYTfmR; spf=pass (imf02.hostedemail.com: domain of chrisl@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=chrisl@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1722408537; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mZMH2Ro7MXV9ZIOqWjTQ/mCW01pKL21+YmWpGPfLWFo=; b=otu4PiYM1twGLvlWznoqBDjjTzu8nOHmeHyqTzo6O4AQhFSrGcJ0y/x5wHvtSZf0RroAaM PhXJq8B0XvJx5WEE7GYCOkdlL50KU0r2ooQJ5Opb65LIl5JexjJ3i81kvSJ+Qxv+BvBPYO vDcgBQSUB7BhNyfZbd5JIM6N2vKpRxw= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=aGUYTfmR; spf=pass (imf02.hostedemail.com: domain of chrisl@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=chrisl@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1722408537; a=rsa-sha256; cv=none; b=lvAluzBeQJgsBUVkZJmBtEMp4ooMV2m6UomIDkNMsIoiTNLA/RTmvXFam/u6FDSJPG8plK g4KktyPFQwFBO+NCZNuzQSg4yCa+dZqWTFHNHU2EmQPOxZqMuON2+L+k+Q86NZRO1mYPzn rK7wxWlYcfECCt3Bvk1tkBuSODo5QOA= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 51895621F7; Wed, 31 Jul 2024 06:49:23 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C268BC4AF0C; Wed, 31 Jul 2024 06:49:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1722408563; bh=LjLGrYfGSpypFUL96oAD4RmEiSL9/y3TWaJKb5zfZhE=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=aGUYTfmR5q32Z0zxb2/wyFW/QbQQ23Efur5PxJUzlVbMYXuYs7ALfTF8G7eFkWuVh mP+TkWiOMDDNrTjs0WIymVBPV06bIR//zu6ftunKi2VIBQCsRK7KW/ZGIO3+ElQwno hy8nsdhEIjtvbtAE395ivBfkcpZT7U2ZkyhmGNiJGk646FEHJQ8MTyCkEGrRgqO312 u3FHrbZ07kBi64GsA+XVa5tnntHpByXK2Vi0uQi50pBcBxBrYw2+FUPXJp6WlknpuW jlbh/caZLD20sJEFqJtxop265nlELUYfZWBrLb3vQbOR3ZVsK/VaL/jAdlS9mm8d0R Li4TyrhURQ42A== From: chrisl@kernel.org Date: Tue, 30 Jul 2024 23:49:17 -0700 Subject: [PATCH v5 5/9] mm: swap: skip slot cache on freeing for mTHP MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20240730-swap-allocator-v5-5-cb9c148b9297@kernel.org> References: <20240730-swap-allocator-v5-0-cb9c148b9297@kernel.org> In-Reply-To: <20240730-swap-allocator-v5-0-cb9c148b9297@kernel.org> To: Andrew Morton Cc: Kairui Song , Hugh Dickins , Ryan Roberts , "Huang, Ying" , Kalesh Singh , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Chris Li , Barry Song X-Mailer: b4 0.13.0 X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: B61D080014 X-Stat-Signature: ty9x5jt747rzh31y8tjjpj8i7y8zjkqh X-HE-Tag: 1722408564-639290 X-HE-Meta: U2FsdGVkX18CnRsi4X58z00tr/yGE1fulANBFoKqeJehP4Lcyj2IMKFldFXOQ+24wbVl6FQdOgN9CS7Oo7QnloyzxmAdyLeOcwKBq1P5j0+mKChH1Lw041h6+6BeiLjRtKymZuXcv1+GuFJ/389wSLcNghOWGxcHw1NI5ZtMR5fbV7f4kgvmTOh52YYX9Q4PQk0PNi/KfIYSm5AbV7Lnq+n6mRArSuAD7ozKGj0FK9PXeL0qYdH8Hfp+2ReFhGfdOBLdFU213XRvDQlzuAevl+pYOz10vKc1FxWOynaFftQJhaqNNA+MXt1C0EhrK/Ry4hFz0YnRVqK+fJm7l1GlhpUM/4ctYoYWDZoT6efr3RQy8NSbZmnZKuKpZdBdY02B5wlYm6WXKZJbK5uFlnFW1znLSPFQIhRuC8VpsbU9LVrXgCHU7wMygYkSp+lHoi0mtutTpV7MW91Ho8u4G17KQPWKqNGxoqLwabzdTEJmrKRgr+tpDsAEmClr9Bj/dA1Trg62LaUsYpleb+IFF+2Wt/GNwPRX1G33LmWCF1S508Y+2/pPHy7rvGc20Mf+hOfG9+9YUOK0Grz7UGCejHEaVznnQGxTn81ZH8Wzi3Ybud6lxt8t2q+yrOEISz8j9fxIqNvvf9fXGadjVc/hQA/3tprCbay27qWAknFHM2m+SpIDcFYDSemJ3YKiJrEIcLWOXL0xof6Vl0L5SKqiTYsGSaX0Dy2llAbZPnzd+zwmjmO0fMZQt45QH4HxB024GzZS/JxGlA3zdTZwUHM8E25gRj8OXrzVWWpRECWnVWPE0iO6CAI4IkKk8y3rs2be0BuweZtQZE0v5IVXn5HXf4Zn4uTMfoqdErC6LPfCHMmD/tPQQLn+2gaya2ao0mHpki36eYaDlKPhQiBMSf7yz7iC/c5t0IGtaXlYC9U4rwGmzj6slqyrbJ+hQ2abIrs8qfMCyphl1vIMdW2QIzzJX4a K1uSpP3d yPGj7E/Z34EDN9c+oTJ50F3vFJ274hYTpGuEGclu+dVxr46h9f38MSNSKk8EeliKMwNsIUDj4Ypnz8RujTL+9dkppIcK5RV/eY0Kc0kMWB861mACjxZo+dkWln0AUNxp1NI1M4KHd69EUrunIFkQ6WzhKtEKERPSjFDnPD6X+wCX0uaEgWE3SZ40KW9u3yZClmWcM+HahxLs0kabo0gvN1Fjct411Ktz2rKtMpjzfUWrcj6CAzZhC5FQYBQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Currently when we are freeing mTHP folios from swap cache, we free then one by one and put each entry into swap slot cache. Slot cache is designed to reduce the overhead by batching the freeing, but mTHP swap entries are already continuous so they can be batch freed without it already, it saves litle overhead, or even increase overhead for larger mTHP. What's more, mTHP entries could stay in swap cache for a while. Contiguous swap entry is an rather rare resource so releasing them directly can help improve mTHP allocation success rate when under pressure. Signed-off-by: Kairui Song --- mm/swapfile.c | 59 ++++++++++++++++++++++++++--------------------------------- 1 file changed, 26 insertions(+), 33 deletions(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index 34e6ea13e8e4..9b63b2262cc2 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -479,20 +479,21 @@ static void inc_cluster_info_page(struct swap_info_struct *p, } /* - * The cluster ci decreases one usage. If the usage counter becomes 0, + * The cluster ci decreases @nr_pages usage. If the usage counter becomes 0, * which means no page in the cluster is in use, we can optionally discard * the cluster and add it to free cluster list. */ -static void dec_cluster_info_page(struct swap_info_struct *p, struct swap_cluster_info *ci) +static void dec_cluster_info_page(struct swap_info_struct *p, + struct swap_cluster_info *ci, int nr_pages) { if (!p->cluster_info) return; - VM_BUG_ON(ci->count == 0); + VM_BUG_ON(ci->count < nr_pages); VM_BUG_ON(cluster_is_free(ci)); lockdep_assert_held(&p->lock); lockdep_assert_held(&ci->lock); - ci->count--; + ci->count -= nr_pages; if (!ci->count) { free_cluster(p, ci); @@ -998,19 +999,6 @@ static int scan_swap_map_slots(struct swap_info_struct *si, return n_ret; } -static void swap_free_cluster(struct swap_info_struct *si, unsigned long idx) -{ - unsigned long offset = idx * SWAPFILE_CLUSTER; - struct swap_cluster_info *ci; - - ci = lock_cluster(si, offset); - memset(si->swap_map + offset, 0, SWAPFILE_CLUSTER); - ci->count = 0; - free_cluster(si, ci); - unlock_cluster(ci); - swap_range_free(si, offset, SWAPFILE_CLUSTER); -} - int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_order) { int order = swap_entry_order(entry_order); @@ -1269,21 +1257,28 @@ static unsigned char __swap_entry_free(struct swap_info_struct *p, return usage; } -static void swap_entry_free(struct swap_info_struct *p, swp_entry_t entry) +/* + * Drop the last HAS_CACHE flag of swap entries, caller have to + * ensure all entries belong to the same cgroup. + */ +static void swap_entry_range_free(struct swap_info_struct *p, swp_entry_t entry, + unsigned int nr_pages) { - struct swap_cluster_info *ci; unsigned long offset = swp_offset(entry); - unsigned char count; + unsigned char *map = p->swap_map + offset; + unsigned char *map_end = map + nr_pages; + struct swap_cluster_info *ci; ci = lock_cluster(p, offset); - count = p->swap_map[offset]; - VM_BUG_ON(count != SWAP_HAS_CACHE); - p->swap_map[offset] = 0; - dec_cluster_info_page(p, ci); + do { + VM_BUG_ON(*map != SWAP_HAS_CACHE); + *map = 0; + } while (++map < map_end); + dec_cluster_info_page(p, ci, nr_pages); unlock_cluster(ci); - mem_cgroup_uncharge_swap(entry, 1); - swap_range_free(p, offset, 1); + mem_cgroup_uncharge_swap(entry, nr_pages); + swap_range_free(p, offset, nr_pages); } static void cluster_swap_free_nr(struct swap_info_struct *sis, @@ -1343,7 +1338,6 @@ void swap_free_nr(swp_entry_t entry, int nr_pages) void put_swap_folio(struct folio *folio, swp_entry_t entry) { unsigned long offset = swp_offset(entry); - unsigned long idx = offset / SWAPFILE_CLUSTER; struct swap_cluster_info *ci; struct swap_info_struct *si; unsigned char *map; @@ -1356,19 +1350,18 @@ void put_swap_folio(struct folio *folio, swp_entry_t entry) return; ci = lock_cluster_or_swap_info(si, offset); - if (size == SWAPFILE_CLUSTER) { + if (size > 1) { map = si->swap_map + offset; - for (i = 0; i < SWAPFILE_CLUSTER; i++) { + for (i = 0; i < size; i++) { val = map[i]; VM_BUG_ON(!(val & SWAP_HAS_CACHE)); if (val == SWAP_HAS_CACHE) free_entries++; } - if (free_entries == SWAPFILE_CLUSTER) { + if (free_entries == size) { unlock_cluster_or_swap_info(si, ci); spin_lock(&si->lock); - mem_cgroup_uncharge_swap(entry, SWAPFILE_CLUSTER); - swap_free_cluster(si, idx); + swap_entry_range_free(si, entry, size); spin_unlock(&si->lock); return; } @@ -1413,7 +1406,7 @@ void swapcache_free_entries(swp_entry_t *entries, int n) for (i = 0; i < n; ++i) { p = swap_info_get_cont(entries[i], prev); if (p) - swap_entry_free(p, entries[i]); + swap_entry_range_free(p, entries[i], 1); prev = p; } if (p) -- 2.46.0.rc1.232.g9752f9e123-goog