From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54846C3271E for ; Thu, 11 Jul 2024 07:29:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D5ED76B009A; Thu, 11 Jul 2024 03:29:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CE7FE6B009D; Thu, 11 Jul 2024 03:29:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A9EC56B009C; Thu, 11 Jul 2024 03:29:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 83BEE6B0099 for ; Thu, 11 Jul 2024 03:29:28 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 40D0C80429 for ; Thu, 11 Jul 2024 07:29:28 +0000 (UTC) X-FDA: 82326646416.26.C977DE9 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf03.hostedemail.com (Postfix) with ESMTP id DD8AD20015 for ; Thu, 11 Jul 2024 07:29:25 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=aeMOCOvu; spf=pass (imf03.hostedemail.com: domain of chrisl@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=chrisl@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720682941; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=o9pmGV8Yo3HxbqV4pHn9jjGUBHGDGWfh98US8kvY0aI=; b=tPJ51jCvxp22XJ1kpFmzwUI8BMrTv+mWBmwx9CY2K24Bez4WezCWsZi4C4XS474xEEmtex Sagct6OzhXtxV3ZsdBXj250XS+WF+H4x8ImSx3m+cvPvaeHLmNzqfGkoX95iFZPNapERYM 8cqhiqFYop/H7dJ7OmA8zkFeLjDYFqs= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=aeMOCOvu; spf=pass (imf03.hostedemail.com: domain of chrisl@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=chrisl@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720682941; a=rsa-sha256; cv=none; b=6c7wF6Uu3OdFK4gi3BtBCAvRdOvoZk/AtZM/FPF6E1FyeK1y8jcD3y0XkmUb24/pHTcwyd s1i7ajQLGMs9gp66yb0mM5X5caqEHNva4jDU5ke0L/88qXiwhf4lpMqYXQGlsaFMYPIBi8 /OHf+ZYNrJgHMIUp/rl17zbYDrGUcGk= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 63D31CE1813; Thu, 11 Jul 2024 07:29:22 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 03A02C4AF0C; Thu, 11 Jul 2024 07:29:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1720682961; bh=GQQhDBkbVm07F633nVzv9d4hoKaeH6pAoRTRgLjaQmQ=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=aeMOCOvupqqC2Ej3StRoP1k7OPZxF8jo4beiiML+npBz7HZKH4qN/FxHXemgCWKl4 kP1rBI3p8z1t3UlHkyhh8gD1rACHQNRI19Z0KACtlrkrp7ShnMo/TtkNlUt9rZOtX1 EVkf9DPfQZIKVM6HBSxl+nULAKk1BtmI3EHx+a6kr56VVk6/wq5DMiFLlJTCIdXUfz XiFaRpxW/32iuxeNLx1GQQr0f/OwPBDnODLaCCGuUo0p8rFa0CcWVn3l9g+4V0f3tX QSWFyjLyxTlUZbH+EXqHPR2f7tFDCsrEkKB0mXK1S5jARo6rTfTHrT3GZPzpfY/gVv vecUSKI2Bfmtw== From: Chris Li Date: Thu, 11 Jul 2024 00:29:06 -0700 Subject: [PATCH v4 2/3] mm: swap: mTHP allocate swap entries from nonfull list MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit Message-Id: <20240711-swap-allocator-v4-2-0295a4d4c7aa@kernel.org> References: <20240711-swap-allocator-v4-0-0295a4d4c7aa@kernel.org> In-Reply-To: <20240711-swap-allocator-v4-0-0295a4d4c7aa@kernel.org> To: Andrew Morton Cc: Kairui Song , Hugh Dickins , Ryan Roberts , "Huang, Ying" , Kalesh Singh , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Chris Li , Barry Song X-Mailer: b4 0.13.0 X-Stat-Signature: e6skrhru9uisefj9i5up8ms1t91w81p7 X-Rspam-User: X-Rspamd-Queue-Id: DD8AD20015 X-Rspamd-Server: rspam02 X-HE-Tag: 1720682965-427172 X-HE-Meta: U2FsdGVkX18Oj4ESbwEvg9v1kc16KSRtKUugk7zpB76GBn1dEnJI6izU2RIwBoSaRtdNp8+RIYJTsOiL8CImH77KNomfX9ZapENINptCTdieXlt+EXjwlOB6hiIA3Jma6xw7FLC+7xS96q3nTmfL/hm8YITZralMlWr1eq8TTt17Z3PMvCG1b114VXWO91kh1hfqZQ4QXm4w12T5m/+Pr8u7fZQB7pj5+a56nQiQGGYTOTjLk9SlxunGUIxrYVNgt+4s9QpNe+vQ/MEeus/avjsOsZO9U/Yqwz7Zec9AWJPQdUrofV2igNM6eUzQkbTQq72vag4erMDwCMbcG6etxngdODEDyo9It+aQp5bGTNZgZbTsgCOqDcpo0zJOFFqlCwgIzA2EPUMBECm3bvt6TA+OGMNMBsc1JfoRqj8WusGVrzqz2x/GqnH612ev47A/uXysPhaeOQ+Tq2jZwmXkmG2dWLdJQSZ5M/jpN3yOZBoVdZwDkUGBKBnwn0WC6ceygzaouQDkW2YgCUVEySmg4395BwdGenHqwlLKqZJ68r6MhCr6AUGppjiD2egqp0EvlAtCa1d5+GmpKjEKUNqWruvMQaCPsjEfO0vH9ijy6uzNAWdH5mLAZnKlpBT+bwOOvR+FN75HIQi3eROaPp/nw/+sLj/Km8U5XLOk+bIVcodZYX19RaOCcwvFvlecWqLQrRkKAJ5Xdke7cIj8ZiGhrdZyeBkjZym9ttQuUBNH0g8+HWo+69QP25msWXBMbDVKi+YIuH+UWdvm/x1lGYiM6B0gvGkOeoKz4sJmLsBhEcH/XiuaTH292UZaO3gJqncw5QkYn/QzKxjmFQWJQ2GR9ipDFnE4Deu/GfRyxwtfTXiFeqB90M+aQqvtI9M2U2OHUYhG9CupbdbuHPAXzgOGS34oU4zlEyE4CmonkJjdWQvhD/w/yeQ96o4aegOrvxwm2FAeWBQGSE/ldf/WMGR 8uRuInfj tBgTWsUsAJWjnreFQSMu8WY7PCTbtWzyFSrUGR6eIidBDjrK7fCqtzRlIdAamoohwl7puZJHi/zSe4pnrxL9urwECBK5rkLFzxP6Nc9rsT+o6z2yp3Btpy6uXHFYxpe6zHniQInvlUp6DwbBzL7gz+GiV0Zpa41pd8U5T7absaqoEnzXVzpkgJSdF3TU0ZPh47LZJAkfEGGtLW2cbUG9SoMvuUfzr9//hRJN81rSvUQXLF3RoOctecSt/N2uCAQuYVdA2nWzn5SUi8gaIAvwvMdS2OQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Track the nonfull cluster as well as the empty cluster on lists. Each order has one nonfull cluster list. The cluster will remember which order it was used during new cluster allocation. When the cluster has free entry, add to the nonfull[order] list.  When the free cluster list is empty, also allocate from the nonempty list of that order. This improves the mTHP swap allocation success rate. There are limitations if the distribution of numbers of different orders of mTHP changes a lot. e.g. there are a lot of nonfull cluster assign to order A while later time there are a lot of order B allocation while very little allocation in order A. Currently the cluster used by order A will not reused by order B unless the cluster is 100% empty. Signed-off-by: Chris Li --- include/linux/swap.h | 4 ++++ mm/swapfile.c | 34 +++++++++++++++++++++++++++++++--- 2 files changed, 35 insertions(+), 3 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index e9be95468fc7..db8d6000c116 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -254,9 +254,11 @@ struct swap_cluster_info { */ u16 count; u8 flags; + u8 order; struct list_head list; }; #define CLUSTER_FLAG_FREE 1 /* This cluster is free */ +#define CLUSTER_FLAG_NONFULL 2 /* This cluster is on nonfull list */ /* @@ -296,6 +298,8 @@ struct swap_info_struct { unsigned long *zeromap; /* vmalloc'ed bitmap to track zero pages */ struct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */ struct list_head free_clusters; /* free clusters list */ + struct list_head nonfull_clusters[SWAP_NR_ORDERS]; + /* list of cluster that contains at least one free slot */ unsigned int lowest_bit; /* index of first free in swap_map */ unsigned int highest_bit; /* index of last free in swap_map */ unsigned int pages; /* total of usable pages of swap */ diff --git a/mm/swapfile.c b/mm/swapfile.c index f70d25005d2c..e13a33664cfa 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -361,14 +361,21 @@ static void swap_cluster_schedule_discard(struct swap_info_struct *si, memset(si->swap_map + idx * SWAPFILE_CLUSTER, SWAP_MAP_BAD, SWAPFILE_CLUSTER); - list_add_tail(&ci->list, &si->discard_clusters); + if (ci->flags) + list_move_tail(&ci->list, &si->discard_clusters); + else + list_add_tail(&ci->list, &si->discard_clusters); + ci->flags = 0; schedule_work(&si->discard_work); } static void __free_cluster(struct swap_info_struct *si, struct swap_cluster_info *ci) { + if (ci->flags & CLUSTER_FLAG_NONFULL) + list_move_tail(&ci->list, &si->free_clusters); + else + list_add_tail(&ci->list, &si->free_clusters); ci->flags = CLUSTER_FLAG_FREE; - list_add_tail(&ci->list, &si->free_clusters); } /* @@ -491,7 +498,12 @@ static void dec_cluster_info_page(struct swap_info_struct *p, struct swap_cluste ci->count--; if (!ci->count) - free_cluster(p, ci); + return free_cluster(p, ci); + + if (!(ci->flags & CLUSTER_FLAG_NONFULL)) { + list_add_tail(&ci->list, &p->nonfull_clusters[ci->order]); + ci->flags |= CLUSTER_FLAG_NONFULL; + } } /* @@ -550,6 +562,18 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, if (tmp == SWAP_NEXT_INVALID) { if (!list_empty(&si->free_clusters)) { ci = list_first_entry(&si->free_clusters, struct swap_cluster_info, list); + list_del(&ci->list); + spin_lock(&ci->lock); + ci->order = order; + ci->flags = 0; + spin_unlock(&ci->lock); + tmp = cluster_index(si, ci) * SWAPFILE_CLUSTER; + } else if (!list_empty(&si->nonfull_clusters[order])) { + ci = list_first_entry(&si->nonfull_clusters[order], struct swap_cluster_info, list); + list_del(&ci->list); + spin_lock(&ci->lock); + ci->flags = 0; + spin_unlock(&ci->lock); tmp = cluster_index(si, ci) * SWAPFILE_CLUSTER; } else if (!list_empty(&si->discard_clusters)) { /* @@ -964,6 +988,7 @@ static void swap_free_cluster(struct swap_info_struct *si, unsigned long idx) ci = lock_cluster(si, offset); memset(si->swap_map + offset, 0, SWAPFILE_CLUSTER); ci->count = 0; + ci->order = 0; ci->flags = 0; free_cluster(si, ci); unlock_cluster(ci); @@ -2919,6 +2944,9 @@ static int setup_swap_map_and_extents(struct swap_info_struct *p, INIT_LIST_HEAD(&p->free_clusters); INIT_LIST_HEAD(&p->discard_clusters); + for (i = 0; i < SWAP_NR_ORDERS; i++) + INIT_LIST_HEAD(&p->nonfull_clusters[i]); + for (i = 0; i < swap_header->info.nr_badpages; i++) { unsigned int page_nr = swap_header->info.badpages[i]; if (page_nr == 0 || page_nr > swap_header->info.last_page) -- 2.45.2.803.g4e1b14247a-goog