From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A02CC3DA7F for ; Wed, 31 Jul 2024 06:49:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AD35C6B0085; Wed, 31 Jul 2024 02:49:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9C8EC6B008A; Wed, 31 Jul 2024 02:49:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7BCB06B0089; Wed, 31 Jul 2024 02:49:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5E5226B0085 for ; Wed, 31 Jul 2024 02:49:26 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id F1B8E120167 for ; Wed, 31 Jul 2024 06:49:25 +0000 (UTC) X-FDA: 82399121490.16.550AF28 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf20.hostedemail.com (Postfix) with ESMTP id 3EAFA1C0003 for ; Wed, 31 Jul 2024 06:49:24 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=Fm0jMWWG; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf20.hostedemail.com: domain of chrisl@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=chrisl@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1722408519; a=rsa-sha256; cv=none; b=hwT/WAlO0lhsYNSfZPBMFILNjFBiC8lVOndN78gswDkcb8XeojJbixjyCytLVuTJFGbLVO VcLHjf+vYER9pN0AwiBMA95niG1CEo3xbTiZLC5ocEpUOLzbYAGg9e+yJL8MsLa2hO+aYr E69UmHVVZR5XEkzRrIB9nMLZvcc3McY= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=Fm0jMWWG; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf20.hostedemail.com: domain of chrisl@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=chrisl@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1722408519; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IoihjcWUIWWf/lqyGzmkdUfeV1jfthpP9BOCdBl91Fw=; b=XOQ81llwLK00odZbD2XAgXMDD+dfmVUMaQ/9/Z0Bs+Byf/VTEgNRbs9P9sUWY7h1n2lu2h oe/KxAmPYPYHjXN2es3Mw/KNmSLPVqqOaPEUmE4AV2etGIKNpY04VL4y2axKp88Z9FvES5 U6ptSbHq+34LJJVXMcYh/iETVu5WMdg= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id BB2F1621F2; Wed, 31 Jul 2024 06:49:21 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 343FAC116B1; Wed, 31 Jul 2024 06:49:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1722408561; bh=hsQxDCZrtRWDWaDs5IVwKPpQSb8Rg3k3cBHZomxM8ak=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=Fm0jMWWG/eAQp2Nr2ryJaZVToF680ojI3W1usxVCU4dItPLJFCXocEnliEzCRHs4m p6QbaxODPsJ7FvmYN2BjjA0YjJ/Sd6rNIBOoxhbKo85vLiD/mzXv0rQnrI2Nu0lAOw BmtbMIw038i/mghBSve4dnQskSg11vp2sOytVDfgYVa5HQuRztLAQ1rOF4f4rAGf32 WAeV6y69sRb+lNtNXKhHemHu7QHS5QKqi8fy9YqeuA19/gAcTKp1FIG8hnwdqc3C1X ZwE9hiHTcF8mC7aNdNM/FtCH+Uk7d0yUYUdI3INqBO0ogYWYmvowlwEcDEakuo9zW1 womIy2jawCs9g== From: Chris Li Date: Tue, 30 Jul 2024 23:49:14 -0700 Subject: [PATCH v5 2/9] mm: swap: mTHP allocate swap entries from nonfull list MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit Message-Id: <20240730-swap-allocator-v5-2-cb9c148b9297@kernel.org> References: <20240730-swap-allocator-v5-0-cb9c148b9297@kernel.org> In-Reply-To: <20240730-swap-allocator-v5-0-cb9c148b9297@kernel.org> To: Andrew Morton Cc: Kairui Song , Hugh Dickins , Ryan Roberts , "Huang, Ying" , Kalesh Singh , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Chris Li , Barry Song X-Mailer: b4 0.13.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 3EAFA1C0003 X-Stat-Signature: u4pggo4j4h5kjtcmx45u34ba3knhb756 X-Rspam-User: X-HE-Tag: 1722408564-303038 X-HE-Meta: U2FsdGVkX18QWmC8XLMnvQtmzJvVsbU0+wxVAFuYY9u3lUoIgItaXrIZrKgdDw+KNw1roEdkEEwch7S/rpKt5IxdRWfUmCA+qtuWvQu7p0cM2MLljq0qpP/UxXhK/CgvwRx4bv7YV9Bflctod9UV40B15xOZRYvA7M2VPWyIbZF0eag2x+YwQUcH5kXvtHP5ssJd2aaENmtiz6EJDIE5v/XoKv5gh7pN89JFn8P643Ny/1qUoxEPRrypGSpd018CD08Q4LZ6I/+0I/8pSyKCkKzkbRkxTKPYQqCo9b/jkXYInAYpwv2nQOJI+vo3bHAR0jz9JIpYVpzcbPX8709bx6c9aw5HCVxPMedIG+2nRhboROjiRwZ/IE6/YBKv19UMEvpcITAeFuyt1nrMaLYDpmbCqs68d+h35R91C11OzNE6HIjwDAxVhXlQByGJpipOXB5LX9SDRZDtL3GWpP9vciUOaPrg1sfnApRV6xHRy83JNEZTryPdRJ1jCTdSMAIl/Rb4PARwHTe1U9SFlCfXnhiBtGJQXVkQVRDj5NnrYgE3iEeo3O4RFltNvW5MAs0CplchmqwxroQvRGm8PGEIDxn/QtEv9QTV0fDILUWPr1JNBfWoijLSJbU/0VrJGweI7DRZVvRUK8pqQqN7T7o09cQ2++T8YbxvGCxf/xGbX/zeLIVoyJA+u5p3d/4RTCSDq8nST31OYhKPsNjI/MiiGXBtjmNUncAImno9AHZlBYl3/cyvUeXi6qe5KapLCBvTvsEii9yWkvUqeys7fu6OrHlqG2aZAc/xUEpyTLSQQxK/dx3S9L4uNtu83+D+CKGpg4KYWhmw63udZ4u8kMddc5wOnQx4NHkHlYtDphCoPktGf/4kU6k0VxIoaAHPvdEB1pRK39RJt+cBVMNvLBydz0RNWn01o3dn1WNZhN08+wYHvbIQrK0BNuL3FXza/ks5ro40F+ot/q9ZWh0cN9W jfGi9DmI T8GZD+NRkXMXuTePriXoPDvO5lQpDwXB7XcJT87QJ/dwmtHyzhRq6XDQuEGzO3DT6ngfwIHZVWWjy6vytk8juREHy1QSjpg0s95Dhxruo+Ny08pJYw+HOniLv+Mdymu475eNCQzjChZZ447/MwTBHJnYJyq1iexdqG14uT9X8PBQoA4CYB0mQ+2yX56461xCiRIIkDpYNIyFigr4iYJKa/wt7jZ7h9ILmxL8N+pvzEzaa/v1poux9aZDIDTlcv0rU/EiIIhMH9bcnKSiCTXZhP/CNOw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Track the nonfull cluster as well as the empty cluster on lists. Each order has one nonfull cluster list. The cluster will remember which order it was used during new cluster allocation. When the cluster has free entry, add to the nonfull[order] list.  When the free cluster list is empty, also allocate from the nonempty list of that order. This improves the mTHP swap allocation success rate. There are limitations if the distribution of numbers of different orders of mTHP changes a lot. e.g. there are a lot of nonfull cluster assign to order A while later time there are a lot of order B allocation while very little allocation in order A. Currently the cluster used by order A will not reused by order B unless the cluster is 100% empty. Signed-off-by: Chris Li --- include/linux/swap.h | 4 ++++ mm/swapfile.c | 38 +++++++++++++++++++++++++++++++++++--- 2 files changed, 39 insertions(+), 3 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index edafd52d7ac4..6716ef236766 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -254,9 +254,11 @@ struct swap_cluster_info { */ u16 count; u8 flags; + u8 order; struct list_head list; }; #define CLUSTER_FLAG_FREE 1 /* This cluster is free */ +#define CLUSTER_FLAG_NONFULL 2 /* This cluster is on nonfull list */ /* * The first page in the swap file is the swap header, which is always marked @@ -295,6 +297,8 @@ struct swap_info_struct { unsigned long *zeromap; /* vmalloc'ed bitmap to track zero pages */ struct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */ struct list_head free_clusters; /* free clusters list */ + struct list_head nonfull_clusters[SWAP_NR_ORDERS]; + /* list of cluster that contains at least one free slot */ unsigned int lowest_bit; /* index of first free in swap_map */ unsigned int highest_bit; /* index of last free in swap_map */ unsigned int pages; /* total of usable pages of swap */ diff --git a/mm/swapfile.c b/mm/swapfile.c index bceead7f9e3c..dcf09eb549db 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -361,14 +361,22 @@ static void swap_cluster_schedule_discard(struct swap_info_struct *si, memset(si->swap_map + idx * SWAPFILE_CLUSTER, SWAP_MAP_BAD, SWAPFILE_CLUSTER); - list_add_tail(&ci->list, &si->discard_clusters); + VM_BUG_ON(ci->flags & CLUSTER_FLAG_FREE); + if (ci->flags & CLUSTER_FLAG_NONFULL) + list_move_tail(&ci->list, &si->discard_clusters); + else + list_add_tail(&ci->list, &si->discard_clusters); + ci->flags = 0; schedule_work(&si->discard_work); } static void __free_cluster(struct swap_info_struct *si, struct swap_cluster_info *ci) { + if (ci->flags & CLUSTER_FLAG_NONFULL) + list_move_tail(&ci->list, &si->free_clusters); + else + list_add_tail(&ci->list, &si->free_clusters); ci->flags = CLUSTER_FLAG_FREE; - list_add_tail(&ci->list, &si->free_clusters); } /* @@ -491,8 +499,15 @@ static void dec_cluster_info_page(struct swap_info_struct *p, struct swap_cluste VM_BUG_ON(ci->count == 0); ci->count--; - if (!ci->count) + if (!ci->count) { free_cluster(p, ci); + return; + } + + if (!(ci->flags & CLUSTER_FLAG_NONFULL)) { + list_add_tail(&ci->list, &p->nonfull_clusters[ci->order]); + ci->flags |= CLUSTER_FLAG_NONFULL; + } } /* @@ -553,6 +568,19 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, if (tmp == SWAP_NEXT_INVALID) { if (!list_empty(&si->free_clusters)) { ci = list_first_entry(&si->free_clusters, struct swap_cluster_info, list); + list_del(&ci->list); + spin_lock(&ci->lock); + ci->order = order; + ci->flags = 0; + spin_unlock(&ci->lock); + tmp = cluster_index(si, ci) * SWAPFILE_CLUSTER; + } else if (!list_empty(&si->nonfull_clusters[order])) { + ci = list_first_entry(&si->nonfull_clusters[order], + struct swap_cluster_info, list); + list_del(&ci->list); + spin_lock(&ci->lock); + ci->flags = 0; + spin_unlock(&ci->lock); tmp = cluster_index(si, ci) * SWAPFILE_CLUSTER; } else if (!list_empty(&si->discard_clusters)) { /* @@ -967,6 +995,7 @@ static void swap_free_cluster(struct swap_info_struct *si, unsigned long idx) ci = lock_cluster(si, offset); memset(si->swap_map + offset, 0, SWAPFILE_CLUSTER); ci->count = 0; + ci->order = 0; ci->flags = 0; free_cluster(si, ci); unlock_cluster(ci); @@ -2922,6 +2951,9 @@ static int setup_swap_map_and_extents(struct swap_info_struct *p, INIT_LIST_HEAD(&p->free_clusters); INIT_LIST_HEAD(&p->discard_clusters); + for (i = 0; i < SWAP_NR_ORDERS; i++) + INIT_LIST_HEAD(&p->nonfull_clusters[i]); + for (i = 0; i < swap_header->info.nr_badpages; i++) { unsigned int page_nr = swap_header->info.badpages[i]; if (page_nr == 0 || page_nr > swap_header->info.last_page) -- 2.46.0.rc1.232.g9752f9e123-goog