From: "Kunwu Chan" <kunwu.chan@linux.dev>
To: "Lian Wang" <lianux.mm@gmail.com>, linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org,
"Andrew Morton" <akpm@linux-foundation.org>,
"Chris Li" <chrisl@kernel.org>, "Nhat Pham" <nphamcs@gmail.com>,
"Baoquan He" <baoquan.he@linux.dev>,
"Barry Song" <baohua@kernel.org>,
"Youngjun Park" <youngjun.park@lge.com>,
"Kemeng Shi" <shikemeng@huaweicloud.com>,
"Kairui Song" <kasong@tencent.com>,
"Jihan LIN" <linjh22s@gmail.com>, "Kees Cook" <kees@kernel.org>,
"Gustavo A. R. Silva" <gustavoars@kernel.org>,
"Thomas Gleixner" <tglx@kernel.org>,
linux-hardening@vger.kernel.org
Subject: Re: [RFC PATCH v4 08/18] mm/swap: change back to use each swap device's percpu cluster
Date: Fri, 20 Nov 2026 04:31:18 +0000 [thread overview]
Message-ID: <7cacd2a09ef50b1480dee84d8a2f160f60e113ab@linux.dev> (raw)
In-Reply-To: <f9cb4b2c57360c07bb1ff0d305e089635cb7e56e.1794666741.git.lianux.mm@gmail.com>
Reviewed-by: Kunwu Chan <kunwu.chan@gmail.com>
Thanks,
Kunwu
>
> From: Youngjun Park <youngjun.park@lge.com>
>
> This reverts commit 1b7e90020eb7 ("mm, swap: use percpu cluster as
> allocation fast path").
>
> The global percpu cluster causes several issues:
>
> 1) It prevents efficient swap allocation for users that do not want
> the default swap device policy. Because the cluster cache sits
> above device selection, users that need a different policy must
> bypass or replace the cache. This complicates incoming ideas like
> swap tiering.
>
> 2) It can bypass a caller's preferred device policy. Consider a future
> policy with per-memcg device restrictions: memcg1 can access devices
> A and B, where A has higher priority; memcg2 can only access B. Memcg2
> can write the global percpu cluster with device B, then memcg1 takes B
> in the fast path even though higher-priority device A is not exhausted.
>
> 3) The fast-path / slow-path design bundled with the global per-CPU
> cache is problematic: only the fast path uses the global cache,
> and the slow path is forced to rotate the allocation list. This
> complicates consumers like discard.
>
> Revert to per-device percpu clusters first. A later patch will
> introduce new infrastructure for a high-performance global
> allocation path.
>
> Suggested-by: Kairui Song <kasong@tencent.com>
> Co-developed-by: Baoquan He <bhe@redhat.com>
> Signed-off-by: Baoquan He <bhe@redhat.com>
> Signed-off-by: Youngjun Park <youngjun.park@lge.com>
> Signed-off-by: Kairui Song <kasong@tencent.com>
> Signed-off-by: Lian Wang <lianux.mm@gmail.com>
> ---
> include/linux/swap.h | 13 +++-
> mm/swapfile.c | 170 ++++++++++++++-----------------------------
> 2 files changed, 63 insertions(+), 120 deletions(-)
>
> diff --git a/include/linux/swap.h b/include/linux/swap.h
> index 717387e16a2d..c8a5684b2f4f 100644
> --- a/include/linux/swap.h
> +++ b/include/linux/swap.h
> @@ -241,10 +241,17 @@ enum {
> #endif
>
> /*
> - * We keep using same cluster for rotational device so IO will be sequential.
> - * The purpose is to optimize SWAP throughput on these device.
> + * We assign a cluster to each CPU, so each CPU can allocate swap entry from
> + * its own cluster and swapout sequentially. The purpose is to optimize swapout
> + * throughput.
> */
> +struct percpu_cluster {
> + local_lock_t lock; /* Protect the percpu_cluster above */
> + unsigned int next[SWAP_NR_ORDERS]; /* Likely next allocation offset */
> +};
> +
> struct swap_sequential_cluster {
> + spinlock_t lock; /* Serialize usage of global cluster */
> unsigned int next[SWAP_NR_ORDERS]; /* Likely next allocation offset */
> };
>
> @@ -267,8 +274,8 @@ struct swap_info_struct {
> /* list of cluster that are fragmented or contented */
> unsigned int pages; /* total of usable pages of swap */
> atomic_long_t inuse_pages; /* number of those currently in use */
> + struct percpu_cluster __percpu *percpu_cluster; /* per cpu's swap location */
> struct swap_sequential_cluster *global_cluster; /* Use one global cluster for rotating device */
> - spinlock_t global_cluster_lock; /* Serialize usage of global cluster */
> struct rb_root swap_extent_root;/* root of the swap extent rbtree */
> struct block_device *bdev; /* swap device or bdev of swap file */
> struct file *swap_file; /* seldom referenced */
> diff --git a/mm/swapfile.c b/mm/swapfile.c
> index aa020f2706b4..c46b43414bf4 100644
> --- a/mm/swapfile.c
> +++ b/mm/swapfile.c
> @@ -145,18 +145,6 @@ static atomic_t proc_poll_event = ATOMIC_INIT(0);
>
> atomic_t nr_rotate_swap = ATOMIC_INIT(0);
>
> -struct percpu_swap_cluster {
> - struct swap_info_struct *si[SWAP_NR_ORDERS];
> - unsigned long offset[SWAP_NR_ORDERS];
> - local_lock_t lock;
> -};
> -
> -static DEFINE_PER_CPU(struct percpu_swap_cluster, percpu_swap_cluster) = {
> - .si = { NULL },
> - .offset = { SWAP_ENTRY_INVALID },
> - .lock = INIT_LOCAL_LOCK(),
> -};
> -
> /* May return NULL on invalid type, caller must check for NULL return */
> static struct swap_info_struct *swap_type_to_info(int type)
> {
> @@ -523,9 +511,10 @@ static int swap_cluster_alloc_table(struct swap_info_struct *si,
> lock:
> if (!locked) {
> if (si) {
> - local_lock(&percpu_swap_cluster.lock);
> - if (!(si->flags & SWP_SOLIDSTATE))
> - spin_lock(&si->global_cluster_lock);
> + if (si->flags & SWP_SOLIDSTATE)
> + local_lock(&si->percpu_cluster->lock);
> + else
> + spin_lock(&si->global_cluster->lock);
> }
> spin_lock(&ci->lock);
> }
> @@ -610,9 +599,10 @@ swap_cluster_populate(struct swap_info_struct *si,
> * Only cluster isolation from the allocator does table allocation.
> * Swap allocator uses percpu clusters and holds the local lock.
> */
> - lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)->lock);
> - if (!(si->flags & SWP_SOLIDSTATE))
> - lockdep_assert_held(&si->global_cluster_lock);
> + if (si->flags & SWP_SOLIDSTATE)
> + lockdep_assert_held(this_cpu_ptr(&si->percpu_cluster->lock));
> + else
> + lockdep_assert_held(&si->global_cluster->lock);
> lockdep_assert_held(&ci->lock);
>
> if (!swap_cluster_alloc_table(si, ci, __GFP_HIGH | __GFP_NOMEMALLOC |
> @@ -625,9 +615,10 @@ swap_cluster_populate(struct swap_info_struct *si,
> * the potential recursive allocation is limited.
> */
> spin_unlock(&ci->lock);
> - if (!(si->flags & SWP_SOLIDSTATE))
> - spin_unlock(&si->global_cluster_lock);
> - local_unlock(&percpu_swap_cluster.lock);
> + if (si->flags & SWP_SOLIDSTATE)
> + local_unlock(&si->percpu_cluster->lock);
> + else
> + spin_unlock(&si->global_cluster->lock);
>
> ret = swap_cluster_alloc_table(si, ci, __GFP_HIGH | __GFP_NOMEMALLOC |
> GFP_KERNEL, false);
> @@ -749,7 +740,7 @@ static bool swap_do_scheduled_discard(struct swap_info_struct *si)
> ci = list_first_entry(&si->discard_clusters, struct swap_cluster_info, list);
> /*
> * Delete the cluster from list to prepare for discard, but keep
> - * the CLUSTER_FLAG_DISCARD flag, percpu_swap_cluster could be
> + * the CLUSTER_FLAG_DISCARD flag, there could be percpu_cluster
> * pointing to it, or ran into by relocate_cluster.
> */
> list_del(&ci->list);
> @@ -1104,12 +1095,10 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
> out:
> relocate_cluster(si, ci);
> swap_cluster_unlock(ci);
> - if (si->flags & SWP_SOLIDSTATE) {
> - this_cpu_write(percpu_swap_cluster.offset[order], next);
> - this_cpu_write(percpu_swap_cluster.si[order], si);
> - } else {
> + if (si->flags & SWP_SOLIDSTATE)
> + this_cpu_write(si->percpu_cluster->next[order], next);
> + else
> si->global_cluster->next[order] = next;
> - }
> return found;
> }
>
> @@ -1210,13 +1199,17 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
> if (order && !(si->flags & SWP_BLKDEV))
> return 0;
>
> - if (!(si->flags & SWP_SOLIDSTATE)) {
> + if (si->flags & SWP_SOLIDSTATE) {
> + /* Fast path using per CPU cluster */
> + local_lock(&si->percpu_cluster->lock);
> + offset = __this_cpu_read(si->percpu_cluster->next[order]);
> + } else {
> /* Serialize HDD SWAP allocation for each device. */
> - spin_lock(&si->global_cluster_lock);
> + spin_lock(&si->global_cluster->lock);
> offset = si->global_cluster->next[order];
> - if (offset == SWAP_ENTRY_INVALID)
> - goto new_cluster;
> + }
>
> + if (offset != SWAP_ENTRY_INVALID) {
> ci = swap_cluster_lock(si, offset);
> /* Cluster could have been used by another order */
> if (cluster_is_usable(ci, order)) {
> @@ -1230,7 +1223,6 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
> goto done;
> }
>
> -new_cluster:
> if (swap_prefer_free_cluster(si)) {
> found = alloc_swap_scan_list(si, &si->free_clusters, folio, false);
> if (found)
> @@ -1283,8 +1275,10 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
> goto done;
> }
> done:
> - if (!(si->flags & SWP_SOLIDSTATE))
> - spin_unlock(&si->global_cluster_lock);
> + if (si->flags & SWP_SOLIDSTATE)
> + local_unlock(&si->percpu_cluster->lock);
> + else
> + spin_unlock(&si->global_cluster->lock);
>
> return found;
> }
> @@ -1440,41 +1434,8 @@ static bool get_swap_device_info(struct swap_info_struct *si)
> return true;
> }
>
> -/*
> - * Fast path try to get swap entries with specified order from current
> - * CPU's swap entry pool (a cluster).
> - */
> -static bool swap_alloc_fast(struct folio *folio)
> -{
> - unsigned int order = folio_order(folio);
> - struct swap_cluster_info *ci;
> - struct swap_info_struct *si;
> - unsigned int offset;
> -
> - /*
> - * Once allocated, swap_info_struct will never be completely freed,
> - * so checking it's liveness by get_swap_device_info is enough.
> - */
> - si = this_cpu_read(percpu_swap_cluster.si[order]);
> - offset = this_cpu_read(percpu_swap_cluster.offset[order]);
> - if (!si || !offset || !get_swap_device_info(si))
> - return false;
> -
> - ci = swap_cluster_lock(si, offset);
> - if (cluster_is_usable(ci, order)) {
> - if (cluster_is_empty(ci))
> - offset = cluster_offset(si, ci);
> - alloc_swap_scan_cluster(si, ci, folio, offset);
> - } else {
> - swap_cluster_unlock(ci);
> - }
> -
> - put_swap_device(si);
> - return folio_test_swapcache(folio);
> -}
> -
> /* Rotate the device and switch to a new cluster */
> -static void swap_alloc_slow(struct folio *folio)
> +static void swap_alloc_entry(struct folio *folio)
> {
> struct swap_info_struct *si, *next;
>
> @@ -1842,10 +1803,7 @@ int folio_alloc_swap(struct folio *folio)
> }
>
> again:
> - local_lock(&percpu_swap_cluster.lock);
> - if (!swap_alloc_fast(folio))
> - swap_alloc_slow(folio);
> - local_unlock(&percpu_swap_cluster.lock);
> + swap_alloc_entry(folio);
>
> if (!order && unlikely(!folio_test_swapcache(folio))) {
> if (swap_sync_discard())
> @@ -2251,31 +2209,14 @@ void swap_put_entries_direct(swp_entry_t entry, int nr)
> */
> swp_entry_t swap_alloc_hibernation_slot(int type)
> {
> - struct swap_info_struct *pcp_si, *si = swap_type_to_info(type);
> - unsigned long pcp_offset, offset = SWAP_ENTRY_INVALID;
> - struct swap_cluster_info *ci;
> + struct swap_info_struct *si = swap_type_to_info(type);
> + unsigned long offset = SWAP_ENTRY_INVALID;
> swp_entry_t entry = {0};
>
> if (!si)
> goto fail;
>
> - /*
> - * Try the local cluster first if it matches the device. If
> - * not, try grab a new cluster and override local cluster.
> - */
> - local_lock(&percpu_swap_cluster.lock);
> - pcp_si = this_cpu_read(percpu_swap_cluster.si[0]);
> - pcp_offset = this_cpu_read(percpu_swap_cluster.offset[0]);
> - if (pcp_si == si && pcp_offset) {
> - ci = swap_cluster_lock(si, pcp_offset);
> - if (cluster_is_usable(ci, 0))
> - offset = alloc_swap_scan_cluster(si, ci, NULL, pcp_offset);
> - else
> - swap_cluster_unlock(ci);
> - }
> - if (!offset)
> - offset = cluster_alloc_swap_entry(si, NULL);
> - local_unlock(&percpu_swap_cluster.lock);
> + offset = cluster_alloc_swap_entry(si, NULL);
> if (offset)
> entry = swp_entry(si->type, offset);
>
> @@ -3253,27 +3194,6 @@ static void free_swap_cluster_info(struct swap_cluster_info *cluster_info,
> kvfree(cluster_info);
> }
>
> -/*
> - * Called after swap device's reference count is dead, so
> - * neither scan nor allocation will use it.
> - */
> -static void flush_percpu_swap_cluster(struct swap_info_struct *si)
> -{
> - int cpu, i;
> - struct swap_info_struct **pcp_si;
> -
> - for_each_possible_cpu(cpu) {
> - pcp_si = per_cpu_ptr(percpu_swap_cluster.si, cpu);
> - /*
> - * Invalidate the percpu swap cluster cache, si->users
> - * is dead, so no new user will point to it, just flush
> - * any existing user.
> - */
> - for (i = 0; i < SWAP_NR_ORDERS; i++)
> - cmpxchg(&pcp_si[i], si, NULL);
> - }
> -}
> -
> SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
> {
> struct swap_info_struct *p = NULL;
> @@ -3325,7 +3245,6 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
>
> flush_work(&p->discard_work);
> flush_work(&p->reclaim_work);
> - flush_percpu_swap_cluster(p);
>
> destroy_swap_extents(p, p->swap_file);
>
> @@ -3346,6 +3265,8 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
> percpu_up_write(&swapon_rwsem);
> arch_swap_invalidate_area(p->type);
> zswap_swapoff(p->type);
> + free_percpu(p->percpu_cluster);
> + p->percpu_cluster = NULL;
> kfree(p->global_cluster);
> p->global_cluster = NULL;
> free_swap_cluster_info(cluster_info, maxpages);
> @@ -3694,7 +3615,7 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
> {
> unsigned long nr_clusters = DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER);
> struct swap_cluster_info *cluster_info;
> - int err = -ENOMEM;
> + int cpu, err = -ENOMEM;
> unsigned long i;
>
> cluster_info = kvzalloc_objs(*cluster_info, nr_clusters);
> @@ -3704,13 +3625,26 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
> for (i = 0; i < nr_clusters; i++)
> spin_lock_init(&cluster_info[i].lock);
>
> - if (!(si->flags & SWP_SOLIDSTATE)) {
> + if (si->flags & SWP_SOLIDSTATE) {
> + si->percpu_cluster = alloc_percpu(struct percpu_cluster);
> + if (!si->percpu_cluster)
> + goto err;
> +
> + for_each_possible_cpu(cpu) {
> + struct percpu_cluster *cluster;
> +
> + cluster = per_cpu_ptr(si->percpu_cluster, cpu);
> + for (i = 0; i < SWAP_NR_ORDERS; i++)
> + cluster->next[i] = SWAP_ENTRY_INVALID;
> + local_lock_init(&cluster->lock);
> + }
> + } else {
> si->global_cluster = kmalloc_obj(*si->global_cluster);
> if (!si->global_cluster)
> goto err;
> for (i = 0; i < SWAP_NR_ORDERS; i++)
> si->global_cluster->next[i] = SWAP_ENTRY_INVALID;
> - spin_lock_init(&si->global_cluster_lock);
> + spin_lock_init(&si->global_cluster->lock);
> }
>
> /*
> @@ -3984,6 +3918,8 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
> bad_swap_unlock_inode:
> inode_unlock(inode);
> bad_swap:
> + free_percpu(si->percpu_cluster);
> + si->percpu_cluster = NULL;
> kfree(si->global_cluster);
> si->global_cluster = NULL;
> inode = NULL;
>
next prev parent reply other threads:[~2026-11-20 4:31 UTC|newest]
Thread overview: 41+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-14 14:47 [RFC PATCH v4 00/18] mm/swap: introduce priority queues for swap device allocation Lian Wang
2026-11-14 14:48 ` [RFC PATCH v4 01/18] mm/swap: remove unused parameter for reading swap header Lian Wang
2026-11-19 15:56 ` Kunwu Chan
2026-11-14 14:49 ` [RFC PATCH v4 02/18] mm/swap: clean up hibernation error handling Lian Wang
2026-11-19 15:57 ` Kunwu Chan
2026-11-14 14:49 ` [RFC PATCH v4 03/18] mm/swap: cleanup and document swap device availability flag usage Lian Wang
2026-11-20 3:58 ` Kunwu Chan
2026-11-14 14:49 ` [RFC PATCH v4 04/18] mm/swap: introduce swap device iteration helper Lian Wang
2026-11-20 3:59 ` Kunwu Chan
2026-11-14 14:49 ` [RFC PATCH v4 05/18] mm/swap: change the swapon lock into a percpu rwsem Lian Wang
2026-11-20 4:12 ` Kunwu Chan
2026-11-14 14:49 ` [RFC PATCH v4 06/18] mm/swap: serialize swap publication before removing swapon mutex Lian Wang
2026-11-20 4:30 ` Kunwu Chan
2026-11-14 14:50 ` [RFC PATCH v4 07/18] mm/swap: consolidate swap inuse accounting helpers Lian Wang
2026-11-20 4:30 ` Kunwu Chan
2026-11-14 14:50 ` [RFC PATCH v4 08/18] mm/swap: change back to use each swap device's percpu cluster Lian Wang
2026-11-20 4:31 ` Kunwu Chan [this message]
2026-11-14 14:50 ` [RFC PATCH v4 09/18] mm/swap: avoid direct reclaim when populating swap cluster metadata Lian Wang
2026-11-20 4:31 ` Kunwu Chan
2026-11-14 14:50 ` [RFC PATCH v4 10/18] mm/swap: add priority queue for swap device allocation Lian Wang
2026-11-16 3:51 ` Kairui Song
2026-11-16 11:11 ` Lian Wang
2026-11-20 4:34 ` Kunwu Chan
2026-11-14 14:51 ` [RFC PATCH v4 11/18] mm/swap: remove available list Lian Wang
2026-11-20 4:36 ` Kunwu Chan
2026-11-14 14:51 ` [RFC PATCH v4 12/18] mm/swap: bound synchronous discard outside the allocation reader Lian Wang
2026-11-20 4:36 ` Kunwu Chan
2026-11-14 14:51 ` [RFC PATCH v4 13/18] mm/swap: drop swap active plist Lian Wang
2026-11-20 4:37 ` Kunwu Chan
2026-11-14 14:51 ` [RFC PATCH v4 14/18] lib/plist.c: remove requeue function Lian Wang
2026-11-14 14:52 ` [RFC PATCH v4 15/18] mm/swap: reuse per-CPU queue readers across priority changes Lian Wang
2026-11-20 4:37 ` Kunwu Chan
2026-11-14 14:52 ` [RFC PATCH v4 16/18] mm/swap: advance the global queue cursor directly between priority rings Lian Wang
2026-11-20 4:38 ` Kunwu Chan
2026-11-14 14:52 ` [RFC PATCH v4 17/18] mm/swap: skip reader rotation for single-device " Lian Wang
2026-11-20 4:38 ` Kunwu Chan
2026-11-14 14:52 ` [RFC PATCH v4 18/18] mm/swap: redistribute queue readers after membership changes Lian Wang
2026-11-16 3:29 ` Kairui Song
2026-11-20 4:38 ` Kunwu Chan
2026-11-16 3:52 ` [RFC PATCH v4 00/18] mm/swap: introduce priority queues for swap device allocation Kairui Song
2026-11-16 11:10 ` Lian Wang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=7cacd2a09ef50b1480dee84d8a2f160f60e113ab@linux.dev \
--to=kunwu.chan@linux.dev \
--cc=akpm@linux-foundation.org \
--cc=baohua@kernel.org \
--cc=baoquan.he@linux.dev \
--cc=chrisl@kernel.org \
--cc=gustavoars@kernel.org \
--cc=kasong@tencent.com \
--cc=kees@kernel.org \
--cc=lianux.mm@gmail.com \
--cc=linjh22s@gmail.com \
--cc=linux-hardening@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=nphamcs@gmail.com \
--cc=shikemeng@huaweicloud.com \
--cc=tglx@kernel.org \
--cc=youngjun.park@lge.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.