The Linux Kernel Mailing List
 help / color / mirror / Atom feed
From: "Kunwu Chan" <kunwu.chan@linux.dev>
To: "Lian Wang" <lianux.mm@gmail.com>, linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org,
	"Andrew Morton" <akpm@linux-foundation.org>,
	"Chris Li" <chrisl@kernel.org>, "Nhat Pham" <nphamcs@gmail.com>,
	"Baoquan He" <baoquan.he@linux.dev>,
	"Barry Song" <baohua@kernel.org>,
	"Youngjun Park" <youngjun.park@lge.com>,
	"Kemeng Shi" <shikemeng@huaweicloud.com>,
	"Kairui Song" <kasong@tencent.com>,
	"Jihan LIN" <linjh22s@gmail.com>, "Kees Cook" <kees@kernel.org>,
	"Gustavo A. R. Silva" <gustavoars@kernel.org>,
	"Thomas Gleixner" <tglx@kernel.org>,
	linux-hardening@vger.kernel.org
Subject: Re: [RFC PATCH v4 08/18] mm/swap: change back to use each swap device's percpu cluster
Date: Fri, 20 Nov 2026 04:31:18 +0000	[thread overview]
Message-ID: <7cacd2a09ef50b1480dee84d8a2f160f60e113ab@linux.dev> (raw)
In-Reply-To: <f9cb4b2c57360c07bb1ff0d305e089635cb7e56e.1794666741.git.lianux.mm@gmail.com>

Reviewed-by: Kunwu Chan <kunwu.chan@gmail.com>

Thanks,
Kunwu


> 
> From: Youngjun Park <youngjun.park@lge.com>
> 
> This reverts commit 1b7e90020eb7 ("mm, swap: use percpu cluster as
> allocation fast path").
> 
> The global percpu cluster causes several issues:
> 
> 1) It prevents efficient swap allocation for users that do not want
>  the default swap device policy. Because the cluster cache sits
>  above device selection, users that need a different policy must
>  bypass or replace the cache. This complicates incoming ideas like
>  swap tiering.
> 
> 2) It can bypass a caller's preferred device policy. Consider a future
>  policy with per-memcg device restrictions: memcg1 can access devices
>  A and B, where A has higher priority; memcg2 can only access B. Memcg2
>  can write the global percpu cluster with device B, then memcg1 takes B
>  in the fast path even though higher-priority device A is not exhausted.
> 
> 3) The fast-path / slow-path design bundled with the global per-CPU
>  cache is problematic: only the fast path uses the global cache,
>  and the slow path is forced to rotate the allocation list. This
>  complicates consumers like discard.
> 
> Revert to per-device percpu clusters first. A later patch will
> introduce new infrastructure for a high-performance global
> allocation path.
> 
> Suggested-by: Kairui Song <kasong@tencent.com>
> Co-developed-by: Baoquan He <bhe@redhat.com>
> Signed-off-by: Baoquan He <bhe@redhat.com>
> Signed-off-by: Youngjun Park <youngjun.park@lge.com>
> Signed-off-by: Kairui Song <kasong@tencent.com>
> Signed-off-by: Lian Wang <lianux.mm@gmail.com>
> ---
>  include/linux/swap.h | 13 +++-
>  mm/swapfile.c | 170 ++++++++++++++-----------------------------
>  2 files changed, 63 insertions(+), 120 deletions(-)
> 
> diff --git a/include/linux/swap.h b/include/linux/swap.h
> index 717387e16a2d..c8a5684b2f4f 100644
> --- a/include/linux/swap.h
> +++ b/include/linux/swap.h
> @@ -241,10 +241,17 @@ enum {
>  #endif
>  
>  /*
> - * We keep using same cluster for rotational device so IO will be sequential.
> - * The purpose is to optimize SWAP throughput on these device.
> + * We assign a cluster to each CPU, so each CPU can allocate swap entry from
> + * its own cluster and swapout sequentially. The purpose is to optimize swapout
> + * throughput.
>  */
> +struct percpu_cluster {
> + local_lock_t lock; /* Protect the percpu_cluster above */
> + unsigned int next[SWAP_NR_ORDERS]; /* Likely next allocation offset */
> +};
> +
>  struct swap_sequential_cluster {
> + spinlock_t lock; /* Serialize usage of global cluster */
>  unsigned int next[SWAP_NR_ORDERS]; /* Likely next allocation offset */
>  };
>  
> @@ -267,8 +274,8 @@ struct swap_info_struct {
>  /* list of cluster that are fragmented or contented */
>  unsigned int pages; /* total of usable pages of swap */
>  atomic_long_t inuse_pages; /* number of those currently in use */
> + struct percpu_cluster __percpu *percpu_cluster; /* per cpu's swap location */
>  struct swap_sequential_cluster *global_cluster; /* Use one global cluster for rotating device */
> - spinlock_t global_cluster_lock; /* Serialize usage of global cluster */
>  struct rb_root swap_extent_root;/* root of the swap extent rbtree */
>  struct block_device *bdev; /* swap device or bdev of swap file */
>  struct file *swap_file; /* seldom referenced */
> diff --git a/mm/swapfile.c b/mm/swapfile.c
> index aa020f2706b4..c46b43414bf4 100644
> --- a/mm/swapfile.c
> +++ b/mm/swapfile.c
> @@ -145,18 +145,6 @@ static atomic_t proc_poll_event = ATOMIC_INIT(0);
>  
>  atomic_t nr_rotate_swap = ATOMIC_INIT(0);
>  
> -struct percpu_swap_cluster {
> - struct swap_info_struct *si[SWAP_NR_ORDERS];
> - unsigned long offset[SWAP_NR_ORDERS];
> - local_lock_t lock;
> -};
> -
> -static DEFINE_PER_CPU(struct percpu_swap_cluster, percpu_swap_cluster) = {
> - .si = { NULL },
> - .offset = { SWAP_ENTRY_INVALID },
> - .lock = INIT_LOCAL_LOCK(),
> -};
> -
>  /* May return NULL on invalid type, caller must check for NULL return */
>  static struct swap_info_struct *swap_type_to_info(int type)
>  {
> @@ -523,9 +511,10 @@ static int swap_cluster_alloc_table(struct swap_info_struct *si,
>  lock:
>  if (!locked) {
>  if (si) {
> - local_lock(&percpu_swap_cluster.lock);
> - if (!(si->flags & SWP_SOLIDSTATE))
> - spin_lock(&si->global_cluster_lock);
> + if (si->flags & SWP_SOLIDSTATE)
> + local_lock(&si->percpu_cluster->lock);
> + else
> + spin_lock(&si->global_cluster->lock);
>  }
>  spin_lock(&ci->lock);
>  }
> @@ -610,9 +599,10 @@ swap_cluster_populate(struct swap_info_struct *si,
>  * Only cluster isolation from the allocator does table allocation.
>  * Swap allocator uses percpu clusters and holds the local lock.
>  */
> - lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)->lock);
> - if (!(si->flags & SWP_SOLIDSTATE))
> - lockdep_assert_held(&si->global_cluster_lock);
> + if (si->flags & SWP_SOLIDSTATE)
> + lockdep_assert_held(this_cpu_ptr(&si->percpu_cluster->lock));
> + else
> + lockdep_assert_held(&si->global_cluster->lock);
>  lockdep_assert_held(&ci->lock);
>  
>  if (!swap_cluster_alloc_table(si, ci, __GFP_HIGH | __GFP_NOMEMALLOC |
> @@ -625,9 +615,10 @@ swap_cluster_populate(struct swap_info_struct *si,
>  * the potential recursive allocation is limited.
>  */
>  spin_unlock(&ci->lock);
> - if (!(si->flags & SWP_SOLIDSTATE))
> - spin_unlock(&si->global_cluster_lock);
> - local_unlock(&percpu_swap_cluster.lock);
> + if (si->flags & SWP_SOLIDSTATE)
> + local_unlock(&si->percpu_cluster->lock);
> + else
> + spin_unlock(&si->global_cluster->lock);
>  
>  ret = swap_cluster_alloc_table(si, ci, __GFP_HIGH | __GFP_NOMEMALLOC |
>  GFP_KERNEL, false);
> @@ -749,7 +740,7 @@ static bool swap_do_scheduled_discard(struct swap_info_struct *si)
>  ci = list_first_entry(&si->discard_clusters, struct swap_cluster_info, list);
>  /*
>  * Delete the cluster from list to prepare for discard, but keep
> - * the CLUSTER_FLAG_DISCARD flag, percpu_swap_cluster could be
> + * the CLUSTER_FLAG_DISCARD flag, there could be percpu_cluster
>  * pointing to it, or ran into by relocate_cluster.
>  */
>  list_del(&ci->list);
> @@ -1104,12 +1095,10 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
>  out:
>  relocate_cluster(si, ci);
>  swap_cluster_unlock(ci);
> - if (si->flags & SWP_SOLIDSTATE) {
> - this_cpu_write(percpu_swap_cluster.offset[order], next);
> - this_cpu_write(percpu_swap_cluster.si[order], si);
> - } else {
> + if (si->flags & SWP_SOLIDSTATE)
> + this_cpu_write(si->percpu_cluster->next[order], next);
> + else
>  si->global_cluster->next[order] = next;
> - }
>  return found;
>  }
>  
> @@ -1210,13 +1199,17 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
>  if (order && !(si->flags & SWP_BLKDEV))
>  return 0;
>  
> - if (!(si->flags & SWP_SOLIDSTATE)) {
> + if (si->flags & SWP_SOLIDSTATE) {
> + /* Fast path using per CPU cluster */
> + local_lock(&si->percpu_cluster->lock);
> + offset = __this_cpu_read(si->percpu_cluster->next[order]);
> + } else {
>  /* Serialize HDD SWAP allocation for each device. */
> - spin_lock(&si->global_cluster_lock);
> + spin_lock(&si->global_cluster->lock);
>  offset = si->global_cluster->next[order];
> - if (offset == SWAP_ENTRY_INVALID)
> - goto new_cluster;
> + }
>  
> + if (offset != SWAP_ENTRY_INVALID) {
>  ci = swap_cluster_lock(si, offset);
>  /* Cluster could have been used by another order */
>  if (cluster_is_usable(ci, order)) {
> @@ -1230,7 +1223,6 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
>  goto done;
>  }
>  
> -new_cluster:
>  if (swap_prefer_free_cluster(si)) {
>  found = alloc_swap_scan_list(si, &si->free_clusters, folio, false);
>  if (found)
> @@ -1283,8 +1275,10 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
>  goto done;
>  }
>  done:
> - if (!(si->flags & SWP_SOLIDSTATE))
> - spin_unlock(&si->global_cluster_lock);
> + if (si->flags & SWP_SOLIDSTATE)
> + local_unlock(&si->percpu_cluster->lock);
> + else
> + spin_unlock(&si->global_cluster->lock);
>  
>  return found;
>  }
> @@ -1440,41 +1434,8 @@ static bool get_swap_device_info(struct swap_info_struct *si)
>  return true;
>  }
>  
> -/*
> - * Fast path try to get swap entries with specified order from current
> - * CPU's swap entry pool (a cluster).
> - */
> -static bool swap_alloc_fast(struct folio *folio)
> -{
> - unsigned int order = folio_order(folio);
> - struct swap_cluster_info *ci;
> - struct swap_info_struct *si;
> - unsigned int offset;
> -
> - /*
> - * Once allocated, swap_info_struct will never be completely freed,
> - * so checking it's liveness by get_swap_device_info is enough.
> - */
> - si = this_cpu_read(percpu_swap_cluster.si[order]);
> - offset = this_cpu_read(percpu_swap_cluster.offset[order]);
> - if (!si || !offset || !get_swap_device_info(si))
> - return false;
> -
> - ci = swap_cluster_lock(si, offset);
> - if (cluster_is_usable(ci, order)) {
> - if (cluster_is_empty(ci))
> - offset = cluster_offset(si, ci);
> - alloc_swap_scan_cluster(si, ci, folio, offset);
> - } else {
> - swap_cluster_unlock(ci);
> - }
> -
> - put_swap_device(si);
> - return folio_test_swapcache(folio);
> -}
> -
>  /* Rotate the device and switch to a new cluster */
> -static void swap_alloc_slow(struct folio *folio)
> +static void swap_alloc_entry(struct folio *folio)
>  {
>  struct swap_info_struct *si, *next;
>  
> @@ -1842,10 +1803,7 @@ int folio_alloc_swap(struct folio *folio)
>  }
>  
>  again:
> - local_lock(&percpu_swap_cluster.lock);
> - if (!swap_alloc_fast(folio))
> - swap_alloc_slow(folio);
> - local_unlock(&percpu_swap_cluster.lock);
> + swap_alloc_entry(folio);
>  
>  if (!order && unlikely(!folio_test_swapcache(folio))) {
>  if (swap_sync_discard())
> @@ -2251,31 +2209,14 @@ void swap_put_entries_direct(swp_entry_t entry, int nr)
>  */
>  swp_entry_t swap_alloc_hibernation_slot(int type)
>  {
> - struct swap_info_struct *pcp_si, *si = swap_type_to_info(type);
> - unsigned long pcp_offset, offset = SWAP_ENTRY_INVALID;
> - struct swap_cluster_info *ci;
> + struct swap_info_struct *si = swap_type_to_info(type);
> + unsigned long offset = SWAP_ENTRY_INVALID;
>  swp_entry_t entry = {0};
>  
>  if (!si)
>  goto fail;
>  
> - /*
> - * Try the local cluster first if it matches the device. If
> - * not, try grab a new cluster and override local cluster.
> - */
> - local_lock(&percpu_swap_cluster.lock);
> - pcp_si = this_cpu_read(percpu_swap_cluster.si[0]);
> - pcp_offset = this_cpu_read(percpu_swap_cluster.offset[0]);
> - if (pcp_si == si && pcp_offset) {
> - ci = swap_cluster_lock(si, pcp_offset);
> - if (cluster_is_usable(ci, 0))
> - offset = alloc_swap_scan_cluster(si, ci, NULL, pcp_offset);
> - else
> - swap_cluster_unlock(ci);
> - }
> - if (!offset)
> - offset = cluster_alloc_swap_entry(si, NULL);
> - local_unlock(&percpu_swap_cluster.lock);
> + offset = cluster_alloc_swap_entry(si, NULL);
>  if (offset)
>  entry = swp_entry(si->type, offset);
>  
> @@ -3253,27 +3194,6 @@ static void free_swap_cluster_info(struct swap_cluster_info *cluster_info,
>  kvfree(cluster_info);
>  }
>  
> -/*
> - * Called after swap device's reference count is dead, so
> - * neither scan nor allocation will use it.
> - */
> -static void flush_percpu_swap_cluster(struct swap_info_struct *si)
> -{
> - int cpu, i;
> - struct swap_info_struct **pcp_si;
> -
> - for_each_possible_cpu(cpu) {
> - pcp_si = per_cpu_ptr(percpu_swap_cluster.si, cpu);
> - /*
> - * Invalidate the percpu swap cluster cache, si->users
> - * is dead, so no new user will point to it, just flush
> - * any existing user.
> - */
> - for (i = 0; i < SWAP_NR_ORDERS; i++)
> - cmpxchg(&pcp_si[i], si, NULL);
> - }
> -}
> -
>  SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
>  {
>  struct swap_info_struct *p = NULL;
> @@ -3325,7 +3245,6 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
>  
>  flush_work(&p->discard_work);
>  flush_work(&p->reclaim_work);
> - flush_percpu_swap_cluster(p);
>  
>  destroy_swap_extents(p, p->swap_file);
>  
> @@ -3346,6 +3265,8 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
>  percpu_up_write(&swapon_rwsem);
>  arch_swap_invalidate_area(p->type);
>  zswap_swapoff(p->type);
> + free_percpu(p->percpu_cluster);
> + p->percpu_cluster = NULL;
>  kfree(p->global_cluster);
>  p->global_cluster = NULL;
>  free_swap_cluster_info(cluster_info, maxpages);
> @@ -3694,7 +3615,7 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
>  {
>  unsigned long nr_clusters = DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER);
>  struct swap_cluster_info *cluster_info;
> - int err = -ENOMEM;
> + int cpu, err = -ENOMEM;
>  unsigned long i;
>  
>  cluster_info = kvzalloc_objs(*cluster_info, nr_clusters);
> @@ -3704,13 +3625,26 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
>  for (i = 0; i < nr_clusters; i++)
>  spin_lock_init(&cluster_info[i].lock);
>  
> - if (!(si->flags & SWP_SOLIDSTATE)) {
> + if (si->flags & SWP_SOLIDSTATE) {
> + si->percpu_cluster = alloc_percpu(struct percpu_cluster);
> + if (!si->percpu_cluster)
> + goto err;
> +
> + for_each_possible_cpu(cpu) {
> + struct percpu_cluster *cluster;
> +
> + cluster = per_cpu_ptr(si->percpu_cluster, cpu);
> + for (i = 0; i < SWAP_NR_ORDERS; i++)
> + cluster->next[i] = SWAP_ENTRY_INVALID;
> + local_lock_init(&cluster->lock);
> + }
> + } else {
>  si->global_cluster = kmalloc_obj(*si->global_cluster);
>  if (!si->global_cluster)
>  goto err;
>  for (i = 0; i < SWAP_NR_ORDERS; i++)
>  si->global_cluster->next[i] = SWAP_ENTRY_INVALID;
> - spin_lock_init(&si->global_cluster_lock);
> + spin_lock_init(&si->global_cluster->lock);
>  }
>  
>  /*
> @@ -3984,6 +3918,8 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
>  bad_swap_unlock_inode:
>  inode_unlock(inode);
>  bad_swap:
> + free_percpu(si->percpu_cluster);
> + si->percpu_cluster = NULL;
>  kfree(si->global_cluster);
>  si->global_cluster = NULL;
>  inode = NULL;
>

  reply	other threads:[~2026-11-20  4:31 UTC|newest]

Thread overview: 38+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-14 14:47 [RFC PATCH v4 00/18] mm/swap: introduce priority queues for swap device allocation Lian Wang
2026-11-14 14:48 ` [RFC PATCH v4 01/18] mm/swap: remove unused parameter for reading swap header Lian Wang
2026-11-19 15:56   ` Kunwu Chan
2026-11-14 14:49 ` [RFC PATCH v4 02/18] mm/swap: clean up hibernation error handling Lian Wang
2026-11-19 15:57   ` Kunwu Chan
2026-11-14 14:49 ` [RFC PATCH v4 03/18] mm/swap: cleanup and document swap device availability flag usage Lian Wang
2026-11-20  3:58   ` Kunwu Chan
2026-11-14 14:49 ` [RFC PATCH v4 04/18] mm/swap: introduce swap device iteration helper Lian Wang
2026-11-20  3:59   ` Kunwu Chan
2026-11-14 14:49 ` [RFC PATCH v4 05/18] mm/swap: change the swapon lock into a percpu rwsem Lian Wang
2026-11-20  4:12   ` Kunwu Chan
2026-11-14 14:49 ` [RFC PATCH v4 06/18] mm/swap: serialize swap publication before removing swapon mutex Lian Wang
2026-11-20  4:30   ` Kunwu Chan
2026-11-14 14:50 ` [RFC PATCH v4 07/18] mm/swap: consolidate swap inuse accounting helpers Lian Wang
2026-11-20  4:30   ` Kunwu Chan
2026-11-14 14:50 ` [RFC PATCH v4 08/18] mm/swap: change back to use each swap device's percpu cluster Lian Wang
2026-11-20  4:31   ` Kunwu Chan [this message]
2026-11-14 14:50 ` [RFC PATCH v4 09/18] mm/swap: avoid direct reclaim when populating swap cluster metadata Lian Wang
2026-11-20  4:31   ` Kunwu Chan
2026-11-14 14:50 ` [RFC PATCH v4 10/18] mm/swap: add priority queue for swap device allocation Lian Wang
2026-11-16  3:51   ` Kairui Song
2026-11-16 11:11     ` Lian Wang
2026-11-20  4:34   ` Kunwu Chan
2026-11-14 14:51 ` [RFC PATCH v4 11/18] mm/swap: remove available list Lian Wang
2026-11-20  4:36   ` Kunwu Chan
2026-11-14 14:51 ` [RFC PATCH v4 12/18] mm/swap: bound synchronous discard outside the allocation reader Lian Wang
2026-11-20  4:36   ` Kunwu Chan
2026-11-14 14:51 ` [RFC PATCH v4 13/18] mm/swap: drop swap active plist Lian Wang
2026-11-20  4:37   ` Kunwu Chan
2026-11-14 14:51 ` [RFC PATCH v4 14/18] lib/plist.c: remove requeue function Lian Wang
2026-11-14 14:52 ` [RFC PATCH v4 15/18] mm/swap: reuse per-CPU queue readers across priority changes Lian Wang
2026-11-20  4:37   ` Kunwu Chan
2026-11-14 14:52 ` [RFC PATCH v4 16/18] mm/swap: advance the global queue cursor directly between priority rings Lian Wang
2026-11-14 14:52 ` [RFC PATCH v4 17/18] mm/swap: skip reader rotation for single-device " Lian Wang
2026-11-14 14:52 ` [RFC PATCH v4 18/18] mm/swap: redistribute queue readers after membership changes Lian Wang
2026-11-16  3:29   ` Kairui Song
2026-11-16  3:52 ` [RFC PATCH v4 00/18] mm/swap: introduce priority queues for swap device allocation Kairui Song
2026-11-16 11:10   ` Lian Wang

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=7cacd2a09ef50b1480dee84d8a2f160f60e113ab@linux.dev \
    --to=kunwu.chan@linux.dev \
    --cc=akpm@linux-foundation.org \
    --cc=baohua@kernel.org \
    --cc=baoquan.he@linux.dev \
    --cc=chrisl@kernel.org \
    --cc=gustavoars@kernel.org \
    --cc=kasong@tencent.com \
    --cc=kees@kernel.org \
    --cc=lianux.mm@gmail.com \
    --cc=linjh22s@gmail.com \
    --cc=linux-hardening@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=nphamcs@gmail.com \
    --cc=shikemeng@huaweicloud.com \
    --cc=tglx@kernel.org \
    --cc=youngjun.park@lge.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox