* [PATCH v3 00/11] Virtual Swap Space (Swap Table Edition)
@ 2026-08-06 18:42 Nhat Pham
2026-08-06 18:42 ` [PATCH v3 01/11] mm, swap: add virtual swap device infrastructure Nhat Pham
` (12 more replies)
0 siblings, 13 replies; 17+ messages in thread
From: Nhat Pham @ 2026-08-06 18:42 UTC (permalink / raw)
To: akpm
Cc: chrisl, kasong, hannes, mhocko, roman.gushchin, shakeel.butt,
yosry, david, muchun.song, shikemeng, baoquan.he, baohua,
youngjun.park, chengming.zhou, ljs, liam, vbabka, rppt, surenb,
qi.zheng, axelrasmussen, yuanchu, weixugc, riel, gourry,
haowenchao22, corbet, kernel-team, nphamcs, linux-mm,
linux-kernel, linux-doc, cgroups
Changelog:
* v2 [v2] -> v3:
* Rebased onto current mm-unstable.
* Add a runtime vm.vswap_enabled sysctl and CONFIG_VSWAP_DEFAULT_ON
to gate vswap allocation.
* More cleanups and small bug fixes.
* Split THP swapin enablement into its own patch (patch 5).
* Add production workload benchmark results, and drop RFC tag.
* v1 [v1] -> v2:
* Rebased to a newer mm-unstable tip.
* Fix a bunch of assorted issues (incorrect zswap store failure
rollback, vswap_init() failure handling, rmap-encoding collision,
etc.) and clean up the code (rename a bunch of functions to
more closely follow existing patterns, etc.).
* Some more code clean up and simplification: some renamings to more
closely follow existing patterns, move vswap backing check to
__swap_cache_add_check, store zero state in the swap_table for
vswap entries, etc.. Many of these are proposed by Kairui Song
in [1].
* Defer memcg_table allocation on physical clusters until the first
vswap-backing slot installs. Saves ~512 bytes per physical cluster
that only serves vswap-backing slots (this is the new patch 8).
* Widen swap_info_struct->max and ->pages (and the swapoff unuse-path
index) so vswap supports ~8 PB of swap space (this is the new
patch 9).
* Split the physical-swap-backend patch into three for reviewability:
the core backend (patch 3), zswap writeback to physical swap
(patch 4), and reclaim of cache-only physical slots (patch 5). No
functional change.
* Add kerneldoc for the vswap API.
* Add some benchmark numbers for zswap case.
I. Context and Motivation
=========================
Currently, when an anon page is swapped out, a slot in a backing swap
device is allocated and stored in the page table entries that refer to
the original page. This slot is also used as the "key" to find the
swapped out content, as well as the index to swap data structures, such
as the swap cache, or the swap cgroup mapping. Tying a swap entry to its
backing slot in this way is performant and efficient when swap is purely
just disk space, and swapoff is rare.
However, the advent of many swap optimizations has exposed major
drawbacks of this design. The first problem is that we occupy a physical
slot in the swap space, even for pages that are NEVER expected to hit
the disk: pages compressed and stored in the zswap pool, zero-filled
pages, or pages rejected by both of these optimizations when zswap
writeback is disabled. This is arguably the central shortcoming of
zswap:
* Resource-wise, it is hugely wasteful in terms of disk usage. At Meta,
we size swapfile in the order of 25-50% of host RAM, depending on flash
availaiblity. This is a lot of flash for a fleet of our size, and
with universal zswap enablement, most of this is wasted for zswap
entries.
* In deployments when no disk space can be afforded for swap (such as
mobile and embedded devices), users cannot adopt zswap, and are forced
to use zram. This is confusing for users, and creates extra burdens
for developers, having to develop and maintain similar features for
two separate swap backends (writeback, cgroup charging, THP support,
etc.). For instance, see the discussion in [2].
* Tying zswap (and more generally, other in-memory swap backends) to
the current physical swapfile infrastructure makes zswap implicitly
statically sized. This does not make sense, as unlike disk swap, in
which we consume a limited resource (disk space or swapfile space) to
save another resource (memory), zswap consumes the same resource it is
saving (memory). The more we zswap, the more memory we have available,
not less. We are not rationing a limited resource when we limit
the size of the zswap pool, but rather we are capping the resource
(memory) saving potential of zswap. Under memory pressure, using
more zswap is almost always better than the alternative (disk IOs, or
even worse, OOMs), and dynamically sizing the zswap pool on demand
allows the system to flexibly respond to these precarious scenarios.
* Operationally, static provisioning the swapfile for zswap poses
significant challenges, because the sysadmin has to prescribe how
much swap is needed a priori, for each combination of
(memory size x disk space x workload usage). It is even more
complicated when we take into account the variance of memory
compression, which changes the reclaim dynamics (and as a result,
swap space size requirement). The problem is further exacerbated for
users who rely on swap utilization (and exhaustion) as an OOM signal.
All of these factors make it very difficult to configure the swapfile
for zswap: too small of a swapfile and we risk preventable OOMs and
limit the memory saving potentials of zswap; too big of a swapfile
and we waste disk space and memory due to swap metadata overhead.
This dilemma becomes more drastic in high memory systems, which can
have up to TBs worth of memory.
Swap virtualization is the answer to these issues, with three properties:
1. Decoupled backends. For zswap in particular, this means we eliminate
the unused storage space, and allows zswap to be used in systems that
do not have enough storage capacity for physical swap (without having
to resort to silly hacks). Zero-filled swap pages and swap-cache-only
folios also benefit here.
2. Dynamic swap space. Since virtual swap is not tied to any physical
resource, we can make it infinite and dynamically grow it on demand.
This massively simplifies operational provisioning, and increases the
utilization of compressed swap backends (zswap). Dynamicity also
reduces overhead on unused swap capacity.
3. Efficient backend transfer. The virtualization scheme should not
introduce PTE/rmap walking overhead for backend transfer. This
is crucial for systems that want to support multiple swap backends
in a tiering fashion (for e.g zswap -> disk swap).
For more historical contexts and references, please take a look at
the cover letter of the older vswap submissions ([3] and [v2]).
II. Design
==========
When we compile kernel with CONFIG_VSWAP, a special vswap device is
allocated at boot time, and all swapped out pages try to allocate from
this device first, falling back to a physical swap device on failure.
Routing can also be turned off at runtime with the vm.vswap_enabled
sysctl, which defaults to 0 unless CONFIG_VSWAP_DEFAULT_ON=y. It is
allocation-only: new swapouts go straight to physical swap, while
entries already backed by vswap keep being served and drain as they
are faulted back in or freed.
These swap entries can subsequently acquire backend on-demand, such as
a zswap entry, or a slot on a physical swap device.
We repurpose much of the existing swap_table infrastructure and
swapfile allocator for this new vswap device, with two notable
differences:
* Clusters are dynamically allocated on demand and managed through
an xarray. This in turn allows us to avoid static provisioning and
let swap space grow dynamically.
* Each cluster of this new vswap device has a virtual_table that stores
the backend information of the entries in the cluster (see below).
Diagrams:
Case 1: vswap entry (virtualized)
PTE swap_cluster_info_dynamic
vswap_entry +---------------------------------+
(swp_entry_t) ------>| swap_cluster_info (ci) |
| +----------------------------+ |
| | swap_table | |
| | PFN / Shadow | |
| | memcg_table | |
| | count,flags,order | |
| | lock, list | |
| +----------------------------+ |
| |
| virtual_table |
| +----------------------------+ |
| | NONE | |
| | SWAPFILE(swp_entry_t) | |
| | ZSWAP(struct zswap_entry*) | |
| +----------------------------+ |
+---------------------------------+
|
| SWAPFILE resolves to
v
PHYSICAL CLUSTER (swap_cluster_info)
+--------------------------+
| swap_table per-slot: |
| NULL - free |
| PFN - cached folio |
| Shadow - swapped out |
| Pointer- vswap rmap |
| Bad - unusable |
| |
| Vswap-backing slot: |
| Pointer(C|swp_entry_t) |
| rmap back to vswap |
+--------------------------+
Case 2: direct-mapped physical entry (no vswap)
PTE PHYSICAL CLUSTER (swap_cluster_info)
phys_entry +--------------------------+
(swp_entry_t) ------>| swap_table per-slot: |
| NULL - free |
| PFN - cached folio |
| Shadow - swapped out |
| Bad - unusable |
+--------------------------+
struct swap_cluster_info_dynamic {
struct swap_cluster_info ci; /* swap_table, lock, etc. */
unsigned int index; /* position in xarray */
struct rcu_head rcu; /* kfree_rcu deferred free */
atomic_long_t *virtual_table; /* backend info, 8 B/slot */
};
Each vswap cluster (swap_cluster_info_dynamic) extends the classic
swap_cluster_info struct with a virtual_table array that stores the
backend information for each virtual swap entry in the cluster. Each
entry is tag-encoded in the low 3 bits to indicate the backend type:
NONE: |----- 0000 ------|000| free / unbacked
SWAPFILE: |- type:5,off:56 -|001| on a physical swapfile
ZSWAP: |--- zswap_entry* |010| compressed in zswap
Other design highlights:
* Note that for the vswap device, we have merged the zswap xarray tree
with the swapfile-level clusters. This means that for zswap only users,
we have negligible extra space overhead.
* Both vswap entries (Case 1) and directly-mapped physical entries
(Case 2) coexist as first-class citizens. When CONFIG_VSWAP=n the
vswap paths compile out.
* Backend transitions in the virtual_table are synchronized through the
swap cache and the folio lock - the same mechanism that already
serializes ordinary swap operations (swapin, swapout, zswap
writeback, swap cache reclaim). IOW, we can only assume that the
backend of a vswap entry is stable through swap cache/folio lock.
Looking at the backend without this should be done at best for
optimization purposes, as there is no guarantee that the backend
will not change under the observer.
* Pointer-tagged swap_table entries on physical clusters provide the
rmap (physical -> virtual) lookup.
* Virtual swap slots not backed by physical swap are not charged to
memcg swap counters - only physical backing is charged (I made the
case for this in [4]).
III. Benchmarks
===============
Note that the goal is not to match vswap performance with baseline on
every single case yet - we still maintain !CONFIG_VSWAP setup. We can
optimize further once we have landed this new feature.
A. Production Workload: Instagram
=================================
To test vswap's stability and performance, I ran an A/B experiment on
Instagram (django) workload, with zswap as the swap backend. On these
hosts, the swapfiles' size is 50% of RAM.
Compared to baseline, vswap gives:
* On par request throughput.
* Lower request serving latency (by about 1-3%).
* Lower memory pressure in the system service cgroups running alongside
the workload. PSI-based proactive reclaimer can therefore recover more
from them, lowering their overall memory footprint, allowing the main
workload to expand.
* Elimination of swapfile footprint for all zswap users in the host.
B. Semi-synthetic Workloads (memhog, usemem, kernel build)
==========================================================
All values are mean +/- standard deviation across rounds.
Test system: x86_64, 52 cores, 64 GB swapfile for all 3 benchmarks.
Swap backend: zswap (zstd) with the traditional active/inactive LRU. We
focus on zswap here because it is the motivating use case for vswap.
For each benchmark, we test 3 kernels:
* Baseline: mm-unstable, no vswap patches.
* VSS off: vswap series applied, CONFIG_VSWAP not set, to verify that
there is no regression to existing swap paths when we disable vswap.
* VSS on: vswap series applied, CONFIG_VSWAP=y.
1. Memhog: single-threaded, 48GB allocation on a host with 16GB RAM,
20 rounds.
Baseline VSS off VSS on
real (s) 131.71 +/- 13.54 132.47 +/- 10.10 120.56 +/- 15.37
sys (s) 114.05 +/- 13.03 115.11 +/- 9.76 103.73 +/- 15.06
user (s) 10.86 +/- 0.13 10.97 +/- 0.10 10.87 +/- 0.11
delta real - +0.6% -8.5%
delta sys - +0.9% -9.1%
Dropping the best and the worst round to reduce variance:
memhog Baseline VSS off VSS on
real (s) 130.24 +/- 8.56 131.51 +/- 5.82 119.39 +/- 12.06
sys (s) 112.58 +/- 7.88 114.26 +/- 5.74 102.63 +/- 11.83
user (s) 10.86 +/- 0.14 10.97 +/- 0.10 10.86 +/- 0.10
delta real - +1.0% -8.3%
delta sys - +1.5% -8.8%
2. Usemem single-threaded: 56GB allocation on a host with 32GB RAM,
16 rounds.
Baseline VSS off VSS on
real (s) 177.14 +/- 7.34 178.20 +/- 5.12 175.83 +/- 6.96
sys (s) 125.30 +/- 7.47 125.19 +/- 5.18 124.09 +/- 7.07
tput (KB/s) 390668 +/- 16840 387878 +/- 11769 390921 +/- 15798
free (ms) 7739 +/- 125 7734 +/- 120 6572 +/- 121
delta real - +0.6% -0.7%
delta sys - -0.1% -1.0%
delta tput - -0.7% +0.1%
delta free - -0.1% -15.1%
3. Kernel build: 52 workers (one per processor), memory.max=3GB, 10 rounds.
Baseline VSS off VSS on
real (s) 168.13 +/- 0.77 168.46 +/- 0.45 167.75 +/- 0.65
sys (s) 772.49 +/- 19.77 781.82 +/- 26.32 763.60 +/- 33.02
user (s) 5128.41 +/- 1.31 5130.64 +/- 1.67 5130.74 +/- 1.66
delta real - +0.2% -0.2%
delta sys - +1.2% -1.1%
delta user - +0.0% +0.0%
For zswap backend, vswap outperforms baseline on usemem freeing, and
memhog benchmark, and is on par with baseline on the rest.
In the RFC v2 ([v2]), I put out several theories for this. I have
done some prototyping to isolate effects, and it turns out the
performance wins come primarily from the elimination of zswap's
xarray and the merging of zswap's metadata to swap device's cluster.
Several code paths are optimized thanks to this - for instance,
in swap_range_free(), we call zswap_invalidate() once for each entry the
range, resulting in multiple xarray tree walks. With vswap, we perform
one single xarray walk to grab a 512-slot cluster, then performs a
flat array scan to free zswap metadata. Similar wins can be observed
in Baoquan's optimization ([8]), which also optimizes away the zswap tree.
IV. References
==============
[v1]: https://lore.kernel.org/all/20260528212955.1912856-1-nphamcs@gmail.com/
[v2]: https://lore.kernel.org/all/20260612193738.2183968-1-nphamcs@gmail.com/
[1]: https://lore.kernel.org/all/CAMgjq7BhOn48xEyC=2j837R7qddfjeBVHMiRqdx8no4ZEBpBLg@mail.gmail.com/
[2]: https://lore.kernel.org/all/Zqe_Nab-Df1CN7iW@infradead.org/
[3]: https://lore.kernel.org/all/20260505153854.1612033-1-nphamcs@gmail.com/
[4]: https://lore.kernel.org/linux-mm/CAKEwX=P4syV38jAVCWq198r2OHXXc=xA-fx1dk6+qYef6yzxWQ@mail.gmail.com/
[5]: https://lore.kernel.org/all/CAKEwX=P50av2rfocpsqZoDQowZ=EEhQ-5vj5tBykbNz8vtKTzA@mail.gmail.com/
[6]: https://lore.kernel.org/all/20260727135029.1059441-1-baoquan.he@linux.dev/
[7]: https://lore.kernel.org/all/20260220-swap-table-p4-v1-15-104795d19815@tencent.com/
[8]: https://lore.kernel.org/all/20260707073215.72183-1-baoquan.he@linux.dev/
Appendix: Alternative Designs and Improvements
==============================================
A. Vmalloc Data Structure:
==========================
This is a promising alternative to the xarray data structure, reducing
the indirection overhead. The initial version relies on userspace knob to
trigger swap address space growth - I have commented on why this is shaky
in [5].
Baoquan has followed-up with a new version (see [6]) that should give us
kernel-driven dynamic growth and (tail-only) shrink. This seems sufficient
for vswap use case, AFAICT - but seems like it would need a couple more
versions to finalize the design.
I think it is better to proceed with the xarray data structure first,
especially since we already see some positive signals on performance
by storing zswap metadata in a per-cluster flat table. With vswap landed,
we will have a concrete setup to show vmalloc data structure's wins.
B. Moving the backend table to struct swap_cluster_info
=======================================================
Another approach Baoquan and I discussed on is to structure vswap patch
series as follows:
1. Moving vtable (renamed to something more generic) to swap cluster,
which removes the xarray.
2. Once vswap is introduced, we simply use this field to store the
backend.
I have a prototype for this, but I ended up scrapping the whole thing, for
the following reasons:
1. It ended up being even more code than what I sent out here - most of
which touches the non-vswap code paths, which we either want to leave
alone (generic swap logic) or want to rip out wholesale down the line
(zswap).
2. There are several fields that are ONLY needed for the vswap clusters
(for instance, rcu_head and index). Shoving them into the shared struct
swap_cluster_info imposes memory and mental overhead for non-vswap
clusters and users.
We can avoid this by simply moving it to the wrapper struct
(swap_cluster_info_dynamic). This is actually Kairui's design
(see [7]), but after trying to deviate from it, I have to conclude
it is the right choice too.
3. Replacing zswap tree with the per-cluster backend table results in
performance wins even when vswap is turned off (this is how I
verified that vswap's performance wins comes from here).
However, it requires more code to make sure this table is not allocated
when not needed. Note that the eventual goal is to make vswap the ONLY
way to use zswap, so we are literally adding complexity and overhead
(even for non-vswap users) to optimize for a code path that is rarely
exercised after vswap lands, and will be ripped out soon after.
That seems very off to me.
To close out, this design brings together the ideas from the earlier
discussions:
1. All of the requirements I set out to solve (dynamicity, backend
decoupling, efficient backend transfer) are implemented.
2. Vswap device now repurpose the swap table design and most of the
generic swap operations.
3. Minimal overhead for non-vswap users, and zswap-no-writeback users.
If you disable writeback, vswap *is* a ghost swapfile.
Nhat Pham (11):
mm, swap: add virtual swap device infrastructure
mm, swap: support zswap and zeroswap as vswap backends
mm, swap: prepare the swap IO path for vswap
mm, swap: support physical swap as a vswap backend
mm, swap: enable THP swapin for vswap entries
mm, swap: write back vswap zswap entries to physical swap
mm, swap: reclaim physical slots backing cache-only vswap entries
mm, swap: only charge physical swap entries
mm, swap: add debugfs counters for vswap
mm, swap: defer memcg_table allocation for physical swap clusters
mm, swap: widen swap_info_struct max/pages to unsigned long
Documentation/admin-guide/sysctl/vm.rst | 16 +
MAINTAINERS | 1 +
include/linux/memcontrol.h | 5 +
include/linux/swap.h | 88 +-
include/linux/zswap.h | 3 +
mm/Kconfig | 21 +
mm/memcontrol.c | 166 +++-
mm/memory.c | 28 +-
mm/page_io.c | 103 +-
mm/shmem.c | 4 +-
mm/swap.h | 55 +-
mm/swap_state.c | 64 +-
mm/swap_table.h | 62 ++
mm/swapfile.c | 1194 +++++++++++++++++++++--
mm/vmscan.c | 14 +-
mm/vswap.h | 454 +++++++++
mm/zswap.c | 140 ++-
17 files changed, 2207 insertions(+), 211 deletions(-)
create mode 100644 mm/vswap.h
base-commit: bacc32cc7de65ffff70080a48eb294f89e434d5e
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH v3 01/11] mm, swap: add virtual swap device infrastructure
2026-08-06 18:42 [PATCH v3 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
@ 2026-08-06 18:42 ` Nhat Pham
2026-08-07 15:49 ` Johannes Weiner
2026-08-06 18:42 ` [PATCH v3 02/11] mm, swap: support zswap and zeroswap as vswap backends Nhat Pham
` (11 subsequent siblings)
12 siblings, 1 reply; 17+ messages in thread
From: Nhat Pham @ 2026-08-06 18:42 UTC (permalink / raw)
To: akpm
Cc: chrisl, kasong, hannes, mhocko, roman.gushchin, shakeel.butt,
yosry, david, muchun.song, shikemeng, baoquan.he, baohua,
youngjun.park, chengming.zhou, ljs, liam, vbabka, rppt, surenb,
qi.zheng, axelrasmussen, yuanchu, weixugc, riel, gourry,
haowenchao22, corbet, kernel-team, nphamcs, linux-mm,
linux-kernel, linux-doc, cgroups
Create a 16 TB virtual swap device at boot, along with the dynamic
cluster infrastructure that the rest of the vswap layer is built on.
swap_cluster_info_dynamic keeps per-cluster info in an xarray, so a
device can be sized without a static cluster_info[] array.
Gated by a new CONFIG_VSWAP (depends on SWAP && 64BIT). For now the
vswap device cannot be swapon'd or swapoff'd. It is created
unconditionally at boot when CONFIG_VSWAP=y and lives for the
lifetime of the kernel. The SWP_VSWAP flag and swap_is_vswap()
helper let hot paths skip per-device bookkeeping that doesn't
apply (avail-list management, percpu_ref get/put, hibernation
target lookup, etc.).
This patch is pure scaffolding. It wires the dynamic-cluster
allocator into cluster_alloc_swap_entry (via an SWP_VSWAP branch
that dispatches to alloc_swap_scan_dynamic), but the branch is
not yet reachable because vswap_si is kept off swap_avail_head
and swap_active_head and folio_alloc_swap has no path that calls
into vswap_si directly. Backends (zswap, zero, physical disk)
and the vswap-aware swap-out / swap-in / writeback paths arrive
in subsequent patches.
Suggested-by: Kairui Song <kasong@tencent.com>
Co-developed-by: Kairui Song <kasong@tencent.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
MAINTAINERS | 1 +
include/linux/swap.h | 16 +++
mm/Kconfig | 10 ++
mm/page_io.c | 15 +++
mm/swap.h | 47 ++++++--
mm/swap_state.c | 41 ++++---
mm/swap_table.h | 2 +
mm/swapfile.c | 270 +++++++++++++++++++++++++++++++++++++++----
mm/vswap.h | 31 +++++
mm/zswap.c | 6 +
10 files changed, 396 insertions(+), 43 deletions(-)
create mode 100644 mm/vswap.h
diff --git a/MAINTAINERS b/MAINTAINERS
index e9c8567308a7..d0da9a29a910 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -17248,6 +17248,7 @@ F: mm/swap.h
F: mm/swap_table.h
F: mm/swap_state.c
F: mm/swapfile.c
+F: mm/vswap.h
MEMORY MANAGEMENT - THP (TRANSPARENT HUGE PAGE)
M: Andrew Morton <akpm@linux-foundation.org>
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 45f301d73e2a..a955bd60dd58 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -207,6 +207,7 @@ enum {
SWP_STABLE_WRITES = (1 << 11), /* no overwrite PG_writeback pages */
SWP_SYNCHRONOUS_IO = (1 << 12), /* synchronous IO is efficient */
SWP_HIBERNATION = (1 << 13), /* pinned for hibernation */
+ SWP_VSWAP = (1 << 14), /* virtual swap device */
/* add others here before... */
};
@@ -276,8 +277,21 @@ struct swap_info_struct {
struct list_head discard_clusters; /* discard clusters list */
struct plist_node avail_list; /* entry in swap_avail_head */
const struct swap_ops *ops;
+ struct xarray cluster_info_pool; /* Xarray for vswap dynamic cluster info */
};
+#ifdef CONFIG_VSWAP
+static inline bool swap_is_vswap(struct swap_info_struct *si)
+{
+ return si->flags & SWP_VSWAP;
+}
+#else
+static inline bool swap_is_vswap(struct swap_info_struct *si)
+{
+ return false;
+}
+#endif
+
static inline swp_entry_t page_swap_entry(struct page *page)
{
struct folio *folio = page_folio(page);
@@ -402,6 +416,8 @@ void swap_free_hibernation_slot(swp_entry_t entry);
static inline void put_swap_device(struct swap_info_struct *si)
{
+ if (swap_is_vswap(si))
+ return;
percpu_ref_put(&si->users);
}
diff --git a/mm/Kconfig b/mm/Kconfig
index 331daf7fcfab..32d38b552845 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -19,6 +19,16 @@ menuconfig SWAP
used to provide more virtual memory than the actual RAM present
in your computer. If unsure say Y.
+config VSWAP
+ bool "Virtual swap device"
+ depends on SWAP && 64BIT
+ help
+ Adds a virtual swap layer that decouples swap entries in page
+ tables from physical backing storage. Swap entries are allocated
+ from a virtual swap device and can be backed by zswap, a physical
+ swapfile, or kept in memory - with the backing changeable at
+ runtime without invalidating page table entries.
+
config ZSWAP
bool "Compressed cache for swap pages"
depends on SWAP
diff --git a/mm/page_io.c b/mm/page_io.c
index e4fa7ffffe8b..fca1718056af 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -27,6 +27,7 @@
#include <linux/zswap.h>
#include "swap.h"
#include "swap_table.h"
+#include "vswap.h"
int generic_swapfile_activate(struct swap_info_struct *sis,
struct file *swap_file,
@@ -247,6 +248,15 @@ int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
}
rcu_read_unlock();
+ /*
+ * A vswap folio that reaches here could not be stored to a backend
+ * (zswap) and has no physical slot to write to, so keep it dirty.
+ */
+ if (is_vswap_entry(folio->swap)) {
+ folio_mark_dirty(folio);
+ return AOP_WRITEPAGE_ACTIVATE;
+ }
+
__swap_writepage(ctx, folio);
return 0;
out_unlock:
@@ -479,6 +489,11 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)
if (zswap_load(folio) != -ENOENT)
goto finish;
+ if (unlikely(swap_is_vswap(sis))) {
+ folio_unlock(folio);
+ goto finish;
+ }
+
/* We have to read from slower devices. Increase zswap protection. */
zswap_folio_swapin(folio);
swap_add_folio(ctx, folio, READ);
diff --git a/mm/swap.h b/mm/swap.h
index ec580c713204..b593ad3214ef 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -66,6 +66,12 @@ struct swap_cluster_info {
struct list_head list;
};
+struct swap_cluster_info_dynamic {
+ struct swap_cluster_info ci;
+ unsigned int index; /* for cluster_index() */
+ struct rcu_head rcu;
+};
+
/* All on-list cluster must have a non-zero flag. */
enum swap_cluster_flags {
CLUSTER_FLAG_NONE = 0, /* For temporary off-list cluster */
@@ -76,6 +82,7 @@ enum swap_cluster_flags {
CLUSTER_FLAG_USABLE = CLUSTER_FLAG_FRAG,
CLUSTER_FLAG_FULL,
CLUSTER_FLAG_DISCARD,
+ CLUSTER_FLAG_DEAD, /* Vswap dynamic cluster pending kfree_rcu */
CLUSTER_FLAG_MAX,
};
@@ -143,9 +150,19 @@ static inline struct swap_info_struct *__swap_entry_to_info(swp_entry_t entry)
static inline struct swap_cluster_info *__swap_offset_to_cluster(
struct swap_info_struct *si, pgoff_t offset)
{
+ unsigned int cluster_idx = offset / SWAPFILE_CLUSTER;
+
VM_WARN_ON_ONCE(percpu_ref_is_zero(&si->users)); /* race with swapoff */
VM_WARN_ON_ONCE(offset >= roundup(si->max, SWAPFILE_CLUSTER));
- return &si->cluster_info[offset / SWAPFILE_CLUSTER];
+
+ if (swap_is_vswap(si)) {
+ struct swap_cluster_info_dynamic *ci_dyn;
+
+ ci_dyn = xa_load(&si->cluster_info_pool, cluster_idx);
+ return ci_dyn ? &ci_dyn->ci : NULL;
+ }
+
+ return &si->cluster_info[cluster_idx];
}
static inline struct swap_cluster_info *__swap_entry_to_cluster(swp_entry_t entry)
@@ -157,7 +174,7 @@ static inline struct swap_cluster_info *__swap_entry_to_cluster(swp_entry_t entr
static __always_inline struct swap_cluster_info *__swap_cluster_lock(
struct swap_info_struct *si, unsigned long offset, bool irq)
{
- struct swap_cluster_info *ci = __swap_offset_to_cluster(si, offset);
+ struct swap_cluster_info *ci;
/*
* Nothing modifies swap cache in an IRQ context. All access to
@@ -170,20 +187,36 @@ static __always_inline struct swap_cluster_info *__swap_cluster_lock(
*/
VM_WARN_ON_ONCE(!in_task());
VM_WARN_ON_ONCE(percpu_ref_is_zero(&si->users)); /* race with swapoff */
- if (irq)
- spin_lock_irq(&ci->lock);
- else
- spin_lock(&ci->lock);
+
+ rcu_read_lock();
+ ci = __swap_offset_to_cluster(si, offset);
+ if (ci) {
+ if (irq)
+ spin_lock_irq(&ci->lock);
+ else
+ spin_lock(&ci->lock);
+
+ if (ci->flags == CLUSTER_FLAG_DEAD) {
+ if (irq)
+ spin_unlock_irq(&ci->lock);
+ else
+ spin_unlock(&ci->lock);
+ ci = NULL;
+ }
+ }
+ rcu_read_unlock();
return ci;
}
/**
* swap_cluster_lock - Lock and return the swap cluster of given offset.
* @si: swap device the cluster belongs to.
- * @offset: the swap entry offset, pointing to a valid slot.
+ * @offset: the swap entry offset.
*
* Context: The caller must ensure the offset is in the valid range and
* protect the swap device with reference count or locks.
+ * Return: the locked cluster, or NULL if it is gone. Only a vswap device
+ * can return NULL, as its clusters are allocated and freed on demand.
*/
static inline struct swap_cluster_info *swap_cluster_lock(
struct swap_info_struct *si, unsigned long offset)
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 5be825911e64..9e0d71fcdc24 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -95,8 +95,10 @@ struct folio *swap_cache_get_folio(swp_entry_t entry)
struct folio *folio;
for (;;) {
+ rcu_read_lock();
swp_tb = swap_table_get(__swap_entry_to_cluster(entry),
swp_cluster_offset(entry));
+ rcu_read_unlock();
if (!swp_tb_is_folio(swp_tb))
return NULL;
folio = swp_tb_to_folio(swp_tb);
@@ -118,8 +120,10 @@ bool swap_cache_has_folio(swp_entry_t entry)
{
unsigned long swp_tb;
+ rcu_read_lock();
swp_tb = swap_table_get(__swap_entry_to_cluster(entry),
swp_cluster_offset(entry));
+ rcu_read_unlock();
return swp_tb_is_folio(swp_tb);
}
@@ -135,8 +139,10 @@ void *swap_cache_get_shadow(swp_entry_t entry)
{
unsigned long swp_tb;
+ rcu_read_lock();
swp_tb = swap_table_get(__swap_entry_to_cluster(entry),
swp_cluster_offset(entry));
+ rcu_read_unlock();
if (swp_tb_is_shadow(swp_tb))
return swp_tb_to_shadow(swp_tb);
return NULL;
@@ -405,14 +411,16 @@ void __swap_cache_replace_folio(struct swap_cluster_info *ci,
* -ENOENT / -EEXIST: Target swap entry is unavailable or cached, the caller
* should abort or try to use the cached folio instead
*/
-static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
- swp_entry_t targ_entry, gfp_t gfp,
+static struct folio *__swap_cache_alloc(swp_entry_t targ_entry, gfp_t gfp,
unsigned int order, struct vm_fault *vmf,
struct mempolicy *mpol, pgoff_t ilx)
{
int err;
swp_entry_t entry;
struct folio *folio;
+ struct swap_cluster_info *ci;
+ struct swap_info_struct *si = __swap_entry_to_info(targ_entry);
+ unsigned long offset = swp_offset(targ_entry);
void *shadow = NULL;
unsigned short memcg_id;
unsigned long address, nr_pages = 1UL << order;
@@ -422,9 +430,12 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
entry.val = round_down(targ_entry.val, nr_pages);
/* Check if the slot and range are available, skip allocation if not */
- spin_lock(&ci->lock);
- err = __swap_cache_add_check(ci, targ_entry, nr_pages, NULL, NULL);
- spin_unlock(&ci->lock);
+ err = -ENOENT;
+ ci = swap_cluster_lock(si, offset);
+ if (ci) {
+ err = __swap_cache_add_check(ci, targ_entry, nr_pages, NULL, NULL);
+ swap_cluster_unlock(ci);
+ }
if (unlikely(err))
return ERR_PTR(err);
@@ -445,10 +456,13 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
return ERR_PTR(-ENOMEM);
/* Double check the range is still not in conflict */
- spin_lock(&ci->lock);
- err = __swap_cache_add_check(ci, targ_entry, nr_pages, &shadow, &memcg_id);
+ err = -ENOENT;
+ ci = swap_cluster_lock(si, offset);
+ if (ci)
+ err = __swap_cache_add_check(ci, targ_entry, nr_pages, &shadow, &memcg_id);
if (unlikely(err)) {
- spin_unlock(&ci->lock);
+ if (ci)
+ swap_cluster_unlock(ci);
folio_put(folio);
return ERR_PTR(err);
}
@@ -456,13 +470,14 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
__folio_set_locked(folio);
__folio_set_swapbacked(folio);
__swap_cache_do_add_folio(ci, folio, entry);
- spin_unlock(&ci->lock);
+ swap_cluster_unlock(ci);
if (mem_cgroup_swapin_charge_folio(folio, memcg_id,
vmf ? vmf->vma->vm_mm : NULL, gfp)) {
- spin_lock(&ci->lock);
+ /* The folio pins the cluster */
+ ci = swap_cluster_lock(si, offset);
__swap_cache_do_del_folio(ci, folio, entry, shadow);
- spin_unlock(&ci->lock);
+ swap_cluster_unlock(ci);
folio_unlock(folio);
/* nr_pages refs from swap cache, 1 from allocation */
folio_put_refs(folio, nr_pages + 1);
@@ -516,9 +531,7 @@ struct folio *swap_cache_alloc_folio(swp_entry_t targ_entry, gfp_t gfp,
{
int order, err;
struct folio *ret;
- struct swap_cluster_info *ci;
- ci = __swap_entry_to_cluster(targ_entry);
order = highest_order(orders);
/* orders must be non-zero, and must not exceed cluster size. */
@@ -526,7 +539,7 @@ struct folio *swap_cache_alloc_folio(swp_entry_t targ_entry, gfp_t gfp,
return ERR_PTR(-EINVAL);
do {
- ret = __swap_cache_alloc(ci, targ_entry, gfp, order,
+ ret = __swap_cache_alloc(targ_entry, gfp, order,
vmf, mpol, ilx);
if (!IS_ERR(ret))
break;
diff --git a/mm/swap_table.h b/mm/swap_table.h
index e6613e62f8d0..fd7f0fb9836a 100644
--- a/mm/swap_table.h
+++ b/mm/swap_table.h
@@ -255,6 +255,8 @@ static inline unsigned long swap_table_get(struct swap_cluster_info *ci,
unsigned long swp_tb;
VM_WARN_ON_ONCE(off >= SWAPFILE_CLUSTER);
+ if (!ci)
+ return SWP_TB_NULL;
rcu_read_lock();
table = rcu_dereference(ci->table);
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 4d4e3e3059f6..fea3a8eccbc1 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -42,10 +42,12 @@
#include <linux/suspend.h>
#include <linux/zswap.h>
#include <linux/plist.h>
+#include <linux/major.h>
#include <asm/tlbflush.h>
#include <linux/leafops.h>
#include "swap_table.h"
+#include "vswap.h"
#include "internal.h"
#include "swap.h"
@@ -401,6 +403,8 @@ static inline bool cluster_is_usable(struct swap_cluster_info *ci, int order)
static inline unsigned int cluster_index(struct swap_info_struct *si,
struct swap_cluster_info *ci)
{
+ if (swap_is_vswap(si))
+ return container_of(ci, struct swap_cluster_info_dynamic, ci)->index;
return ci - si->cluster_info;
}
@@ -712,6 +716,34 @@ static void swap_users_ref_free(struct percpu_ref *ref)
complete(&si->comp);
}
+#ifdef CONFIG_VSWAP
+static void vswap_free_cluster(struct swap_info_struct *si,
+ struct swap_cluster_info *ci)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ if (ci->flags != CLUSTER_FLAG_NONE) {
+ spin_lock(&si->lock);
+ list_del(&ci->list);
+ spin_unlock(&si->lock);
+ }
+ swap_cluster_free_table(ci);
+ /*
+ * Ordering vs the RCU cluster lookup: erase from the xarray first
+ * (new lookups miss it), mark DEAD under the held ci->lock (a lookup
+ * that already has ci sees DEAD on relock and bails), then kfree_rcu
+ * so the cluster outlives any reader still in its RCU section.
+ */
+ xa_erase(&si->cluster_info_pool, ci_dyn->index);
+ ci->flags = CLUSTER_FLAG_DEAD;
+ kfree_rcu(ci_dyn, rcu);
+}
+#else
+static inline void vswap_free_cluster(struct swap_info_struct *si,
+ struct swap_cluster_info *ci) {}
+#endif
+
/*
* Must be called after freeing if ci->count == 0, moves the cluster to free
* or discard list.
@@ -733,6 +765,11 @@ static void free_cluster(struct swap_info_struct *si, struct swap_cluster_info *
return;
}
+ if (swap_is_vswap(si)) {
+ vswap_free_cluster(si, ci);
+ return;
+ }
+
__free_cluster(si, ci);
}
@@ -835,14 +872,21 @@ static int swap_cluster_setup_bad_slot(struct swap_info_struct *si,
* stolen by a lower order). @usable will be set to false if that happens.
*/
static bool cluster_reclaim_range(struct swap_info_struct *si,
- struct swap_cluster_info *ci,
+ struct swap_cluster_info **pcip,
unsigned long start, unsigned int order,
bool *usable)
{
+ struct swap_cluster_info *ci = *pcip;
unsigned int nr_pages = 1 << order;
unsigned long offset = start, end = start + nr_pages;
unsigned long swp_tb;
+ /*
+ * Take RCU read lock before releasing the cluster lock to keep ci
+ * alive - for vswap dynamic clusters, ci is freed via kfree_rcu
+ * and the grace period could otherwise elapse in the window.
+ */
+ rcu_read_lock();
spin_unlock(&ci->lock);
do {
swp_tb = swap_table_get(ci, offset % SWAPFILE_CLUSTER);
@@ -852,7 +896,15 @@ static bool cluster_reclaim_range(struct swap_info_struct *si,
if (__try_to_reclaim_swap(si, offset, TTRS_ANYWAY) < 0)
break;
} while (++offset < end);
- spin_lock(&ci->lock);
+ rcu_read_unlock();
+
+ /* Re-lookup: dynamic cluster may have been freed while lock was dropped */
+ ci = swap_cluster_lock(si, start);
+ *pcip = ci;
+ if (!ci) {
+ *usable = false;
+ return false;
+ }
/*
* We just dropped ci->lock so cluster could be used by another
@@ -983,7 +1035,8 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
if (!cluster_scan_range(si, ci, offset, nr_pages, &need_reclaim))
continue;
if (need_reclaim) {
- ret = cluster_reclaim_range(si, ci, offset, order, &usable);
+ ret = cluster_reclaim_range(si, &ci, offset, order,
+ &usable);
if (!usable)
goto out;
if (cluster_is_empty(ci))
@@ -1001,8 +1054,10 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
break;
}
out:
- relocate_cluster(si, ci);
- swap_cluster_unlock(ci);
+ if (ci) {
+ relocate_cluster(si, ci);
+ swap_cluster_unlock(ci);
+ }
if (si->flags & SWP_SOLIDSTATE) {
this_cpu_write(percpu_swap_cluster.offset[order], next);
this_cpu_write(percpu_swap_cluster.si[order], si);
@@ -1034,6 +1089,41 @@ static unsigned int alloc_swap_scan_list(struct swap_info_struct *si,
return found;
}
+static unsigned int alloc_swap_scan_dynamic(struct swap_info_struct *si,
+ struct folio *folio)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ struct swap_cluster_info *ci;
+ unsigned long offset;
+
+ VM_WARN_ON(!swap_is_vswap(si));
+
+ ci_dyn = kzalloc_obj(*ci_dyn, GFP_ATOMIC);
+ if (!ci_dyn)
+ return SWAP_ENTRY_INVALID;
+
+ spin_lock_init(&ci_dyn->ci.lock);
+ INIT_LIST_HEAD(&ci_dyn->ci.list);
+
+ if (swap_cluster_alloc_table(&ci_dyn->ci, GFP_ATOMIC)) {
+ kfree(ci_dyn);
+ return SWAP_ENTRY_INVALID;
+ }
+
+ if (xa_alloc(&si->cluster_info_pool, &ci_dyn->index, ci_dyn,
+ XA_LIMIT(1, DIV_ROUND_UP(si->max, SWAPFILE_CLUSTER) - 1),
+ GFP_ATOMIC)) {
+ swap_cluster_free_table(&ci_dyn->ci);
+ kfree(ci_dyn);
+ return SWAP_ENTRY_INVALID;
+ }
+
+ ci = &ci_dyn->ci;
+ spin_lock(&ci->lock);
+ offset = cluster_offset(si, ci);
+ return alloc_swap_scan_cluster(si, ci, folio, offset);
+}
+
static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
{
long to_scan = 1;
@@ -1056,7 +1146,9 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
spin_unlock(&ci->lock);
nr_reclaim = __try_to_reclaim_swap(si, offset,
TTRS_ANYWAY);
- spin_lock(&ci->lock);
+ ci = swap_cluster_lock(si, offset);
+ if (!ci)
+ goto next;
if (nr_reclaim) {
offset += abs(nr_reclaim);
continue;
@@ -1070,6 +1162,7 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
relocate_cluster(si, ci);
swap_cluster_unlock(ci);
+next:
if (to_scan <= 0)
break;
@@ -1146,6 +1239,12 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
goto done;
}
+ if (swap_is_vswap(si)) {
+ found = alloc_swap_scan_dynamic(si, folio);
+ if (found)
+ goto done;
+ }
+
if (!(si->flags & SWP_PAGE_DISCARD)) {
found = alloc_swap_scan_list(si, &si->free_clusters, folio, false);
if (found)
@@ -1264,6 +1363,13 @@ static void add_to_avail_list(struct swap_info_struct *si, bool swapon)
goto skip;
}
+ /*
+ * Keep vswap off the avail list - it is not allocated from by
+ * the physical swap allocator (swap_alloc_fast/slow).
+ */
+ if (swap_is_vswap(si))
+ goto skip;
+
plist_add(&si->avail_list, &swap_avail_head);
skip:
@@ -1280,10 +1386,10 @@ static bool swap_usage_add(struct swap_info_struct *si, unsigned int nr_entries)
long val = atomic_long_add_return_relaxed(nr_entries, &si->inuse_pages);
/*
- * If device is full, and SWAP_USAGE_OFFLIST_BIT is not set,
- * remove it from the plist.
+ * If device is full, and SWAP_USAGE_OFFLIST_BIT is not set, remove it
+ * from the plist. Vswap is never on the avail list, so skip it.
*/
- if (unlikely(val == si->pages)) {
+ if (unlikely(val == si->pages) && !swap_is_vswap(si)) {
del_from_avail_list(si, false);
return true;
}
@@ -1296,10 +1402,10 @@ static void swap_usage_sub(struct swap_info_struct *si, unsigned int nr_entries)
long val = atomic_long_sub_return_relaxed(nr_entries, &si->inuse_pages);
/*
- * If device is not full, and SWAP_USAGE_OFFLIST_BIT is set,
- * add it to the plist.
+ * If device is not full, and SWAP_USAGE_OFFLIST_BIT is set, add it to
+ * the plist. Vswap is never on the avail list, so skip it.
*/
- if (unlikely(val & SWAP_USAGE_OFFLIST_BIT))
+ if (unlikely(val & SWAP_USAGE_OFFLIST_BIT) && !swap_is_vswap(si))
add_to_avail_list(si, false);
}
@@ -1346,6 +1452,10 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
static bool get_swap_device_info(struct swap_info_struct *si)
{
+ /* vswap device is always alive - no ref counting needed */
+ if (swap_is_vswap(si))
+ return true;
+
if (!percpu_ref_tryget_live(&si->users))
return false;
/*
@@ -1381,11 +1491,11 @@ static bool swap_alloc_fast(struct folio *folio)
return false;
ci = swap_cluster_lock(si, offset);
- if (cluster_is_usable(ci, order)) {
+ if (ci && cluster_is_usable(ci, order)) {
if (cluster_is_empty(ci))
offset = cluster_offset(si, ci);
alloc_swap_scan_cluster(si, ci, folio, offset);
- } else {
+ } else if (ci) {
swap_cluster_unlock(ci);
}
@@ -1507,6 +1617,7 @@ int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp)
if (!si)
return 0;
+ /* Entry is in use (being faulted in), so its cluster is alive. */
ci = __swap_offset_to_cluster(si, offset);
ret = swap_extend_table_alloc(si, ci, swp_cluster_offset(entry), gfp);
@@ -1742,6 +1853,7 @@ int folio_alloc_swap(struct folio *folio)
unsigned int order = folio_order(folio);
unsigned int size = 1 << order;
+ VM_WARN_ON_FOLIO(folio_test_swapcache(folio), folio);
VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
VM_BUG_ON_FOLIO(!folio_test_uptodate(folio), folio);
@@ -1904,7 +2016,8 @@ struct swap_info_struct *get_swap_device(swp_entry_t entry)
return NULL;
put_out:
pr_err("%s: %s%08lx\n", __func__, Bad_offset, entry.val);
- percpu_ref_put(&si->users);
+ if (!swap_is_vswap(si))
+ percpu_ref_put(&si->users);
return NULL;
}
@@ -2036,6 +2149,7 @@ static bool folio_maybe_swapped(struct folio *folio)
VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);
+ /* Folio is locked and in swap cache, so ci->count > 0: cluster is alive. */
ci = __swap_entry_to_cluster(entry);
ci_off = swp_cluster_offset(entry);
ci_end = ci_off + folio_nr_pages(folio);
@@ -2223,6 +2337,9 @@ static int __find_hibernation_swap_type(dev_t device, sector_t offset)
if (!(sis->flags & SWP_WRITEOK))
continue;
+ /* vswap has no bdev - never a hibernation target */
+ if (swap_is_vswap(sis))
+ continue;
if (device == sis->bdev->bd_dev) {
struct swap_extent *se = first_se(sis);
@@ -2349,6 +2466,9 @@ int find_first_swap(dev_t *device)
if (!(sis->flags & SWP_WRITEOK))
continue;
+ /* vswap has no bdev - never a hibernation target */
+ if (swap_is_vswap(sis))
+ continue;
*device = sis->bdev->bd_dev;
spin_unlock(&swap_lock);
return type;
@@ -2565,8 +2685,10 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
&vmf);
}
if (!folio) {
+ rcu_read_lock();
swp_tb = swap_table_get(__swap_entry_to_cluster(entry),
swp_cluster_offset(entry));
+ rcu_read_unlock();
if (swp_tb_get_count(swp_tb) <= 0)
continue;
return -ENOMEM;
@@ -2712,8 +2834,10 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si,
* allocations from this area (while holding swap_lock).
*/
for (i = prev + 1; i < si->max; i++) {
+ rcu_read_lock();
swp_tb = swap_table_get(__swap_offset_to_cluster(si, i),
i % SWAPFILE_CLUSTER);
+ rcu_read_unlock();
if (!swp_tb_is_null(swp_tb) && !swp_tb_is_bad(swp_tb))
break;
if ((i % LATENCY_LIMIT) == 0)
@@ -2952,6 +3076,11 @@ static int setup_swap_extents(struct swap_info_struct *sis,
struct inode *inode = mapping->host;
int ret;
+ if (swap_is_vswap(sis)) {
+ *span = 0;
+ return 0;
+ }
+
ret = sio_pool_init();
if (ret)
return ret;
@@ -2977,15 +3106,24 @@ static int setup_swap_extents(struct swap_info_struct *sis,
static void _enable_swap_info(struct swap_info_struct *si)
{
- atomic_long_add(si->pages, &nr_swap_pages);
- total_swap_pages += si->pages;
+ if (!swap_is_vswap(si)) {
+ atomic_long_add(si->pages, &nr_swap_pages);
+ total_swap_pages += si->pages;
+ }
assert_spin_locked(&swap_lock);
- plist_add(&si->list, &swap_active_head);
+ /*
+ * Vswap has no backing file and no swapoff support - keep it
+ * off swap_active_head (used by swapoff filename lookup and
+ * swap_sync_discard) and swap_avail_head (physical allocator).
+ */
+ if (!swap_is_vswap(si)) {
+ plist_add(&si->list, &swap_active_head);
- /* Add back to available list */
- add_to_avail_list(si, true);
+ /* Add back to available list */
+ add_to_avail_list(si, true);
+ }
}
/*
@@ -3022,6 +3160,8 @@ static void wait_for_allocation(struct swap_info_struct *si)
struct swap_cluster_info *ci;
BUG_ON(si->flags & SWP_WRITEOK);
+ if (swap_is_vswap(si))
+ return;
for (offset = 0; offset < end; offset += SWAPFILE_CLUSTER) {
ci = swap_cluster_lock(si, offset);
@@ -3528,10 +3668,43 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
unsigned long maxpages)
{
unsigned long nr_clusters = DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER);
- struct swap_cluster_info *cluster_info;
+ struct swap_cluster_info *cluster_info = NULL;
+ struct swap_cluster_info_dynamic *ci_dyn;
int err = -ENOMEM;
unsigned long i;
+ /* For SWP_VSWAP files, initialize Xarray pool instead of static array */
+ if (swap_is_vswap(si)) {
+ /*
+ * Pre-allocate cluster 0 and mark slot 0 (header page)
+ * as bad so the allocator never hands out page offset 0.
+ */
+ ci_dyn = kzalloc_obj(*ci_dyn, GFP_KERNEL);
+ if (!ci_dyn)
+ goto err;
+ spin_lock_init(&ci_dyn->ci.lock);
+ INIT_LIST_HEAD(&ci_dyn->ci.list);
+
+ nr_clusters = 0;
+ xa_init_flags(&si->cluster_info_pool, XA_FLAGS_ALLOC);
+ err = xa_insert(&si->cluster_info_pool, 0, ci_dyn, GFP_KERNEL);
+ if (err) {
+ kfree(ci_dyn);
+ goto err;
+ }
+
+ err = swap_cluster_setup_bad_slot(si, &ci_dyn->ci, 0, false);
+ if (err) {
+ xa_erase(&si->cluster_info_pool, 0);
+ swap_cluster_free_table(&ci_dyn->ci);
+ kfree(ci_dyn);
+ xa_destroy(&si->cluster_info_pool);
+ goto err;
+ }
+
+ goto setup_cluster_info;
+ }
+
cluster_info = kvzalloc_objs(*cluster_info, nr_clusters);
if (!cluster_info)
goto err;
@@ -3556,6 +3729,10 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
err = swap_cluster_setup_bad_slot(si, cluster_info, 0, false);
if (err)
goto err;
+
+ if (!swap_header)
+ goto setup_cluster_info;
+
for (i = 0; i < swap_header->info.nr_badpages; i++) {
unsigned int page_nr = swap_header->info.badpages[i];
@@ -3575,6 +3752,7 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
goto err;
}
+setup_cluster_info:
INIT_LIST_HEAD(&si->free_clusters);
INIT_LIST_HEAD(&si->full_clusters);
INIT_LIST_HEAD(&si->discard_clusters);
@@ -3611,7 +3789,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
struct dentry *dentry;
int prio;
int error;
- union swap_header *swap_header;
+ union swap_header *swap_header = NULL;
int nr_extents;
sector_t span;
unsigned long maxpages;
@@ -3949,3 +4127,51 @@ static int __init swapfile_init(void)
return 0;
}
subsys_initcall(swapfile_init);
+
+#ifdef CONFIG_VSWAP
+struct swap_info_struct *vswap_si;
+
+/* vswap does no IO on its own. */
+static const struct swap_ops vswap_ops = { };
+
+static int __init vswap_init(void)
+{
+ struct swap_info_struct *si;
+ unsigned long maxpages;
+ int err;
+
+ si = alloc_swap_info();
+ if (IS_ERR(si))
+ return PTR_ERR(si);
+
+ maxpages = min(swapfile_maximum_size,
+ ALIGN_DOWN((unsigned long)UINT_MAX, SWAPFILE_CLUSTER));
+ si->flags |= SWP_VSWAP | SWP_SOLIDSTATE | SWP_WRITEOK;
+ si->ops = &vswap_ops;
+ si->bdev = NULL;
+ si->max = maxpages;
+ si->pages = maxpages - 1;
+ si->prio = SHRT_MAX;
+ si->list.prio = -si->prio;
+ si->avail_list.prio = -si->prio;
+
+ err = setup_swap_clusters_info(si, NULL, maxpages);
+ if (err)
+ goto fail;
+
+ mutex_lock(&swapon_mutex);
+ enable_swap_info(si);
+ mutex_unlock(&swapon_mutex);
+
+ vswap_si = si;
+ pr_info("vswap: created virtual swap device (%lu pages)\n", maxpages);
+ return 0;
+
+fail:
+ spin_lock(&swap_lock);
+ si->flags = 0;
+ spin_unlock(&swap_lock);
+ return err;
+}
+late_initcall(vswap_init);
+#endif
diff --git a/mm/vswap.h b/mm/vswap.h
new file mode 100644
index 000000000000..5641692f5be3
--- /dev/null
+++ b/mm/vswap.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Virtual swap space
+ *
+ * Copyright (C) 2026 Nhat Pham
+ */
+#ifndef _MM_VSWAP_H
+#define _MM_VSWAP_H
+
+#include <linux/swap.h>
+#include "swap.h"
+
+#ifdef CONFIG_VSWAP
+
+extern struct swap_info_struct *vswap_si;
+
+static inline bool is_vswap_entry(swp_entry_t entry)
+{
+ return swap_is_vswap(__swap_entry_to_info(entry));
+}
+
+#else
+
+static inline bool is_vswap_entry(swp_entry_t entry)
+{
+ return false;
+}
+
+#endif /* CONFIG_VSWAP */
+
+#endif /* _MM_VSWAP_H */
diff --git a/mm/zswap.c b/mm/zswap.c
index f7c9c89f6449..354bf8bd7482 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -1000,6 +1000,12 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
if (!si)
return -EEXIST;
+ /* Vswap entries have no physical backing to write to. */
+ if (swap_is_vswap(si)) {
+ put_swap_device(si);
+ return -EINVAL;
+ }
+
mpol = get_task_policy(current);
folio = swap_cache_alloc_folio(swpentry, GFP_KERNEL, BIT(0), NULL, mpol,
NO_INTERLEAVE_INDEX);
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v3 02/11] mm, swap: support zswap and zeroswap as vswap backends
2026-08-06 18:42 [PATCH v3 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
2026-08-06 18:42 ` [PATCH v3 01/11] mm, swap: add virtual swap device infrastructure Nhat Pham
@ 2026-08-06 18:42 ` Nhat Pham
2026-08-06 18:42 ` [PATCH v3 03/11] mm, swap: prepare the swap IO path for vswap Nhat Pham
` (10 subsequent siblings)
12 siblings, 0 replies; 17+ messages in thread
From: Nhat Pham @ 2026-08-06 18:42 UTC (permalink / raw)
To: akpm
Cc: chrisl, kasong, hannes, mhocko, roman.gushchin, shakeel.butt,
yosry, david, muchun.song, shikemeng, baoquan.he, baohua,
youngjun.park, chengming.zhou, ljs, liam, vbabka, rppt, surenb,
qi.zheng, axelrasmussen, yuanchu, weixugc, riel, gourry,
haowenchao22, corbet, kernel-team, nphamcs, linux-mm,
linux-kernel, linux-doc, cgroups
Build the virtual swap layer on top of the swap-table infrastructure.
Virtual swap entries decouple PTE swap entries from physical backing,
allowing pages to be compressed by zswap (or detected as zero-filled)
without pre-allocating a physical swap slot.
This patch only supports zswap and zero-page backends. If zswap_store
fails, the page stays dirty in the swap cache. Physical disk backing
arrives in the next patch.
Zswap writeback of vswap-backed entries is also disabled: they have no
physical slot to write back to yet, so the zswap shrinker (both the
dynamic count path and the pool-full worker path) is skipped while
vswap is enabled. Physical backing and real writeback come in later
patches.
THP swapin is disabled for vswap entries for now.
Add a /proc/sys/vm/vswap_enabled sysctl and a CONFIG_VSWAP_DEFAULT_ON
build option so vswap allocation can be enabled and disabled at
runtime, defaulting off unless CONFIG_VSWAP_DEFAULT_ON=y. The knob
only gates vswap_alloc(), so existing virtual entries keep resolving
their backend and drain naturally when it is turned off.
Suggested-by: Kairui Song <kasong@tencent.com>
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
Documentation/admin-guide/sysctl/vm.rst | 16 ++
include/linux/zswap.h | 3 +
mm/Kconfig | 11 ++
mm/memcontrol.c | 8 +
mm/memory.c | 18 +-
mm/page_io.c | 12 +-
mm/shmem.c | 4 +-
mm/swap.h | 1 +
mm/swap_state.c | 8 +
mm/swapfile.c | 242 ++++++++++++++++++++++--
mm/vmscan.c | 14 +-
mm/vswap.h | 206 +++++++++++++++++++-
mm/zswap.c | 56 ++++--
13 files changed, 561 insertions(+), 38 deletions(-)
diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst
index 5b318d17aa4b..50b41f292631 100644
--- a/Documentation/admin-guide/sysctl/vm.rst
+++ b/Documentation/admin-guide/sysctl/vm.rst
@@ -74,6 +74,7 @@ Currently, these files are in /proc/sys/vm:
- user_reserve_kbytes
- vfs_cache_pressure
- vfs_cache_pressure_denom
+- vswap_enabled
- watermark_boost_factor
- watermark_scale_factor
- zone_reclaim_mode
@@ -1152,6 +1153,21 @@ vfs_cache_pressure_denom
Defaults to 100 (minimum allowed value). Requires corresponding
vfs_cache_pressure setting to take effect.
+vswap_enabled
+=============
+
+Controls whether new swapouts are routed through the virtual swap layer
+(only present when the kernel is built with CONFIG_VSWAP). Set to 1 to
+route swapouts through vswap, 0 to send them straight to the physical
+swap device.
+
+The default is 0 unless the kernel was built with
+CONFIG_VSWAP_DEFAULT_ON=y.
+
+Disabling is allocation-only: it only stops new swapouts from using
+vswap. Swap entries already backed by vswap keep being served and drain
+naturally as they are faulted back in or freed.
+
watermark_boost_factor
======================
diff --git a/include/linux/zswap.h b/include/linux/zswap.h
index 30c193a1207e..4b4f211f3301 100644
--- a/include/linux/zswap.h
+++ b/include/linux/zswap.h
@@ -6,6 +6,7 @@
#include <linux/mm_types.h>
struct lruvec;
+struct zswap_entry;
extern atomic_long_t zswap_stored_pages;
@@ -28,6 +29,7 @@ unsigned long zswap_total_pages(void);
bool zswap_store(struct folio *folio);
int zswap_load(struct folio *folio);
void zswap_invalidate(swp_entry_t swp);
+void zswap_entry_free(struct zswap_entry *entry);
int zswap_swapon(int type, unsigned long nr_pages);
void zswap_swapoff(int type);
void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg);
@@ -50,6 +52,7 @@ static inline int zswap_load(struct folio *folio)
}
static inline void zswap_invalidate(swp_entry_t swp) {}
+static inline void zswap_entry_free(struct zswap_entry *entry) {}
static inline int zswap_swapon(int type, unsigned long nr_pages)
{
return 0;
diff --git a/mm/Kconfig b/mm/Kconfig
index 32d38b552845..8d147c0483ef 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -29,6 +29,17 @@ config VSWAP
swapfile, or kept in memory - with the backing changeable at
runtime without invalidating page table entries.
+config VSWAP_DEFAULT_ON
+ bool "Route swapouts through virtual swap by default"
+ depends on VSWAP
+ default n
+ help
+ Say Y to route swapouts through the virtual swap layer from
+ boot.
+
+ Say N (default) to leave vswap off until it is enabled at
+ runtime via /proc/sys/vm/vswap_enabled.
+
config ZSWAP
bool "Compressed cache for swap pages"
depends on SWAP
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 77582acd8ee5..7a426db06222 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -65,6 +65,7 @@
#include "internal.h"
#include "swap.h"
#include "swap_table.h"
+#include "vswap.h"
#include <net/sock.h>
#include <net/ip.h>
#include "slab.h"
@@ -5728,6 +5729,13 @@ long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg)
{
long nr_swap_pages = get_nr_swap_pages();
+ /*
+ * vswap zswap-backed swapout needs no physical slot, so gate anon
+ * reclaim on the swap.max headroom instead of the physical free count.
+ */
+ if (vswap_is_enabled() && zswap_is_enabled())
+ nr_swap_pages = PAGE_COUNTER_MAX;
+
if (mem_cgroup_disabled() || do_memsw_account())
return nr_swap_pages;
for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg))
diff --git a/mm/memory.c b/mm/memory.c
index 6ae52e3869b1..de3573b7c6b1 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -89,6 +89,7 @@
#include "pgalloc-track.h"
#include "internal.h"
#include "swap.h"
+#include "vswap.h"
#if defined(LAST_CPUPID_NOT_IN_PAGE_FLAGS) && !defined(CONFIG_COMPILE_TEST)
#warning Unfortunate NUMA and NUMA Balancing config, growing page-frame for last_cpupid.
@@ -4657,6 +4658,12 @@ static inline bool should_try_to_free_swap(struct swap_info_struct *si,
*/
if (data_race(si->flags & SWP_SYNCHRONOUS_IO))
return true;
+ /*
+ * Non-swapfile backends cannot be reused for future swapouts.
+ * Free the swap slot unless backed by contiguous physical swap.
+ */
+ if (is_vswap_entry(folio->swap))
+ return true;
if (mem_cgroup_swap_full(folio) || (vma->vm_flags & VM_LOCKED) ||
folio_test_mlocked(folio))
return true;
@@ -4805,15 +4812,16 @@ static unsigned long thp_swapin_suitable_orders(struct vm_fault *vmf)
if (unlikely(userfaultfd_armed(vma)))
return 0;
+ entry = softleaf_from_pte(vmf->orig_pte);
+
/*
- * A large swapped out folio could be partially or fully in zswap. We
- * lack handling for such cases, so fallback to swapping in order-0
- * folio.
+ * THP swapin for vswap is not supported yet. Also, a large swapped
+ * out folio could be partially or fully in zswap, which we lack
+ * handling for. In both cases, fall back to order-0 swapin.
*/
- if (!zswap_never_enabled())
+ if (is_vswap_entry(entry) || !zswap_never_enabled())
return 0;
- entry = softleaf_from_pte(vmf->orig_pte);
/*
* Get a list of all the (large) orders below PMD_ORDER that are enabled
* and suitable for swapping THP.
diff --git a/mm/page_io.c b/mm/page_io.c
index fca1718056af..b1894cd014b3 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -160,14 +160,19 @@ static void swap_zeromap_folio_set(struct folio *folio)
struct obj_cgroup *objcg = get_obj_cgroup_from_folio(folio);
int nr_pages = folio_nr_pages(folio);
struct swap_cluster_info *ci;
+ unsigned int voff, i;
swp_entry_t entry;
- unsigned int i;
VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);
VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
ci = swap_cluster_get_and_lock(folio);
- for (i = 0; i < folio_nr_pages(folio); i++) {
+ if (is_vswap_entry(folio->swap)) {
+ /* Free any prior backing (e.g. ZSWAP entry from earlier swapout) */
+ voff = swp_cluster_offset(folio->swap);
+ __vswap_release_backing(ci, voff, nr_pages);
+ }
+ for (i = 0; i < nr_pages; i++) {
entry = page_swap_entry(folio_page(folio, i));
__swap_table_set_zero(ci, swp_cluster_offset(entry));
}
@@ -235,6 +240,9 @@ int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
*/
swap_zeromap_folio_clear(folio);
+ if (is_vswap_entry(folio->swap))
+ folio_release_vswap_backing(folio);
+
if (zswap_store(folio)) {
count_mthp_stat(folio_order(folio), MTHP_STAT_ZSWPOUT);
goto out_unlock;
diff --git a/mm/shmem.c b/mm/shmem.c
index 2e4dacdcce11..153ac7433fb0 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -85,6 +85,7 @@ static struct vfsmount *shm_mnt __ro_after_init;
#include <linux/uaccess.h>
#include "internal.h"
+#include "vswap.h"
#define VM_ACCT(size) (PAGE_ALIGN(size) >> PAGE_SHIFT)
@@ -1617,7 +1618,8 @@ int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio,
if ((info->flags & SHMEM_F_LOCKED) || sbinfo->noswap)
goto redirty;
- if (!total_swap_pages)
+ /* vswap doesn't contribute to total_swap_pages */
+ if (!total_swap_pages && !(vswap_is_enabled() && zswap_is_enabled()))
goto redirty;
/*
diff --git a/mm/swap.h b/mm/swap.h
index b593ad3214ef..d241e81e967e 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -70,6 +70,7 @@ struct swap_cluster_info_dynamic {
struct swap_cluster_info ci;
unsigned int index; /* for cluster_index() */
struct rcu_head rcu;
+ atomic_long_t *virtual_table; /* Backing pointers for vswap slots */
};
/* All on-list cluster must have a non-zero flag. */
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 9e0d71fcdc24..9f6377b32911 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -26,6 +26,7 @@
#include "internal.h"
#include "swap_table.h"
#include "swap.h"
+#include "vswap.h"
/* Swap readahead cluster size, as a power of 2 pages. */
static int page_cluster;
@@ -196,6 +197,13 @@ static int __swap_cache_add_check(struct swap_cluster_info *ci,
if (nr == 1)
return 0;
+ /*
+ * THP swapin for vswap is not supported yet; reject the batch so
+ * swap_cache_alloc_folio falls back to order 0.
+ */
+ if (is_vswap_entry(targ_entry))
+ return -EBUSY;
+
is_zero = __swap_table_test_zero(ci, ci_off);
ci_off = round_down(ci_off, nr);
ci_end = ci_off + nr;
diff --git a/mm/swapfile.c b/mm/swapfile.c
index fea3a8eccbc1..a26cfe5751c8 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -23,6 +23,7 @@
#include <linux/random.h>
#include <linux/writeback.h>
#include <linux/proc_fs.h>
+#include <linux/sysctl.h>
#include <linux/seq_file.h>
#include <linux/init.h>
#include <linux/ksm.h>
@@ -131,6 +132,29 @@ static DEFINE_PER_CPU(struct percpu_swap_cluster, percpu_swap_cluster) = {
.lock = INIT_LOCAL_LOCK(),
};
+#ifdef CONFIG_VSWAP
+static int sysctl_vswap_enabled = IS_ENABLED(CONFIG_VSWAP_DEFAULT_ON);
+
+bool vswap_is_enabled(void)
+{
+ return sysctl_vswap_enabled;
+}
+
+struct percpu_vswap_cluster {
+ unsigned long offset[SWAP_NR_ORDERS];
+ local_lock_t lock;
+};
+
+static DEFINE_PER_CPU(struct percpu_vswap_cluster, percpu_vswap_cluster) = {
+ .offset = { [0 ... SWAP_NR_ORDERS - 1] = SWAP_ENTRY_INVALID },
+ .lock = INIT_LOCAL_LOCK(),
+};
+
+static bool vswap_alloc(struct folio *folio);
+#else
+static inline bool vswap_alloc(struct folio *folio) { return false; }
+#endif
+
/* May return NULL on invalid type, caller must check for NULL return */
static struct swap_info_struct *swap_type_to_info(int type)
{
@@ -236,7 +260,8 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si,
need_reclaim = ((flags & TTRS_ANYWAY) ||
((flags & TTRS_UNMAPPED) && !folio_mapped(folio)) ||
- ((flags & TTRS_FULL) && mem_cgroup_swap_full(folio)));
+ ((flags & TTRS_FULL) && mem_cgroup_swap_full(folio) &&
+ !is_vswap_entry(folio->swap)));
if (!need_reclaim || !folio_swapcache_freeable(folio))
goto out_unlock;
@@ -537,7 +562,12 @@ swap_cluster_populate(struct swap_info_struct *si,
* Only cluster isolation from the allocator does table allocation.
* Swap allocator uses percpu clusters and holds the local lock.
*/
- lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)->lock);
+#ifdef CONFIG_VSWAP
+ if (swap_is_vswap(si))
+ lockdep_assert_held(&this_cpu_ptr(&percpu_vswap_cluster)->lock);
+#endif
+ if (!swap_is_vswap(si))
+ lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)->lock);
if (!(si->flags & SWP_SOLIDSTATE))
lockdep_assert_held(&si->global_cluster_lock);
lockdep_assert_held(&ci->lock);
@@ -554,7 +584,12 @@ swap_cluster_populate(struct swap_info_struct *si,
spin_unlock(&ci->lock);
if (!(si->flags & SWP_SOLIDSTATE))
spin_unlock(&si->global_cluster_lock);
- local_unlock(&percpu_swap_cluster.lock);
+#ifdef CONFIG_VSWAP
+ if (swap_is_vswap(si))
+ local_unlock(&percpu_vswap_cluster.lock);
+#endif
+ if (!swap_is_vswap(si))
+ local_unlock(&percpu_swap_cluster.lock);
ret = swap_cluster_alloc_table(ci, __GFP_HIGH | __GFP_NOMEMALLOC |
GFP_KERNEL);
@@ -567,7 +602,12 @@ swap_cluster_populate(struct swap_info_struct *si,
* could happen with ignoring the percpu cluster is fragmentation,
* which is acceptable since this fallback and race is rare.
*/
- local_lock(&percpu_swap_cluster.lock);
+#ifdef CONFIG_VSWAP
+ if (swap_is_vswap(si))
+ local_lock(&percpu_vswap_cluster.lock);
+#endif
+ if (!swap_is_vswap(si))
+ local_lock(&percpu_swap_cluster.lock);
if (!(si->flags & SWP_SOLIDSTATE))
spin_lock(&si->global_cluster_lock);
spin_lock(&ci->lock);
@@ -729,6 +769,7 @@ static void vswap_free_cluster(struct swap_info_struct *si,
spin_unlock(&si->lock);
}
swap_cluster_free_table(ci);
+ vswap_cluster_free_vtable(ci);
/*
* Ordering vs the RCU cluster lookup: erase from the xarray first
* (new lookups miss it), mark DEAD under the held ci->lock (a lookup
@@ -765,6 +806,10 @@ static void free_cluster(struct swap_info_struct *si, struct swap_cluster_info *
return;
}
+ /*
+ * Vswap dynamic clusters need explicit cleanup (xarray erase,
+ * kfree_rcu, virtual_table free if allocated).
+ */
if (swap_is_vswap(si)) {
vswap_free_cluster(si, ci);
return;
@@ -947,7 +992,8 @@ static bool cluster_scan_range(struct swap_info_struct *si,
if (swp_tb_is_null(swp_tb))
continue;
if (swp_tb_is_folio(swp_tb) && !__swp_tb_get_count(swp_tb)) {
- if (!vm_swap_full())
+ /* vswap slots are unlimited; never reclaim to reuse one */
+ if (swap_is_vswap(si) || !vm_swap_full())
return false;
*need_reclaim = true;
continue;
@@ -1015,7 +1061,8 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct *si,
/* Try use a new cluster for current CPU and allocate from it. */
static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
struct swap_cluster_info *ci,
- struct folio *folio, unsigned long offset)
+ struct folio *folio,
+ unsigned long offset)
{
unsigned int next = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;
unsigned long start = ALIGN_DOWN(offset, SWAPFILE_CLUSTER);
@@ -1058,6 +1105,12 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
relocate_cluster(si, ci);
swap_cluster_unlock(ci);
}
+#ifdef CONFIG_VSWAP
+ if (swap_is_vswap(si)) {
+ this_cpu_write(percpu_vswap_cluster.offset[order], next);
+ return found;
+ }
+#endif
if (si->flags & SWP_SOLIDSTATE) {
this_cpu_write(percpu_swap_cluster.offset[order], next);
this_cpu_write(percpu_swap_cluster.si[order], si);
@@ -1110,10 +1163,17 @@ static unsigned int alloc_swap_scan_dynamic(struct swap_info_struct *si,
return SWAP_ENTRY_INVALID;
}
+ if (vswap_cluster_alloc_vtable(ci_dyn)) {
+ swap_cluster_free_table(&ci_dyn->ci);
+ kfree(ci_dyn);
+ return SWAP_ENTRY_INVALID;
+ }
+
if (xa_alloc(&si->cluster_info_pool, &ci_dyn->index, ci_dyn,
XA_LIMIT(1, DIV_ROUND_UP(si->max, SWAPFILE_CLUSTER) - 1),
GFP_ATOMIC)) {
swap_cluster_free_table(&ci_dyn->ci);
+ vswap_cluster_free_vtable(&ci_dyn->ci);
kfree(ci_dyn);
return SWAP_ENTRY_INVALID;
}
@@ -1199,7 +1259,7 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
* Swapfile is not block device so unable
* to allocate large entries.
*/
- if (order && !(si->flags & SWP_BLKDEV))
+ if (order && !(si->flags & SWP_BLKDEV) && !swap_is_vswap(si))
return 0;
if (!(si->flags & SWP_SOLIDSTATE)) {
@@ -1252,7 +1312,7 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
}
/* Try reclaim full clusters if free and nonfull lists are drained */
- if (vm_swap_full())
+ if (!swap_is_vswap(si) && vm_swap_full())
swap_reclaim_full_clusters(si, false);
if (order < PMD_ORDER) {
@@ -1416,7 +1476,8 @@ static void swap_range_alloc(struct swap_info_struct *si,
if (vm_swap_full())
schedule_work(&si->reclaim_work);
}
- atomic_long_sub(nr_entries, &nr_swap_pages);
+ if (!swap_is_vswap(si))
+ atomic_long_sub(nr_entries, &nr_swap_pages);
}
static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
@@ -1426,8 +1487,10 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
void (*swap_slot_free_notify)(struct block_device *, unsigned long);
unsigned int i;
- for (i = 0; i < nr_entries; i++)
- zswap_invalidate(swp_entry(si->type, offset + i));
+ if (!swap_is_vswap(si)) {
+ for (i = 0; i < nr_entries; i++)
+ zswap_invalidate(swp_entry(si->type, offset + i));
+ }
if (si->flags & SWP_BLKDEV)
swap_slot_free_notify =
@@ -1446,7 +1509,8 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
* only after the above cleanups are done.
*/
smp_wmb();
- atomic_long_add(nr_entries, &nr_swap_pages);
+ if (!swap_is_vswap(si))
+ atomic_long_add(nr_entries, &nr_swap_pages);
swap_usage_sub(si, nr_entries);
}
@@ -1838,6 +1902,49 @@ static int swap_dup_entries_cluster(struct swap_info_struct *si,
return err;
}
+#ifdef CONFIG_VSWAP
+static bool vswap_alloc(struct folio *folio)
+{
+ unsigned int order = folio_order(folio);
+ struct swap_cluster_info *ci;
+ unsigned long offset;
+
+ if (!sysctl_vswap_enabled)
+ return false;
+
+ /* vswap_init failed: fall back to direct physical swap */
+ if (!vswap_si)
+ return false;
+
+ local_lock(&percpu_vswap_cluster.lock);
+ offset = this_cpu_read(percpu_vswap_cluster.offset[order]);
+
+ if (offset != SWAP_ENTRY_INVALID) {
+ ci = swap_cluster_lock(vswap_si, offset);
+ if (ci && cluster_is_usable(ci, order)) {
+ if (cluster_is_empty(ci))
+ offset = cluster_offset(vswap_si, ci);
+ alloc_swap_scan_cluster(vswap_si, ci, folio, offset);
+ } else if (ci) {
+ swap_cluster_unlock(ci);
+ }
+ }
+
+ if (!folio_test_swapcache(folio))
+ cluster_alloc_swap_entry(vswap_si, folio);
+
+ if (folio_test_swapcache(folio)) {
+ /* alloc_swap_scan_cluster updated percpu offset already */
+ local_unlock(&percpu_vswap_cluster.lock);
+ return true;
+ }
+
+ this_cpu_write(percpu_vswap_cluster.offset[order], SWAP_ENTRY_INVALID);
+ local_unlock(&percpu_vswap_cluster.lock);
+ return false;
+}
+#endif
+
/**
* folio_alloc_swap - allocate swap space for a folio
* @folio: folio we want to move to swap
@@ -1875,12 +1982,17 @@ int folio_alloc_swap(struct folio *folio)
}
}
+ /* Without zswap a vswap entry has nowhere to go on writeout. */
+ if (zswap_is_enabled() && vswap_alloc(folio))
+ goto done;
+
again:
local_lock(&percpu_swap_cluster.lock);
if (!swap_alloc_fast(folio))
swap_alloc_slow(folio);
local_unlock(&percpu_swap_cluster.lock);
+done:
if (!order && unlikely(!folio_test_swapcache(folio))) {
if (swap_sync_discard())
goto again;
@@ -1896,6 +2008,80 @@ int folio_alloc_swap(struct folio *folio)
return 0;
}
+#ifdef CONFIG_VSWAP
+
+/**
+ * __vswap_release_backing - release the backing of a range of vtable slots
+ * @ci: the locked vswap cluster
+ * @ci_start: first slot offset within @ci
+ * @nr: number of slots
+ *
+ * Releases each slot in [@ci_start, @ci_start + @nr): physical swap slots,
+ * zswap entries, etc. Clears the zero marks if set.
+ *
+ * Context: caller must hold @ci->lock. The entire range must belong to the
+ * same memcg.
+ */
+void __vswap_release_backing(struct swap_cluster_info *ci,
+ unsigned int ci_start, unsigned int nr)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ unsigned int ci_off;
+ unsigned long vt;
+
+ lockdep_assert_held(&ci->lock);
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+
+ for (ci_off = ci_start; ci_off < ci_start + nr; ci_off++) {
+ vt = __vtable_get(ci_dyn, ci_off);
+
+ switch (vtable_type(vt)) {
+ case VSWAP_ZSWAP:
+ zswap_entry_free(vtable_to_zswap(vt));
+ break;
+ case VSWAP_NONE:
+ break;
+ default:
+ /* VSWAP_ZERO/VSWAP_FOLIO are return-only, not vtable tags */
+ break;
+ }
+
+ __vtable_set(ci_dyn, ci_off, VSWAP_NONE);
+ /* Zero-backed state lives in swap_table; clear it too. */
+ if (__swap_table_test_zero(ci, ci_off))
+ __swap_table_clear_zero(ci, ci_off);
+ }
+}
+
+/**
+ * folio_release_vswap_backing() - Drop all backing for a folio's vswap entry.
+ * @folio: the folio, occupying a virtual swap entry.
+ *
+ * Release whatever backing the folio's virtual swap slots currently hold and
+ * reset them to empty, so a fresh backing can be installed. Used when a
+ * folio's swap backend is replaced.
+ *
+ * Context: Caller must hold the folio lock; @folio must be in the swap cache
+ * and occupy a virtual swap entry.
+ */
+void folio_release_vswap_backing(struct folio *folio)
+{
+ struct swap_cluster_info *ci;
+ int nr = folio_nr_pages(folio);
+ unsigned int voff;
+
+ ci = __swap_entry_to_cluster(folio->swap);
+ if (!ci)
+ return;
+ voff = swp_cluster_offset(folio->swap);
+
+ spin_lock(&ci->lock);
+ __vswap_release_backing(ci, voff, nr);
+ spin_unlock(&ci->lock);
+}
+
+#endif /* CONFIG_VSWAP */
+
/**
* folio_dup_swap() - Increase swap count of swap entries of a folio.
* @folio: folio with swap entries bounded.
@@ -2037,6 +2223,9 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,
VM_WARN_ON(ci->count < nr_pages);
+ if (swap_is_vswap(si))
+ __vswap_release_backing(ci, ci_start, nr_pages);
+
ci->count -= nr_pages;
do {
old_tb = __swap_table_get(ci, ci_off);
@@ -2907,6 +3096,7 @@ static int try_to_unuse(unsigned int type)
(i = find_next_to_unuse(si, i)) != 0) {
entry = swp_entry(type, i);
+
folio = swap_cache_get_folio(entry);
if (!folio)
continue;
@@ -4134,6 +4324,18 @@ struct swap_info_struct *vswap_si;
/* vswap does no IO on its own. */
static const struct swap_ops vswap_ops = { };
+static const struct ctl_table vswap_sysctls[] = {
+ {
+ .procname = "vswap_enabled",
+ .data = &sysctl_vswap_enabled,
+ .maxlen = sizeof(sysctl_vswap_enabled),
+ .mode = 0644,
+ .proc_handler = proc_dointvec_minmax,
+ .extra1 = SYSCTL_ZERO,
+ .extra2 = SYSCTL_ONE,
+ },
+};
+
static int __init vswap_init(void)
{
struct swap_info_struct *si;
@@ -4141,8 +4343,12 @@ static int __init vswap_init(void)
int err;
si = alloc_swap_info();
- if (IS_ERR(si))
- return PTR_ERR(si);
+ if (IS_ERR(si)) {
+ pr_warn("vswap: alloc_swap_info failed (%ld); vswap disabled, swapout falls back to direct physical swap\n",
+ PTR_ERR(si));
+ sysctl_vswap_enabled = 0;
+ return 0;
+ }
maxpages = min(swapfile_maximum_size,
ALIGN_DOWN((unsigned long)UINT_MAX, SWAPFILE_CLUSTER));
@@ -4164,14 +4370,20 @@ static int __init vswap_init(void)
mutex_unlock(&swapon_mutex);
vswap_si = si;
+
+ register_sysctl_init("vm", vswap_sysctls);
+
pr_info("vswap: created virtual swap device (%lu pages)\n", maxpages);
return 0;
fail:
+ pr_warn("vswap: setup_swap_clusters_info failed (%d); vswap disabled, swapout falls back to direct physical swap\n",
+ err);
+ sysctl_vswap_enabled = 0;
spin_lock(&swap_lock);
si->flags = 0;
spin_unlock(&swap_lock);
- return err;
+ return 0;
}
late_initcall(vswap_init);
#endif
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 17d2b793cbfc..78ec51f53757 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -68,6 +68,7 @@
#include "internal.h"
#include "page_alloc.h"
#include "swap.h"
+#include "vswap.h"
#define CREATE_TRACE_POINTS
#include <trace/events/vmscan.h>
@@ -352,6 +353,9 @@ static inline bool can_reclaim_anon_pages(struct mem_cgroup *memcg,
*/
if (get_nr_swap_pages() > 0)
return true;
+ /* vswap doesn't contribute to nr_swap_pages */
+ if (vswap_is_enabled() && zswap_is_enabled())
+ return true;
} else {
/* Is the memcg below its swap limit? */
if (mem_cgroup_get_nr_swap_pages(memcg) > 0)
@@ -1521,9 +1525,13 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
nr_pages = 1;
}
activate_locked:
- /* Not a candidate for swapping, so reclaim swap space. */
+ /*
+ * Not a candidate for swapping, so reclaim physical swap
+ * space if we are running out.
+ */
if (folio_test_swapcache(folio) &&
- (mem_cgroup_swap_full(folio) || folio_test_mlocked(folio)))
+ ((mem_cgroup_swap_full(folio) && !is_vswap_entry(folio->swap)) ||
+ folio_test_mlocked(folio)))
folio_free_swap(folio);
VM_BUG_ON_FOLIO(folio_test_active(folio), folio);
if (!folio_test_mlocked(folio)) {
@@ -2680,7 +2688,7 @@ static bool can_age_anon_pages(struct lruvec *lruvec,
struct scan_control *sc)
{
/* Aging the anon LRU is valuable if swap is present: */
- if (total_swap_pages > 0)
+ if (total_swap_pages > 0 || (vswap_is_enabled() && zswap_is_enabled()))
return true;
/* Also valuable if anon pages can be demoted: */
diff --git a/mm/vswap.h b/mm/vswap.h
index 5641692f5be3..6d25e0911fa9 100644
--- a/mm/vswap.h
+++ b/mm/vswap.h
@@ -10,8 +10,23 @@
#include <linux/swap.h>
#include "swap.h"
+struct zswap_entry;
+
+/*
+ * VSWAP_ZERO and VSWAP_FOLIO are return-only values synthesized from
+ * swap_table state; the rest are stored in the vtable per slot.
+ */
+enum vswap_backing_type {
+ VSWAP_NONE = 0,
+ VSWAP_ZSWAP = 1,
+ VSWAP_ZERO,
+ VSWAP_FOLIO,
+};
+
#ifdef CONFIG_VSWAP
+#include "swap_table.h"
+
extern struct swap_info_struct *vswap_si;
static inline bool is_vswap_entry(swp_entry_t entry)
@@ -19,13 +34,202 @@ static inline bool is_vswap_entry(swp_entry_t entry)
return swap_is_vswap(__swap_entry_to_info(entry));
}
-#else
+bool vswap_is_enabled(void);
+
+/*
+ * Virtual table entry encoding for vswap clusters.
+ *
+ * Each entry in ci_dyn->virtual_table stores the backing type and
+ * pointer for a virtual swap slot. Tag in low 3 bits, payload in
+ * upper 61 bits.
+ *
+ * NONE: |----- 0000 ------|000| - no separate backend pointer
+ * ZSWAP: |--- zswap_entry* |001| - compressed in zswap (tag in low bits)
+ *
+ * Pointer payloads (ZSWAP) are stored directly with the tag OR'd into the
+ * low bits (kernel pointers are >= 8-byte aligned, same approach as xarray).
+ *
+ * vtable[i] = NONE does not by itself mean "free". The swap_table entry
+ * and the per-slot zero flag carry the rest of the state. The full
+ * per-slot state table is:
+ *
+ * vtable[i] | swap_table[i] | zero | meaning
+ * ----------+---------------+-------+--------------------------------
+ * NONE | NULL | clear | truly free / unbacked
+ * NONE | PFN | clear | folio cached, no backing
+ * NONE | shadow | clear | folio evicted, no backing (bug)
+ * NONE | * | set | zero-backed; cached if PFN set
+ * ZSWAP | PFN | clear | folio cached + zswap entry
+ * ZSWAP | shadow / NULL | clear | evicted, only in zswap
+ *
+ * Locking: a slot's vtable entry (the vswap entry's backend) is only
+ * stable while the caller owns and holds the lock on that entry's swap
+ * cache folio. The cluster lock (ci_dyn->ci.lock) only makes an individual
+ * vtable read atomic, and by itself does not give the caller the right to
+ * change the backend. A backend read without the folio lock is
+ * best-effort and must be re-validated under the folio lock before
+ * being acted on.
+ *
+ * Zero-backed slots use the swap_table per-slot zero flag (same as
+ * direct-mapped physical swap), since CONFIG_VSWAP requires 64BIT and
+ * SWAP_TABLE_HAS_ZEROFLAG is always true on 64-bit. Cached folios are
+ * read out of the swap_table PFN entry; there is no separate FOLIO
+ * vtable type because the folio pointer would duplicate that PFN and
+ * would go stale on folio migration / split.
+ */
+
+#define VTABLE_TAG_BITS 3
+#define VTABLE_TAG_MASK ((1UL << VTABLE_TAG_BITS) - 1)
+
+static inline enum vswap_backing_type vtable_type(unsigned long vt)
+{
+ return vt & VTABLE_TAG_MASK;
+}
+
+static inline struct zswap_entry *vtable_to_zswap(unsigned long vt)
+{
+ VM_WARN_ON(vtable_type(vt) != VSWAP_ZSWAP);
+ return (struct zswap_entry *)(vt & ~VTABLE_TAG_MASK);
+}
+
+/* Virtual table accessors */
+
+static inline unsigned long __vtable_get(struct swap_cluster_info_dynamic *ci_dyn,
+ unsigned int off)
+{
+ VM_WARN_ON_ONCE(off >= SWAPFILE_CLUSTER);
+ return atomic_long_read(&ci_dyn->virtual_table[off]);
+}
+
+static inline void __vtable_set(struct swap_cluster_info_dynamic *ci_dyn,
+ unsigned int off, unsigned long vt)
+{
+ VM_WARN_ON_ONCE(off >= SWAPFILE_CLUSTER);
+ atomic_long_set(&ci_dyn->virtual_table[off], vt);
+}
+
+/**
+ * vswap_lock_cluster - look up and lock the vswap cluster for an entry
+ * @entry: the virtual swap entry
+ * @voff: out param, receives @entry's slot offset within the cluster
+ *
+ * Return: the locked vswap cluster, or NULL if no cluster is found for @entry.
+ */
+static inline struct swap_cluster_info_dynamic *
+vswap_lock_cluster(swp_entry_t entry, unsigned int *voff)
+{
+ struct swap_cluster_info *ci;
+ struct swap_cluster_info_dynamic *ci_dyn;
+
+ ci = __swap_entry_to_cluster(entry);
+ if (!ci)
+ return NULL;
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ *voff = swp_cluster_offset(entry);
+ spin_lock(&ci->lock);
+ return ci_dyn;
+}
+
+void __vswap_release_backing(struct swap_cluster_info *ci,
+ unsigned int ci_start, unsigned int nr);
+
+/**
+ * vswap_zswap_store - record a zswap entry as the backing for a vswap entry.
+ * @entry: the vswap entry
+ * @ze: the zswap entry now holding @entry's compressed data
+ *
+ * Releases @entry's previous backing, and sets the zswap entry @ze as the new
+ * backing.
+ *
+ * Context: takes and drops the vswap cluster lock internally.
+ */
+static inline void vswap_zswap_store(swp_entry_t entry,
+ struct zswap_entry *ze)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ unsigned int voff;
+
+ ci_dyn = vswap_lock_cluster(entry, &voff);
+ if (!ci_dyn)
+ return;
+ __vswap_release_backing(&ci_dyn->ci, voff, 1);
+ __vtable_set(ci_dyn, voff, (unsigned long)ze | VSWAP_ZSWAP);
+ spin_unlock(&ci_dyn->ci.lock);
+}
+
+/**
+ * vswap_zswap_load - return the zswap entry backing a vswap entry
+ * @entry: the virtual swap entry
+ *
+ * Context: takes and drops the vswap cluster lock internally.
+ * Return: the backing zswap entry, or NULL if @entry is not zswap-backed.
+ */
+static inline struct zswap_entry *vswap_zswap_load(swp_entry_t entry)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ unsigned int voff;
+ unsigned long vt;
+
+ ci_dyn = vswap_lock_cluster(entry, &voff);
+ if (!ci_dyn)
+ return NULL;
+ vt = __vtable_get(ci_dyn, voff);
+ spin_unlock(&ci_dyn->ci.lock);
+
+ if (vtable_type(vt) != VSWAP_ZSWAP)
+ return NULL;
+ return vtable_to_zswap(vt);
+}
+
+void folio_release_vswap_backing(struct folio *folio);
+
+static inline int vswap_cluster_alloc_vtable(struct swap_cluster_info_dynamic *ci_dyn)
+{
+ ci_dyn->virtual_table = kcalloc(SWAPFILE_CLUSTER,
+ sizeof(*ci_dyn->virtual_table),
+ GFP_ATOMIC);
+ return ci_dyn->virtual_table ? 0 : -ENOMEM;
+}
+
+static inline void vswap_cluster_free_vtable(struct swap_cluster_info *ci)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ kfree(ci_dyn->virtual_table);
+ ci_dyn->virtual_table = NULL;
+}
+
+#else /* !CONFIG_VSWAP */
static inline bool is_vswap_entry(swp_entry_t entry)
{
return false;
}
+static inline bool vswap_is_enabled(void) { return false; }
+
+static inline void __vswap_release_backing(struct swap_cluster_info *ci,
+ unsigned int ci_start,
+ unsigned int nr) {}
+
+static inline void vswap_zswap_store(swp_entry_t entry,
+ struct zswap_entry *ze) {}
+
+static inline struct zswap_entry *vswap_zswap_load(swp_entry_t entry)
+{
+ return NULL;
+}
+
+static inline void folio_release_vswap_backing(struct folio *folio) {}
+
+static inline int vswap_cluster_alloc_vtable(struct swap_cluster_info_dynamic *ci_dyn)
+{
+ return 0;
+}
+
+static inline void vswap_cluster_free_vtable(struct swap_cluster_info *ci) {}
+
#endif /* CONFIG_VSWAP */
#endif /* _MM_VSWAP_H */
diff --git a/mm/zswap.c b/mm/zswap.c
index 354bf8bd7482..e19bde9df722 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -38,6 +38,7 @@
#include <linux/zsmalloc.h>
#include "swap.h"
+#include "vswap.h"
#include "internal.h"
/*********************************
@@ -234,6 +235,25 @@ static inline struct xarray *swap_zswap_tree(swp_entry_t swp)
>> ZSWAP_ADDRESS_SPACE_SHIFT];
}
+static struct zswap_entry *zswap_entry_load(swp_entry_t swp)
+{
+ if (is_vswap_entry(swp))
+ return vswap_zswap_load(swp);
+ return xa_load(swap_zswap_tree(swp), swp_offset(swp));
+}
+
+static struct zswap_entry *zswap_entry_store(swp_entry_t swp,
+ struct zswap_entry *entry)
+{
+ if (is_vswap_entry(swp)) {
+ vswap_zswap_store(swp, entry);
+ return NULL;
+ }
+
+ return xa_store(swap_zswap_tree(swp), swp_offset(swp), entry,
+ GFP_KERNEL);
+}
+
#define zswap_pool_debug(msg, p) \
pr_debug("%s pool %s\n", msg, (p)->tfm_name)
@@ -762,7 +782,7 @@ static void zswap_entry_cache_free(struct zswap_entry *entry)
* Carries out the common pattern of freeing an entry's zsmalloc allocation,
* freeing the entry itself, and decrementing the number of stored pages.
*/
-static void zswap_entry_free(struct zswap_entry *entry)
+void zswap_entry_free(struct zswap_entry *entry)
{
zswap_lru_del(entry);
zs_free(entry->pool->zs_pool, entry->handle);
@@ -1208,6 +1228,9 @@ static unsigned long zswap_shrinker_count(struct shrinker *shrinker,
if (!zswap_shrinker_enabled || !mem_cgroup_zswap_writeback_enabled(memcg))
return 0;
+ if (vswap_is_enabled())
+ return 0;
+
/*
* The shrinker resumes swap writeback, which will enter block
* and may enter fs. XXX: Harmonize with vmscan.c __GFP_FS
@@ -1290,6 +1313,9 @@ static int shrink_memcg(struct mem_cgroup *memcg)
if (!mem_cgroup_zswap_writeback_enabled(memcg))
return -ENOENT;
+ if (vswap_is_enabled())
+ return -ENOENT;
+
/*
* Skip zombies because their LRUs are reparented and we would be
* reclaiming from the parent instead of the dead memcg.
@@ -1418,9 +1444,7 @@ static bool zswap_store_page(struct page *page,
if (!zswap_compress(page, entry, pool))
goto compress_failed;
- old = xa_store(swap_zswap_tree(page_swpentry),
- swp_offset(page_swpentry),
- entry, GFP_KERNEL);
+ old = zswap_entry_store(page_swpentry, entry);
if (xa_is_err(old)) {
int err = xa_err(old);
@@ -1489,7 +1513,7 @@ bool zswap_store(struct folio *folio)
struct mem_cgroup *memcg = NULL;
struct zswap_pool *pool;
bool ret = false;
- long index;
+ long index = 0;
VM_WARN_ON_ONCE(!folio_test_locked(folio));
VM_WARN_ON_ONCE(!folio_test_swapcache(folio));
@@ -1544,13 +1568,19 @@ bool zswap_store(struct folio *folio)
if (!ret && zswap_pool_reached_full)
queue_work(shrink_wq, &zswap_shrink_work);
check_old:
+ if (ret)
+ return ret;
+
/*
* If the zswap store fails or zswap is disabled, we must invalidate
* the possibly stale entries which were previously stored at the
* offsets corresponding to each page of the folio. Otherwise,
* writeback could overwrite the new data in the swapfile.
*/
- if (!ret) {
+ if (is_vswap_entry(swp)) {
+ if (index > 0)
+ folio_release_vswap_backing(folio);
+ } else {
unsigned type = swp_type(swp);
pgoff_t offset = swp_offset(swp);
struct zswap_entry *entry;
@@ -1590,8 +1620,7 @@ bool zswap_store(struct folio *folio)
int zswap_load(struct folio *folio)
{
swp_entry_t swp = folio->swap;
- pgoff_t offset = swp_offset(swp);
- struct xarray *tree = swap_zswap_tree(swp);
+ struct swap_info_struct *si = __swap_entry_to_info(swp);
struct zswap_entry *entry;
VM_WARN_ON_ONCE(!folio_test_locked(folio));
@@ -1610,7 +1639,7 @@ int zswap_load(struct folio *folio)
return -EINVAL;
}
- entry = xa_load(tree, offset);
+ entry = zswap_entry_load(swp);
if (!entry)
return -ENOENT;
@@ -1633,8 +1662,13 @@ int zswap_load(struct folio *folio)
* compression work.
*/
folio_mark_dirty(folio);
- xa_erase(tree, offset);
- zswap_entry_free(entry);
+
+ if (swap_is_vswap(si)) {
+ folio_release_vswap_backing(folio);
+ } else {
+ xa_erase(swap_zswap_tree(swp), swp_offset(swp));
+ zswap_entry_free(entry);
+ }
folio_unlock(folio);
return 0;
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v3 03/11] mm, swap: prepare the swap IO path for vswap
2026-08-06 18:42 [PATCH v3 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
2026-08-06 18:42 ` [PATCH v3 01/11] mm, swap: add virtual swap device infrastructure Nhat Pham
2026-08-06 18:42 ` [PATCH v3 02/11] mm, swap: support zswap and zeroswap as vswap backends Nhat Pham
@ 2026-08-06 18:42 ` Nhat Pham
2026-08-06 18:42 ` [PATCH v3 04/11] mm, swap: support physical swap as a vswap backend Nhat Pham
` (9 subsequent siblings)
12 siblings, 0 replies; 17+ messages in thread
From: Nhat Pham @ 2026-08-06 18:42 UTC (permalink / raw)
To: akpm
Cc: chrisl, kasong, hannes, mhocko, roman.gushchin, shakeel.butt,
yosry, david, muchun.song, shikemeng, baoquan.he, baohua,
youngjun.park, chengming.zhou, ljs, liam, vbabka, rppt, surenb,
qi.zheng, axelrasmussen, yuanchu, weixugc, riel, gourry,
haowenchao22, corbet, kernel-team, nphamcs, linux-mm,
linux-kernel, linux-doc, cgroups
In preparation for adding a physical swap backend for vswap, make the
swap IO path able to submit IO for a swap entry other than folio->swap.
swap_add_folio() and __swap_writepage() derive the target device and
sector from folio->swap. For a vswap folio backed by a physical slot
that entry is virtual, so it identifies neither the backing device nor
the sector to submit IO against.
Compute the sector from an explicit entry (swap_folio_sector becomes
swap_entry_sector), thread that entry through swap_add_folio,
__swap_writepage and ops->can_merge, and stash it in swap_iocb so the
submit path addresses the IO from it rather than from folio->swap.
This lets the batching path serve both vswap entries (backed by a
physical slot) and physical entries mapped directly into PTEs.
All callers pass folio->swap for now, so there is no functional change.
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
include/linux/swap.h | 2 +-
mm/page_io.c | 51 ++++++++++++++++++++++----------------------
mm/swap.h | 7 +++---
mm/swapfile.c | 6 +++---
mm/zswap.c | 2 +-
5 files changed, 35 insertions(+), 33 deletions(-)
diff --git a/include/linux/swap.h b/include/linux/swap.h
index a955bd60dd58..8359134d08cb 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -389,7 +389,7 @@ extern int __swap_count(swp_entry_t entry);
extern bool swap_entry_swapped(struct swap_info_struct *si, swp_entry_t entry);
extern int swp_swapcount(swp_entry_t entry);
extern struct swap_info_struct *get_swap_device(swp_entry_t entry);
-sector_t swap_folio_sector(struct folio *folio);
+sector_t swap_entry_sector(swp_entry_t entry);
/*
* If there is an existing swap slot reference (swap entry) and the caller
diff --git a/mm/page_io.c b/mm/page_io.c
index b1894cd014b3..5c780bda92bb 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -265,7 +265,7 @@ int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
return AOP_WRITEPAGE_ACTIVATE;
}
- __swap_writepage(ctx, folio);
+ __swap_writepage(ctx, folio, folio->swap);
return 0;
out_unlock:
folio_unlock(folio);
@@ -326,6 +326,7 @@ struct swap_iocb {
struct bio_vec bvecs[SWAP_CLUSTER_MAX];
int nr_bvecs;
int len;
+ swp_entry_t entry; /* first slot in the batch; addresses the IO */
};
static mempool_t *sio_pool;
@@ -343,24 +344,22 @@ int sio_pool_init(void)
}
static bool swap_can_merge(struct swap_io_ctx *ctx, struct folio *folio,
- int rw)
+ swp_entry_t phys, int rw)
{
- struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
- struct bio_vec *last_bv = &ctx->sio->bvecs[ctx->sio->nr_bvecs - 1];
- struct folio *prev_folio = bvec_folio(last_bv);
- size_t prev_folio_size = folio_size(prev_folio);
+ struct swap_info_struct *sis = __swap_entry_to_info(phys);
if (ctx->sis != sis)
return false;
- return sis->ops->can_merge(folio, prev_folio, prev_folio_size, rw);
+ return sis->ops->can_merge(folio, phys, ctx->sio, rw);
}
-static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, int rw)
+static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio,
+ swp_entry_t phys, int rw)
{
- struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
+ struct swap_info_struct *sis = __swap_entry_to_info(phys);
struct swap_iocb *sio = ctx->sio;
- if (sio && !swap_can_merge(ctx, folio, rw)) {
+ if (sio && !swap_can_merge(ctx, folio, phys, rw)) {
if (rw == WRITE)
swap_write_submit(ctx);
else
@@ -373,6 +372,7 @@ static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, int rw)
ctx->sio = sio = mempool_alloc(sio_pool, GFP_NOIO);
sio->nr_bvecs = 0;
sio->len = 0;
+ sio->entry = phys;
}
bvec_set_folio(&sio->bvecs[sio->nr_bvecs], folio, folio_size(folio), 0);
sio->len += folio_size(folio);
@@ -384,7 +384,8 @@ static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, int rw)
}
}
-void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio)
+void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio,
+ swp_entry_t phys)
{
VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio);
@@ -400,7 +401,7 @@ void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio)
folio_start_writeback(folio);
folio_unlock(folio);
- swap_add_folio(ctx, folio, WRITE);
+ swap_add_folio(ctx, folio, phys, WRITE);
}
/*
@@ -504,7 +505,7 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)
/* We have to read from slower devices. Increase zswap protection. */
zswap_folio_swapin(folio);
- swap_add_folio(ctx, folio, READ);
+ swap_add_folio(ctx, folio, folio->swap, READ);
finish:
if (workingset) {
@@ -617,7 +618,7 @@ static void swap_bdev_submit_write(struct swap_io_ctx *ctx)
bio_init(bio, ctx->sis->bdev, sio->bvecs, ARRAY_SIZE(sio->bvecs),
REQ_OP_WRITE | REQ_SWAP);
bio->bi_iter.bi_size = sio->len;
- bio->bi_iter.bi_sector = swap_folio_sector(bio_first_folio_all(bio));
+ bio->bi_iter.bi_sector = swap_entry_sector(sio->entry);
bio_associate_blkg_from_page(bio, bio_first_folio_all(bio));
if (ctx->sis->flags & SWP_SYNCHRONOUS_IO) {
@@ -637,7 +638,7 @@ static void swap_bdev_submit_read(struct swap_io_ctx *ctx)
bio_init(bio, ctx->sis->bdev, sio->bvecs, ARRAY_SIZE(sio->bvecs),
REQ_OP_READ);
bio->bi_iter.bi_size = sio->len;
- bio->bi_iter.bi_sector = swap_folio_sector(bio_first_folio_all(bio));
+ bio->bi_iter.bi_sector = swap_entry_sector(sio->entry);
if (ctx->sis->flags & SWP_SYNCHRONOUS_IO) {
/*
@@ -655,13 +656,14 @@ static void swap_bdev_submit_read(struct swap_io_ctx *ctx)
}
}
-static bool swap_bdev_can_merge(struct folio *folio, struct folio *prev_folio,
- size_t prev_folio_size, int rw)
+static bool swap_bdev_can_merge(struct folio *folio, swp_entry_t phys,
+ struct swap_iocb *sio, int rw)
{
- if (swap_folio_sector(folio) !=
- swap_folio_sector(prev_folio) + (prev_folio_size >> SECTOR_SHIFT))
+ if (swap_entry_sector(phys) !=
+ swap_entry_sector(sio->entry) + (sio->len >> SECTOR_SHIFT))
return false;
- if (rw == WRITE && !folio_blkg_can_merge(folio, prev_folio))
+ if (rw == WRITE && !folio_blkg_can_merge(folio,
+ bvec_folio(&sio->bvecs[sio->nr_bvecs - 1])))
return false;
return true;
}
@@ -679,7 +681,7 @@ static void swap_fs_submit(struct swap_io_ctx *ctx, int rw)
int ret;
init_sync_kiocb(&sio->iocb, ctx->sis->swap_file);
- sio->iocb.ki_pos = swap_dev_pos(bvec_folio(&sio->bvecs[0])->swap);
+ sio->iocb.ki_pos = swap_dev_pos(sio->entry);
if (rw == WRITE)
sio->iocb.ki_complete = swap_fs_write_complete;
else
@@ -702,11 +704,10 @@ static void swap_fs_submit_read(struct swap_io_ctx *ctx)
swap_fs_submit(ctx, READ);
}
-static bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio,
- size_t prev_folio_size, int rw)
+static bool swap_fs_can_merge(struct folio *folio, swp_entry_t phys,
+ struct swap_iocb *sio, int rw)
{
- return swap_dev_pos(folio->swap) ==
- swap_dev_pos(prev_folio->swap) + prev_folio_size;
+ return swap_dev_pos(phys) == swap_dev_pos(sio->entry) + sio->len;
}
static const struct swap_ops swap_fs_ops = {
diff --git a/mm/swap.h b/mm/swap.h
index d241e81e967e..88ca9be71b7e 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -103,8 +103,8 @@ struct swap_io_ctx {
struct swap_ops {
unsigned int flags;
- bool (*can_merge)(struct folio *folio, struct folio *prev_folio,
- size_t prev_folio_size, int rw);
+ bool (*can_merge)(struct folio *folio, swp_entry_t phys,
+ struct swap_iocb *sio, int rw);
void (*submit_write)(struct swap_io_ctx *ctx);
void (*submit_read)(struct swap_io_ctx *ctx);
};
@@ -313,7 +313,8 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio);
void swap_read_submit(struct swap_io_ctx *ctx);
void swap_write_submit(struct swap_io_ctx *ctx);
int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio);
-void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio);
+void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio,
+ swp_entry_t phys);
/* linux/mm/swap_state.c */
extern struct address_space swap_space __read_mostly;
diff --git a/mm/swapfile.c b/mm/swapfile.c
index a26cfe5751c8..b8fdb426514c 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -355,14 +355,14 @@ offset_to_swap_extent(struct swap_info_struct *sis, unsigned long offset)
BUG();
}
-sector_t swap_folio_sector(struct folio *folio)
+sector_t swap_entry_sector(swp_entry_t entry)
{
- struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
+ struct swap_info_struct *sis = __swap_entry_to_info(entry);
struct swap_extent *se;
sector_t sector;
pgoff_t offset;
- offset = swp_offset(folio->swap);
+ offset = swp_offset(entry);
se = offset_to_swap_extent(sis, offset);
sector = se->start_block + (offset - se->start_page);
return sector << (PAGE_SHIFT - 9);
diff --git a/mm/zswap.c b/mm/zswap.c
index e19bde9df722..789079c3945b 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -1076,7 +1076,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
folio_set_reclaim(folio);
/* start writeback */
- __swap_writepage(&ctx, folio);
+ __swap_writepage(&ctx, folio, folio->swap);
swap_write_submit(&ctx);
out:
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v3 04/11] mm, swap: support physical swap as a vswap backend
2026-08-06 18:42 [PATCH v3 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
` (2 preceding siblings ...)
2026-08-06 18:42 ` [PATCH v3 03/11] mm, swap: prepare the swap IO path for vswap Nhat Pham
@ 2026-08-06 18:42 ` Nhat Pham
2026-08-06 18:42 ` [PATCH v3 05/11] mm, swap: enable THP swapin for vswap entries Nhat Pham
` (8 subsequent siblings)
12 siblings, 0 replies; 17+ messages in thread
From: Nhat Pham @ 2026-08-06 18:42 UTC (permalink / raw)
To: akpm
Cc: chrisl, kasong, hannes, mhocko, roman.gushchin, shakeel.butt,
yosry, david, muchun.song, shikemeng, baoquan.he, baohua,
youngjun.park, chengming.zhou, ljs, liam, vbabka, rppt, surenb,
qi.zheng, axelrasmussen, yuanchu, weixugc, riel, gourry,
haowenchao22, corbet, kernel-team, nphamcs, linux-mm,
linux-kernel, linux-doc, cgroups
Add physical swap as a backend for the virtual swap layer. When zswap
declines a page, the swapout path allocates a physical slot on demand
for swap out.
Each vswap entry's physical slot is tracked via a pointer-tagged
swap_table entry on the physical cluster (an rmap back to the vswap
entry).
Physical readahead scans a whole offset window and would trip over
these rmap slots, so __swap_cache_add_check() now skips
swp_tb_is_pointer() entries. Nothing is lost: a backing slot is
faulted through its owning vswap entry, never through the physical
offset.
Writeback of zswap-backed vswap entries to physical swap, and reclaim
of physical slots backing cache-only vswap entries, are added in the
following patches.
Suggested-by: Kairui Song <kasong@tencent.com>
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
include/linux/swap.h | 9 ++
mm/memory.c | 8 +-
mm/page_io.c | 43 ++++--
mm/swap_state.c | 6 +-
mm/swap_table.h | 55 +++++++
mm/swapfile.c | 352 +++++++++++++++++++++++++++++++++++++++----
mm/vmscan.c | 2 +-
mm/vswap.h | 198 +++++++++++++++++++++++-
mm/zswap.c | 2 +-
9 files changed, 627 insertions(+), 48 deletions(-)
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 8359134d08cb..2b2bd56afffa 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -391,6 +391,15 @@ extern int swp_swapcount(swp_entry_t entry);
extern struct swap_info_struct *get_swap_device(swp_entry_t entry);
sector_t swap_entry_sector(swp_entry_t entry);
+#ifdef CONFIG_VSWAP
+swp_entry_t folio_realloc_swap(struct folio *folio);
+#else
+static inline swp_entry_t folio_realloc_swap(struct folio *folio)
+{
+ return (swp_entry_t){};
+}
+#endif
+
/*
* If there is an existing swap slot reference (swap entry) and the caller
* guarantees that there is no race modification of it (e.g., PTL
diff --git a/mm/memory.c b/mm/memory.c
index de3573b7c6b1..ba84565605a1 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4656,13 +4656,13 @@ static inline bool should_try_to_free_swap(struct swap_info_struct *si,
* are fast, and meanwhile, swap cache pinning the slot deferring the
* release of metadata or fragmentation is a more critical issue.
*/
- if (data_race(si->flags & SWP_SYNCHRONOUS_IO))
+ if (swap_entry_backend_has_flag(si, folio->swap, SWP_SYNCHRONOUS_IO))
return true;
/*
* Non-swapfile backends cannot be reused for future swapouts.
* Free the swap slot unless backed by contiguous physical swap.
*/
- if (is_vswap_entry(folio->swap))
+ if (!folio_phys_swap_backed(folio))
return true;
if (mem_cgroup_swap_full(folio) || (vma->vm_flags & VM_LOCKED) ||
folio_test_mlocked(folio))
@@ -4968,7 +4968,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
swap_update_readahead(folio, vma, vmf->address);
if (!folio) {
/* Swapin bypasses readahead for SWP_SYNCHRONOUS_IO devices */
- if (data_race(si->flags & SWP_SYNCHRONOUS_IO))
+ if (swap_entry_backend_has_flag(si, entry, SWP_SYNCHRONOUS_IO))
folio = swapin_sync(entry, GFP_HIGHUSER_MOVABLE,
thp_swapin_suitable_orders(vmf) | BIT(0),
vmf, NULL, 0);
@@ -5133,7 +5133,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
*/
exclusive = true;
} else if (exclusive && folio_test_writeback(folio) &&
- data_race(si->flags & SWP_STABLE_WRITES)) {
+ swap_entry_backend_has_flag(si, entry, SWP_STABLE_WRITES)) {
/*
* This is tricky: not all swap backends support
* concurrent page modifications while under writeback.
diff --git a/mm/page_io.c b/mm/page_io.c
index 5c780bda92bb..605a66a32604 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -208,6 +208,7 @@ static void swap_zeromap_folio_clear(struct folio *folio)
*/
int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
{
+ swp_entry_t phys;
int ret = 0;
if (folio_free_swap(folio))
@@ -240,8 +241,14 @@ int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
*/
swap_zeromap_folio_clear(folio);
+ /*
+ * For vswap: release stale non-swapfile backings (e.g. ZSWAP from a
+ * previous swapout cycle) so zswap_store or folio_realloc_swap
+ * starts on clean slots. Contiguous PHYS backing is preserved for
+ * reuse by folio_realloc_swap.
+ */
if (is_vswap_entry(folio->swap))
- folio_release_vswap_backing(folio);
+ folio_release_non_phys_swap_backing(folio);
if (zswap_store(folio)) {
count_mthp_stat(folio_order(folio), MTHP_STAT_ZSWPOUT);
@@ -257,12 +264,19 @@ int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
rcu_read_unlock();
/*
- * A vswap folio that reaches here could not be stored to a backend
- * (zswap) and has no physical slot to write to, so keep it dirty.
+ * A vswap folio with no backend needs a physical slot to write to.
+ * zswap_store rolled back any partial vtable state on failure, so
+ * PHYS backing from a prior cycle is still there to reuse. If none
+ * is free, keep it dirty.
*/
if (is_vswap_entry(folio->swap)) {
- folio_mark_dirty(folio);
- return AOP_WRITEPAGE_ACTIVATE;
+ phys = folio_realloc_swap(folio);
+ if (!phys.val) {
+ folio_mark_dirty(folio);
+ return AOP_WRITEPAGE_ACTIVATE;
+ }
+ __swap_writepage(ctx, folio, phys);
+ return 0;
}
__swap_writepage(ctx, folio, folio->swap);
@@ -474,6 +488,7 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)
bool workingset = folio_test_workingset(folio);
unsigned long pflags;
bool in_thrashing;
+ swp_entry_t phys;
VM_BUG_ON_FOLIO(!folio_test_swapcache(folio) && !synchronous, folio);
VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
@@ -498,14 +513,24 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)
if (zswap_load(folio) != -ENOENT)
goto finish;
- if (unlikely(swap_is_vswap(sis))) {
- folio_unlock(folio);
- goto finish;
+ /*
+ * Resolve the physical slot to read from. A vswap entry keeps
+ * folio->swap virtual, so map it to its physical backing; a folio with
+ * no backing has nothing to read.
+ */
+ if (swap_is_vswap(sis)) {
+ phys = vswap_to_phys(folio->swap);
+ if (!phys.val) {
+ folio_unlock(folio);
+ goto finish;
+ }
+ } else {
+ phys = folio->swap;
}
/* We have to read from slower devices. Increase zswap protection. */
zswap_folio_swapin(folio);
- swap_add_folio(ctx, folio, folio->swap, READ);
+ swap_add_folio(ctx, folio, phys, READ);
finish:
if (workingset) {
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 9f6377b32911..c61bb3eef62a 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -185,6 +185,9 @@ static int __swap_cache_add_check(struct swap_cluster_info *ci,
return -ENOENT;
ci_off = swp_cluster_offset(targ_entry);
old_tb = __swap_table_get(ci, ci_off);
+ /* Physical readahead can hit a vswap-backing rmap slot; skip it. */
+ if (swp_tb_is_pointer(old_tb))
+ return -ENOENT;
if (swp_tb_is_folio(old_tb))
return -EEXIST;
if (!__swp_tb_get_count(old_tb))
@@ -209,7 +212,8 @@ static int __swap_cache_add_check(struct swap_cluster_info *ci,
ci_end = ci_off + nr;
do {
old_tb = __swap_table_get(ci, ci_off);
- if (unlikely(swp_tb_is_folio(old_tb) ||
+ if (unlikely(swp_tb_is_pointer(old_tb) ||
+ swp_tb_is_folio(old_tb) ||
!__swp_tb_get_count(old_tb) ||
is_zero != __swap_table_test_zero(ci, ci_off) ||
(memcg_id && *memcg_id != __swap_cgroup_get(ci, ci_off))))
diff --git a/mm/swap_table.h b/mm/swap_table.h
index fd7f0fb9836a..5b0eca07a821 100644
--- a/mm/swap_table.h
+++ b/mm/swap_table.h
@@ -4,8 +4,11 @@
#include <linux/rcupdate.h>
#include <linux/atomic.h>
+#include <linux/swapops.h>
#include "swap.h"
+struct zswap_entry;
+
/* A typical flat array in each cluster as swap table */
struct swap_table {
atomic_long_t entries[SWAPFILE_CLUSTER];
@@ -368,4 +371,56 @@ static inline unsigned short __swap_cgroup_clear(struct swap_cluster_info *ci,
}
#endif
+/*
+ * Pointer-tagged swap table entry: rmap for vswap-backing physical slots.
+ *
+ * On physical clusters, a Pointer-tagged entry stores the offset of the
+ * vswap entry that owns this physical slot (the reverse map). Only the
+ * offset is stored; the swap type is implicit (always vswap_si->type,
+ * since there is exactly one vswap device).
+ *
+ * Pointer: |---- vswap offset ----|100|
+ */
+#ifdef CONFIG_VSWAP
+extern struct swap_info_struct *vswap_si;
+
+#define SWP_TB_PTR_MARK_BITS 3
+#define SWP_TB_PTR_MARK 0b100UL
+#define SWP_TB_PTR_MARK_MASK ((1UL << SWP_TB_PTR_MARK_BITS) - 1)
+#define SWP_RMAP_ENTRY_MASK (~SWP_TB_PTR_MARK_MASK)
+
+static inline bool swp_tb_is_pointer(unsigned long swp_tb)
+{
+ return (swp_tb & SWP_TB_PTR_MARK_MASK) == SWP_TB_PTR_MARK;
+}
+
+static inline unsigned long swp_entry_to_swp_tb_ptr(swp_entry_t entry)
+{
+ return (swp_offset(entry) << SWP_TB_PTR_MARK_BITS) | SWP_TB_PTR_MARK;
+}
+
+static inline swp_entry_t swp_tb_ptr_to_swp_entry(unsigned long swp_tb)
+{
+ unsigned long offset;
+
+ VM_WARN_ON(!swp_tb_is_pointer(swp_tb));
+ offset = (swp_tb & SWP_RMAP_ENTRY_MASK) >> SWP_TB_PTR_MARK_BITS;
+ return swp_entry(vswap_si->type, offset);
+}
+#else
+static inline bool swp_tb_is_pointer(unsigned long swp_tb)
+{
+ return false;
+}
+static inline unsigned long swp_entry_to_swp_tb_ptr(swp_entry_t entry)
+{
+ return 0;
+}
+static inline swp_entry_t swp_tb_ptr_to_swp_entry(unsigned long swp_tb)
+{
+ return (swp_entry_t){};
+}
+
+#endif /* CONFIG_VSWAP */
+
#endif
diff --git a/mm/swapfile.c b/mm/swapfile.c
index b8fdb426514c..0874f57d3124 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -261,7 +261,7 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si,
need_reclaim = ((flags & TTRS_ANYWAY) ||
((flags & TTRS_UNMAPPED) && !folio_mapped(folio)) ||
((flags & TTRS_FULL) && mem_cgroup_swap_full(folio) &&
- !is_vswap_entry(folio->swap)));
+ folio_phys_swap_backed(folio)));
if (!need_reclaim || !folio_swapcache_freeable(folio))
goto out_unlock;
@@ -1013,6 +1013,8 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct *si,
{
unsigned int order;
unsigned long nr_pages;
+ swp_entry_t vswap_entry, v;
+ unsigned int i;
lockdep_assert_held(&ci->lock);
@@ -1032,8 +1034,26 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct *si,
order = folio_order(folio);
nr_pages = 1 << order;
swap_cluster_assert_empty(ci, ci_off, nr_pages, false);
- __swap_cache_add_folio(ci, folio, swp_entry(si->type,
- ci_off + cluster_offset(si, ci)));
+ if (folio_test_swapcache(folio)) {
+ /*
+ * Folio already in the swap cache: we are allocating
+ * physical backing for its vswap entry. Point each
+ * physical slot back at its own vswap entry
+ * (Pointer-tagged rmap).
+ */
+ VM_WARN_ON(!is_vswap_entry(folio->swap));
+ vswap_entry = folio->swap;
+ for (i = 0; i < nr_pages; i++) {
+ v = vswap_entry;
+ v.val += i;
+ __swap_table_set(ci, ci_off + i,
+ swp_entry_to_swp_tb_ptr(v));
+ }
+ } else {
+ __swap_cache_add_folio(ci, folio,
+ swp_entry(si->type,
+ ci_off + cluster_offset(si, ci)));
+ }
} else if (IS_ENABLED(CONFIG_HIBERNATION)) {
order = 0;
nr_pages = 1;
@@ -1538,12 +1558,14 @@ static bool get_swap_device_info(struct swap_info_struct *si)
* Fast path try to get swap entries with specified order from current
* CPU's swap entry pool (a cluster).
*/
-static bool swap_alloc_fast(struct folio *folio)
+static swp_entry_t swap_alloc_fast(struct folio *folio)
{
unsigned int order = folio_order(folio);
struct swap_cluster_info *ci;
struct swap_info_struct *si;
- unsigned int offset;
+ unsigned long offset, found = 0;
+
+ lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)->lock);
/*
* Once allocated, swap_info_struct will never be completely freed,
@@ -1552,25 +1574,28 @@ static bool swap_alloc_fast(struct folio *folio)
si = this_cpu_read(percpu_swap_cluster.si[order]);
offset = this_cpu_read(percpu_swap_cluster.offset[order]);
if (!si || !offset || !get_swap_device_info(si))
- return false;
+ return (swp_entry_t){};
ci = swap_cluster_lock(si, offset);
if (ci && cluster_is_usable(ci, order)) {
if (cluster_is_empty(ci))
offset = cluster_offset(si, ci);
- alloc_swap_scan_cluster(si, ci, folio, offset);
+ found = alloc_swap_scan_cluster(si, ci, folio, offset);
} else if (ci) {
swap_cluster_unlock(ci);
}
put_swap_device(si);
- return folio_test_swapcache(folio);
+ if (found)
+ return swp_entry(si->type, found);
+ return (swp_entry_t){};
}
/* Rotate the device and switch to a new cluster */
-static void swap_alloc_slow(struct folio *folio)
+static swp_entry_t swap_alloc_slow(struct folio *folio)
{
struct swap_info_struct *si, *next;
+ unsigned long found;
spin_lock(&swap_avail_lock);
start_over:
@@ -1579,12 +1604,12 @@ static void swap_alloc_slow(struct folio *folio)
plist_requeue(&si->avail_list, &swap_avail_head);
spin_unlock(&swap_avail_lock);
if (get_swap_device_info(si)) {
- cluster_alloc_swap_entry(si, folio);
+ found = cluster_alloc_swap_entry(si, folio);
put_swap_device(si);
- if (folio_test_swapcache(folio))
- return;
+ if (found)
+ return swp_entry(si->type, found);
if (folio_test_large(folio))
- return;
+ return (swp_entry_t){};
}
spin_lock(&swap_avail_lock);
@@ -1602,6 +1627,7 @@ static void swap_alloc_slow(struct folio *folio)
goto start_over;
}
spin_unlock(&swap_avail_lock);
+ return (swp_entry_t){};
}
/*
@@ -1982,13 +2008,12 @@ int folio_alloc_swap(struct folio *folio)
}
}
- /* Without zswap a vswap entry has nowhere to go on writeout. */
- if (zswap_is_enabled() && vswap_alloc(folio))
+ if (vswap_alloc(folio))
goto done;
again:
local_lock(&percpu_swap_cluster.lock);
- if (!swap_alloc_fast(folio))
+ if (!swap_alloc_fast(folio).val)
swap_alloc_slow(folio);
local_unlock(&percpu_swap_cluster.lock);
@@ -2010,6 +2035,11 @@ int folio_alloc_swap(struct folio *folio)
#ifdef CONFIG_VSWAP
+static void __swap_cluster_free_phys_backing(struct swap_info_struct *psi,
+ struct swap_cluster_info *pci,
+ unsigned int ci_start,
+ unsigned int nr_pages);
+
/**
* __vswap_release_backing - release the backing of a range of vtable slots
* @ci: the locked vswap cluster
@@ -2026,8 +2056,12 @@ void __vswap_release_backing(struct swap_cluster_info *ci,
unsigned int ci_start, unsigned int nr)
{
struct swap_cluster_info_dynamic *ci_dyn;
+ struct swap_info_struct *psi;
+ unsigned long phys_start = 0, phys_end = 0;
+ unsigned int phys_type = 0;
unsigned int ci_off;
unsigned long vt;
+ swp_entry_t phys;
lockdep_assert_held(&ci->lock);
ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
@@ -2035,7 +2069,37 @@ void __vswap_release_backing(struct swap_cluster_info *ci,
for (ci_off = ci_start; ci_off < ci_start + nr; ci_off++) {
vt = __vtable_get(ci_dyn, ci_off);
+ /*
+ * Flush batched physical slots when the next entry
+ * breaks contiguity, changes type/device, or would
+ * cross a SWAPFILE_CLUSTER boundary (the free helper
+ * operates on a single cluster).
+ */
+ if (phys_start != phys_end &&
+ (vtable_type(vt) != VSWAP_SWAPFILE ||
+ swp_type(vtable_to_phys(vt)) != phys_type ||
+ swp_offset(vtable_to_phys(vt)) != phys_end ||
+ phys_end % SWAPFILE_CLUSTER == 0)) {
+ psi = __swap_type_to_info(phys_type);
+ __swap_cluster_free_phys_backing(psi,
+ __swap_entry_to_cluster(
+ swp_entry(phys_type, phys_start)),
+ phys_start % SWAPFILE_CLUSTER,
+ phys_end - phys_start);
+ phys_start = phys_end = 0;
+ }
+
switch (vtable_type(vt)) {
+ case VSWAP_SWAPFILE:
+ if (phys_start == phys_end) {
+ phys = vtable_to_phys(vt);
+ phys_start = swp_offset(phys);
+ phys_end = phys_start + 1;
+ phys_type = swp_type(phys);
+ } else {
+ phys_end++;
+ }
+ break;
case VSWAP_ZSWAP:
zswap_entry_free(vtable_to_zswap(vt));
break;
@@ -2051,6 +2115,15 @@ void __vswap_release_backing(struct swap_cluster_info *ci,
if (__swap_table_test_zero(ci, ci_off))
__swap_table_clear_zero(ci, ci_off);
}
+
+ if (phys_start != phys_end) {
+ psi = __swap_type_to_info(phys_type);
+ __swap_cluster_free_phys_backing(psi,
+ __swap_entry_to_cluster(
+ swp_entry(phys_type, phys_start)),
+ phys_start % SWAPFILE_CLUSTER,
+ phys_end - phys_start);
+ }
}
/**
@@ -2080,6 +2153,106 @@ void folio_release_vswap_backing(struct folio *folio)
spin_unlock(&ci->lock);
}
+/**
+ * folio_release_non_phys_swap_backing() - Drop a folio's non-physical vswap backing.
+ * @folio: the folio, occupying a virtual swap entry.
+ *
+ * Release any ZSWAP or zero-filled backing recorded for @folio's virtual
+ * swap entry, leaving the slots empty so the writeout path can install fresh
+ * physical backing. If the first slot is already VSWAP_SWAPFILE or
+ * VSWAP_NONE, nothing is released: physical backing is kept for reuse.
+ *
+ * Context: Caller must hold the folio lock; @folio must be in the swap cache
+ * and occupy a virtual swap entry.
+ */
+void folio_release_non_phys_swap_backing(struct folio *folio)
+{
+ struct swap_cluster_info *ci;
+ struct swap_cluster_info_dynamic *ci_dyn;
+ int nr = folio_nr_pages(folio);
+ unsigned int voff;
+ unsigned long vt;
+ enum vswap_backing_type type;
+
+ ci = __swap_entry_to_cluster(folio->swap);
+ if (!ci)
+ return;
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ voff = swp_cluster_offset(folio->swap);
+
+ spin_lock(&ci->lock);
+ vt = __vtable_get(ci_dyn, voff);
+ type = vtable_type(vt);
+
+ if (type == VSWAP_SWAPFILE || type == VSWAP_NONE) {
+ spin_unlock(&ci->lock);
+ return;
+ }
+
+ __vswap_release_backing(ci, voff, nr);
+ spin_unlock(&ci->lock);
+}
+
+/**
+ * folio_realloc_swap() - Back a virtual swap folio with a physical swap slot.
+ * @folio: the folio, occupying a virtual swap entry.
+ *
+ * Ensure @folio's virtual swap entry has physical (swapfile) backing,
+ * allocating a physical slot on demand if it has none. Called from the
+ * writeout path and from zswap writeback to move a vswap entry onto a real
+ * swapfile slot. If @folio is already physically backed, the existing
+ * physical entry is returned unchanged.
+ *
+ * Context: Caller must hold the folio lock; @folio must be in the swap cache
+ * and occupy a virtual swap entry.
+ * Return: The physical swap entry now backing @folio, or an empty entry
+ * (.val == 0) on failure.
+ */
+swp_entry_t folio_realloc_swap(struct folio *folio)
+{
+ swp_entry_t vswap_entry = folio->swap;
+ struct swap_cluster_info *ci;
+ struct swap_cluster_info_dynamic *ci_dyn;
+ unsigned int voff;
+ swp_entry_t phys_entry = {};
+ swp_entry_t pe;
+ int i, nr = folio_nr_pages(folio);
+
+ VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
+ VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio);
+ VM_WARN_ON(!is_vswap_entry(vswap_entry));
+
+ phys_entry = vswap_to_phys(vswap_entry);
+ if (phys_entry.val)
+ return phys_entry;
+
+ local_lock(&percpu_swap_cluster.lock);
+ phys_entry = swap_alloc_fast(folio);
+ if (!phys_entry.val)
+ phys_entry = swap_alloc_slow(folio);
+ local_unlock(&percpu_swap_cluster.lock);
+
+ if (!phys_entry.val)
+ return (swp_entry_t){};
+
+ voff = swp_cluster_offset(vswap_entry);
+
+ ci = __swap_entry_to_cluster(vswap_entry);
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ spin_lock(&ci->lock);
+ /*
+ * Install PHYS backing without freeing any prior contents of the
+ * vtable. Releasing the old backing is the caller's job: it may
+ * still need the slot, or may have released it already.
+ */
+ for (i = 0; i < nr; i++) {
+ pe.val = phys_entry.val + i;
+ __vtable_set(ci_dyn, voff + i, vtable_mk_phys(pe));
+ }
+ spin_unlock(&ci->lock);
+
+ return phys_entry;
+}
#endif /* CONFIG_VSWAP */
/**
@@ -2207,6 +2380,63 @@ struct swap_info_struct *get_swap_device(swp_entry_t entry)
return NULL;
}
+#ifdef CONFIG_VSWAP
+/*
+ * Clear swap table entries to NULL and reset zero flags.
+ * Does not touch memcg or count - caller handles those.
+ */
+static void __swap_cluster_clear_table(struct swap_cluster_info *ci,
+ unsigned int ci_start,
+ unsigned int nr_pages)
+{
+ unsigned int ci_off;
+
+ lockdep_assert_held(&ci->lock);
+ for (ci_off = ci_start; ci_off < ci_start + nr_pages; ci_off++) {
+ __swap_table_set(ci, ci_off, null_to_swp_tb());
+ if (!SWAP_TABLE_HAS_ZEROFLAG)
+ __swap_table_clear_zero(ci, ci_off);
+ }
+}
+#endif
+
+/*
+ * Common tail for freeing swap slots: device-level accounting
+ * and cluster list management.
+ */
+static void __swap_cluster_finish_free(struct swap_info_struct *si,
+ struct swap_cluster_info *ci,
+ unsigned int ci_start,
+ unsigned int nr_pages)
+{
+ lockdep_assert_held(&ci->lock);
+ swap_range_free(si, cluster_offset(si, ci) + ci_start, nr_pages);
+ swap_cluster_assert_empty(ci, ci_start, nr_pages, false);
+
+ if (!ci->count)
+ free_cluster(si, ci);
+ else
+ partial_free_cluster(si, ci);
+}
+
+#ifdef CONFIG_VSWAP
+/*
+ * Free physical swap slots that were backing vswap entries (Pointer-tagged).
+ */
+static void __swap_cluster_free_phys_backing(struct swap_info_struct *psi,
+ struct swap_cluster_info *pci,
+ unsigned int ci_start,
+ unsigned int nr_pages)
+{
+ spin_lock_nested(&pci->lock, SINGLE_DEPTH_NESTING);
+ VM_WARN_ON(pci->count < nr_pages);
+ pci->count -= nr_pages;
+ __swap_cluster_clear_table(pci, ci_start, nr_pages);
+ __swap_cluster_finish_free(psi, pci, ci_start, nr_pages);
+ swap_cluster_unlock(pci);
+}
+#endif
+
/*
* Free a set of swap slots after their swap count dropped to zero, or will be
* zero after putting the last ref (saves one __swap_cluster_put_entry call).
@@ -2218,7 +2448,6 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,
unsigned long old_tb;
unsigned short batch_id = 0, id_cur;
unsigned int ci_off = ci_start, ci_end = ci_start + nr_pages;
- unsigned long ci_head = cluster_offset(si, ci);
unsigned int batch_off = ci_off;
VM_WARN_ON(ci->count < nr_pages);
@@ -2256,13 +2485,7 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,
if (batch_id)
mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);
- swap_range_free(si, ci_head + ci_start, nr_pages);
- swap_cluster_assert_empty(ci, ci_start, nr_pages, false);
-
- if (!ci->count)
- free_cluster(si, ci);
- else
- partial_free_cluster(si, ci);
+ __swap_cluster_finish_free(si, ci, ci_start, nr_pages);
}
int __swap_count(swp_entry_t entry)
@@ -3041,19 +3264,88 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si,
static int try_to_unuse(unsigned int type)
{
+ struct mempolicy *mpol = get_task_policy(current);
struct mm_struct *prev_mm;
struct mm_struct *mm;
struct list_head *p;
int retval = 0;
struct swap_info_struct *si = swap_info[type];
struct folio *folio;
- swp_entry_t entry;
- unsigned int i;
+ struct swap_io_ctx ctx;
+ swp_entry_t entry, vswap_entry;
+ unsigned long swp_tb;
+ unsigned int i, j;
if (!swap_usage_in_pages(si))
goto success;
retry:
+ /*
+ * Free vswap-backing slots (Pointer-tagged) first. Walk physical
+ * clusters, read the vswap entry from the rmap, ensure the data
+ * is in the swap cache, and transition PHYS to FOLIO. No page table
+ * walk needed - just free the physical backing.
+ */
+ i = 0;
+ while (IS_ENABLED(CONFIG_VSWAP) &&
+ swap_usage_in_pages(si) &&
+ !signal_pending(current) &&
+ (i = find_next_to_unuse(si, i)) != 0) {
+ swp_entry_t phys;
+
+ swp_tb = swap_table_get(__swap_offset_to_cluster(si, i),
+ i % SWAPFILE_CLUSTER);
+ if (!swp_tb_is_pointer(swp_tb))
+ continue;
+
+ vswap_entry = swp_tb_ptr_to_swp_entry(swp_tb);
+
+ folio = swap_cache_get_folio(vswap_entry);
+ if (!folio) {
+ folio = swap_cache_alloc_folio(vswap_entry,
+ GFP_KERNEL, BIT(0), NULL,
+ mpol, NO_INTERLEAVE_INDEX);
+ if (IS_ERR(folio))
+ continue;
+ ctx = (struct swap_io_ctx){};
+ swap_read_folio(&ctx, folio);
+ swap_read_submit(&ctx);
+ folio_lock(folio);
+ } else {
+ folio_lock(folio);
+ }
+
+ if (!folio_matches_swap_entry(folio, vswap_entry)) {
+ folio_unlock(folio);
+ folio_put(folio);
+ continue;
+ }
+
+ /*
+ * Re-validate under folio lock: rmap holds folio->swap + j
+ * for some j in [0, nr_pages). Check folio->swap still maps
+ * to the contiguous physical run that includes our slot i.
+ */
+ j = vswap_entry.val - folio->swap.val;
+ phys = vswap_to_phys(folio->swap);
+ if (!phys.val || swp_type(phys) != type ||
+ swp_offset(phys) + j != i ||
+ j >= folio_nr_pages(folio)) {
+ folio_unlock(folio);
+ folio_put(folio);
+ continue;
+ }
+
+ folio_wait_writeback(folio);
+ folio_release_vswap_backing(folio);
+ folio_mark_dirty(folio);
+ folio_unlock(folio);
+ folio_put(folio);
+ }
+
+ if (!swap_usage_in_pages(si))
+ goto success;
+
retval = shmem_unuse(type);
if (retval)
return retval;
@@ -3097,6 +3389,14 @@ static int try_to_unuse(unsigned int type)
entry = swp_entry(type, i);
+ if (IS_ENABLED(CONFIG_VSWAP)) {
+ swp_tb = swap_table_get(
+ __swap_offset_to_cluster(si, i),
+ i % SWAPFILE_CLUSTER);
+ if (swp_tb_is_pointer(swp_tb))
+ continue;
+ }
+
folio = swap_cache_get_folio(entry);
if (!folio)
continue;
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 78ec51f53757..f3f9e3993215 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1530,7 +1530,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
* space if we are running out.
*/
if (folio_test_swapcache(folio) &&
- ((mem_cgroup_swap_full(folio) && !is_vswap_entry(folio->swap)) ||
+ ((mem_cgroup_swap_full(folio) && folio_phys_swap_backed(folio)) ||
folio_test_mlocked(folio)))
folio_free_swap(folio);
VM_BUG_ON_FOLIO(folio_test_active(folio), folio);
diff --git a/mm/vswap.h b/mm/vswap.h
index 6d25e0911fa9..239b47b577d5 100644
--- a/mm/vswap.h
+++ b/mm/vswap.h
@@ -19,6 +19,7 @@ struct zswap_entry;
enum vswap_backing_type {
VSWAP_NONE = 0,
VSWAP_ZSWAP = 1,
+ VSWAP_SWAPFILE = 2,
VSWAP_ZERO,
VSWAP_FOLIO,
};
@@ -27,8 +28,6 @@ enum vswap_backing_type {
#include "swap_table.h"
-extern struct swap_info_struct *vswap_si;
-
static inline bool is_vswap_entry(swp_entry_t entry)
{
return swap_is_vswap(__swap_entry_to_info(entry));
@@ -43,11 +42,15 @@ bool vswap_is_enabled(void);
* pointer for a virtual swap slot. Tag in low 3 bits, payload in
* upper 61 bits.
*
- * NONE: |----- 0000 ------|000| - no separate backend pointer
- * ZSWAP: |--- zswap_entry* |001| - compressed in zswap (tag in low bits)
+ * NONE: |----- 0000 ------|000| - no separate backend pointer
+ * ZSWAP: |--- zswap_entry* |001| - compressed in zswap (tag in low bits)
+ * SWAPFILE: |- type:5,off:56 -|010| - on a physical swapfile
*
- * Pointer payloads (ZSWAP) are stored directly with the tag OR'd into the
- * low bits (kernel pointers are >= 8-byte aligned, same approach as xarray).
+ * SWAPFILE packs swp_type in the top MAX_SWAPFILES_SHIFT bits and swp_offset in
+ * the middle VTABLE_PHYS_OFF_BITS bits, both above the tag, so the type is
+ * not shifted off the word. Pointer payloads (ZSWAP) are stored directly with
+ * the tag OR'd into the low bits (kernel pointers are >= 8-byte aligned, same
+ * approach as xarray).
*
* vtable[i] = NONE does not by itself mean "free". The swap_table entry
* and the per-slot zero flag carry the rest of the state. The full
@@ -86,6 +89,23 @@ static inline enum vswap_backing_type vtable_type(unsigned long vt)
return vt & VTABLE_TAG_MASK;
}
+/* swp_offset field width in a physical backend slot; layout described above. */
+#define VTABLE_PHYS_OFF_BITS (BITS_PER_LONG - VTABLE_TAG_BITS - MAX_SWAPFILES_SHIFT)
+
+static inline unsigned long vtable_mk_phys(swp_entry_t entry)
+{
+ VM_WARN_ON_ONCE(swp_offset(entry) >> VTABLE_PHYS_OFF_BITS);
+ return ((unsigned long)swp_type(entry) << (VTABLE_TAG_BITS + VTABLE_PHYS_OFF_BITS)) |
+ (swp_offset(entry) << VTABLE_TAG_BITS) | VSWAP_SWAPFILE;
+}
+
+static inline swp_entry_t vtable_to_phys(unsigned long vt)
+{
+ VM_WARN_ON(vtable_type(vt) != VSWAP_SWAPFILE);
+ return swp_entry(vt >> (VTABLE_TAG_BITS + VTABLE_PHYS_OFF_BITS),
+ (vt >> VTABLE_TAG_BITS) & ((1UL << VTABLE_PHYS_OFF_BITS) - 1));
+}
+
static inline struct zswap_entry *vtable_to_zswap(unsigned long vt)
{
VM_WARN_ON(vtable_type(vt) != VSWAP_ZSWAP);
@@ -130,6 +150,33 @@ vswap_lock_cluster(swp_entry_t entry, unsigned int *voff)
return ci_dyn;
}
+/**
+ * vswap_to_phys - resolve a vswap entry's physical swap backing
+ * @entry: the virtual swap entry
+ *
+ * Context: takes and drops the vswap cluster lock internally.
+ * Return: the backing physical swp_entry_t, or the null entry (.val == 0)
+ * when @entry has no physical backing (NONE/ZSWAP/ZERO).
+ */
+static inline swp_entry_t vswap_to_phys(swp_entry_t entry)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ unsigned int voff;
+ unsigned long vt;
+
+ ci_dyn = vswap_lock_cluster(entry, &voff);
+ if (!ci_dyn)
+ return (swp_entry_t){};
+
+ vt = __vtable_get(ci_dyn, voff);
+ spin_unlock(&ci_dyn->ci.lock);
+
+ if (vtable_type(vt) != VSWAP_SWAPFILE)
+ return (swp_entry_t){};
+
+ return vtable_to_phys(vt);
+}
+
void __vswap_release_backing(struct swap_cluster_info *ci,
unsigned int ci_start, unsigned int nr);
@@ -182,6 +229,103 @@ static inline struct zswap_entry *vswap_zswap_load(swp_entry_t entry)
}
void folio_release_vswap_backing(struct folio *folio);
+void folio_release_non_phys_swap_backing(struct folio *folio);
+
+/*
+ * Walk nr vtable slots starting at voff in ci_dyn. Returns the prefix
+ * length of slots sharing one effective backing type. For SWAPFILE,
+ * the prefix is also restricted to contiguous offsets in the same
+ * swapfile.
+ *
+ * Effective type per slot:
+ * vtable=NONE + zero flag set -> VSWAP_ZERO
+ * vtable=NONE + swap_table PFN tag -> VSWAP_FOLIO
+ * vtable=NONE + neither -> VSWAP_NONE
+ * vtable=SWAPFILE -> VSWAP_SWAPFILE
+ * vtable=ZSWAP -> VSWAP_ZSWAP
+ *
+ * *typep returns the effective type of slot 0. Caller holds
+ * ci_dyn->ci.lock.
+ */
+static inline int __vswap_check_backing(struct swap_cluster_info_dynamic *ci_dyn,
+ unsigned int voff, int nr,
+ enum vswap_backing_type *typep)
+{
+ enum vswap_backing_type first_type = VSWAP_NONE;
+ enum vswap_backing_type slot_type;
+ swp_entry_t first_phys = {};
+ unsigned long vt, swap_tb;
+ int i;
+
+ lockdep_assert_held(&ci_dyn->ci.lock);
+
+ for (i = 0; i < nr; i++) {
+ vt = __vtable_get(ci_dyn, voff + i);
+ if (vtable_type(vt) == VSWAP_NONE) {
+ swap_tb = __swap_table_get(&ci_dyn->ci, voff + i);
+ if (__swap_table_test_zero(&ci_dyn->ci, voff + i))
+ slot_type = VSWAP_ZERO;
+ else if (swp_tb_is_folio(swap_tb))
+ slot_type = VSWAP_FOLIO;
+ else
+ slot_type = VSWAP_NONE;
+ } else {
+ slot_type = vtable_type(vt);
+ }
+
+ if (!i) {
+ first_type = slot_type;
+ if (first_type == VSWAP_SWAPFILE)
+ first_phys = vtable_to_phys(vt);
+ } else if (slot_type != first_type) {
+ break;
+ } else if (first_type == VSWAP_SWAPFILE &&
+ vtable_to_phys(vt).val != first_phys.val + i) {
+ break;
+ }
+ }
+
+ if (typep)
+ *typep = first_type;
+ return i;
+}
+
+static inline int vswap_check_backing(swp_entry_t entry, int nr,
+ enum vswap_backing_type *typep)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ unsigned int voff;
+ int ret;
+
+ ci_dyn = vswap_lock_cluster(entry, &voff);
+ if (!ci_dyn) {
+ if (typep)
+ *typep = VSWAP_NONE;
+ return 0;
+ }
+ ret = __vswap_check_backing(ci_dyn, voff, nr, typep);
+ spin_unlock(&ci_dyn->ci.lock);
+ return ret;
+}
+
+/**
+ * folio_phys_swap_backed - test whether a folio is backed by a contiguous
+ * range of physical swap slots.
+ * @folio: a swap-cache resident folio
+ *
+ * Return: %true if @folio->swap is not a vswap entry, or if these vswap
+ * entries are backed by a contiguous range of physical slots.
+ */
+static inline bool folio_phys_swap_backed(struct folio *folio)
+{
+ swp_entry_t entry = folio->swap;
+ int nr = folio_nr_pages(folio);
+ enum vswap_backing_type type;
+
+ return !is_vswap_entry(entry) ||
+ (vswap_check_backing(entry, nr, &type) == nr &&
+ type == VSWAP_SWAPFILE);
+}
static inline int vswap_cluster_alloc_vtable(struct swap_cluster_info_dynamic *ci_dyn)
{
@@ -209,6 +353,16 @@ static inline bool is_vswap_entry(swp_entry_t entry)
static inline bool vswap_is_enabled(void) { return false; }
+static inline swp_entry_t vswap_to_phys(swp_entry_t entry)
+{
+ return (swp_entry_t){};
+}
+
+static inline bool folio_phys_swap_backed(struct folio *folio)
+{
+ return true;
+}
+
static inline void __vswap_release_backing(struct swap_cluster_info *ci,
unsigned int ci_start,
unsigned int nr) {}
@@ -222,6 +376,7 @@ static inline struct zswap_entry *vswap_zswap_load(swp_entry_t entry)
}
static inline void folio_release_vswap_backing(struct folio *folio) {}
+static inline void folio_release_non_phys_swap_backing(struct folio *folio) {}
static inline int vswap_cluster_alloc_vtable(struct swap_cluster_info_dynamic *ci_dyn)
{
@@ -232,4 +387,35 @@ static inline void vswap_cluster_free_vtable(struct swap_cluster_info *ci) {}
#endif /* CONFIG_VSWAP */
+/*
+ * Test a per-backend swap flag (SWP_SYNCHRONOUS_IO, SWP_STABLE_WRITES, ...)
+ * for @entry. For a vswap entry the property belongs to the current
+ * physical backing rather than vswap_si itself; resolve to the backing
+ * and test there. Returns false for zswap/zero/unbacked vswap entries
+ * as they don't have a backing bdev.
+ */
+static inline bool swap_entry_backend_has_flag(struct swap_info_struct *si,
+ swp_entry_t entry,
+ unsigned long flag)
+{
+ struct swap_info_struct *phys_si;
+ swp_entry_t phys;
+ bool has_flag;
+
+ if (!swap_is_vswap(si))
+ return data_race(si->flags & flag);
+
+ phys = vswap_to_phys(entry);
+ if (!phys.val)
+ return false;
+
+ phys_si = get_swap_device(phys);
+ if (!phys_si)
+ return false;
+
+ has_flag = data_race(phys_si->flags & flag);
+ put_swap_device(phys_si);
+ return has_flag;
+}
+
#endif /* _MM_VSWAP_H */
diff --git a/mm/zswap.c b/mm/zswap.c
index 789079c3945b..d0c6ce2aa092 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -1579,7 +1579,7 @@ bool zswap_store(struct folio *folio)
*/
if (is_vswap_entry(swp)) {
if (index > 0)
- folio_release_vswap_backing(folio);
+ folio_release_non_phys_swap_backing(folio);
} else {
unsigned type = swp_type(swp);
pgoff_t offset = swp_offset(swp);
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v3 05/11] mm, swap: enable THP swapin for vswap entries
2026-08-06 18:42 [PATCH v3 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
` (3 preceding siblings ...)
2026-08-06 18:42 ` [PATCH v3 04/11] mm, swap: support physical swap as a vswap backend Nhat Pham
@ 2026-08-06 18:42 ` Nhat Pham
2026-08-06 18:42 ` [PATCH v3 06/11] mm, swap: write back vswap zswap entries to physical swap Nhat Pham
` (7 subsequent siblings)
12 siblings, 0 replies; 17+ messages in thread
From: Nhat Pham @ 2026-08-06 18:42 UTC (permalink / raw)
To: akpm
Cc: chrisl, kasong, hannes, mhocko, roman.gushchin, shakeel.butt,
yosry, david, muchun.song, shikemeng, baoquan.he, baohua,
youngjun.park, chengming.zhou, ljs, liam, vbabka, rppt, surenb,
qi.zheng, axelrasmussen, yuanchu, weixugc, riel, gourry,
haowenchao22, corbet, kernel-team, nphamcs, linux-mm,
linux-kernel, linux-doc, cgroups
Swap a large folio back in as a unit when its vswap entries share a
THP-amenable backing (a contiguous physical run, or all zero-filled),
instead of always falling back to order-0 faults.
A zswap-backed or mixed-backing batch is still refused, and the fault
retries at a smaller order.
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
mm/memory.c | 12 ++++++++----
mm/swap_state.c | 17 +++++++++++++----
mm/vswap.h | 7 +++++++
mm/zswap.c | 18 ++++++++++++------
4 files changed, 40 insertions(+), 14 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index ba84565605a1..a1e106a5c5e6 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4815,11 +4815,15 @@ static unsigned long thp_swapin_suitable_orders(struct vm_fault *vmf)
entry = softleaf_from_pte(vmf->orig_pte);
/*
- * THP swapin for vswap is not supported yet. Also, a large swapped
- * out folio could be partially or fully in zswap, which we lack
- * handling for. In both cases, fall back to order-0 swapin.
+ * A large swapped out folio could be partially or fully in zswap.
+ * For vswap entries the THP-amenability of the backing is checked
+ * later under the cluster lock in __swap_cache_add_check, which
+ * rejects ZSWAP and mixed batches via -EBUSY and triggers
+ * order-fallback. For non-vswap entries we still need the
+ * zswap_never_enabled() bail: zswap_load rejects large folios with
+ * -EINVAL, which would SIGBUS the fault.
*/
- if (is_vswap_entry(entry) || !zswap_never_enabled())
+ if (!is_vswap_entry(entry) && !zswap_never_enabled())
return 0;
/*
diff --git a/mm/swap_state.c b/mm/swap_state.c
index c61bb3eef62a..479814d19f50 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -173,6 +173,9 @@ static int __swap_cache_add_check(struct swap_cluster_info *ci,
unsigned int ci_off, ci_end;
unsigned long old_tb;
bool is_zero;
+ struct swap_cluster_info_dynamic *ci_dyn;
+ enum vswap_backing_type type;
+ int ret;
lockdep_assert_held(&ci->lock);
@@ -201,11 +204,17 @@ static int __swap_cache_add_check(struct swap_cluster_info *ci,
return 0;
/*
- * THP swapin for vswap is not supported yet; reject the batch so
- * swap_cache_alloc_folio falls back to order 0.
+ * For a vswap entry batch, reject if the backing is not THP-amenable
+ * (e.g. uniformly ZSWAP, or mixed). The order-fallback loop in
+ * swap_cache_alloc_folio will retry with a smaller order on -EBUSY.
*/
- if (is_vswap_entry(targ_entry))
- return -EBUSY;
+ if (is_vswap_entry(targ_entry)) {
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ ret = __vswap_check_backing(ci_dyn, round_down(ci_off, nr),
+ nr, &type);
+ if (ret != nr || type == VSWAP_ZSWAP)
+ return -EBUSY;
+ }
is_zero = __swap_table_test_zero(ci, ci_off);
ci_off = round_down(ci_off, nr);
diff --git a/mm/vswap.h b/mm/vswap.h
index 239b47b577d5..a921620f08be 100644
--- a/mm/vswap.h
+++ b/mm/vswap.h
@@ -378,6 +378,13 @@ static inline struct zswap_entry *vswap_zswap_load(swp_entry_t entry)
static inline void folio_release_vswap_backing(struct folio *folio) {}
static inline void folio_release_non_phys_swap_backing(struct folio *folio) {}
+static inline int __vswap_check_backing(struct swap_cluster_info_dynamic *ci_dyn,
+ unsigned int voff, int nr,
+ enum vswap_backing_type *typep)
+{
+ return 0;
+}
+
static inline int vswap_cluster_alloc_vtable(struct swap_cluster_info_dynamic *ci_dyn)
{
return 0;
diff --git a/mm/zswap.c b/mm/zswap.c
index d0c6ce2aa092..5dc338188a29 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -1630,13 +1630,19 @@ int zswap_load(struct folio *folio)
return -ENOENT;
/*
- * Large folios should not be swapped in while zswap is being used, as
- * they are not properly handled. Zswap does not properly load large
- * folios, and a large folio may only be partially in zswap.
+ * zswap_load() does not support large folios. For non-vswap
+ * entries this is unexpected on the swapin path: WARN and
+ * sigbus. For vswap entries __swap_cache_add_check() has already
+ * filtered out ZSWAP-backed THPs under the cluster lock, so the
+ * large folio here is zero- or phys-backed; return -ENOENT so the
+ * phys/zero IO path handles it.
*/
- if (WARN_ON_ONCE(folio_test_large(folio))) {
- folio_unlock(folio);
- return -EINVAL;
+ if (folio_test_large(folio)) {
+ if (WARN_ON_ONCE(!swap_is_vswap(si))) {
+ folio_unlock(folio);
+ return -EINVAL;
+ }
+ return -ENOENT;
}
entry = zswap_entry_load(swp);
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v3 06/11] mm, swap: write back vswap zswap entries to physical swap
2026-08-06 18:42 [PATCH v3 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
` (4 preceding siblings ...)
2026-08-06 18:42 ` [PATCH v3 05/11] mm, swap: enable THP swapin for vswap entries Nhat Pham
@ 2026-08-06 18:42 ` Nhat Pham
2026-08-06 18:42 ` [PATCH v3 07/11] mm, swap: reclaim physical slots backing cache-only vswap entries Nhat Pham
` (6 subsequent siblings)
12 siblings, 0 replies; 17+ messages in thread
From: Nhat Pham @ 2026-08-06 18:42 UTC (permalink / raw)
To: akpm
Cc: chrisl, kasong, hannes, mhocko, roman.gushchin, shakeel.butt,
yosry, david, muchun.song, shikemeng, baoquan.he, baohua,
youngjun.park, chengming.zhou, ljs, liam, vbabka, rppt, surenb,
qi.zheng, axelrasmussen, yuanchu, weixugc, riel, gourry,
haowenchao22, corbet, kernel-team, nphamcs, linux-mm,
linux-kernel, linux-doc, cgroups
Add support for writing back zswap-backed vswap entries to physical
swap. The mechanism mirrors the existing zswap writeback path, except
the backing physical slot is allocated on demand at writeback time
rather than already being pinned by the PTE.
Now that vswap entries can be written back, relax the zswap shrinker
gate: replace the blanket "skip while vswap is enabled" check with
can_zswap_writeback(), which only skips when no physical slot is free to
allocate on demand. This keeps the shrinker off futile writeback when no
physical swap is available while letting it drain vswap entries
otherwise.
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
mm/zswap.c | 76 +++++++++++++++++++++++++++++++++++++++---------------
1 file changed, 55 insertions(+), 21 deletions(-)
diff --git a/mm/zswap.c b/mm/zswap.c
index 5dc338188a29..128309d5063b 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -1007,12 +1007,12 @@ static bool zswap_decompress(struct zswap_entry *entry, struct folio *folio)
static int zswap_writeback_entry(struct zswap_entry *entry,
swp_entry_t swpentry)
{
- struct xarray *tree;
pgoff_t offset = swp_offset(swpentry);
struct folio *folio;
struct mempolicy *mpol;
struct swap_info_struct *si;
struct swap_io_ctx ctx = {};
+ swp_entry_t phys = {};
int ret = 0;
/* try to allocate swap cache folio */
@@ -1020,12 +1020,6 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
if (!si)
return -EEXIST;
- /* Vswap entries have no physical backing to write to. */
- if (swap_is_vswap(si)) {
- put_swap_device(si);
- return -EINVAL;
- }
-
mpol = get_task_policy(current);
folio = swap_cache_alloc_folio(swpentry, GFP_KERNEL, BIT(0), NULL, mpol,
NO_INTERLEAVE_INDEX);
@@ -1044,41 +1038,71 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
/*
* folio is locked, and the swapcache is now secured against
* concurrent swapping to and from the slot, and concurrent
- * swapoff so we can safely dereference the zswap tree here.
- * Verify that the swap entry hasn't been invalidated and recycled
- * behind our backs, to avoid overwriting a new swap folio with
- * old compressed data. Only when this is successful can the entry
- * be dereferenced.
+ * swapoff so we can safely dereference the zswap tree (or vswap
+ * vtable) here. Verify that the swap entry hasn't been
+ * invalidated and recycled behind our backs, to avoid overwriting
+ * a new swap folio with old compressed data. Only when this is
+ * successful can the entry be dereferenced.
*/
- tree = swap_zswap_tree(swpentry);
- if (entry != xa_load(tree, offset)) {
+ if (entry != zswap_entry_load(swpentry)) {
ret = -ENOMEM;
goto out;
}
+ if (swap_is_vswap(si)) {
+ /*
+ * Allocate physical backing before decompress so a failure
+ * wastes no work. folio_realloc_swap retags the vtable to
+ * PHYS, leaving the entry pointer held only by the caller.
+ */
+ phys = folio_realloc_swap(folio);
+ if (!phys.val) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ }
+
if (!zswap_decompress(entry, folio)) {
ret = -EIO;
+ /*
+ * For vswap: folio_realloc_swap already moved the entry
+ * out of the vtable. Restore it via vswap_zswap_store so
+ * the entry stays tracked (and the just-allocated PHYS
+ * slot is freed). For non-vswap: entry is still in the
+ * zswap tree.
+ */
+ if (swap_is_vswap(si) && phys.val)
+ vswap_zswap_store(swpentry, entry);
goto out;
}
- xa_erase(tree, offset);
+ if (!swap_is_vswap(si))
+ xa_erase(swap_zswap_tree(swpentry), offset);
count_vm_event(ZSWPWB);
if (entry->objcg)
count_objcg_events(entry->objcg, ZSWPWB, 1);
- zswap_entry_free(entry);
-
/* folio is up to date */
folio_mark_uptodate(folio);
/* move it to the tail of the inactive list after end_writeback */
folio_set_reclaim(folio);
- /* start writeback */
- __swap_writepage(&ctx, folio, folio->swap);
+ /*
+ * Start writeback. The entry has been moved out of its prior location
+ * (vtable PHYS for vswap, removed from the tree otherwise), so we own
+ * the free. vswap writes to the on-demand physical slot; others write
+ * to the folio's own entry.
+ */
+ if (swap_is_vswap(si))
+ __swap_writepage(&ctx, folio, phys);
+ else
+ __swap_writepage(&ctx, folio, folio->swap);
swap_write_submit(&ctx);
+ zswap_entry_free(entry);
+
out:
if (ret) {
swap_cache_del_folio(folio);
@@ -1091,6 +1115,16 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
/*********************************
* shrinker functions
**********************************/
+/*
+ * vswap zswap entries need a physical slot allocated on demand (via
+ * folio_realloc_swap) for writeback; if none is free, writeback fails, so
+ * skip the shrinker to avoid spinning on entries we cannot drain.
+ */
+static bool can_zswap_writeback(void)
+{
+ return !vswap_is_enabled() || get_nr_swap_pages();
+}
+
/*
* The dynamic shrinker is modulated by the following factors:
*
@@ -1228,7 +1262,7 @@ static unsigned long zswap_shrinker_count(struct shrinker *shrinker,
if (!zswap_shrinker_enabled || !mem_cgroup_zswap_writeback_enabled(memcg))
return 0;
- if (vswap_is_enabled())
+ if (!can_zswap_writeback())
return 0;
/*
@@ -1313,7 +1347,7 @@ static int shrink_memcg(struct mem_cgroup *memcg)
if (!mem_cgroup_zswap_writeback_enabled(memcg))
return -ENOENT;
- if (vswap_is_enabled())
+ if (!can_zswap_writeback())
return -ENOENT;
/*
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v3 07/11] mm, swap: reclaim physical slots backing cache-only vswap entries
2026-08-06 18:42 [PATCH v3 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
` (5 preceding siblings ...)
2026-08-06 18:42 ` [PATCH v3 06/11] mm, swap: write back vswap zswap entries to physical swap Nhat Pham
@ 2026-08-06 18:42 ` Nhat Pham
2026-08-06 18:42 ` [PATCH v3 08/11] mm, swap: only charge physical swap entries Nhat Pham
` (5 subsequent siblings)
12 siblings, 0 replies; 17+ messages in thread
From: Nhat Pham @ 2026-08-06 18:42 UTC (permalink / raw)
To: akpm
Cc: chrisl, kasong, hannes, mhocko, roman.gushchin, shakeel.butt,
yosry, david, muchun.song, shikemeng, baoquan.he, baohua,
youngjun.park, chengming.zhou, ljs, liam, vbabka, rppt, surenb,
qi.zheng, axelrasmussen, yuanchu, weixugc, riel, gourry,
haowenchao22, corbet, kernel-team, nphamcs, linux-mm,
linux-kernel, linux-doc, cgroups
A vswap entry backed by a physical slot can become cache-only: its
swap_count drops to 0 while the folio is still in the swap cache, so the
physical slot is redundant and reclaimable. Until now such a slot was
only freed when the vswap entry itself was freed, pinning otherwise
reclaimable physical capacity.
Reclaim such slots from the physical reclaim scanner, once swap is more
than half used (vm_swap_full()), to free physical capacity for new
allocations.
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
mm/swap_table.h | 11 +++-
mm/swapfile.c | 142 ++++++++++++++++++++++++++++++++++++++++++++++++
mm/vswap.h | 26 +++++++++
3 files changed, 176 insertions(+), 3 deletions(-)
diff --git a/mm/swap_table.h b/mm/swap_table.h
index 5b0eca07a821..b50ebcd9e4de 100644
--- a/mm/swap_table.h
+++ b/mm/swap_table.h
@@ -377,9 +377,12 @@ static inline unsigned short __swap_cgroup_clear(struct swap_cluster_info *ci,
* On physical clusters, a Pointer-tagged entry stores the offset of the
* vswap entry that owns this physical slot (the reverse map). Only the
* offset is stored; the swap type is implicit (always vswap_si->type,
- * since there is exactly one vswap device).
+ * since there is exactly one vswap device). The top bit is reserved as
+ * a cache-only flag, set when vswap swap_count drops to 0 but the folio
+ * is still in swap cache.
*
- * Pointer: |---- vswap offset ----|100|
+ * Pointer: |C|---- vswap offset ----|100|
+ * C = SWP_RMAP_CACHE_ONLY (bit 63)
*/
#ifdef CONFIG_VSWAP
extern struct swap_info_struct *vswap_si;
@@ -387,7 +390,8 @@ extern struct swap_info_struct *vswap_si;
#define SWP_TB_PTR_MARK_BITS 3
#define SWP_TB_PTR_MARK 0b100UL
#define SWP_TB_PTR_MARK_MASK ((1UL << SWP_TB_PTR_MARK_BITS) - 1)
-#define SWP_RMAP_ENTRY_MASK (~SWP_TB_PTR_MARK_MASK)
+#define SWP_RMAP_CACHE_ONLY (1UL << (BITS_PER_LONG - 1))
+#define SWP_RMAP_ENTRY_MASK (~(SWP_RMAP_CACHE_ONLY | SWP_TB_PTR_MARK_MASK))
static inline bool swp_tb_is_pointer(unsigned long swp_tb)
{
@@ -408,6 +412,7 @@ static inline swp_entry_t swp_tb_ptr_to_swp_entry(unsigned long swp_tb)
return swp_entry(vswap_si->type, offset);
}
#else
+#define SWP_RMAP_CACHE_ONLY 0UL
static inline bool swp_tb_is_pointer(unsigned long swp_tb)
{
return false;
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 0874f57d3124..ab4bb57707e6 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -151,8 +151,20 @@ static DEFINE_PER_CPU(struct percpu_vswap_cluster, percpu_vswap_cluster) = {
};
static bool vswap_alloc(struct folio *folio);
+static void vswap_mark_cache_only(struct swap_info_struct *si,
+ struct swap_cluster_info *ci,
+ unsigned int ci_off);
+static void vswap_clear_cache_only(struct swap_info_struct *si,
+ struct swap_cluster_info *ci,
+ unsigned int ci_start, int nr);
#else
static inline bool vswap_alloc(struct folio *folio) { return false; }
+static inline void vswap_mark_cache_only(struct swap_info_struct *si,
+ struct swap_cluster_info *ci,
+ unsigned int ci_off) {}
+static inline void vswap_clear_cache_only(struct swap_info_struct *si,
+ struct swap_cluster_info *ci,
+ unsigned int ci_start, int nr) {}
#endif
/* May return NULL on invalid type, caller must check for NULL return */
@@ -912,6 +924,59 @@ static int swap_cluster_setup_bad_slot(struct swap_info_struct *si,
return ret;
}
+/*
+ * Try to reclaim a Pointer-tagged physical slot backing a vswap entry.
+ * The physical cluster lock must NOT be held. Returns the number of physical
+ * slots reclaimed (the backing folio's page count), or < 0 on failure.
+ */
+static int try_to_reclaim_vswap_backing(struct swap_info_struct *si,
+ unsigned long offset,
+ swp_entry_t vswap_entry)
+{
+ swp_entry_t phys_base;
+ struct folio *folio;
+ unsigned int i;
+ int ret;
+
+ folio = swap_cache_get_folio(vswap_entry);
+ if (!folio)
+ return -1;
+
+ if (!folio_trylock(folio)) {
+ folio_put(folio);
+ return -1;
+ }
+
+ if (!folio_matches_swap_entry(folio, vswap_entry)) {
+ folio_unlock(folio);
+ folio_put(folio);
+ return -1;
+ }
+
+ /*
+ * Re-validate under folio lock. The folio's first vswap entry is
+ * folio->swap; the rmap value we just read is folio->swap + i for
+ * some i in [0, nr_pages). Check the folio's first entry still maps
+ * to the contiguous physical run that includes our target offset.
+ */
+ i = vswap_entry.val - folio->swap.val;
+ phys_base = vswap_to_phys(folio->swap);
+ if (!phys_base.val || swp_type(phys_base) != si->type ||
+ swp_offset(phys_base) + i != offset ||
+ i >= folio_nr_pages(folio)) {
+ folio_unlock(folio);
+ folio_put(folio);
+ return -1;
+ }
+
+ ret = folio_nr_pages(folio);
+ if (!folio_free_swap(folio))
+ ret = -1;
+ folio_unlock(folio);
+ folio_put(folio);
+ return ret;
+}
+
/*
* Reclaim drops the ci lock, so the cluster may become unusable (freed or
* stolen by a lower order). @usable will be set to false if that happens.
@@ -935,6 +1000,16 @@ static bool cluster_reclaim_range(struct swap_info_struct *si,
spin_unlock(&ci->lock);
do {
swp_tb = swap_table_get(ci, offset % SWAPFILE_CLUSTER);
+ if (swp_tb_is_pointer(swp_tb)) {
+ rcu_read_unlock();
+ if (!(swp_tb & SWP_RMAP_CACHE_ONLY))
+ goto relock;
+ if (try_to_reclaim_vswap_backing(si, offset,
+ swp_tb_ptr_to_swp_entry(swp_tb)) < 0)
+ goto relock;
+ rcu_read_lock();
+ continue;
+ }
if (swp_tb_get_count(swp_tb))
break;
if (swp_tb_is_folio(swp_tb))
@@ -942,6 +1017,7 @@ static bool cluster_reclaim_range(struct swap_info_struct *si,
break;
} while (++offset < end);
rcu_read_unlock();
+relock:
/* Re-lookup: dynamic cluster may have been freed while lock was dropped */
ci = swap_cluster_lock(si, start);
@@ -1209,6 +1285,7 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
long to_scan = 1;
unsigned long offset, end;
struct swap_cluster_info *ci;
+ swp_entry_t vswap_entry;
unsigned long swp_tb;
int nr_reclaim;
@@ -1233,6 +1310,19 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
offset += abs(nr_reclaim);
continue;
}
+ } else if (swp_tb_is_pointer(swp_tb) &&
+ (swp_tb & SWP_RMAP_CACHE_ONLY)) {
+ vswap_entry = swp_tb_ptr_to_swp_entry(swp_tb);
+ spin_unlock(&ci->lock);
+ nr_reclaim = try_to_reclaim_vswap_backing(si, offset,
+ vswap_entry);
+ ci = swap_cluster_lock(si, offset);
+ if (!ci)
+ goto next;
+ if (nr_reclaim > 0) {
+ offset += nr_reclaim;
+ continue;
+ }
}
offset++;
}
@@ -1812,6 +1902,8 @@ static void swap_put_entries_cluster(struct swap_info_struct *si,
}
/* count will be 0 after put, slot can be reclaimed */
need_reclaim = true;
+ if (swap_is_vswap(si))
+ vswap_mark_cache_only(si, ci, ci_off);
}
/*
* A count != 1 or cached slot can't be freed. Put its swap
@@ -1918,6 +2010,7 @@ static int swap_dup_entries_cluster(struct swap_info_struct *si,
goto failed;
}
} while (++ci_off < ci_end);
+ vswap_clear_cache_only(si, ci, ci_start, nr);
swap_cluster_unlock(ci);
return 0;
failed:
@@ -2034,6 +2127,55 @@ int folio_alloc_swap(struct folio *folio)
}
#ifdef CONFIG_VSWAP
+static void vswap_mark_cache_only(struct swap_info_struct *si,
+ struct swap_cluster_info *ci,
+ unsigned int ci_off)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ struct swap_cluster_info *pci;
+ swp_entry_t phys;
+ unsigned long vt;
+
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ vt = __vtable_get(ci_dyn, ci_off);
+
+ if (vtable_type(vt) == VSWAP_SWAPFILE) {
+ phys = vtable_to_phys(vt);
+ pci = __swap_entry_to_cluster(phys);
+ swap_rmap_mark_cache_only(pci, swp_cluster_offset(phys));
+ }
+}
+
+/*
+ * Clear the cache-only rmap hint for entries re-referenced from count 0 to 1
+ * (no longer reclaimable), so the physical reclaim scanner skips them.
+ */
+static void vswap_clear_cache_only(struct swap_info_struct *si,
+ struct swap_cluster_info *ci,
+ unsigned int ci_start, int nr)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ struct swap_cluster_info *pci;
+ unsigned long swp_tb, vt;
+ swp_entry_t phys;
+ unsigned int off;
+
+ if (!swap_is_vswap(si))
+ return;
+
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ for (off = ci_start; off < ci_start + nr; off++) {
+ swp_tb = __swap_table_get(ci, off);
+ if (!swp_tb_is_folio(swp_tb) || swp_tb_get_count(swp_tb) != 1)
+ continue;
+ vt = __vtable_get(ci_dyn, off);
+ if (vtable_type(vt) != VSWAP_SWAPFILE)
+ continue;
+ phys = vtable_to_phys(vt);
+ pci = __swap_entry_to_cluster(phys);
+ swap_rmap_clear_cache_only(pci, swp_cluster_offset(phys));
+ }
+}
static void __swap_cluster_free_phys_backing(struct swap_info_struct *psi,
struct swap_cluster_info *pci,
diff --git a/mm/vswap.h b/mm/vswap.h
index a921620f08be..803e9a3271fe 100644
--- a/mm/vswap.h
+++ b/mm/vswap.h
@@ -35,6 +35,32 @@ static inline bool is_vswap_entry(swp_entry_t entry)
bool vswap_is_enabled(void);
+/*
+ * Rmap cache-only helpers for physical cluster Pointer-tagged entries.
+ * SWP_RMAP_CACHE_ONLY records, inline on the physical swap_table entry,
+ * that the backing vswap entry has swap_count == 0 (swap-cache-only, so
+ * reclaimable). The physical reclaim scanner reads it directly instead of
+ * chasing the rmap into the vswap layer and paying the cluster-lookup
+ * indirection.
+ */
+static inline void swap_rmap_mark_cache_only(struct swap_cluster_info *ci,
+ unsigned int off)
+{
+ atomic_long_t *table;
+
+ table = rcu_dereference_check(ci->table, true);
+ atomic_long_or(SWP_RMAP_CACHE_ONLY, &table[off]);
+}
+
+static inline void swap_rmap_clear_cache_only(struct swap_cluster_info *ci,
+ unsigned int off)
+{
+ atomic_long_t *table;
+
+ table = rcu_dereference_check(ci->table, true);
+ atomic_long_and(~SWP_RMAP_CACHE_ONLY, &table[off]);
+}
+
/*
* Virtual table entry encoding for vswap clusters.
*
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v3 08/11] mm, swap: only charge physical swap entries
2026-08-06 18:42 [PATCH v3 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
` (6 preceding siblings ...)
2026-08-06 18:42 ` [PATCH v3 07/11] mm, swap: reclaim physical slots backing cache-only vswap entries Nhat Pham
@ 2026-08-06 18:42 ` Nhat Pham
2026-08-07 16:31 ` Johannes Weiner
2026-08-06 18:42 ` [PATCH v3 09/11] mm, swap: add debugfs counters for vswap Nhat Pham
` (4 subsequent siblings)
12 siblings, 1 reply; 17+ messages in thread
From: Nhat Pham @ 2026-08-06 18:42 UTC (permalink / raw)
To: akpm
Cc: chrisl, kasong, hannes, mhocko, roman.gushchin, shakeel.butt,
yosry, david, muchun.song, shikemeng, baoquan.he, baohua,
youngjun.park, chengming.zhou, ljs, liam, vbabka, rppt, surenb,
qi.zheng, axelrasmussen, yuanchu, weixugc, riel, gourry,
haowenchao22, corbet, kernel-team, nphamcs, linux-mm,
linux-kernel, linux-doc, cgroups
Charge memcg->swap when a vswap entry acquires physical backing rather
than when it is allocated, so memory.swap.current tracks on-disk swap
usage. Zswap-backed and zero-filled pages occupy no swap space but were
charged as though they did.
memory.swap.current therefore no longer counts them, and a cgroup whose
pages all land in zswap can now reclaim anon memory with memory.swap.max
set to 0.
Direct-mapped physical swap charging is unchanged.
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
include/linux/memcontrol.h | 5 ++
include/linux/swap.h | 57 +++++++++++++
mm/memcontrol.c | 166 ++++++++++++++++++++++++++++++++-----
mm/swapfile.c | 111 +++++++++++++++++++++----
4 files changed, 303 insertions(+), 36 deletions(-)
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index e78bc98ab229..0a2f85ac7b6a 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -1900,6 +1900,7 @@ static inline bool memcg_is_dying(struct mem_cgroup *memcg)
#if defined(CONFIG_MEMCG) && defined(CONFIG_ZSWAP)
bool obj_cgroup_may_zswap(struct obj_cgroup *objcg);
+bool mem_cgroup_may_zswap(struct mem_cgroup *memcg, bool may_flush);
void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size);
void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size);
bool mem_cgroup_zswap_writeback_enabled(struct mem_cgroup *memcg);
@@ -1908,6 +1909,10 @@ static inline bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)
{
return true;
}
+static inline bool mem_cgroup_may_zswap(struct mem_cgroup *memcg, bool may_flush)
+{
+ return true;
+}
static inline void obj_cgroup_charge_zswap(struct obj_cgroup *objcg,
size_t size)
{
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 2b2bd56afffa..19a703510675 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -523,6 +523,43 @@ static inline int mem_cgroup_try_charge_swap(struct folio *folio)
return __mem_cgroup_try_charge_swap(folio);
}
+extern void __mem_cgroup_record_swap(struct folio *folio);
+static inline void mem_cgroup_record_swap(struct folio *folio)
+{
+ if (mem_cgroup_disabled())
+ return;
+ __mem_cgroup_record_swap(folio);
+}
+
+extern int __mem_cgroup_charge_backing_phys_swap(struct mem_cgroup *memcg,
+ unsigned int nr_pages);
+static inline int mem_cgroup_charge_backing_phys_swap(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
+{
+ if (mem_cgroup_disabled())
+ return 0;
+ return __mem_cgroup_charge_backing_phys_swap(memcg, nr_pages);
+}
+
+extern void __mem_cgroup_uncharge_backing_phys_swap(struct mem_cgroup *memcg,
+ unsigned int nr_pages);
+static inline void mem_cgroup_uncharge_backing_phys_swap(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
+{
+ if (mem_cgroup_disabled())
+ return;
+ __mem_cgroup_uncharge_backing_phys_swap(memcg, nr_pages);
+}
+
+extern void __mem_cgroup_id_put_swap(unsigned short id, unsigned int nr_pages);
+static inline void mem_cgroup_id_put_swap(unsigned short id,
+ unsigned int nr_pages)
+{
+ if (mem_cgroup_disabled())
+ return;
+ __mem_cgroup_id_put_swap(id, nr_pages);
+}
+
extern void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages);
static inline void mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages)
{
@@ -539,6 +576,26 @@ static inline int mem_cgroup_try_charge_swap(struct folio *folio)
return 0;
}
+static inline void mem_cgroup_record_swap(struct folio *folio)
+{
+}
+
+static inline int mem_cgroup_charge_backing_phys_swap(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
+{
+ return 0;
+}
+
+static inline void mem_cgroup_uncharge_backing_phys_swap(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
+{
+}
+
+static inline void mem_cgroup_id_put_swap(unsigned short id,
+ unsigned int nr_pages)
+{
+}
+
static inline void mem_cgroup_uncharge_swap(unsigned short id,
unsigned int nr_pages)
{
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 7a426db06222..f6aee32ef542 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -48,6 +48,7 @@
#include <linux/rbtree.h>
#include <linux/slab.h>
#include <linux/swapops.h>
+#include <linux/zswap.h>
#include <linux/spinlock.h>
#include <linux/fs.h>
#include <linux/seq_file.h>
@@ -5701,6 +5702,116 @@ int __mem_cgroup_try_charge_swap(struct folio *folio)
return 0;
}
+/**
+ * __mem_cgroup_record_swap - record memcg for swap without charging
+ * @folio: folio being added to swap
+ *
+ * Pin the memcg private ID ref and record it in the swap cgroup table
+ * without charging memcg->swap; the charge is deferred to physical-backing
+ * allocation (vswap).
+ */
+void __mem_cgroup_record_swap(struct folio *folio)
+{
+ unsigned int nr_pages = folio_nr_pages(folio);
+ struct swap_cluster_info *ci;
+ struct mem_cgroup *memcg;
+ struct obj_cgroup *objcg;
+
+ if (do_memsw_account())
+ return;
+
+ objcg = folio_objcg(folio);
+ VM_WARN_ON_ONCE_FOLIO(!objcg, folio);
+ if (!objcg)
+ return;
+
+ rcu_read_lock();
+ memcg = obj_cgroup_memcg(objcg);
+ if (!folio_test_swapcache(folio)) {
+ rcu_read_unlock();
+ return;
+ }
+
+ memcg = mem_cgroup_private_id_get_online(memcg, nr_pages);
+ rcu_read_unlock();
+
+ ci = swap_cluster_get_and_lock(folio);
+ __swap_cgroup_set(ci, swp_cluster_offset(folio->swap), nr_pages,
+ mem_cgroup_private_id(memcg));
+ swap_cluster_unlock(ci);
+}
+
+/**
+ * __mem_cgroup_charge_backing_phys_swap - charge memcg->swap
+ * @memcg: the mem_cgroup to charge (may be NULL)
+ * @nr_pages: number of physical swap pages to charge
+ *
+ * Charge the swap counter when a vswap entry gains physical backing. The
+ * private ID ref is already held (pinned by __mem_cgroup_record_swap() at
+ * vswap allocation), so this only moves the counter.
+ *
+ * Return: 0 on success, -ENOMEM on failure.
+ */
+int __mem_cgroup_charge_backing_phys_swap(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
+{
+ struct page_counter *counter;
+
+ if (do_memsw_account())
+ return 0;
+ if (!memcg)
+ return 0;
+
+ if (!mem_cgroup_is_root(memcg) &&
+ !page_counter_try_charge(&memcg->swap, nr_pages, &counter)) {
+ memcg_memory_event(memcg, MEMCG_SWAP_MAX);
+ memcg_memory_event(memcg, MEMCG_SWAP_FAIL);
+ return -ENOMEM;
+ }
+ mod_memcg_state(memcg, MEMCG_SWAP, nr_pages);
+ return 0;
+}
+
+/**
+ * __mem_cgroup_uncharge_backing_phys_swap - uncharge memcg->swap counter
+ * @memcg: the mem_cgroup to uncharge (may be NULL)
+ * @nr_pages: number of physical swap pages to uncharge
+ *
+ * Uncharge the swap counter on physical backing release for a vswap entry.
+ * The private ID ref is dropped separately via __mem_cgroup_id_put_swap() when
+ * the vswap entry is freed.
+ */
+void __mem_cgroup_uncharge_backing_phys_swap(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
+{
+ if (!memcg)
+ return;
+
+ if (!mem_cgroup_is_root(memcg)) {
+ if (do_memsw_account())
+ page_counter_uncharge(&memcg->memsw, nr_pages);
+ else
+ page_counter_uncharge(&memcg->swap, nr_pages);
+ }
+ mod_memcg_state(memcg, MEMCG_SWAP, -nr_pages);
+}
+
+/**
+ * __mem_cgroup_id_put_swap - drop memcg private ID ref without uncharging
+ * @id: cgroup private id
+ * @nr_pages: number of refs to drop
+ */
+void __mem_cgroup_id_put_swap(unsigned short id, unsigned int nr_pages)
+{
+ struct mem_cgroup *memcg;
+
+ rcu_read_lock();
+ memcg = mem_cgroup_from_private_id(id);
+ if (memcg)
+ mem_cgroup_private_id_put(memcg, nr_pages);
+ rcu_read_unlock();
+}
+
/**
* __mem_cgroup_uncharge_swap - uncharge swap space
* @id: cgroup id to uncharge
@@ -5727,15 +5838,21 @@ void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages)
long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg)
{
- long nr_swap_pages = get_nr_swap_pages();
+ long nr_swap_pages;
/*
- * vswap zswap-backed swapout needs no physical slot, so gate anon
- * reclaim on the swap.max headroom instead of the physical free count.
+ * vswap charges only physical backing (folio_realloc_swap), not
+ * allocation. For a zswap-capable memcg virtual swap is unbounded, so
+ * the swap.max walk below would underestimate it and starve anon
+ * reclaim; report unbounded. swap.max is still enforced at
+ * phys-backing charge time.
*/
- if (vswap_is_enabled() && zswap_is_enabled())
- nr_swap_pages = PAGE_COUNTER_MAX;
+ if (vswap_is_enabled() && zswap_is_enabled() &&
+ (mem_cgroup_disabled() || do_memsw_account() ||
+ mem_cgroup_may_zswap(memcg, false)))
+ return PAGE_COUNTER_MAX;
+ nr_swap_pages = get_nr_swap_pages();
if (mem_cgroup_disabled() || do_memsw_account())
return nr_swap_pages;
for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg))
@@ -5907,8 +6024,10 @@ static struct cftype swap_files[] = {
#ifdef CONFIG_ZSWAP
/**
- * obj_cgroup_may_zswap - check if this cgroup can zswap
- * @objcg: the object cgroup
+ * mem_cgroup_may_zswap - check if this cgroup hierarchy can zswap
+ * @original_memcg: the memcg to query
+ * @may_flush: force-flush stats for an accurate check (sleeps). Pass false
+ * from atomic contexts; the check is then best-effort.
*
* Check if the hierarchical zswap limit has been reached.
*
@@ -5918,15 +6037,13 @@ static struct cftype swap_files[] = {
* spending cycles on compression when there is already no room left
* or zswap is disabled altogether somewhere in the hierarchy.
*/
-bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)
+bool mem_cgroup_may_zswap(struct mem_cgroup *original_memcg, bool may_flush)
{
- struct mem_cgroup *memcg, *original_memcg;
- bool ret = true;
+ struct mem_cgroup *memcg;
if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
return true;
- original_memcg = get_mem_cgroup_from_objcg(objcg);
for (memcg = original_memcg; !mem_cgroup_is_root(memcg);
memcg = parent_mem_cgroup(memcg)) {
unsigned long max = READ_ONCE(memcg->zswap_max);
@@ -5934,20 +6051,27 @@ bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)
if (max == PAGE_COUNTER_MAX)
continue;
- if (max == 0) {
- ret = false;
- break;
- }
+ if (max == 0)
+ return false;
/* Force flush to get accurate stats for charging */
- __mem_cgroup_flush_stats(memcg, true);
+ if (may_flush)
+ __mem_cgroup_flush_stats(memcg, true);
pages = memcg_page_state(memcg, MEMCG_ZSWAP_B) / PAGE_SIZE;
- if (pages < max)
- continue;
- ret = false;
- break;
+ if (pages >= max)
+ return false;
}
- mem_cgroup_put(original_memcg);
+ return true;
+}
+
+bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)
+{
+ struct mem_cgroup *memcg;
+ bool ret;
+
+ memcg = get_mem_cgroup_from_objcg(objcg);
+ ret = mem_cgroup_may_zswap(memcg, true);
+ mem_cgroup_put(memcg);
return ret;
}
diff --git a/mm/swapfile.c b/mm/swapfile.c
index ab4bb57707e6..1a2d9d9625fc 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -47,6 +47,7 @@
#include <asm/tlbflush.h>
#include <linux/leafops.h>
+#include "memcontrol-v1.h"
#include "swap_table.h"
#include "vswap.h"
#include "internal.h"
@@ -2116,8 +2117,16 @@ int folio_alloc_swap(struct folio *folio)
goto again;
}
- /* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. */
- if (unlikely(mem_cgroup_try_charge_swap(folio)))
+ /*
+ * A vswap entry has no physical swap yet, so only record the memcg;
+ * folio_realloc_swap() charges once backing is allocated.
+ *
+ * Need to call this even if allocation failed, for MEMCG_SWAP_FAIL.
+ */
+ if (folio_test_swapcache(folio) &&
+ is_vswap_entry(folio->swap))
+ mem_cgroup_record_swap(folio);
+ else if (unlikely(mem_cgroup_try_charge_swap(folio)))
swap_cache_del_folio(folio);
if (unlikely(!folio_test_swapcache(folio)))
@@ -2182,6 +2191,28 @@ static void __swap_cluster_free_phys_backing(struct swap_info_struct *psi,
unsigned int ci_start,
unsigned int nr_pages);
+static void vswap_uncharge_cgroup_batch(unsigned short memcg_id,
+ unsigned int batch_nr,
+ unsigned int batch_nr_swapfile)
+{
+ struct mem_cgroup *memcg;
+ unsigned int n;
+
+ /*
+ * v1 (memsw): __memcg1_swapout() charges memsw for every swapped-out
+ * entry regardless of backing, so uncharge all of them. v2: only
+ * swapfile-backed entries are charged, so uncharge just those.
+ */
+ n = do_memsw_account() ? batch_nr : batch_nr_swapfile;
+ if (!n)
+ return;
+
+ rcu_read_lock();
+ memcg = memcg_id ? mem_cgroup_from_private_id(memcg_id) : NULL;
+ rcu_read_unlock();
+ mem_cgroup_uncharge_backing_phys_swap(memcg, n);
+}
+
/**
* __vswap_release_backing - release the backing of a range of vtable slots
* @ci: the locked vswap cluster
@@ -2191,8 +2222,7 @@ static void __swap_cluster_free_phys_backing(struct swap_info_struct *psi,
* Releases each slot in [@ci_start, @ci_start + @nr): physical swap slots,
* zswap entries, etc. Clears the zero marks if set.
*
- * Context: caller must hold @ci->lock. The entire range must belong to the
- * same memcg.
+ * Context: caller must hold @ci->lock.
*/
void __vswap_release_backing(struct swap_cluster_info *ci,
unsigned int ci_start, unsigned int nr)
@@ -2204,12 +2234,27 @@ void __vswap_release_backing(struct swap_cluster_info *ci,
unsigned int ci_off;
unsigned long vt;
swp_entry_t phys;
+ unsigned short batch_id;
+ unsigned int batch_nr = 0, batch_nr_swapfile = 0;
lockdep_assert_held(&ci->lock);
ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ batch_id = __swap_cgroup_get(ci, ci_start);
for (ci_off = ci_start; ci_off < ci_start + nr; ci_off++) {
+ unsigned short cur_id;
+
vt = __vtable_get(ci_dyn, ci_off);
+ cur_id = __swap_cgroup_get(ci, ci_off);
+
+ if (cur_id != batch_id) {
+ vswap_uncharge_cgroup_batch(batch_id, batch_nr,
+ batch_nr_swapfile);
+ batch_id = cur_id;
+ batch_nr = 0;
+ batch_nr_swapfile = 0;
+ }
+ batch_nr++;
/*
* Flush batched physical slots when the next entry
@@ -2233,6 +2278,7 @@ void __vswap_release_backing(struct swap_cluster_info *ci,
switch (vtable_type(vt)) {
case VSWAP_SWAPFILE:
+ batch_nr_swapfile++;
if (phys_start == phys_end) {
phys = vtable_to_phys(vt);
phys_start = swp_offset(phys);
@@ -2266,6 +2312,8 @@ void __vswap_release_backing(struct swap_cluster_info *ci,
phys_start % SWAPFILE_CLUSTER,
phys_end - phys_start);
}
+
+ vswap_uncharge_cgroup_batch(batch_id, batch_nr, batch_nr_swapfile);
}
/**
@@ -2355,7 +2403,10 @@ swp_entry_t folio_realloc_swap(struct folio *folio)
swp_entry_t vswap_entry = folio->swap;
struct swap_cluster_info *ci;
struct swap_cluster_info_dynamic *ci_dyn;
+ struct mem_cgroup *memcg;
unsigned int voff;
+ unsigned long vt;
+ unsigned short memcg_id;
swp_entry_t phys_entry = {};
swp_entry_t pe;
int i, nr = folio_nr_pages(folio);
@@ -2364,9 +2415,18 @@ swp_entry_t folio_realloc_swap(struct folio *folio)
VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio);
VM_WARN_ON(!is_vswap_entry(vswap_entry));
- phys_entry = vswap_to_phys(vswap_entry);
- if (phys_entry.val)
- return phys_entry;
+ voff = swp_cluster_offset(vswap_entry);
+ ci = __swap_entry_to_cluster(vswap_entry);
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+
+ spin_lock(&ci->lock);
+ vt = __vtable_get(ci_dyn, voff);
+ if (vtable_type(vt) == VSWAP_SWAPFILE) {
+ spin_unlock(&ci->lock);
+ return vtable_to_phys(vt);
+ }
+ memcg_id = __swap_cgroup_get(ci, voff);
+ spin_unlock(&ci->lock);
local_lock(&percpu_swap_cluster.lock);
phys_entry = swap_alloc_fast(folio);
@@ -2377,10 +2437,20 @@ swp_entry_t folio_realloc_swap(struct folio *folio)
if (!phys_entry.val)
return (swp_entry_t){};
- voff = swp_cluster_offset(vswap_entry);
+ rcu_read_lock();
+ memcg = folio_memcg(folio);
+ if (!memcg || mem_cgroup_private_id(memcg) != memcg_id)
+ memcg = memcg_id ? mem_cgroup_from_private_id(memcg_id) : NULL;
+ rcu_read_unlock();
+
+ if (mem_cgroup_charge_backing_phys_swap(memcg, nr)) {
+ __swap_cluster_free_phys_backing(
+ __swap_entry_to_info(phys_entry),
+ __swap_entry_to_cluster(phys_entry),
+ swp_cluster_offset(phys_entry), nr);
+ return (swp_entry_t){};
+ }
- ci = __swap_entry_to_cluster(vswap_entry);
- ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
spin_lock(&ci->lock);
/*
* Install PHYS backing without freeing any prior contents of the
@@ -2591,10 +2661,11 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,
unsigned short batch_id = 0, id_cur;
unsigned int ci_off = ci_start, ci_end = ci_start + nr_pages;
unsigned int batch_off = ci_off;
+ bool is_vswap = swap_is_vswap(si);
VM_WARN_ON(ci->count < nr_pages);
- if (swap_is_vswap(si))
+ if (is_vswap)
__vswap_release_backing(ci, ci_start, nr_pages);
ci->count -= nr_pages;
@@ -2614,18 +2685,28 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,
/*
* Uncharge swap slots by memcg in batches. Consecutive
* slots with the same cgroup id are uncharged together.
+ * For vswap, only drop the ID ref - physical swap was
+ * already uncharged in __vswap_release_backing above.
*/
id_cur = __swap_cgroup_clear(ci, ci_off, 1);
if (batch_id != id_cur) {
- if (batch_id)
- mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);
+ if (batch_id) {
+ if (is_vswap)
+ mem_cgroup_id_put_swap(batch_id, ci_off - batch_off);
+ else
+ mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);
+ }
batch_id = id_cur;
batch_off = ci_off;
}
} while (++ci_off < ci_end);
- if (batch_id)
- mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);
+ if (batch_id) {
+ if (is_vswap)
+ mem_cgroup_id_put_swap(batch_id, ci_off - batch_off);
+ else
+ mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);
+ }
__swap_cluster_finish_free(si, ci, ci_start, nr_pages);
}
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v3 09/11] mm, swap: add debugfs counters for vswap
2026-08-06 18:42 [PATCH v3 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
` (7 preceding siblings ...)
2026-08-06 18:42 ` [PATCH v3 08/11] mm, swap: only charge physical swap entries Nhat Pham
@ 2026-08-06 18:42 ` Nhat Pham
2026-08-06 18:42 ` [PATCH v3 10/11] mm, swap: defer memcg_table allocation for physical swap clusters Nhat Pham
` (3 subsequent siblings)
12 siblings, 0 replies; 17+ messages in thread
From: Nhat Pham @ 2026-08-06 18:42 UTC (permalink / raw)
To: akpm
Cc: chrisl, kasong, hannes, mhocko, roman.gushchin, shakeel.butt,
yosry, david, muchun.song, shikemeng, baoquan.he, baohua,
youngjun.park, chengming.zhou, ljs, liam, vbabka, rppt, surenb,
qi.zheng, axelrasmussen, yuanchu, weixugc, riel, gourry,
haowenchao22, corbet, kernel-team, nphamcs, linux-mm,
linux-kernel, linux-doc, cgroups
Add /sys/kernel/debug/vswap/ with two counters:
* used: virtual swap slots (pages) currently allocated
* alloc_reject: cumulative pages that failed to get a vswap slot
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
mm/swapfile.c | 15 ++++++++++++++-
1 file changed, 14 insertions(+), 1 deletion(-)
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 1a2d9d9625fc..b4d7af21ca1c 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -7,6 +7,7 @@
*/
#include <linux/blkdev.h>
+#include <linux/debugfs.h>
#include <linux/mm.h>
#include <linux/sched/mm.h>
#include <linux/sched/task.h>
@@ -133,6 +134,9 @@ static DEFINE_PER_CPU(struct percpu_swap_cluster, percpu_swap_cluster) = {
.lock = INIT_LOCAL_LOCK(),
};
+static atomic_t __maybe_unused vswap_used = ATOMIC_INIT(0);
+static atomic_t __maybe_unused vswap_alloc_reject = ATOMIC_INIT(0);
+
#ifdef CONFIG_VSWAP
static int sysctl_vswap_enabled = IS_ENABLED(CONFIG_VSWAP_DEFAULT_ON);
@@ -2056,11 +2060,13 @@ static bool vswap_alloc(struct folio *folio)
if (folio_test_swapcache(folio)) {
/* alloc_swap_scan_cluster updated percpu offset already */
local_unlock(&percpu_vswap_cluster.lock);
+ atomic_add(folio_nr_pages(folio), &vswap_used);
return true;
}
this_cpu_write(percpu_vswap_cluster.offset[order], SWAP_ENTRY_INVALID);
local_unlock(&percpu_vswap_cluster.lock);
+ atomic_add(folio_nr_pages(folio), &vswap_alloc_reject);
return false;
}
#endif
@@ -2665,8 +2671,10 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,
VM_WARN_ON(ci->count < nr_pages);
- if (is_vswap)
+ if (is_vswap) {
__vswap_release_backing(ci, ci_start, nr_pages);
+ atomic_sub(nr_pages, &vswap_used);
+ }
ci->count -= nr_pages;
do {
@@ -4862,6 +4870,7 @@ static const struct ctl_table vswap_sysctls[] = {
static int __init vswap_init(void)
{
struct swap_info_struct *si;
+ struct dentry *root;
unsigned long maxpages;
int err;
@@ -4896,6 +4905,10 @@ static int __init vswap_init(void)
register_sysctl_init("vm", vswap_sysctls);
+ root = debugfs_create_dir("vswap", NULL);
+ debugfs_create_atomic_t("used", 0444, root, &vswap_used);
+ debugfs_create_atomic_t("alloc_reject", 0444, root, &vswap_alloc_reject);
+
pr_info("vswap: created virtual swap device (%lu pages)\n", maxpages);
return 0;
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v3 10/11] mm, swap: defer memcg_table allocation for physical swap clusters
2026-08-06 18:42 [PATCH v3 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
` (8 preceding siblings ...)
2026-08-06 18:42 ` [PATCH v3 09/11] mm, swap: add debugfs counters for vswap Nhat Pham
@ 2026-08-06 18:42 ` Nhat Pham
2026-08-06 18:42 ` [PATCH v3 11/11] mm, swap: widen swap_info_struct max/pages to unsigned long Nhat Pham
` (2 subsequent siblings)
12 siblings, 0 replies; 17+ messages in thread
From: Nhat Pham @ 2026-08-06 18:42 UTC (permalink / raw)
To: akpm
Cc: chrisl, kasong, hannes, mhocko, roman.gushchin, shakeel.butt,
yosry, david, muchun.song, shikemeng, baoquan.he, baohua,
youngjun.park, chengming.zhou, ljs, liam, vbabka, rppt, surenb,
qi.zheng, axelrasmussen, yuanchu, weixugc, riel, gourry,
haowenchao22, corbet, kernel-team, nphamcs, linux-mm,
linux-kernel, linux-doc, cgroups
Stop allocating a memcg table for every physical swap cluster that only
ever holds vswap backings. The table costs SWAPFILE_CLUSTER *
sizeof(unsigned short) per cluster, 1 KB per 2 MB of swap on a 64-bit
kernel with 4 KB pages. On a vswap-heavy workload, where zswap writeback
is the only consumer of physical swap, that is the common case.
Such clusters never have their memcg_table read or written: vswap-layer
charging records on the vswap cluster's table, not the physical one.
Allocate eagerly only when the cluster is known to need a table: any
cluster in a !CONFIG_VSWAP build, or any vswap cluster. For physical
clusters in CONFIG_VSWAP builds, defer to alloc_swap_scan_cluster(),
which allocates on the first direct-use slot and skips entirely when the
cluster only holds pointer-tagged vswap backings.
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
mm/swapfile.c | 40 ++++++++++++++++++++++++++++++++--------
1 file changed, 32 insertions(+), 8 deletions(-)
diff --git a/mm/swapfile.c b/mm/swapfile.c
index b4d7af21ca1c..65559647eeb4 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -492,7 +492,8 @@ static void swap_cluster_free_table(struct swap_cluster_info *ci)
swap_cluster_free_table_folio_rcu_cb);
}
-static int swap_cluster_alloc_table(struct swap_cluster_info *ci, gfp_t gfp)
+static int swap_cluster_alloc_table(struct swap_info_struct *si,
+ struct swap_cluster_info *ci, gfp_t gfp)
{
struct swap_table *table = NULL;
struct folio *folio;
@@ -515,7 +516,16 @@ static int swap_cluster_alloc_table(struct swap_cluster_info *ci, gfp_t gfp)
rcu_assign_pointer(ci->table, table);
#ifdef CONFIG_MEMCG
- if (!mem_cgroup_disabled()) {
+ /*
+ * Allocate memcg_table eagerly only when we know it will be used:
+ * any cluster in a !CONFIG_VSWAP build (all slots are direct use),
+ * or any vswap cluster (every vswap alloc records memcg). Physical
+ * clusters in a CONFIG_VSWAP build defer to alloc_swap_scan_cluster,
+ * which allocates on the first direct-use slot and skips entirely
+ * when the cluster only holds Pointer-tagged vswap backings.
+ */
+ if ((!IS_ENABLED(CONFIG_VSWAP) || swap_is_vswap(si)) &&
+ !mem_cgroup_disabled()) {
VM_WARN_ON_ONCE(ci->memcg_table);
ci->memcg_table = kzalloc_obj(*ci->memcg_table, gfp);
if (!ci->memcg_table) {
@@ -589,8 +599,8 @@ swap_cluster_populate(struct swap_info_struct *si,
lockdep_assert_held(&si->global_cluster_lock);
lockdep_assert_held(&ci->lock);
- if (!swap_cluster_alloc_table(ci, __GFP_HIGH | __GFP_NOMEMALLOC |
- __GFP_NOWARN))
+ if (!swap_cluster_alloc_table(si, ci, __GFP_HIGH | __GFP_NOMEMALLOC |
+ __GFP_NOWARN))
return ci;
/*
@@ -608,8 +618,8 @@ swap_cluster_populate(struct swap_info_struct *si,
if (!swap_is_vswap(si))
local_unlock(&percpu_swap_cluster.lock);
- ret = swap_cluster_alloc_table(ci, __GFP_HIGH | __GFP_NOMEMALLOC |
- GFP_KERNEL);
+ ret = swap_cluster_alloc_table(si, ci, __GFP_HIGH | __GFP_NOMEMALLOC |
+ GFP_KERNEL);
/*
* Back to atomic context. We might have migrated to a new CPU with a
@@ -911,7 +921,7 @@ static int swap_cluster_setup_bad_slot(struct swap_info_struct *si,
ci = cluster_info + idx;
/* Need to allocate swap table first for initial bad slot marking. */
- if (!ci->count && swap_cluster_alloc_table(ci, GFP_KERNEL))
+ if (!ci->count && swap_cluster_alloc_table(si, ci, GFP_KERNEL))
return -ENOMEM;
spin_lock(&ci->lock);
/* Check for duplicated bad swap slots. */
@@ -1193,6 +1203,20 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
if (!ret)
continue;
}
+#ifdef CONFIG_MEMCG
+ /*
+ * Lazy-allocate memcg_table on the first direct-use slot of a
+ * physical cluster.
+ */
+ if (IS_ENABLED(CONFIG_VSWAP) && folio &&
+ !folio_test_swapcache(folio) && !mem_cgroup_disabled() &&
+ !ci->memcg_table) {
+ ci->memcg_table = kzalloc_obj(*ci->memcg_table,
+ GFP_ATOMIC | __GFP_NOWARN);
+ if (!ci->memcg_table)
+ goto out;
+ }
+#endif
if (!__swap_cluster_alloc_entries(si, ci, folio, offset % SWAPFILE_CLUSTER))
break;
found = offset;
@@ -1259,7 +1283,7 @@ static unsigned int alloc_swap_scan_dynamic(struct swap_info_struct *si,
spin_lock_init(&ci_dyn->ci.lock);
INIT_LIST_HEAD(&ci_dyn->ci.list);
- if (swap_cluster_alloc_table(&ci_dyn->ci, GFP_ATOMIC)) {
+ if (swap_cluster_alloc_table(si, &ci_dyn->ci, GFP_ATOMIC)) {
kfree(ci_dyn);
return SWAP_ENTRY_INVALID;
}
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v3 11/11] mm, swap: widen swap_info_struct max/pages to unsigned long
2026-08-06 18:42 [PATCH v3 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
` (9 preceding siblings ...)
2026-08-06 18:42 ` [PATCH v3 10/11] mm, swap: defer memcg_table allocation for physical swap clusters Nhat Pham
@ 2026-08-06 18:42 ` Nhat Pham
2026-08-07 5:26 ` [syzbot ci] Re: Virtual Swap Space (Swap Table Edition) syzbot ci
2026-08-07 9:07 ` [PATCH v3 00/11] " Chris Li
12 siblings, 0 replies; 17+ messages in thread
From: Nhat Pham @ 2026-08-06 18:42 UTC (permalink / raw)
To: akpm
Cc: chrisl, kasong, hannes, mhocko, roman.gushchin, shakeel.butt,
yosry, david, muchun.song, shikemeng, baoquan.he, baohua,
youngjun.park, chengming.zhou, ljs, liam, vbabka, rppt, surenb,
qi.zheng, axelrasmussen, yuanchu, weixugc, riel, gourry,
haowenchao22, corbet, kernel-team, nphamcs, linux-mm,
linux-kernel, linux-doc, cgroups
Widen swap_info_struct->max and ->pages from unsigned int to
unsigned long so the vswap device can exceed the current 16 TB
cap (ALIGN_DOWN(UINT_MAX, SWAPFILE_CLUSTER) pages).
Physical swap is unaffected; backing files/bdevs continue to bound
it independently of the field width.
The new vswap cap is the cluster_info_pool xarray's allocator
limit. XA_FLAGS_ALLOC stores allocated IDs in u32, so
max_pages = UINT_MAX * SWAPFILE_CLUSTER (~8 PB at the typical
SWAPFILE_CLUSTER=512 layout).
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
include/linux/swap.h | 4 +--
mm/swapfile.c | 62 +++++++++++++++++++++++---------------------
2 files changed, 34 insertions(+), 32 deletions(-)
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 19a703510675..44353eb554da 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -246,7 +246,7 @@ struct swap_info_struct {
signed short prio; /* swap priority of this type */
struct plist_node list; /* entry in swap_active_head */
signed char type; /* strange name for an index */
- unsigned int max; /* size of this swap device */
+ unsigned long max; /* size of this swap device */
struct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */
struct list_head free_clusters; /* free clusters list */
struct list_head full_clusters; /* full clusters list */
@@ -254,7 +254,7 @@ struct swap_info_struct {
/* list of cluster that contains at least one free slot */
struct list_head frag_clusters[SWAP_NR_ORDERS];
/* list of cluster that are fragmented or contented */
- unsigned int pages; /* total of usable pages of swap */
+ unsigned long pages; /* total of usable pages of swap */
atomic_long_t inuse_pages; /* number of those currently in use */
struct swap_sequential_cluster *global_cluster; /* Use one global cluster for rotating device */
spinlock_t global_cluster_lock; /* Serialize usage of global cluster */
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 65559647eeb4..6db605594d48 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -450,10 +450,10 @@ static inline unsigned int cluster_index(struct swap_info_struct *si,
return ci - si->cluster_info;
}
-static inline unsigned int cluster_offset(struct swap_info_struct *si,
- struct swap_cluster_info *ci)
+static inline unsigned long cluster_offset(struct swap_info_struct *si,
+ struct swap_cluster_info *ci)
{
- return cluster_index(si, ci) * SWAPFILE_CLUSTER;
+ return (unsigned long)cluster_index(si, ci) * SWAPFILE_CLUSTER;
}
static void swap_cluster_free_table_folio_rcu_cb(struct rcu_head *head)
@@ -904,7 +904,7 @@ static int swap_cluster_setup_bad_slot(struct swap_info_struct *si,
/* si->max may got shrunk by swap swap_activate() */
if (offset >= si->max && !mask) {
- pr_debug("Ignoring bad slot %u (max: %u)\n", offset, si->max);
+ pr_debug("Ignoring bad slot %u (max: %lu)\n", offset, si->max);
return 0;
}
/*
@@ -1170,12 +1170,12 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct *si,
}
/* Try use a new cluster for current CPU and allocate from it. */
-static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
- struct swap_cluster_info *ci,
- struct folio *folio,
- unsigned long offset)
+static unsigned long alloc_swap_scan_cluster(struct swap_info_struct *si,
+ struct swap_cluster_info *ci,
+ struct folio *folio,
+ unsigned long offset)
{
- unsigned int next = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;
+ unsigned long next = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;
unsigned long start = ALIGN_DOWN(offset, SWAPFILE_CLUSTER);
unsigned int order = likely(folio) ? folio_order(folio) : 0;
unsigned long end = start + SWAPFILE_CLUSTER;
@@ -1245,12 +1245,12 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
return found;
}
-static unsigned int alloc_swap_scan_list(struct swap_info_struct *si,
- struct list_head *list,
- struct folio *folio,
- bool scan_all)
+static unsigned long alloc_swap_scan_list(struct swap_info_struct *si,
+ struct list_head *list,
+ struct folio *folio,
+ bool scan_all)
{
- unsigned int found = SWAP_ENTRY_INVALID;
+ unsigned long found = SWAP_ENTRY_INVALID;
do {
struct swap_cluster_info *ci = isolate_lock_cluster(si, list);
@@ -1267,8 +1267,8 @@ static unsigned int alloc_swap_scan_list(struct swap_info_struct *si,
return found;
}
-static unsigned int alloc_swap_scan_dynamic(struct swap_info_struct *si,
- struct folio *folio)
+static unsigned long alloc_swap_scan_dynamic(struct swap_info_struct *si,
+ struct folio *folio)
{
struct swap_cluster_info_dynamic *ci_dyn;
struct swap_cluster_info *ci;
@@ -1392,7 +1392,7 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
{
struct swap_cluster_info *ci;
unsigned int order = likely(folio) ? folio_order(folio) : 0;
- unsigned int offset = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;
+ unsigned long offset = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;
/*
* Swapfile is not block device so unable
@@ -3488,10 +3488,10 @@ static int unuse_mm(struct mm_struct *mm, unsigned int type)
* Return 0 if there are no inuse entries after prev till end of
* the map.
*/
-static unsigned int find_next_to_unuse(struct swap_info_struct *si,
- unsigned int prev)
+static unsigned long find_next_to_unuse(struct swap_info_struct *si,
+ unsigned long prev)
{
- unsigned int i;
+ unsigned long i;
unsigned long swp_tb;
/*
@@ -3529,7 +3529,8 @@ static int try_to_unuse(unsigned int type)
struct swap_io_ctx ctx;
swp_entry_t entry, vswap_entry;
unsigned long swp_tb;
- unsigned int i, j;
+ unsigned long i;
+ unsigned int j;
if (!swap_usage_in_pages(si))
goto success;
@@ -3964,7 +3965,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
struct file *swap_file, *victim;
struct address_space *mapping;
struct inode *inode;
- unsigned int maxpages;
+ unsigned long maxpages;
int err, found = 0;
if (!capable(CAP_SYS_ADMIN))
@@ -4386,12 +4387,8 @@ static unsigned long read_swap_header(struct swap_info_struct *si,
pr_warn("Truncating oversized swap area, only using %luk out of %luk\n",
K(maxpages), K(last_page));
}
- if (maxpages > last_page) {
+ if (maxpages > last_page)
maxpages = last_page + 1;
- /* p->max is an unsigned int: don't overflow it */
- if ((unsigned int)maxpages == 0)
- maxpages = UINT_MAX;
- }
if (!maxpages)
return 0;
@@ -4630,7 +4627,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
goto bad_swap_unlock_inode;
}
if (si->pages != si->max - 1) {
- pr_err("swap:%u != (max:%u - 1)\n", si->pages, si->max);
+ pr_err("swap:%lu != (max:%lu - 1)\n", si->pages, si->max);
error = -EINVAL;
goto bad_swap_unlock_inode;
}
@@ -4718,7 +4715,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
/* Sets SWP_WRITEOK, resurrect the percpu ref, expose the swap device */
enable_swap_info(si);
- pr_info("Adding %uk swap on %s. Priority:%d extents:%d across:%lluk %s%s%s%s\n",
+ pr_info("Adding %luk swap on %s. Priority:%d extents:%d across:%lluk %s%s%s%s\n",
K(si->pages), name->name, si->prio, nr_extents,
K((unsigned long long)span),
(si->flags & SWP_SOLIDSTATE) ? "SS" : "",
@@ -4906,8 +4903,13 @@ static int __init vswap_init(void)
return 0;
}
+ /*
+ * Cap at the cluster_info_pool xarray's allocator limit
+ * (XA_FLAGS_ALLOC stores IDs in u32, tops out at UINT_MAX).
+ */
maxpages = min(swapfile_maximum_size,
- ALIGN_DOWN((unsigned long)UINT_MAX, SWAPFILE_CLUSTER));
+ ALIGN_DOWN((unsigned long)UINT_MAX * SWAPFILE_CLUSTER,
+ SWAPFILE_CLUSTER));
si->flags |= SWP_VSWAP | SWP_SOLIDSTATE | SWP_WRITEOK;
si->ops = &vswap_ops;
si->bdev = NULL;
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [syzbot ci] Re: Virtual Swap Space (Swap Table Edition)
2026-08-06 18:42 [PATCH v3 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
` (10 preceding siblings ...)
2026-08-06 18:42 ` [PATCH v3 11/11] mm, swap: widen swap_info_struct max/pages to unsigned long Nhat Pham
@ 2026-08-07 5:26 ` syzbot ci
2026-08-07 7:21 ` Chris Li
2026-08-07 9:07 ` [PATCH v3 00/11] " Chris Li
12 siblings, 1 reply; 17+ messages in thread
From: syzbot ci @ 2026-08-07 5:26 UTC (permalink / raw)
To: akpm, axelrasmussen, baohua, baoquan.he, cgroups, chengming.zhou,
chrisl, corbet, david, gourry, hannes, haowenchao22, kasong,
kernel-team, liam, linux-doc, linux-kernel, linux-mm, ljs, mhocko,
muchun.song, nphamcs, qi.zheng, riel, roman.gushchin, rppt,
shakeel.butt, shikemeng, surenb, vbabka, weixugc, yosry,
youngjun.park, yuanchu
Cc: syzbot, syzkaller-bugs
syzbot ci has tested the following series
[v3] Virtual Swap Space (Swap Table Edition)
https://lore.kernel.org/all/20260806184254.3790858-1-nphamcs@gmail.com
* [PATCH v3 01/11] mm, swap: add virtual swap device infrastructure
* [PATCH v3 02/11] mm, swap: support zswap and zeroswap as vswap backends
* [PATCH v3 03/11] mm, swap: prepare the swap IO path for vswap
* [PATCH v3 04/11] mm, swap: support physical swap as a vswap backend
* [PATCH v3 05/11] mm, swap: enable THP swapin for vswap entries
* [PATCH v3 06/11] mm, swap: write back vswap zswap entries to physical swap
* [PATCH v3 07/11] mm, swap: reclaim physical slots backing cache-only vswap entries
* [PATCH v3 08/11] mm, swap: only charge physical swap entries
* [PATCH v3 09/11] mm, swap: add debugfs counters for vswap
* [PATCH v3 10/11] mm, swap: defer memcg_table allocation for physical swap clusters
* [PATCH v3 11/11] mm, swap: widen swap_info_struct max/pages to unsigned long
and found the following issue:
KASAN: null-ptr-deref Read in swap_entry_backend_has_flag
Full report is available here:
https://ci.syzbot.org/series/e7da1097-3230-4e50-80cb-1dfafeebea40
***
KASAN: null-ptr-deref Read in swap_entry_backend_has_flag
tree: linux-next
URL: https://kernel.googlesource.com/pub/scm/linux/kernel/git/next/linux-next
base: bacc32cc7de65ffff70080a48eb294f89e434d5e
arch: amd64
compiler: Debian clang version 22.1.8 (++20260613092233+e80beda6e255-1~exp1~20260613092250.77), Debian LLD 22.1.8
config: https://ci.syzbot.org/builds/4f943657-9481-4637-9b1f-9be1bff18f92/config
syz repro: https://ci.syzbot.org/findings/f6c4f1ab-951c-4759-b397-e22c23c3bf32/syz_repro
==================================================================
BUG: KASAN: null-ptr-deref in instrument_atomic_read include/linux/instrumented.h:82 [inline]
BUG: KASAN: null-ptr-deref in atomic_long_read include/linux/atomic/atomic-instrumented.h:3188 [inline]
BUG: KASAN: null-ptr-deref in __vtable_get mm/vswap.h:147 [inline]
BUG: KASAN: null-ptr-deref in vswap_to_phys mm/vswap.h:197 [inline]
BUG: KASAN: null-ptr-deref in swap_entry_backend_has_flag+0xfe/0x220 mm/vswap.h:441
Read of size 8 at addr 0000000000000000 by task syz.2.20/5818
CPU: 1 UID: 0 PID: 5818 Comm: syz.2.20 Not tainted syzkaller #0 PREEMPT(full)
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0xe8/0x150 lib/dump_stack.c:120
kasan_report+0x117/0x150 mm/kasan/report.c:595
check_region_inline mm/kasan/generic.c:-1 [inline]
kasan_check_range+0x264/0x2c0 mm/kasan/generic.c:200
instrument_atomic_read include/linux/instrumented.h:82 [inline]
atomic_long_read include/linux/atomic/atomic-instrumented.h:3188 [inline]
__vtable_get mm/vswap.h:147 [inline]
vswap_to_phys mm/vswap.h:197 [inline]
swap_entry_backend_has_flag+0xfe/0x220 mm/vswap.h:441
do_swap_page+0x3c0/0x5620 mm/memory.c:4975
__collapse_huge_page_swapin mm/khugepaged.c:1202 [inline]
collapse_huge_page mm/khugepaged.c:1325 [inline]
mthp_collapse mm/khugepaged.c:1524 [inline]
collapse_scan_pmd mm/khugepaged.c:1786 [inline]
collapse_single_pmd+0x24c6/0x3da0 mm/khugepaged.c:2803
madvise_collapse+0x2cf/0x790 mm/khugepaged.c:3237
madvise_vma_behavior+0x115f/0x4170 mm/madvise.c:1363
madvise_walk_vmas+0x576/0xb00 mm/madvise.c:1712
madvise_do_behavior+0x385/0x540 mm/madvise.c:1907
do_madvise+0x327/0x3a0 mm/madvise.c:2005
__do_sys_madvise mm/madvise.c:2014 [inline]
__se_sys_madvise mm/madvise.c:2012 [inline]
__x64_sys_madvise+0xa6/0xc0 mm/madvise.c:2012
do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
do_syscall_64+0x174/0x580 arch/x86/entry/syscall_64.c:94
entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7fbcad79e019
Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 e8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007fbcae5c4028 EFLAGS: 00000246 ORIG_RAX: 000000000000001c
RAX: ffffffffffffffda RBX: 00007fbcada25fa0 RCX: 00007fbcad79e019
RDX: 0000000000000019 RSI: 0000000000c00000 RDI: 0000200000000000
RBP: 00007fbcad83500c R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007fbcada26038 R14: 00007fbcada25fa0 R15: 00007fff095fa0b8
</TASK>
==================================================================
***
If these findings have caused you to resend the series or submit a
separate fix, please add the following tag to your commit message:
Tested-by: syzbot@syzkaller.appspotmail.com
---
This report is generated by a bot. It may contain errors.
syzbot ci engineers can be reached at syzkaller@googlegroups.com.
To test a patch for this bug, please reply with `#syz test`
(should be on a separate line).
The patch should be attached to the email.
Note: arguments like custom git repos and branches are not supported.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [syzbot ci] Re: Virtual Swap Space (Swap Table Edition)
2026-08-07 5:26 ` [syzbot ci] Re: Virtual Swap Space (Swap Table Edition) syzbot ci
@ 2026-08-07 7:21 ` Chris Li
0 siblings, 0 replies; 17+ messages in thread
From: Chris Li @ 2026-08-07 7:21 UTC (permalink / raw)
To: Nhat Pham
Cc: akpm, axelrasmussen, baohua, baoquan.he, cgroups, chengming.zhou,
corbet, david, gourry, hannes, haowenchao22, kasong, kernel-team,
liam, linux-doc, linux-kernel, linux-mm, ljs, mhocko, muchun.song,
qi.zheng, riel, roman.gushchin, rppt, shakeel.butt, shikemeng,
surenb, vbabka, weixugc, yosry, youngjun.park, yuanchu, syzbot,
syzkaller-bugs
Hi Nhat,
On Thu, Aug 6, 2026 at 10:26 PM syzbot ci
<syzbot+cif942a042aaa8793c@syzkaller.appspotmail.com> wrote:
>
> syzbot ci has tested the following series
>
> [v3] Virtual Swap Space (Swap Table Edition)
> https://lore.kernel.org/all/20260806184254.3790858-1-nphamcs@gmail.com
> * [PATCH v3 01/11] mm, swap: add virtual swap device infrastructure
> * [PATCH v3 02/11] mm, swap: support zswap and zeroswap as vswap backends
> * [PATCH v3 03/11] mm, swap: prepare the swap IO path for vswap
> * [PATCH v3 04/11] mm, swap: support physical swap as a vswap backend
> * [PATCH v3 05/11] mm, swap: enable THP swapin for vswap entries
> * [PATCH v3 06/11] mm, swap: write back vswap zswap entries to physical swap
> * [PATCH v3 07/11] mm, swap: reclaim physical slots backing cache-only vswap entries
> * [PATCH v3 08/11] mm, swap: only charge physical swap entries
> * [PATCH v3 09/11] mm, swap: add debugfs counters for vswap
> * [PATCH v3 10/11] mm, swap: defer memcg_table allocation for physical swap clusters
> * [PATCH v3 11/11] mm, swap: widen swap_info_struct max/pages to unsigned long
>
> and found the following issue:
> KASAN: null-ptr-deref Read in swap_entry_backend_has_flag
>
> Full report is available here:
> https://ci.syzbot.org/series/e7da1097-3230-4e50-80cb-1dfafeebea40
Please take a look at the syzbot report and let us know what you think.
Chris
>
> ***
>
> KASAN: null-ptr-deref Read in swap_entry_backend_has_flag
>
> tree: linux-next
> URL: https://kernel.googlesource.com/pub/scm/linux/kernel/git/next/linux-next
> base: bacc32cc7de65ffff70080a48eb294f89e434d5e
> arch: amd64
> compiler: Debian clang version 22.1.8 (++20260613092233+e80beda6e255-1~exp1~20260613092250.77), Debian LLD 22.1.8
> config: https://ci.syzbot.org/builds/4f943657-9481-4637-9b1f-9be1bff18f92/config
> syz repro: https://ci.syzbot.org/findings/f6c4f1ab-951c-4759-b397-e22c23c3bf32/syz_repro
>
> ==================================================================
> BUG: KASAN: null-ptr-deref in instrument_atomic_read include/linux/instrumented.h:82 [inline]
> BUG: KASAN: null-ptr-deref in atomic_long_read include/linux/atomic/atomic-instrumented.h:3188 [inline]
> BUG: KASAN: null-ptr-deref in __vtable_get mm/vswap.h:147 [inline]
> BUG: KASAN: null-ptr-deref in vswap_to_phys mm/vswap.h:197 [inline]
> BUG: KASAN: null-ptr-deref in swap_entry_backend_has_flag+0xfe/0x220 mm/vswap.h:441
> Read of size 8 at addr 0000000000000000 by task syz.2.20/5818
>
> CPU: 1 UID: 0 PID: 5818 Comm: syz.2.20 Not tainted syzkaller #0 PREEMPT(full)
> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
> Call Trace:
> <TASK>
> dump_stack_lvl+0xe8/0x150 lib/dump_stack.c:120
> kasan_report+0x117/0x150 mm/kasan/report.c:595
> check_region_inline mm/kasan/generic.c:-1 [inline]
> kasan_check_range+0x264/0x2c0 mm/kasan/generic.c:200
> instrument_atomic_read include/linux/instrumented.h:82 [inline]
> atomic_long_read include/linux/atomic/atomic-instrumented.h:3188 [inline]
> __vtable_get mm/vswap.h:147 [inline]
> vswap_to_phys mm/vswap.h:197 [inline]
> swap_entry_backend_has_flag+0xfe/0x220 mm/vswap.h:441
> do_swap_page+0x3c0/0x5620 mm/memory.c:4975
> __collapse_huge_page_swapin mm/khugepaged.c:1202 [inline]
> collapse_huge_page mm/khugepaged.c:1325 [inline]
> mthp_collapse mm/khugepaged.c:1524 [inline]
> collapse_scan_pmd mm/khugepaged.c:1786 [inline]
> collapse_single_pmd+0x24c6/0x3da0 mm/khugepaged.c:2803
> madvise_collapse+0x2cf/0x790 mm/khugepaged.c:3237
> madvise_vma_behavior+0x115f/0x4170 mm/madvise.c:1363
> madvise_walk_vmas+0x576/0xb00 mm/madvise.c:1712
> madvise_do_behavior+0x385/0x540 mm/madvise.c:1907
> do_madvise+0x327/0x3a0 mm/madvise.c:2005
> __do_sys_madvise mm/madvise.c:2014 [inline]
> __se_sys_madvise mm/madvise.c:2012 [inline]
> __x64_sys_madvise+0xa6/0xc0 mm/madvise.c:2012
> do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
> do_syscall_64+0x174/0x580 arch/x86/entry/syscall_64.c:94
> entry_SYSCALL_64_after_hwframe+0x77/0x7f
> RIP: 0033:0x7fbcad79e019
> Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 e8 ff ff ff f7 d8 64 89 01 48
> RSP: 002b:00007fbcae5c4028 EFLAGS: 00000246 ORIG_RAX: 000000000000001c
> RAX: ffffffffffffffda RBX: 00007fbcada25fa0 RCX: 00007fbcad79e019
> RDX: 0000000000000019 RSI: 0000000000c00000 RDI: 0000200000000000
> RBP: 00007fbcad83500c R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
> R13: 00007fbcada26038 R14: 00007fbcada25fa0 R15: 00007fff095fa0b8
> </TASK>
> ==================================================================
>
>
> ***
>
> If these findings have caused you to resend the series or submit a
> separate fix, please add the following tag to your commit message:
> Tested-by: syzbot@syzkaller.appspotmail.com
>
> ---
> This report is generated by a bot. It may contain errors.
> syzbot ci engineers can be reached at syzkaller@googlegroups.com.
>
> To test a patch for this bug, please reply with `#syz test`
> (should be on a separate line).
>
> The patch should be attached to the email.
> Note: arguments like custom git repos and branches are not supported.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH v3 00/11] Virtual Swap Space (Swap Table Edition)
2026-08-06 18:42 [PATCH v3 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
` (11 preceding siblings ...)
2026-08-07 5:26 ` [syzbot ci] Re: Virtual Swap Space (Swap Table Edition) syzbot ci
@ 2026-08-07 9:07 ` Chris Li
12 siblings, 0 replies; 17+ messages in thread
From: Chris Li @ 2026-08-07 9:07 UTC (permalink / raw)
To: Nhat Pham
Cc: akpm, kasong, hannes, mhocko, roman.gushchin, shakeel.butt, yosry,
david, muchun.song, shikemeng, baoquan.he, baohua, youngjun.park,
chengming.zhou, ljs, liam, vbabka, rppt, surenb, qi.zheng,
axelrasmussen, yuanchu, weixugc, riel, gourry, haowenchao22,
corbet, kernel-team, linux-mm, linux-kernel, linux-doc, cgroups
Hi Nhat,
First of all, thank you very much for addressing the feedback
regarding the swap metadata size concern and for stopping the
punishment of zram usage. I'm unsure how to proceed with your earlier
VS series (before swap table version V2), given the previous concerns.
I was a bit nervous when you reverted the swap table and replaced it
with something that performed worse in earlier series. I'm not
attached to the swap table. The performance regression for existing
use cases simply doesn't make sense to me. Thanks again that is no
longer the case.
On Thu, Aug 6, 2026 at 11:43 AM Nhat Pham <nphamcs@gmail.com> wrote:
>
> Changelog:
> * v2 [v2] -> v3:
> * Rebased onto current mm-unstable.
> * Add a runtime vm.vswap_enabled sysctl and CONFIG_VSWAP_DEFAULT_ON
> to gate vswap allocation.
> * More cleanups and small bug fixes.
> * Split THP swapin enablement into its own patch (patch 5).
> * Add production workload benchmark results, and drop RFC tag.
> * v1 [v1] -> v2:
> * Rebased to a newer mm-unstable tip.
> * Fix a bunch of assorted issues (incorrect zswap store failure
> rollback, vswap_init() failure handling, rmap-encoding collision,
> etc.) and clean up the code (rename a bunch of functions to
> more closely follow existing patterns, etc.).
> * Some more code clean up and simplification: some renamings to more
> closely follow existing patterns, move vswap backing check to
> __swap_cache_add_check, store zero state in the swap_table for
> vswap entries, etc.. Many of these are proposed by Kairui Song
> in [1].
> * Defer memcg_table allocation on physical clusters until the first
> vswap-backing slot installs. Saves ~512 bytes per physical cluster
> that only serves vswap-backing slots (this is the new patch 8).
> * Widen swap_info_struct->max and ->pages (and the swapoff unuse-path
> index) so vswap supports ~8 PB of swap space (this is the new
> patch 9).
> * Split the physical-swap-backend patch into three for reviewability:
> the core backend (patch 3), zswap writeback to physical swap
> (patch 4), and reclaim of cache-only physical slots (patch 5). No
> functional change.
> * Add kerneldoc for the vswap API.
> * Add some benchmark numbers for zswap case.
>
>
> I. Context and Motivation
> =========================
>
> Currently, when an anon page is swapped out, a slot in a backing swap
> device is allocated and stored in the page table entries that refer to
> the original page. This slot is also used as the "key" to find the
> swapped out content, as well as the index to swap data structures, such
> as the swap cache, or the swap cgroup mapping. Tying a swap entry to its
> backing slot in this way is performant and efficient when swap is purely
> just disk space, and swapoff is rare.
>
> However, the advent of many swap optimizations has exposed major
> drawbacks of this design. The first problem is that we occupy a physical
> slot in the swap space, even for pages that are NEVER expected to hit
> the disk: pages compressed and stored in the zswap pool, zero-filled
> pages, or pages rejected by both of these optimizations when zswap
> writeback is disabled. This is arguably the central shortcoming of
> zswap:
> * Resource-wise, it is hugely wasteful in terms of disk usage. At Meta,
> we size swapfile in the order of 25-50% of host RAM, depending on flash
> availaiblity. This is a lot of flash for a fleet of our size, and
> with universal zswap enablement, most of this is wasted for zswap
> entries.
>
> * In deployments when no disk space can be afforded for swap (such as
> mobile and embedded devices), users cannot adopt zswap, and are forced
> to use zram. This is confusing for users, and creates extra burdens
> for developers, having to develop and maintain similar features for
> two separate swap backends (writeback, cgroup charging, THP support,
> etc.). For instance, see the discussion in [2].
>
> * Tying zswap (and more generally, other in-memory swap backends) to
> the current physical swapfile infrastructure makes zswap implicitly
> statically sized. This does not make sense, as unlike disk swap, in
> which we consume a limited resource (disk space or swapfile space) to
> save another resource (memory), zswap consumes the same resource it is
> saving (memory). The more we zswap, the more memory we have available,
> not less. We are not rationing a limited resource when we limit
> the size of the zswap pool, but rather we are capping the resource
> (memory) saving potential of zswap. Under memory pressure, using
> more zswap is almost always better than the alternative (disk IOs, or
> even worse, OOMs), and dynamically sizing the zswap pool on demand
> allows the system to flexibly respond to these precarious scenarios.
>
> * Operationally, static provisioning the swapfile for zswap poses
> significant challenges, because the sysadmin has to prescribe how
> much swap is needed a priori, for each combination of
> (memory size x disk space x workload usage). It is even more
> complicated when we take into account the variance of memory
> compression, which changes the reclaim dynamics (and as a result,
> swap space size requirement). The problem is further exacerbated for
> users who rely on swap utilization (and exhaustion) as an OOM signal.
>
> All of these factors make it very difficult to configure the swapfile
> for zswap: too small of a swapfile and we risk preventable OOMs and
> limit the memory saving potentials of zswap; too big of a swapfile
> and we waste disk space and memory due to swap metadata overhead.
> This dilemma becomes more drastic in high memory systems, which can
> have up to TBs worth of memory.
>
> Swap virtualization is the answer to these issues, with three properties:
>
> 1. Decoupled backends. For zswap in particular, this means we eliminate
> the unused storage space, and allows zswap to be used in systems that
> do not have enough storage capacity for physical swap (without having
> to resort to silly hacks). Zero-filled swap pages and swap-cache-only
> folios also benefit here.
>
> 2. Dynamic swap space. Since virtual swap is not tied to any physical
> resource, we can make it infinite and dynamically grow it on demand.
> This massively simplifies operational provisioning, and increases the
> utilization of compressed swap backends (zswap). Dynamicity also
> reduces overhead on unused swap capacity.
>
> 3. Efficient backend transfer. The virtualization scheme should not
> introduce PTE/rmap walking overhead for backend transfer. This
> is crucial for systems that want to support multiple swap backends
> in a tiering fashion (for e.g zswap -> disk swap).
>
> For more historical contexts and references, please take a look at
> the cover letter of the older vswap submissions ([3] and [v2]).
>
> II. Design
> ==========
>
> When we compile kernel with CONFIG_VSWAP, a special vswap device is
Does the CONFIG_VSWAP only make sense for zswap right now? No other
swap usage can benifit from CONFIG_VSWAP.
> allocated at boot time, and all swapped out pages try to allocate from
> this device first, falling back to a physical swap device on failure.
Does it create a new user visible behavior change where users don't
need to swapon and can start using VSWAP for zswap?
That is a user-visible behavior change and we need to be more cautious about it.
I think a system should not use zswap or any type of swap if no device
is swapped on.
Have vm.vswap_enabled is no the answer to address the new API change
because existing distro that use fstab to control swap will need to
jump through hooks.
Previously, using fstab to control was at least consistent for all swap types.
> Routing can also be turned off at runtime with the vm.vswap_enabled
> sysctl, which defaults to 0 unless CONFIG_VSWAP_DEFAULT_ON=y. It is
> allocation-only: new swapouts go straight to physical swap, while
> entries already backed by vswap keep being served and drain as they
> are faulted back in or freed.
>
> These swap entries can subsequently acquire backend on-demand, such as
What do "These" refer to? Are they entries already backed by vswap?
> a zswap entry, or a slot on a physical swap device.
>
> We repurpose much of the existing swap_table infrastructure and
> swapfile allocator for this new vswap device, with two notable
> differences:
> * Clusters are dynamically allocated on demand and managed through
> an xarray. This in turn allows us to avoid static provisioning and
> let swap space grow dynamically.
>
> * Each cluster of this new vswap device has a virtual_table that stores
> the backend information of the entries in the cluster (see below).
>
> Diagrams:
>
> Case 1: vswap entry (virtualized)
>
> PTE swap_cluster_info_dynamic
> vswap_entry +---------------------------------+
> (swp_entry_t) ------>| swap_cluster_info (ci) |
> | +----------------------------+ |
> | | swap_table | |
> | | PFN / Shadow | |
> | | memcg_table | |
> | | count,flags,order | |
> | | lock, list | |
> | +----------------------------+ |
> | |
> | virtual_table |
> | +----------------------------+ |
> | | NONE | |
> | | SWAPFILE(swp_entry_t) | |
> | | ZSWAP(struct zswap_entry*) | |
> | +----------------------------+ |
> +---------------------------------+
> |
> | SWAPFILE resolves to
> v
> PHYSICAL CLUSTER (swap_cluster_info)
> +--------------------------+
> | swap_table per-slot: |
> | NULL - free |
> | PFN - cached folio |
> | Shadow - swapped out |
> | Pointer- vswap rmap |
> | Bad - unusable |
> | |
> | Vswap-backing slot: |
> | Pointer(C|swp_entry_t) |
> | rmap back to vswap |
> +--------------------------+
>
> Case 2: direct-mapped physical entry (no vswap)
>
> PTE PHYSICAL CLUSTER (swap_cluster_info)
> phys_entry +--------------------------+
> (swp_entry_t) ------>| swap_table per-slot: |
> | NULL - free |
> | PFN - cached folio |
> | Shadow - swapped out |
> | Bad - unusable |
> +--------------------------+
>
> struct swap_cluster_info_dynamic {
> struct swap_cluster_info ci; /* swap_table, lock, etc. */
> unsigned int index; /* position in xarray */
> struct rcu_head rcu; /* kfree_rcu deferred free */
> atomic_long_t *virtual_table; /* backend info, 8 B/slot */
> };
No a big fan of this two personality data structure thing depending on
whether it is VS or not.
If ci is the common part, I prefer to keep it separate and leave it alone.
Also the extension is too vswap specific, it does not apply to other
swap device types that might need their own private extension.
You can take the VFS layer as an example. There is a VFS layer generic
inode, which is common and shared by all file systems. And then you
have filesystem-specific inodes as extensions, e.g. ext4_inode. The
ext4_inode does not contain VFS inode. You don't see VFS having a code
path like: if it is ext4, get the inode this way, else if f2fs, get
the inode that way.
In the first swap abstraction LPC talk, where I co-hosted with Yosry,
I talked about the alternative approach: "VFS-like swap layers". That
is exactly what I have in mind. We are getting very close to
fulfilling that promise via swap ops and xswap extension interfaces.
I think implementing the generic interface first is simpler than
implementing the non-generic vswap interface, ripping it out to
replace it with a generic interface, and then putting back the generic
modified version of vswap.
If the two personality vswap xarray lookup gets in first, it will
ultimately take more work to achieve the desired VFS-like extendable
swap operations.
I am happy to spend some time working with you to discuss the generic
adopted version of vswap, if you are open to it. Or if you don't want
to waste time on it. I can have someone else or myself come up with
the generic adopted version of vswap for you to review, which I prefer
less.
Another piece of feedback is to please come up with a plan to submit
your vswap changes piecemeal rather than as one long series. There is
a lot of change like swap charging, that deserves a separate
discussion before it gets merged. Look, the swap table changes took
four phases. Each phase achieved a smaller milestone, with four of
them ultimately reaching the finish line. I wish vswap had a similar
piecemeal plan.
Sorry I have to crash now, to be continued...
> Each vswap cluster (swap_cluster_info_dynamic) extends the classic
> swap_cluster_info struct with a virtual_table array that stores the
> backend information for each virtual swap entry in the cluster. Each
> entry is tag-encoded in the low 3 bits to indicate the backend type:
>
> NONE: |----- 0000 ------|000| free / unbacked
> SWAPFILE: |- type:5,off:56 -|001| on a physical swapfile
> ZSWAP: |--- zswap_entry* |010| compressed in zswap
>
> Other design highlights:
>
> * Note that for the vswap device, we have merged the zswap xarray tree
> with the swapfile-level clusters. This means that for zswap only users,
> we have negligible extra space overhead.
>
> * Both vswap entries (Case 1) and directly-mapped physical entries
> (Case 2) coexist as first-class citizens. When CONFIG_VSWAP=n the
> vswap paths compile out.
>
> * Backend transitions in the virtual_table are synchronized through the
> swap cache and the folio lock - the same mechanism that already
> serializes ordinary swap operations (swapin, swapout, zswap
> writeback, swap cache reclaim). IOW, we can only assume that the
> backend of a vswap entry is stable through swap cache/folio lock.
> Looking at the backend without this should be done at best for
> optimization purposes, as there is no guarantee that the backend
> will not change under the observer.
>
> * Pointer-tagged swap_table entries on physical clusters provide the
> rmap (physical -> virtual) lookup.
>
> * Virtual swap slots not backed by physical swap are not charged to
> memcg swap counters - only physical backing is charged (I made the
> case for this in [4]).
>
>
> III. Benchmarks
> ===============
>
> Note that the goal is not to match vswap performance with baseline on
> every single case yet - we still maintain !CONFIG_VSWAP setup. We can
> optimize further once we have landed this new feature.
>
> A. Production Workload: Instagram
> =================================
>
> To test vswap's stability and performance, I ran an A/B experiment on
> Instagram (django) workload, with zswap as the swap backend. On these
> hosts, the swapfiles' size is 50% of RAM.
>
> Compared to baseline, vswap gives:
>
> * On par request throughput.
> * Lower request serving latency (by about 1-3%).
> * Lower memory pressure in the system service cgroups running alongside
> the workload. PSI-based proactive reclaimer can therefore recover more
> from them, lowering their overall memory footprint, allowing the main
> workload to expand.
> * Elimination of swapfile footprint for all zswap users in the host.
>
> B. Semi-synthetic Workloads (memhog, usemem, kernel build)
> ==========================================================
>
> All values are mean +/- standard deviation across rounds.
>
> Test system: x86_64, 52 cores, 64 GB swapfile for all 3 benchmarks.
> Swap backend: zswap (zstd) with the traditional active/inactive LRU. We
> focus on zswap here because it is the motivating use case for vswap.
>
> For each benchmark, we test 3 kernels:
> * Baseline: mm-unstable, no vswap patches.
> * VSS off: vswap series applied, CONFIG_VSWAP not set, to verify that
> there is no regression to existing swap paths when we disable vswap.
> * VSS on: vswap series applied, CONFIG_VSWAP=y.
>
> 1. Memhog: single-threaded, 48GB allocation on a host with 16GB RAM,
> 20 rounds.
>
> Baseline VSS off VSS on
> real (s) 131.71 +/- 13.54 132.47 +/- 10.10 120.56 +/- 15.37
> sys (s) 114.05 +/- 13.03 115.11 +/- 9.76 103.73 +/- 15.06
> user (s) 10.86 +/- 0.13 10.97 +/- 0.10 10.87 +/- 0.11
> delta real - +0.6% -8.5%
> delta sys - +0.9% -9.1%
>
> Dropping the best and the worst round to reduce variance:
>
> memhog Baseline VSS off VSS on
> real (s) 130.24 +/- 8.56 131.51 +/- 5.82 119.39 +/- 12.06
> sys (s) 112.58 +/- 7.88 114.26 +/- 5.74 102.63 +/- 11.83
> user (s) 10.86 +/- 0.14 10.97 +/- 0.10 10.86 +/- 0.10
> delta real - +1.0% -8.3%
> delta sys - +1.5% -8.8%
>
>
> 2. Usemem single-threaded: 56GB allocation on a host with 32GB RAM,
> 16 rounds.
>
> Baseline VSS off VSS on
> real (s) 177.14 +/- 7.34 178.20 +/- 5.12 175.83 +/- 6.96
> sys (s) 125.30 +/- 7.47 125.19 +/- 5.18 124.09 +/- 7.07
> tput (KB/s) 390668 +/- 16840 387878 +/- 11769 390921 +/- 15798
> free (ms) 7739 +/- 125 7734 +/- 120 6572 +/- 121
> delta real - +0.6% -0.7%
> delta sys - -0.1% -1.0%
> delta tput - -0.7% +0.1%
> delta free - -0.1% -15.1%
>
> 3. Kernel build: 52 workers (one per processor), memory.max=3GB, 10 rounds.
>
> Baseline VSS off VSS on
> real (s) 168.13 +/- 0.77 168.46 +/- 0.45 167.75 +/- 0.65
> sys (s) 772.49 +/- 19.77 781.82 +/- 26.32 763.60 +/- 33.02
> user (s) 5128.41 +/- 1.31 5130.64 +/- 1.67 5130.74 +/- 1.66
> delta real - +0.2% -0.2%
> delta sys - +1.2% -1.1%
> delta user - +0.0% +0.0%
>
>
> For zswap backend, vswap outperforms baseline on usemem freeing, and
> memhog benchmark, and is on par with baseline on the rest.
>
> In the RFC v2 ([v2]), I put out several theories for this. I have
> done some prototyping to isolate effects, and it turns out the
> performance wins come primarily from the elimination of zswap's
> xarray and the merging of zswap's metadata to swap device's cluster.
> Several code paths are optimized thanks to this - for instance,
> in swap_range_free(), we call zswap_invalidate() once for each entry the
> range, resulting in multiple xarray tree walks. With vswap, we perform
> one single xarray walk to grab a 512-slot cluster, then performs a
> flat array scan to free zswap metadata. Similar wins can be observed
> in Baoquan's optimization ([8]), which also optimizes away the zswap tree.
>
> IV. References
> ==============
>
> [v1]: https://lore.kernel.org/all/20260528212955.1912856-1-nphamcs@gmail.com/
> [v2]: https://lore.kernel.org/all/20260612193738.2183968-1-nphamcs@gmail.com/
> [1]: https://lore.kernel.org/all/CAMgjq7BhOn48xEyC=2j837R7qddfjeBVHMiRqdx8no4ZEBpBLg@mail.gmail.com/
> [2]: https://lore.kernel.org/all/Zqe_Nab-Df1CN7iW@infradead.org/
> [3]: https://lore.kernel.org/all/20260505153854.1612033-1-nphamcs@gmail.com/
> [4]: https://lore.kernel.org/linux-mm/CAKEwX=P4syV38jAVCWq198r2OHXXc=xA-fx1dk6+qYef6yzxWQ@mail.gmail.com/
> [5]: https://lore.kernel.org/all/CAKEwX=P50av2rfocpsqZoDQowZ=EEhQ-5vj5tBykbNz8vtKTzA@mail.gmail.com/
> [6]: https://lore.kernel.org/all/20260727135029.1059441-1-baoquan.he@linux.dev/
> [7]: https://lore.kernel.org/all/20260220-swap-table-p4-v1-15-104795d19815@tencent.com/
> [8]: https://lore.kernel.org/all/20260707073215.72183-1-baoquan.he@linux.dev/
>
>
> Appendix: Alternative Designs and Improvements
> ==============================================
>
> A. Vmalloc Data Structure:
> ==========================
>
> This is a promising alternative to the xarray data structure, reducing
> the indirection overhead. The initial version relies on userspace knob to
> trigger swap address space growth - I have commented on why this is shaky
> in [5].
>
> Baoquan has followed-up with a new version (see [6]) that should give us
> kernel-driven dynamic growth and (tail-only) shrink. This seems sufficient
> for vswap use case, AFAICT - but seems like it would need a couple more
> versions to finalize the design.
>
> I think it is better to proceed with the xarray data structure first,
> especially since we already see some positive signals on performance
> by storing zswap metadata in a per-cluster flat table. With vswap landed,
> we will have a concrete setup to show vmalloc data structure's wins.
>
> B. Moving the backend table to struct swap_cluster_info
> =======================================================
>
> Another approach Baoquan and I discussed on is to structure vswap patch
> series as follows:
>
> 1. Moving vtable (renamed to something more generic) to swap cluster,
> which removes the xarray.
>
> 2. Once vswap is introduced, we simply use this field to store the
> backend.
>
> I have a prototype for this, but I ended up scrapping the whole thing, for
> the following reasons:
>
> 1. It ended up being even more code than what I sent out here - most of
> which touches the non-vswap code paths, which we either want to leave
> alone (generic swap logic) or want to rip out wholesale down the line
> (zswap).
>
> 2. There are several fields that are ONLY needed for the vswap clusters
> (for instance, rcu_head and index). Shoving them into the shared struct
> swap_cluster_info imposes memory and mental overhead for non-vswap
> clusters and users.
>
> We can avoid this by simply moving it to the wrapper struct
> (swap_cluster_info_dynamic). This is actually Kairui's design
> (see [7]), but after trying to deviate from it, I have to conclude
> it is the right choice too.
Kairui took a shortcut to deliver a simpler demo RFC, it is not ready
for merge as is. Please take a look at the VFS inode vs
ext4_inode_info for the effect I am trying to get to.
I am really crashing now.
Chris
>
> 3. Replacing zswap tree with the per-cluster backend table results in
> performance wins even when vswap is turned off (this is how I
> verified that vswap's performance wins comes from here).
>
> However, it requires more code to make sure this table is not allocated
> when not needed. Note that the eventual goal is to make vswap the ONLY
> way to use zswap, so we are literally adding complexity and overhead
> (even for non-vswap users) to optimize for a code path that is rarely
> exercised after vswap lands, and will be ripped out soon after.
> That seems very off to me.
>
> To close out, this design brings together the ideas from the earlier
> discussions:
>
> 1. All of the requirements I set out to solve (dynamicity, backend
> decoupling, efficient backend transfer) are implemented.
>
> 2. Vswap device now repurpose the swap table design and most of the
> generic swap operations.
>
> 3. Minimal overhead for non-vswap users, and zswap-no-writeback users.
> If you disable writeback, vswap *is* a ghost swapfile.
>
> Nhat Pham (11):
> mm, swap: add virtual swap device infrastructure
> mm, swap: support zswap and zeroswap as vswap backends
> mm, swap: prepare the swap IO path for vswap
> mm, swap: support physical swap as a vswap backend
> mm, swap: enable THP swapin for vswap entries
> mm, swap: write back vswap zswap entries to physical swap
> mm, swap: reclaim physical slots backing cache-only vswap entries
> mm, swap: only charge physical swap entries
> mm, swap: add debugfs counters for vswap
> mm, swap: defer memcg_table allocation for physical swap clusters
> mm, swap: widen swap_info_struct max/pages to unsigned long
>
> Documentation/admin-guide/sysctl/vm.rst | 16 +
> MAINTAINERS | 1 +
> include/linux/memcontrol.h | 5 +
> include/linux/swap.h | 88 +-
> include/linux/zswap.h | 3 +
> mm/Kconfig | 21 +
> mm/memcontrol.c | 166 +++-
> mm/memory.c | 28 +-
> mm/page_io.c | 103 +-
> mm/shmem.c | 4 +-
> mm/swap.h | 55 +-
> mm/swap_state.c | 64 +-
> mm/swap_table.h | 62 ++
> mm/swapfile.c | 1194 +++++++++++++++++++++--
> mm/vmscan.c | 14 +-
> mm/vswap.h | 454 +++++++++
> mm/zswap.c | 140 ++-
> 17 files changed, 2207 insertions(+), 211 deletions(-)
> create mode 100644 mm/vswap.h
>
>
> base-commit: bacc32cc7de65ffff70080a48eb294f89e434d5e
> --
> 2.53.0-Meta
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH v3 01/11] mm, swap: add virtual swap device infrastructure
2026-08-06 18:42 ` [PATCH v3 01/11] mm, swap: add virtual swap device infrastructure Nhat Pham
@ 2026-08-07 15:49 ` Johannes Weiner
0 siblings, 0 replies; 17+ messages in thread
From: Johannes Weiner @ 2026-08-07 15:49 UTC (permalink / raw)
To: Nhat Pham
Cc: akpm, chrisl, kasong, mhocko, roman.gushchin, shakeel.butt, yosry,
david, muchun.song, shikemeng, baoquan.he, baohua, youngjun.park,
chengming.zhou, ljs, liam, vbabka, rppt, surenb, qi.zheng,
axelrasmussen, yuanchu, weixugc, riel, gourry, haowenchao22,
corbet, kernel-team, linux-mm, linux-kernel, linux-doc, cgroups
Hello Nhat,
This looks pretty clean to me overall. Nice. A few comments inline:
On Thu, Aug 06, 2026 at 11:42:44AM -0700, Nhat Pham wrote:
> @@ -276,8 +277,21 @@ struct swap_info_struct {
> struct list_head discard_clusters; /* discard clusters list */
> struct plist_node avail_list; /* entry in swap_avail_head */
> const struct swap_ops *ops;
> + struct xarray cluster_info_pool; /* Xarray for vswap dynamic cluster info */
> };
>
> +#ifdef CONFIG_VSWAP
> +static inline bool swap_is_vswap(struct swap_info_struct *si)
> +{
> + return si->flags & SWP_VSWAP;
> +}
> +#else
I think what could make sense is have
- CONFIG_VSWAP
- CONFIG_VSWAP_DEFAULT_ENABLED
- a boot flag that's = CONFIG_VSWAP_DEFAULT_ENABLED
Then you can use jump labels to cut runtime overhead to 0 if it isn't
enabled at boot.
This way distributions can ship it with no overhead out of the box,
while allowing users to opt in for early testing.
IMO this is more valuable than a runtime toggle.
It should be part of this patch if you add it.
> +static inline bool swap_is_vswap(struct swap_info_struct *si)
> +{
> + return false;
> +}
> +#endif
> +
> static inline swp_entry_t page_swap_entry(struct page *page)
> {
> struct folio *folio = page_folio(page);
> @@ -402,6 +416,8 @@ void swap_free_hibernation_slot(swp_entry_t entry);
>
> static inline void put_swap_device(struct swap_info_struct *si)
> {
> + if (swap_is_vswap(si))
> + return;
> percpu_ref_put(&si->users);
> }
>
> diff --git a/mm/Kconfig b/mm/Kconfig
> index 331daf7fcfab..32d38b552845 100644
> --- a/mm/Kconfig
> +++ b/mm/Kconfig
> @@ -19,6 +19,16 @@ menuconfig SWAP
> used to provide more virtual memory than the actual RAM present
> in your computer. If unsure say Y.
>
> +config VSWAP
> + bool "Virtual swap device"
> + depends on SWAP && 64BIT
> + help
> + Adds a virtual swap layer that decouples swap entries in page
> + tables from physical backing storage. Swap entries are allocated
> + from a virtual swap device and can be backed by zswap, a physical
> + swapfile, or kept in memory - with the backing changeable at
> + runtime without invalidating page table entries.
IMO this is difficult for a user to decide.
If we want to make this as a config, then IMO the help text should be
about not requiring a swapfile to use zswap, zerofill pages etc.
From where I'm standing, though, this code is no longer a separate
vswap.c feature like it used to be. It's pretty integrated into the
swapcode, and it looks like a good chunk of the core datastructures
aren't conditional either.
At that point, the usefulness of the config option is limited. If you
go with the jump label and there is no runtime overhead, it might make
sense to just keep the option on what the JL should default to.
> @@ -143,9 +150,19 @@ static inline struct swap_info_struct *__swap_entry_to_info(swp_entry_t entry)
> static inline struct swap_cluster_info *__swap_offset_to_cluster(
> struct swap_info_struct *si, pgoff_t offset)
> {
> + unsigned int cluster_idx = offset / SWAPFILE_CLUSTER;
> +
> VM_WARN_ON_ONCE(percpu_ref_is_zero(&si->users)); /* race with swapoff */
> VM_WARN_ON_ONCE(offset >= roundup(si->max, SWAPFILE_CLUSTER));
> - return &si->cluster_info[offset / SWAPFILE_CLUSTER];
> +
> + if (swap_is_vswap(si)) {
> + struct swap_cluster_info_dynamic *ci_dyn;
> +
> + ci_dyn = xa_load(&si->cluster_info_pool, cluster_idx);
> + return ci_dyn ? &ci_dyn->ci : NULL;
> + }
> +
> + return &si->cluster_info[cluster_idx];
> }
This now requires rcu. IMO a kdoc explaining that would be good.
> @@ -733,6 +765,11 @@ static void free_cluster(struct swap_info_struct *si, struct swap_cluster_info *
> return;
> }
>
> + if (swap_is_vswap(si)) {
> + vswap_free_cluster(si, ci);
> + return;
> + }
> +
> __free_cluster(si, ci);
Should this be part of __free_cluster() instead?
- swap_cluster_assert_empty() seems useful.
- move_cluster() handles the list linkage and could do the list_del().
- swap_cluster_free_table() is the same.
The other caller is the discard path. That one isn't relevant, but it
shouldn't hurt. The layering would still be a bit cleaner this way.
> @@ -835,14 +872,21 @@ static int swap_cluster_setup_bad_slot(struct swap_info_struct *si,
> * stolen by a lower order). @usable will be set to false if that happens.
> */
> static bool cluster_reclaim_range(struct swap_info_struct *si,
> - struct swap_cluster_info *ci,
> + struct swap_cluster_info **pcip,
> unsigned long start, unsigned int order,
> bool *usable)
> {
> + struct swap_cluster_info *ci = *pcip;
> unsigned int nr_pages = 1 << order;
> unsigned long offset = start, end = start + nr_pages;
> unsigned long swp_tb;
>
> + /*
> + * Take RCU read lock before releasing the cluster lock to keep ci
> + * alive - for vswap dynamic clusters, ci is freed via kfree_rcu
> + * and the grace period could otherwise elapse in the window.
> + */
> + rcu_read_lock();
> spin_unlock(&ci->lock);
Since you replace the naked spin_lock() below with
swap_cluster_lock(), change this to swap_cluster_unlock() as well?
> do {
> swp_tb = swap_table_get(ci, offset % SWAPFILE_CLUSTER);
> @@ -852,7 +896,15 @@ static bool cluster_reclaim_range(struct swap_info_struct *si,
> if (__try_to_reclaim_swap(si, offset, TTRS_ANYWAY) < 0)
> break;
> } while (++offset < end);
> - spin_lock(&ci->lock);
> + rcu_read_unlock();
> +
> + /* Re-lookup: dynamic cluster may have been freed while lock was dropped */
> + ci = swap_cluster_lock(si, start);
> + *pcip = ci;
> + if (!ci) {
> + *usable = false;
> + return false;
> + }
>
> /*
> * We just dropped ci->lock so cluster could be used by another
> @@ -1146,6 +1239,12 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
> goto done;
> }
>
> + if (swap_is_vswap(si)) {
> + found = alloc_swap_scan_dynamic(si, folio);
> + if (found)
> + goto done;
> + }
The name and the composability look off to me.
Can you make this a pure allocation function - vswap_alloc_cluster()
or something - that links it to &si->free_clusters, then jump back and
use the existing alloc_swap_scan_list() sites?
There is a new_cluster: label upstream you could use for the retry.
> +
> if (!(si->flags & SWP_PAGE_DISCARD)) {
> found = alloc_swap_scan_list(si, &si->free_clusters, folio, false);
> if (found)
> @@ -1264,6 +1363,13 @@ static void add_to_avail_list(struct swap_info_struct *si, bool swapon)
> goto skip;
> }
>
> + /*
> + * Keep vswap off the avail list - it is not allocated from by
> + * the physical swap allocator (swap_alloc_fast/slow).
> + */
> + if (swap_is_vswap(si))
> + goto skip;
This is describing what the code does, followed by a why not.
Describe the why: Vswap space is only allocated through [...], not [...]
> @@ -3528,10 +3668,43 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
> unsigned long maxpages)
> {
> unsigned long nr_clusters = DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER);
> - struct swap_cluster_info *cluster_info;
> + struct swap_cluster_info *cluster_info = NULL;
> + struct swap_cluster_info_dynamic *ci_dyn;
> int err = -ENOMEM;
> unsigned long i;
>
> + /* For SWP_VSWAP files, initialize Xarray pool instead of static array */
> + if (swap_is_vswap(si)) {
> + /*
> + * Pre-allocate cluster 0 and mark slot 0 (header page)
> + * as bad so the allocator never hands out page offset 0.
> + */
> + ci_dyn = kzalloc_obj(*ci_dyn, GFP_KERNEL);
> + if (!ci_dyn)
> + goto err;
> + spin_lock_init(&ci_dyn->ci.lock);
> + INIT_LIST_HEAD(&ci_dyn->ci.list);
> +
> + nr_clusters = 0;
But it's one, not zero. And only the first slot is bad. Why is the
rest not usable and the cluster on the nonfull_clusters list?
I think you can make the integration a bit more organic in general:
Don't do the DIV_ROUND_UP per default only to overwrite it here again
e.g.
If you come out of this branch with cluster_info == &ci_dyn->ci, you
should be able to reuse the existing swap_cluster_setup_bad_slot(0).
swap_header && i < swap_header->info.nr_badpages should skip that loop
as well.
Your maxpages is SWAPFILE_CLUSTER aligned, so that
for (i = maxpages; i < round_up(maxpages, SWAPFILE_CLUSTER); i++) {
loop is skipped as well.
> + xa_init_flags(&si->cluster_info_pool, XA_FLAGS_ALLOC);
> + err = xa_insert(&si->cluster_info_pool, 0, ci_dyn, GFP_KERNEL);
> + if (err) {
> + kfree(ci_dyn);
> + goto err;
> + }
> +
> + err = swap_cluster_setup_bad_slot(si, &ci_dyn->ci, 0, false);
> + if (err) {
> + xa_erase(&si->cluster_info_pool, 0);
> + swap_cluster_free_table(&ci_dyn->ci);
> + kfree(ci_dyn);
IMO it would be nicer to make this part of free_swap_cluster_info()
instead of duplicating it.
> + xa_destroy(&si->cluster_info_pool);
That one probably doesn't matter. If these allocations fail at boot,
the system is in trouble.
> + goto err;
> + }
> +
> + goto setup_cluster_info;
> + }
> +
> cluster_info = kvzalloc_objs(*cluster_info, nr_clusters);
> if (!cluster_info)
> goto err;
> @@ -3556,6 +3729,10 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
> err = swap_cluster_setup_bad_slot(si, cluster_info, 0, false);
> if (err)
> goto err;
> +
> + if (!swap_header)
> + goto setup_cluster_info;
Is this reachable?
> @@ -3949,3 +4127,51 @@ static int __init swapfile_init(void)
> return 0;
> }
> subsys_initcall(swapfile_init);
> +
> +#ifdef CONFIG_VSWAP
> +struct swap_info_struct *vswap_si;
> +
> +/* vswap does no IO on its own. */
> +static const struct swap_ops vswap_ops = { };
> +
> +static int __init vswap_init(void)
> +{
> + struct swap_info_struct *si;
> + unsigned long maxpages;
> + int err;
> +
> + si = alloc_swap_info();
> + if (IS_ERR(si))
> + return PTR_ERR(si);
> +
> + maxpages = min(swapfile_maximum_size,
> + ALIGN_DOWN((unsigned long)UINT_MAX, SWAPFILE_CLUSTER));
> + si->flags |= SWP_VSWAP | SWP_SOLIDSTATE | SWP_WRITEOK;
Please add a comment on the flag choices besides SWP_VSWAP.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH v3 08/11] mm, swap: only charge physical swap entries
2026-08-06 18:42 ` [PATCH v3 08/11] mm, swap: only charge physical swap entries Nhat Pham
@ 2026-08-07 16:31 ` Johannes Weiner
0 siblings, 0 replies; 17+ messages in thread
From: Johannes Weiner @ 2026-08-07 16:31 UTC (permalink / raw)
To: Nhat Pham
Cc: akpm, chrisl, kasong, mhocko, roman.gushchin, shakeel.butt, yosry,
david, muchun.song, shikemeng, baoquan.he, baohua, youngjun.park,
chengming.zhou, ljs, liam, vbabka, rppt, surenb, qi.zheng,
axelrasmussen, yuanchu, weixugc, riel, gourry, haowenchao22,
corbet, kernel-team, linux-mm, linux-kernel, linux-doc, cgroups
On Thu, Aug 06, 2026 at 11:42:51AM -0700, Nhat Pham wrote:
> Charge memcg->swap when a vswap entry acquires physical backing rather
> than when it is allocated, so memory.swap.current tracks on-disk swap
> usage. Zswap-backed and zero-filled pages occupy no swap space but were
> charged as though they did.
>
> memory.swap.current therefore no longer counts them, and a cgroup whose
> pages all land in zswap can now reclaim anon memory with memory.swap.max
> set to 0.
>
> Direct-mapped physical swap charging is unchanged.
>
> Signed-off-by: Nhat Pham <nphamcs@gmail.com>
To head off any uncertainty about this: this is exactly what needs to
happen in terms of cgroup semantics.
memory.swap.* are about physical swap space. They track, control, and
enforce fairness for a finite resource that is separate from memory.
When a user switches on vswsap and a bunch of empty pages are stored
inside the zeromap without consuming swapfile space, these counters
must be 0.
When a user switches on vswap to use zswap without a backing file,
these counters must be 0.
When a user switches on vswap to use zswap with writeback, only the
pages that get written to the swapfile must be tracked and controlled
by these counters.
A few inline comments on the implementation:
> @@ -5701,6 +5702,116 @@ int __mem_cgroup_try_charge_swap(struct folio *folio)
> return 0;
> }
>
> +/**
> + * __mem_cgroup_record_swap - record memcg for swap without charging
> + * @folio: folio being added to swap
> + *
> + * Pin the memcg private ID ref and record it in the swap cgroup table
> + * without charging memcg->swap; the charge is deferred to physical-backing
> + * allocation (vswap).
> + */
> +void __mem_cgroup_record_swap(struct folio *folio)
> +{
> + unsigned int nr_pages = folio_nr_pages(folio);
> + struct swap_cluster_info *ci;
> + struct mem_cgroup *memcg;
> + struct obj_cgroup *objcg;
> +
> + if (do_memsw_account())
> + return;
> +
> + objcg = folio_objcg(folio);
> + VM_WARN_ON_ONCE_FOLIO(!objcg, folio);
> + if (!objcg)
> + return;
> +
> + rcu_read_lock();
> + memcg = obj_cgroup_memcg(objcg);
> + if (!folio_test_swapcache(folio)) {
> + rcu_read_unlock();
> + return;
> + }
> +
> + memcg = mem_cgroup_private_id_get_online(memcg, nr_pages);
> + rcu_read_unlock();
> +
> + ci = swap_cluster_get_and_lock(folio);
> + __swap_cgroup_set(ci, swp_cluster_offset(folio->swap), nr_pages,
> + mem_cgroup_private_id(memcg));
> + swap_cluster_unlock(ci);
> +}
> +
> +/**
> + * __mem_cgroup_charge_backing_phys_swap - charge memcg->swap
> + * @memcg: the mem_cgroup to charge (may be NULL)
> + * @nr_pages: number of physical swap pages to charge
> + *
> + * Charge the swap counter when a vswap entry gains physical backing. The
> + * private ID ref is already held (pinned by __mem_cgroup_record_swap() at
> + * vswap allocation), so this only moves the counter.
> + *
> + * Return: 0 on success, -ENOMEM on failure.
> + */
> +int __mem_cgroup_charge_backing_phys_swap(struct mem_cgroup *memcg,
> + unsigned int nr_pages)
> +{
> + struct page_counter *counter;
> +
> + if (do_memsw_account())
> + return 0;
> + if (!memcg)
> + return 0;
> +
> + if (!mem_cgroup_is_root(memcg) &&
> + !page_counter_try_charge(&memcg->swap, nr_pages, &counter)) {
> + memcg_memory_event(memcg, MEMCG_SWAP_MAX);
> + memcg_memory_event(memcg, MEMCG_SWAP_FAIL);
> + return -ENOMEM;
> + }
> + mod_memcg_state(memcg, MEMCG_SWAP, nr_pages);
> + return 0;
> +}
These functions are just __mem_cgroup_try_charge_swap() in two acts :-)
Please refactor this properly:
__mem_cgroup_swap_record()
__mem_cgroup_swap_charge()
> + * __mem_cgroup_uncharge_backing_phys_swap - uncharge memcg->swap counter
> + * @memcg: the mem_cgroup to uncharge (may be NULL)
> + * @nr_pages: number of physical swap pages to uncharge
> + *
> + * Uncharge the swap counter on physical backing release for a vswap entry.
> + * The private ID ref is dropped separately via __mem_cgroup_id_put_swap() when
> + * the vswap entry is freed.
> + */
> +void __mem_cgroup_uncharge_backing_phys_swap(struct mem_cgroup *memcg,
> + unsigned int nr_pages)
Same on the uncharge side...
__mem_cgroup_swap_uncharge()
> +{
> + if (!memcg)
> + return;
> +
> + if (!mem_cgroup_is_root(memcg)) {
> + if (do_memsw_account())
> + page_counter_uncharge(&memcg->memsw, nr_pages);
> + else
> + page_counter_uncharge(&memcg->swap, nr_pages);
> + }
> + mod_memcg_state(memcg, MEMCG_SWAP, -nr_pages);
> +}
> +
> +/**
> + * __mem_cgroup_id_put_swap - drop memcg private ID ref without uncharging
> + * @id: cgroup private id
> + * @nr_pages: number of refs to drop
> + */
> +void __mem_cgroup_id_put_swap(unsigned short id, unsigned int nr_pages)
> +{
> + struct mem_cgroup *memcg;
> +
> + rcu_read_lock();
> + memcg = mem_cgroup_from_private_id(id);
> + if (memcg)
> + mem_cgroup_private_id_put(memcg, nr_pages);
> + rcu_read_unlock();
> +}
__mem_cgroup_swap_put()
and then remove __mem_cgroup_uncharge_swap(). Handle this split the
same way as on the charge path.
> @@ -2116,8 +2117,16 @@ int folio_alloc_swap(struct folio *folio)
> goto again;
> }
>
> - /* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. */
> - if (unlikely(mem_cgroup_try_charge_swap(folio)))
> + /*
> + * A vswap entry has no physical swap yet, so only record the memcg;
> + * folio_realloc_swap() charges once backing is allocated.
> + *
> + * Need to call this even if allocation failed, for MEMCG_SWAP_FAIL.
> + */
> + if (folio_test_swapcache(folio) &&
> + is_vswap_entry(folio->swap))
> + mem_cgroup_record_swap(folio);
> + else if (unlikely(mem_cgroup_try_charge_swap(folio)))
> swap_cache_del_folio(folio);
This becomes:
if (!vswap && mem_cgroup_swap_try_charge())
abort
mem_cgroup_swap_record()
> @@ -2614,18 +2685,28 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,
> /*
> * Uncharge swap slots by memcg in batches. Consecutive
> * slots with the same cgroup id are uncharged together.
> + * For vswap, only drop the ID ref - physical swap was
> + * already uncharged in __vswap_release_backing above.
> */
> id_cur = __swap_cgroup_clear(ci, ci_off, 1);
> if (batch_id != id_cur) {
> - if (batch_id)
> - mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);
> + if (batch_id) {
> + if (is_vswap)
> + mem_cgroup_id_put_swap(batch_id, ci_off - batch_off);
> + else
> + mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);
> + }
And this becomes:
if (!vswap)
mem_cgroup_swap_uncharge()
mem_cgroup_swap_put()
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2026-08-07 16:31 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-08-06 18:42 [PATCH v3 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
2026-08-06 18:42 ` [PATCH v3 01/11] mm, swap: add virtual swap device infrastructure Nhat Pham
2026-08-07 15:49 ` Johannes Weiner
2026-08-06 18:42 ` [PATCH v3 02/11] mm, swap: support zswap and zeroswap as vswap backends Nhat Pham
2026-08-06 18:42 ` [PATCH v3 03/11] mm, swap: prepare the swap IO path for vswap Nhat Pham
2026-08-06 18:42 ` [PATCH v3 04/11] mm, swap: support physical swap as a vswap backend Nhat Pham
2026-08-06 18:42 ` [PATCH v3 05/11] mm, swap: enable THP swapin for vswap entries Nhat Pham
2026-08-06 18:42 ` [PATCH v3 06/11] mm, swap: write back vswap zswap entries to physical swap Nhat Pham
2026-08-06 18:42 ` [PATCH v3 07/11] mm, swap: reclaim physical slots backing cache-only vswap entries Nhat Pham
2026-08-06 18:42 ` [PATCH v3 08/11] mm, swap: only charge physical swap entries Nhat Pham
2026-08-07 16:31 ` Johannes Weiner
2026-08-06 18:42 ` [PATCH v3 09/11] mm, swap: add debugfs counters for vswap Nhat Pham
2026-08-06 18:42 ` [PATCH v3 10/11] mm, swap: defer memcg_table allocation for physical swap clusters Nhat Pham
2026-08-06 18:42 ` [PATCH v3 11/11] mm, swap: widen swap_info_struct max/pages to unsigned long Nhat Pham
2026-08-07 5:26 ` [syzbot ci] Re: Virtual Swap Space (Swap Table Edition) syzbot ci
2026-08-07 7:21 ` Chris Li
2026-08-07 9:07 ` [PATCH v3 00/11] " Chris Li
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox