Path forward for Virtualized Swap?

Path forward for Virtualized Swap?

[Nhat Pham]

To Kairui, Johannes, Yosry, Chris, David, Andrew, and other (z)swap folks -
I need some help here.

I'd like to ask for a decision, or failing that, the specific thing that
would get us to one. I'd much rather work against a concrete objection than
keep respinning into silence.

Quick recap: today a swapped-out page holds a disk slot even when the data
never reaches disk, like zswap or zero-filled pages. My patch series
(latest version at [10]) puts a virtual entry in the page table instead and
resolves the backing separately, so zswap stops reserving disk it never
uses and can work on machines with no storage swap space at all.

I've been working on this since April 2025, going through 10 revisions (6
on the original design ([1] through [6]) and 4 on the swap-table-based
rewrite ([7] through [10])). I also gave a talk on it at LPC [11].

Throughout the process, I've been very receptive to reviewers - including
parties whose use cases are very different from my own, like zram users and
the ghost-swapfile-style deployments that want zswap with no backing disk
at all - and the current design reflects that:

* Kairui asked that this be optional, not a replacement [12]. It is now:
  vswap is opt-in behind the vswap= cmdline parameter, and with vswap=off
  the existing swap paths aren't touched at all.

* Kairui also pointed out that if I kept shrinking the metadata I'd "end up
  reimplementing the swap table format" [12]. I took that feedback, and
  reworked the entire series on top of swap table.

* Per Chris' requests, I've figured out ways to minimize memory overhead
  [13]. On a zswap-only setup there's almost 0 overhead now, because the
  vtable is merged with zswap's xarray rather than sitting on top of it.
  The indirection is paid for by the tree we delete.

* Johannes pointed out that a physical cluster backing only vswap entries
  still had to allocate a memcg_table, which was an overhead regression
  against my older design on the zswap writeback case [14] (since a swap
  entry occupies multiple levels). I've also fixed this.

* I've spent a lot of time chasing regressions on setups we don't even run
  at Meta - zram in particular [6]. And if anyone's still worried after all
  that, vswap=off keeps the old swap code available.

Numbers are in the v4 cover letter [10]. In terms of performance, vswap is
on par or better than baseline on zswap across memhog, usemem and kernel
build, and vswap=off tracks baseline within noise. I've even included an
A/B experiment on a real production workload (Instagram/django): throughput
on par, latency slightly better, and the swapfile footprint for every zswap
user on the host goes away.

Now, on xswap. Baoquan's working on a series [15] that covers some of the
same ground, and the VM_SPARSE cluster_info idea in it is genuinely good.
I've been reviewing that lineage since July [16] and I'd like whatever
lands to end up with the best parts of both. From my perspective the
differences are:

1. Userspace knobs. xswap asks the admin for a size (a percent of RAM) plus
   a per-device limit to tune afterwards. I'm not aware of any use case
   that needs those, and I don't think users have a good way to answer the
   question anyway - sizing swap for compressed memory depends on memory
   size, workload, and compression ratio all at once. That's precisely the
   provisioning problem vswap exists to remove. The kernel should be as
   transparent and dynamic as possible here, and not add knobs unless
   there's a use case for them.

2. Writeback support. Writeback is core functionality for zswap, not an
   add-on, and a design needs to account for it from the start. This came
   up before, in the discussion around Chris' ghost swapfile RFC [17]: for
   a solution here to be acceptable, it has to work with the primary
   usecase and support disk writeback. Without it, whatever zswap won't
   take (incompressible pages especially) has nowhere to go, and cold
   compressed data can never leave RAM.

3. Cgroup charging behavior. vswap/xswap shouldn't be charged against the
   swap usage counter. It's fundamentally a different resource from
   physical swapfile space, and memory.swap.* should read 0 when nothing is
   on disk [18]. I made the longer argument for this in [19].

4. Data structure (xarray vs sparse vmalloc array). Even with xarray, vswap
   is already on par with or beating baseline. I like the sparse array
   idea, but why are we landing an optimization before the feature itself,
   without any A/B data showing the difference matters?

One thing I do want to be clear about: I'm glad other people care about
this problem. Chris' ghost swapfile and Baoquan's xswap are both going
after the same set of problems, and that's a good sign. It means this is
real and shared, not something only Meta runs into.

What's been harder is the shape of the engagement. Alternatives keep
getting posted and pushed that don't cover all the requirements, while this
series sits without review. I don't think I'm owed anyone's interest in the
problems I care about. But I do think working code, with benchmarks and
production exposure behind it, deserves a fair hearing next to in-progress
proposals.

So what I'm asking for: I'd like us to converge rather than keep two series
in flight. My preference is that we land vswap first, then build Baoquan's
sparse array on top of it as an optimization. That gets the feature in, and
by then we'd have the A/B data to show whether the sparse array actually
beats the xarray.

If you think that's the wrong order, I'd genuinely like to understand why -
after 17 months and 10 revisions I still don't have a clear picture of the
objection.

[1] https://lore.kernel.org/all/20250407234223.1059191-1-nphamcs@gmail.com/
[2] https://lore.kernel.org/all/20250429233848.3093350-1-nphamcs@gmail.com/
[3] https://lore.kernel.org/all/20260208215839.87595-1-nphamcs@gmail.com/
[4] https://lore.kernel.org/all/20260318222953.441758-1-nphamcs@gmail.com/
[5] https://lore.kernel.org/all/20260320192735.748051-1-nphamcs@gmail.com/
[6] https://lore.kernel.org/all/20260505153854.1612033-1-nphamcs@gmail.com/
[7] https://lore.kernel.org/all/20260528212955.1912856-1-nphamcs@gmail.com/
[8] https://lore.kernel.org/all/20260612193738.2183968-1-nphamcs@gmail.com/
[9] https://lore.kernel.org/all/20260806184254.3790858-1-nphamcs@gmail.com/
[10] https://lore.kernel.org/all/20260825153238.2695446-1-nphamcs@gmail.com/
[11] https://lwn.net/Articles/1016136/
[12] https://lore.kernel.org/all/CAMgjq7AQNGK-a=AOgvn4-V+zGO21QMbMTVbrYSW_R2oDSLoC+A@mail.gmail.com/
[13] https://lore.kernel.org/all/CACePvbVXQWgcPD-bgK7iDba4NFLo2tT89ZbLOa03maJU4er4ag@mail.gmail.com/
[14] https://lore.kernel.org/all/aZyFxKGXc8J6PIij@cmpxchg.org/
[15] https://lore.kernel.org/all/20260827094509.1016740-1-hebaoquan@kylinos.cn/
[16] https://lore.kernel.org/lkml/CAKEwX=Pe+qMZd2xhnU-PAGQtgXkp56c-JwYCbt2Lux9htgB67Q@mail.gmail.com/
[17] https://lore.kernel.org/all/20251121114011.GA71307@cmpxchg.org/
[18] https://lore.kernel.org/all/anYIboHEUZb4fhHv@cmpxchg.org/
[19] https://lore.kernel.org/all/CAKEwX=P4syV38jAVCWq198r2OHXXc=xA-fx1dk6+qYef6yzxWQ@mail.gmail.com/

Re: Path forward for Virtualized Swap?

[Kairui Song]

On Sat, Sep 5, 2026 at 5:14 AM Nhat Pham <nphamcs@gmail.com> wrote:
>
> To Kairui, Johannes, Yosry, Chris, David, Andrew, and other (z)swap folks -
> I need some help here.

Hello Nhat,

>
> I'd like to ask for a decision, or failing that, the specific thing that
> would get us to one. I'd much rather work against a concrete objection than
> keep respinning into silence.
>
> Quick recap: today a swapped-out page holds a disk slot even when the data
> never reaches disk, like zswap or zero-filled pages. My patch series
> (latest version at [10]) puts a virtual entry in the page table instead and
> resolves the backing separately, so zswap stops reserving disk it never
> uses and can work on machines with no storage swap space at all.
>
> I've been working on this since April 2025, going through 10 revisions (6
> on the original design ([1] through [6]) and 4 on the swap-table-based
> rewrite ([7] through [10])). I also gave a talk on it at LPC [11].

Thanks for the hard work! And I mean that. Upstream work at this scale
does take a long time and it isn't a reflection on the series: the swap
cache optimization idea also took me over 2 years and changed direction
multiple times and split multiple times before it landed as
the swap table series, and the readahead unification part is
still in flight after nearly 3 years. It's frustrating and I don't think
anyone enjoys it. I mentioned this once with a longer history:

https://lore.kernel.org/linux-mm/CAMgjq7CF9RgnZCAS-+Gv0LAvkzzHk4jiok+_6-KOFw-o+s8E_g@mail.gmail.com/

There is also a quote from LSF/MM/BPF saying it's for improvements to
"find their way into the mainline kernel and Linux distributions in the
next 24-48 months.", suffering souls everywhere :P

>
> Throughout the process, I've been very receptive to reviewers - including
> parties whose use cases are very different from my own, like zram users and
> the ghost-swapfile-style deployments that want zswap with no backing disk
> at all - and the current design reflects that:
>
> * Kairui asked that this be optional, not a replacement [12]. It is now:
>   vswap is opt-in behind the vswap= cmdline parameter, and with vswap=off
>   the existing swap paths aren't touched at all.
>
> * Kairui also pointed out that if I kept shrinking the metadata I'd "end up
>   reimplementing the swap table format" [12]. I took that feedback, and
>   reworked the entire series on top of swap table.
>
> * Per Chris' requests, I've figured out ways to minimize memory overhead
>   [13]. On a zswap-only setup there's almost 0 overhead now, because the
>   vtable is merged with zswap's xarray rather than sitting on top of it.
>   The indirection is paid for by the tree we delete.
>
> * Johannes pointed out that a physical cluster backing only vswap entries
>   still had to allocate a memcg_table, which was an overhead regression
>   against my older design on the zswap writeback case [14] (since a swap
>   entry occupies multiple levels). I've also fixed this.
>
> * I've spent a lot of time chasing regressions on setups we don't even run
>   at Meta - zram in particular [6]. And if anyone's still worried after all
>   that, vswap=off keeps the old swap code available.

Acknowledged, specifically reworking the whole series on top of swap
table, the memcg table fix, and chasing zram regressions you don't even
run were all real work on cases that don't benefit your fleet. Thank
you!

Roughly 80% of my own time also goes to things that don't run on my
fleet either. Desktop and Android (I do use them though), lowend and even
32bits. Upstream covers every usage case as the baseline and it's tiring,
I know what it costs. :)

> Numbers are in the v4 cover letter [10]. In terms of performance, vswap is
> on par or better than baseline on zswap across memhog, usemem and kernel
> build, and vswap=off tracks baseline within noise. I've even included an
> A/B experiment on a real production workload (Instagram/django): throughput
> on par, latency slightly better, and the swapfile footprint for every zswap
> user on the host goes away.
>
> Now, on xswap. Baoquan's working on a series [15] that covers some of the
> same ground, and the VM_SPARSE cluster_info idea in it is genuinely good.

My read is that xswap and vswap differ mostly in the interface, and that
both could sit on VM_SPARSE for cluster/metadata management.
Simpler and faster. There are some other issues though, see below.

> I've been reviewing that lineage since July [16] and I'd like whatever
> lands to end up with the best parts of both. From my perspective the
> differences are:
>
> 1. Userspace knobs. xswap asks the admin for a size (a percent of RAM) plus
>    a per-device limit to tune afterwards. I'm not aware of any use case
>    that needs those, and I don't think users have a good way to answer the
>    question anyway - sizing swap for compressed memory depends on memory
>    size, workload, and compression ratio all at once. That's precisely the
>    provisioning problem vswap exists to remove. The kernel should be as
>    transparent and dynamic as possible here, and not add knobs unless
>    there's a use case for them.

I asked Baoquan the same thing, see the "Why not make both args
optional" part:

https://lore.kernel.org/linux-mm/apaw_XmqnEcY8sv7@MiWiFi-R3L-srv/

Where I've ended up is that unbounded growth is a real concern. On a
host with no memcg limit (root cgroup, and most desktop and embedded
setups), an unlimited pool means usage can keep growing, with no
admin visible ceiling at all. I'm not attached to xswap's percent of RAM
knob specifically, but I do think some kind of bound makes sense.

> 2. Writeback support. Writeback is core functionality for zswap, not an
>    add-on, and a design needs to account for it from the start. This came
>    up before, in the discussion around Chris' ghost swapfile RFC [17]: for
>    a solution here to be acceptable, it has to work with the primary
>    usecase and support disk writeback. Without it, whatever zswap won't
>    take (incompressible pages especially) has nowhere to go, and cold
>    compressed data can never leave RAM.

Agreed completely, and I think this one is settled: whatever lands
has to support disk writeback in the end. Baoquan mentioned that
what he sent is just a foundation though. I know talking is cheap
but still, could it be combined with the writeback support from the
other side? Or am I missing anything here?

> 3. Cgroup charging behavior. vswap/xswap shouldn't be charged against the
>    swap usage counter. It's fundamentally a different resource from
>    physical swapfile space, and memory.swap.* should read 0 when nothing is
>    on disk [18]. I made the longer argument for this in [19].

I personally don't have a strong opinion on this. But I think this
needs an explicit decision from the group rather than being
inherited from whichever series merges first. Skipping the charge
when a device has a special flag isn't hard to do if the code
abstraction is done well, so this is more of a design decision
than a technical problem.

> 4. Data structure (xarray vs sparse vmalloc array). Even with xarray, vswap
>    is already on par with or beating baseline. I like the sparse array
>    idea, but why are we landing an optimization before the feature itself,
>    without any A/B data showing the difference matters?

This is where I'd push back a little. To be clear, my concern isn't the
size of the win, it's that VM_SPARSE isn't purely an optimization. It
changes the cluster/metadata layout and avoids RCU conventions,
could help avoid some future sleep while atomic limitations and
it's tangled with the initial size issue too. So bolting it on afterwards
could mean rewriting the parts of vswap that touch it and could be
costly some conflict in ideas. That's why I'd rather we agree on
it now, not that it has to land first.

Using VM_SPARSE does outsmart my earlier xarray suggestion :) There
is an interesting early benchmark from Baoquan, VM_SPARSE wasn't there
yet, but with it we can use clusters natively in the same way. It shows
about 20% latency cut and 10% throughput gain, which seems a major
improvement, and that's serial performance, not even mentioning
scalability yet:
https://lore.kernel.org/linux-mm/20260707073215.72183-1-baoquan.he@linux.dev/

And it matters more than most micro-optimizations: under heavy
pressure many workloads spend less than half of system time in
compression itself, the rest is mm/swap. Swap has had a terrible
reputation for good reasons, and even with zswap/zram users
have been scared of the performance.

After the recent refactor and optimization we already noticed many
users can now run much heavier workloads using compressed memory only
or fast storages, which is a real breakthrough. Every chunk of
overhead we remove there unblocks another class of user from using
compressed memory as a "secondary memory".

>
> One thing I do want to be clear about: I'm glad other people care about
> this problem. Chris' ghost swapfile and Baoquan's xswap are both going
> after the same set of problems, and that's a good sign. It means this is
> real and shared, not something only Meta runs into.
>
> What's been harder is the shape of the engagement. Alternatives keep
> getting posted and pushed that don't cover all the requirements, while this
> series sits without review. I don't think I'm owed anyone's interest in the
> problems I care about. But I do think working code, with benchmarks and
> production exposure behind it, deserves a fair hearing next to in-progress
> proposals.

I'm sorry it has felt that way, and I don't want to hand wave it. For
what it's worth, I did review it, and I did think it could be merged
with some adjustments:

https://lore.kernel.org/linux-mm/CAMgjq7D4XsAD4NGDL7FC2kaYAQAP8PDJdn4bpzGZwXYtjEpJ6w@mail.gmail.com/#t

We were also discussing the limitation of the global cluster cache in
that thread, so I paused my other work for a while and posted the swap
queue series first. That one is orthogonal on purpose, and helps both
swap tier and vswap, it reduces the overhead enough to cancel out
the layering cost due to missing caching for either:
https://lore.kernel.org/linux-mm/20260829-swap-pcp-priq-v2-resend-0-68d3d925578c@gmail.com/
(this is V2, V1 is earlier - thanks to Lian for keeping this going!)

Then xswap appeared with the VM_SPARSE idea and exploring it seems
necessary because getting the cluster foundation right affects
everything built above where it went. Not because the vswap series
wasn't worth reviewing.

> So what I'm asking for: I'd like us to converge rather than keep two series
> in flight. My preference is that we land vswap first, then build Baoquan's
> sparse array on top of it as an optimization. That gets the feature in, and
> by then we'd have the A/B data to show whether the sparse array actually
> beats the xarray.
>
> If you think that's the wrong order, I'd genuinely like to understand why -
> after 17 months and 10 revisions I still don't have a clear picture of the
> objection.

I fully agree we must converge. At the moment I don't think one can
can settle that alone as there are several of us reviewing both. I'll try
to be more concrete about what I think has to be agreed first.

The two interfaces aren't equivalent, xswap has its limitations, but a
couple of xswap's properties look genuinely better to me. Runtime
on/off rather than a boot parameter is better, more flexibility and
doesn't conflict with what vswap is trying to do. Some other of these
differences involve design decisions rather than just surface knobs,
and those are the parts we're stuck with once they ship.

So a few current interface difference of vswap and xswap I've noticed are:

 - default size: somehow also tangled with VM_SPARSE, which
needs an initial virtual value. It's super cheap if not zero cost
(the only cost is occupying some vmalloc area, which is fine
I guess?). But we still have to pick a number, could be large
enough to satisfy everyone?
 - boot vs runtime toggle: I believe a runtime toggle is better. It
can still be tuned on boot via cmdline, but it offers the extra
ability to turn it on without modifying a base VM boot image or
kernel.
 - growth / limits: as talked above.
 - charging semantics
 - there could be other longterm design decisions on top of these as
we talked in above.

Once these settle, I think the picture will be clean. Your writeback
logic and swap table integration are solid, while Baoquan's VM_SPARSE,
previous preparation with swap ops give us a structural foundation. A
clean long-term architecture means getting the core structure right,
let's try not treating this as a race between series.

Re: Path forward for Virtualized Swap?

[David Hildenbrand (Arm)]

On 9/4/26 23:14, Nhat Pham wrote:

Hi,

> To Kairui, Johannes, Yosry, Chris, David, Andrew, and other (z)swap folks -
> I need some help here.

I'll mostly ignore swap-related stuff because (a) -EBUSY and (b) we have quite
some swap experts around :)

Reading below, I do wonder if the swap subsystem needs some more coordination
and collaboration?

For example, maybe having bi-weekly/monthly meetings to get everybody aligned
and uptodate could be something reasonable?

This approach has proven fairly helpful in areas where (1) multiple companies
are involved (2) many concurrent development is happening, and (3) people have
different priorities and requirements.

Is there already something like that to organize development? Should we get
started with that?

-- 
Cheers,

David

Re: Path forward for Virtualized Swap?

[Nhat Pham]

On Sun, Sep 6, 2026 at 10:52 PM Kairui Song <ryncsn@gmail.com> wrote:

Hi Kairui,

Thank you for the kind words and encouragement :) It means a lot
coming from you.

My main concern so far has been - it seems like every time I satisfied
one of your guys' requirements, another pre-requisite is added to the
list. I mean the performance regression and memory overhead concerns
make a lot of sense, especially when vswap is non-optional. But now it
just seems like we're delaying real use case for the fabled perfect
swap system. It gets very exhausting trying to keep expanding the
scope of this.

>
> I asked Baoquan the same thing, see the "Why not make both args
> optional" part:
>
> https://lore.kernel.org/linux-mm/apaw_XmqnEcY8sv7@MiWiFi-R3L-srv/
>
> Where I've ended up is that unbounded growth is a real concern. On a
> host with no memcg limit (root cgroup, and most desktop and embedded
> setups), an unlimited pool means usage can keep growing, with no
> admin visible ceiling at all. I'm not attached to xswap's percent of RAM
> knob specifically, but I do think some kind of bound makes sense.

Technically speaking, virtual swap *is* bounded, i.e 8PB (xarray
induced a limit to the number of clusters it can allocate). But more
importantly, it's effectively capped by the backend it can take:
zswap, swap cache, disk swap, etc.

I'm not entirely certain this will be problem in practice...? Besides,
we can always add new knobs if needs arise (especially since the old
design still exists if unlimited swapping is somehow real), but if we
started out with some knobs that turns out to be unnecessary or
undesirable, removing them and/or educating users to be careful with
using them is harder, in my opinion and experience deploying zswap
across the fleet in the past 4 years.

>
> > 2. Writeback support. Writeback is core functionality for zswap, not an
> >    add-on, and a design needs to account for it from the start. This came
> >    up before, in the discussion around Chris' ghost swapfile RFC [17]: for
> >    a solution here to be acceptable, it has to work with the primary
> >    usecase and support disk writeback. Without it, whatever zswap won't
> >    take (incompressible pages especially) has nowhere to go, and cold
> >    compressed data can never leave RAM.
>
> Agreed completely, and I think this one is settled: whatever lands
> has to support disk writeback in the end. Baoquan mentioned that
> what he sent is just a foundation though. I know talking is cheap
> but still, could it be combined with the writeback support from the
> other side? Or am I missing anything here?

I'm just very nervous about any design that doesn't quite take that
into account and handwave it away into a follow-up :)

>
> > 3. Cgroup charging behavior. vswap/xswap shouldn't be charged against the
> >    swap usage counter. It's fundamentally a different resource from
> >    physical swapfile space, and memory.swap.* should read 0 when nothing is
> >    on disk [18]. I made the longer argument for this in [19].
>
> I personally don't have a strong opinion on this. But I think this
> needs an explicit decision from the group rather than being
> inherited from whichever series merges first. Skipping the charge
> when a device has a special flag isn't hard to do if the code
> abstraction is done well, so this is more of a design decision
> than a technical problem.

I'm fine with discussing about this. I just want to point out that
this is not something I snuck in :) This has been the behavior since
probably the first or second RFC of the first design though, and in
most of them, it is a front-and-centre patch of its own, not hushed
away in one big patch. Folks have had 1.5 years to comment on it (and
still not too late - nothing is merged yet!)

I'd love to hear opinions from other zswap users regarding this :)

>
> > 4. Data structure (xarray vs sparse vmalloc array). Even with xarray, vswap
> >    is already on par with or beating baseline. I like the sparse array
> >    idea, but why are we landing an optimization before the feature itself,
> >    without any A/B data showing the difference matters?
>
> This is where I'd push back a little. To be clear, my concern isn't the
> size of the win, it's that VM_SPARSE isn't purely an optimization. It
> changes the cluster/metadata layout and avoids RCU conventions,
> could help avoid some future sleep while atomic limitations and
> it's tangled with the initial size issue too. So bolting it on afterwards
> could mean rewriting the parts of vswap that touch it and could be
> costly some conflict in ideas. That's why I'd rather we agree on
> it now, not that it has to land first.

I don't think the current code differs significantly from the existing
model, no?

For the most part, RCU played quite nicely with the current swap code.
Practically, it's just:

1. If you have a reference to the swap entry, you can assume entry and
the cluster owning it is safe.

2. If you don't, then do not assume it's safe once you exit the rcu
read section.

What conflicts specifically do you have in mind here :)

>
> Using VM_SPARSE does outsmart my earlier xarray suggestion :) There
> is an interesting early benchmark from Baoquan, VM_SPARSE wasn't there
> yet, but with it we can use clusters natively in the same way. It shows
> about 20% latency cut and 10% throughput gain, which seems a major
> improvement, and that's serial performance, not even mentioning
> scalability yet:
> https://lore.kernel.org/linux-mm/20260707073215.72183-1-baoquan.he@linux.dev/

I think a lot of these wins come from the fact that the zswap's xarray
specifically is *really* bad in certain paths. For instance, in the
free path (swap_range_free() -> zswap_invalidate()), there is no
batching behavior, so we walk the tree from scratch for each entry
rather than using an xas iterator for the entire batch. That goes away
in the current vswap model, because the xarray manages cluster rather
than entries, so you just do one tree walk + one flat array walk.

You can see some performance numbers in my cover letter - it's
promising on both micro benchmarks, and a production workload that I
throw vswap on :) I've also done an ablative study, where I still let
zswap and swapfile coupled, but only remove the xarray and replace it
with the flat array in swapfile's cluster struct. I observe the same
win, basically (especially on the usemem benchmark).

>
> And it matters more than most micro-optimizations: under heavy
> pressure many workloads spend less than half of system time in
> compression itself, the rest is mm/swap. Swap has had a terrible
> reputation for good reasons, and even with zswap/zram users
> have been scared of the performance.
>
> After the recent refactor and optimization we already noticed many
> users can now run much heavier workloads using compressed memory only
> or fast storages, which is a real breakthrough. Every chunk of
> overhead we remove there unblocks another class of user from using
> compressed memory as a "secondary memory".

I don't necessarily disagree with any of the points you made here.
FWIW, I like the vmalloc array idea too. I have actually put in work
to review Baoquan's patch series, because I'm hopeful that it will be
superior in the end (we're using this infra too so it helps us too :))

I just want to re-iterate that:

1. Vswap is already decently performant.

2. We're not removing the old code, so no one will regress with vswap
(they can just disable it for now). Optimization can be done on top of
it after we land the use cases (which also unblock zswap usage for
many parties who are capped by the storage limitations).

>
> >
> > One thing I do want to be clear about: I'm glad other people care about
> > this problem. Chris' ghost swapfile and Baoquan's xswap are both going
> > after the same set of problems, and that's a good sign. It means this is
> > real and shared, not something only Meta runs into.
> >
> > What's been harder is the shape of the engagement. Alternatives keep
> > getting posted and pushed that don't cover all the requirements, while this
> > series sits without review. I don't think I'm owed anyone's interest in the
> > problems I care about. But I do think working code, with benchmarks and
> > production exposure behind it, deserves a fair hearing next to in-progress
> > proposals.
>
> I'm sorry it has felt that way, and I don't want to hand wave it. For
> what it's worth, I did review it, and I did think it could be merged
> with some adjustments:
>
> https://lore.kernel.org/linux-mm/CAMgjq7D4XsAD4NGDL7FC2kaYAQAP8PDJdn4bpzGZwXYtjEpJ6w@mail.gmail.com/#t
>
> We were also discussing the limitation of the global cluster cache in
> that thread, so I paused my other work for a while and posted the swap
> queue series first. That one is orthogonal on purpose, and helps both
> swap tier and vswap, it reduces the overhead enough to cancel out
> the layering cost due to missing caching for either:
> https://lore.kernel.org/linux-mm/20260829-swap-pcp-priq-v2-resend-0-68d3d925578c@gmail.com/
> (this is V2, V1 is earlier - thanks to Lian for keeping this going!)

Ah that! I sort of tabled that as a nice-idea-but-still-optimization :P

I'll spend some time thinking about the design and reviewing it. In
the long run, it will get things more simple, but another thing that
should not be a blocker, correct?

>
> Then xswap appeared with the VM_SPARSE idea and exploring it seems
> necessary because getting the cluster foundation right affects
> everything built above where it went. Not because the vswap series
> wasn't worth reviewing.
>
> > So what I'm asking for: I'd like us to converge rather than keep two series
> > in flight. My preference is that we land vswap first, then build Baoquan's
> > sparse array on top of it as an optimization. That gets the feature in, and
> > by then we'd have the A/B data to show whether the sparse array actually
> > beats the xarray.
> >
> > If you think that's the wrong order, I'd genuinely like to understand why -
> > after 17 months and 10 revisions I still don't have a clear picture of the
> > objection.
>
> I fully agree we must converge. At the moment I don't think one can
> can settle that alone as there are several of us reviewing both. I'll try
> to be more concrete about what I think has to be agreed first.
>
> The two interfaces aren't equivalent, xswap has its limitations, but a
> couple of xswap's properties look genuinely better to me. Runtime
> on/off rather than a boot parameter is better, more flexibility and

I would also like to point out - I *do* have a runtime parameter in an
older version of vswap (v3). There is also not a lot of engagement
from folks in that version - no one expressed any interest in that
runtime knob in particular. Only Johannes made the case that a boot
parameter is superior, so I switched over. I think his reasoning makes
sense at the time FWIW, and it also removes the need for CONFIG_VSWAP
(which in turns remove a bunch of useless stubs - saving 100 LoCs), so
I figure folks will prefer less code and less userspace decisions :)

v3: https://lore.kernel.org/all/20260806184254.3790858-1-nphamcs@gmail.com/

Regardless - do you have a specific use case in mind where a runtime
knob would be useful?

> doesn't conflict with what vswap is trying to do. Some other of these
> differences involve design decisions rather than just surface knobs,
> and those are the parts we're stuck with once they ship.
>
> So a few current interface difference of vswap and xswap I've noticed are:
>
>  - default size: somehow also tangled with VM_SPARSE, which
> needs an initial virtual value. It's super cheap if not zero cost
> (the only cost is occupying some vmalloc area, which is fine
> I guess?). But we still have to pick a number, could be large
> enough to satisfy everyone?

8PB? ;)

I think choosing a meaningful number that covers from O(dozen of GB)
to O(terrabytes) scale machine, with different workload shapes, access
patterns, memory compressibility, etc. is an unsolvable problem. If
it's cheap, why don't we just max out?

>  - boot vs runtime toggle: I believe a runtime toggle is better. It
> can still be tuned on boot via cmdline, but it offers the extra
> ability to turn it on without modifying a base VM boot image or
> kernel.
>  - growth / limits: as talked above.
>  - charging semantics
>  - there could be other longterm design decisions on top of these as
> we talked in above.
>
> Once these settle, I think the picture will be clean. Your writeback
> logic and swap table integration are solid, while Baoquan's VM_SPARSE,
> previous preparation with swap ops give us a structural foundation. A
> clean long-term architecture means getting the core structure right,
> let's try not treating this as a race between series.

I've reached out to Baoquan as well, but I figured there's value in
both discussions in big and small groups, and especially since there's
a chance some of us might not be coming to LPC. Let's see what we come
up with.

Re: Path forward for Virtualized Swap?

[Nhat Pham]

On Mon, Sep 7, 2026 at 4:30 AM David Hildenbrand (Arm) <david@kernel.org> wrote:
>
> On 9/4/26 23:14, Nhat Pham wrote:
>
> Hi,
>
> > To Kairui, Johannes, Yosry, Chris, David, Andrew, and other (z)swap folks -
> > I need some help here.
>
> I'll mostly ignore swap-related stuff because (a) -EBUSY and (b) we have quite
> some swap experts around :)
>
> Reading below, I do wonder if the swap subsystem needs some more coordination
> and collaboration?
>
> For example, maybe having bi-weekly/monthly meetings to get everybody aligned
> and uptodate could be something reasonable?
>
> This approach has proven fairly helpful in areas where (1) multiple companies
> are involved (2) many concurrent development is happening, and (3) people have
> different priorities and requirements.
>
> Is there already something like that to organize development? Should we get
> started with that?

Hi David!

Thank you for your suggestions :) More collaboration and communication
is definitely needed here. Conference is a good place for this, but
it's not something we can do very frequently, and many folks could not
attend (visa issues, etc.).

A bi-weekly/monthly series of meetings sound very appealing. Swap
contributors are very spread out (China, NorAm, Europe, etc.), so
coordination will be tricky, but let us figure out something.

Re: Path forward for Virtualized Swap?

[Johannes Weiner]

On Mon, Sep 07, 2026 at 01:51:31PM +0800, Kairui Song wrote:
> Where I've ended up is that unbounded growth is a real concern. On a
> host with no memcg limit (root cgroup, and most desktop and embedded
> setups), an unlimited pool means usage can keep growing, with no
> admin visible ceiling at all. I'm not attached to xswap's percent of RAM
> knob specifically, but I do think some kind of bound makes sense.

Swap space is just process virtual address space, no?

Swap entries already have one or more page table entries pointing to
them, which in turn are managed by trees of vm_area_structs. That
means rlimits apply, overcommit protection applies, and OOM killer
attribution works as well (oom_badness()).

Shmem has its own defaults and limit interface on address space.

I'm not quite seeing how the swap space needs an additional limit. How
could users break things in unique new ways without it?

It would be good to spell out that vector before discussing numbers :)

Re: Path forward for Virtualized Swap?

[Nhat Pham]

On Tue, Sep 8, 2026 at 11:30 AM Johannes Weiner <hannes@cmpxchg.org> wrote:
>
> On Mon, Sep 07, 2026 at 01:51:31PM +0800, Kairui Song wrote:
> > Where I've ended up is that unbounded growth is a real concern. On a
> > host with no memcg limit (root cgroup, and most desktop and embedded
> > setups), an unlimited pool means usage can keep growing, with no
> > admin visible ceiling at all. I'm not attached to xswap's percent of RAM
> > knob specifically, but I do think some kind of bound makes sense.
>
> Swap space is just process virtual address space, no?
>
> Swap entries already have one or more page table entries pointing to
> them, which in turn are managed by trees of vm_area_structs. That
> means rlimits apply, overcommit protection applies, and OOM killer
> attribution works as well (oom_badness()).

I tested this theory. I spinned up a process, and let it spam 0-filled
memory + swap these pages out continually.

As you predicted, oom-killer picked it up eventually. The host was
(and is) intact otherwise :)

Re: Path forward for Virtualized Swap?

[Nhat Pham]

On Wed, Sep 9, 2026 at 9:41 AM Nhat Pham <nphamcs@gmail.com> wrote:
>
> On Tue, Sep 8, 2026 at 11:30 AM Johannes Weiner <hannes@cmpxchg.org> wrote:
> >
> > On Mon, Sep 07, 2026 at 01:51:31PM +0800, Kairui Song wrote:
> > > Where I've ended up is that unbounded growth is a real concern. On a
> > > host with no memcg limit (root cgroup, and most desktop and embedded
> > > setups), an unlimited pool means usage can keep growing, with no
> > > admin visible ceiling at all. I'm not attached to xswap's percent of RAM
> > > knob specifically, but I do think some kind of bound makes sense.
> >
> > Swap space is just process virtual address space, no?
> >
> > Swap entries already have one or more page table entries pointing to
> > them, which in turn are managed by trees of vm_area_structs. That
> > means rlimits apply, overcommit protection applies, and OOM killer
> > attribution works as well (oom_badness()).
>
> I tested this theory. I spinned up a process, and let it spam 0-filled
> memory + swap these pages out continually.
>
> As you predicted, oom-killer picked it up eventually. The host was
> (and is) intact otherwise :)

Oh digging through dmesg - I did find something quite interesting:

[12462.749663] ------------[ cut here ]------------
[12462.749670] refcount_t: saturated; leaking memory.
[12462.749817] ------------[ cut here ]------------
[12462.759267] WARNING: lib/refcount.c:19 at
refcount_warn_saturate+0x1b/0x70, CPU#17: zeroleak/76075
[12462.759276] refcount_t: underflow; use-after-free.
[12462.777208] Modules linked in: sch_fq(E) tcp_diag(E) inet_diag(E)
act_gact(E) cls_bpf(E)
[12462.786790] WARNING: lib/refcount.c:28 at
refcount_warn_saturate+0x42/0x70, CPU#25: dmesg/76074

Seems like it managed to oversaturate mem_cgroup_private_id:

struct mem_cgroup_private_id {
    int id;
    refcount_t ref;
};

We should either fix mem_cgroup_private_id to use 64-bit refererence
count here, or hardcap the swap usage to below the saturation point of
refcount_t ref. I'm leaning towards the former in the long run.

This is sort of an argument for handling all of this transparently
though - expecting users to know what the cap should be based on a
myriad of architectural and arcane implementation details (which can
change arbitrarily in the future) is very unreasonable.

Re: Path forward for Virtualized Swap?

[Baoquan He]

Hi Nhat,

On 09/04/26 at 02:14pm, Nhat Pham wrote:
.....snip...
> Now, on xswap. Baoquan's working on a series [15] that covers some of the
> same ground, and the VM_SPARSE cluster_info idea in it is genuinely good.
> I've been reviewing that lineage since July [16] and I'd like whatever
> lands to end up with the best parts of both. From my perspective the
> differences are:
> 
> 1. Userspace knobs. xswap asks the admin for a size (a percent of RAM) plus
>    a per-device limit to tune afterwards. I'm not aware of any use case
>    that needs those, and I don't think users have a good way to answer the
>    question anyway - sizing swap for compressed memory depends on memory
>    size, workload, and compression ratio all at once. That's precisely the
>    provisioning problem vswap exists to remove. The kernel should be as
>    transparent and dynamic as possible here, and not add knobs unless
>    there's a use case for them.
> 
> 2. Writeback support. Writeback is core functionality for zswap, not an
>    add-on, and a design needs to account for it from the start. This came
>    up before, in the discussion around Chris' ghost swapfile RFC [17]: for
>    a solution here to be acceptable, it has to work with the primary
>    usecase and support disk writeback. Without it, whatever zswap won't
>    take (incompressible pages especially) has nowhere to go, and cold
>    compressed data can never leave RAM.
> 
> 3. Cgroup charging behavior. vswap/xswap shouldn't be charged against the
>    swap usage counter. It's fundamentally a different resource from
>    physical swapfile space, and memory.swap.* should read 0 when nothing is
>    on disk [18]. I made the longer argument for this in [19].
> 
> 4. Data structure (xarray vs sparse vmalloc array). Even with xarray, vswap
>    is already on par with or beating baseline. I like the sparse array
>    idea, but why are we landing an optimization before the feature itself,
>    without any A/B data showing the difference matters?


Thanks for laying this out, and for the honest push to converge. Let me
be equally direct about the ordering: I think the xswap base should land
first, and the things vswap demonstrates -  writeback, rmap lookup, the
charging semantics, later THP -- should be built on top of it. Because
it is the foundation that keeps the swap core simpler, and the first thing
to merge should be the one that doesn't have to be redone.

The VM_SPARSE array is not an optimization to bolt on later; it is a
structural choice, and the code reflects it. In vswap, the cluster
metadata lives in a dynamically-allocated xarray.

struct swap_cluster_info_dynamic {
        struct swap_cluster_info ci;
        unsigned int index;             /* for cluster_index() */
        struct rcu_head rcu;
        atomic_long_t *virtual_table;   /* Backing pointers for vswap slots */
};

To support dynamic growth and shrink, vswap stores its cluster metadata
in an xarray, and that forces two things the plain swap_cluster_info[]
array never needed:

1. Every cluster has to carry an extra index and an rcu_head —
24 bytes per cluster — purely so the xarray can locate it and free
it safely.
2. To keep that bookkeeping from leaking into the normal-swap code, the
cluster had to be wrapped in a container, swap_cluster_info_dynamic,
so the xarray holds a pointer to the wrapper instead of an inline
array element.

So in vswap, every cluster access in the shared hot path has to answer
"is this a vswap device?" and take a separate branch:

- swap_is_vswap() is checked in 36 places across page_io.c, swapfile.c,
  zswap.c and swap.h;
- __swap_offset_to_cluster() branches into xa_load() for vswap vs the
  flat array otherwise, and the xarray path can return NULL (a cluster
  can be torn down);
- __swap_cluster_lock() branches into __vswap_cluster_lock(), which
  wraps every access in rcu_read_lock() and a CLUSTER_FLAG_DEAD check,
  plus kfree_rcu()/container_of()/rcu_head plumbing for node lifetime.

With VM_SPARSE, xswap's cluster access is exactly the plain-array line the
rest of swap already uses:

    return &si->cluster_info[offset / SWAPFILE_CLUSTER];

no branch, no RCU discipline, no tear-down state machine, and no NULL
return. So VM_SPARSE doesn't add complexity to close a gap; it lets the
cluster layer stay as simple as it already is, which is precisely the
part later work (writeback, rmap lookup, memcg charging, THP) has to sit
on.

I'm not going to claim xswap wins on throughput. I measured it:
on a 64G/64-thread swapout, xswap, vswap and plain swap+zswap are all
within ~2-3% of each other, effectively identical. Because the cost is
dominated by zswap compression, not the cluster table. So the ordering
question is not "which is faster" but "which structure should the use-case
layer be built on". If vswap is chosen, the xarray-based table is an
intermediate form. Your own ablative study already showed the flat array
wins, and VM_SPARSE is exactly that flat array plus lazy mapping.
 
On the metadata side I want to be precise, because it is easy to overstate:
the per-cluster cost that xswap saves is the xarray-induced index + rcu_head
(20 bytes/cluster or 24bytes for alignment), small. 

On writeback: agreed it is required in the end. But it is a consumer of
the foundation, not a reason to pick a different one. xswap is deliberately
the base; 
- writeback
- rmap lookup
- THP support
- memcg accounting
All these can land on top of the xswap base rather than be stranded on a table
we later replace. As we have discussed and I have been mentioning in each
cover-letter, I didn't touch these core changes, glad to see your work built
on top of it.

On the interface, xswap v2 drops the percent knob entirely (your point about
"why not just max out" is taken): create now takes only an optional
priority, a device starts at full RAM, which is free because the VM_SPARSE
area is mapped lazily, with an optional per-device size limit for admins
who want a ceiling. More importantly, xswap keeps per-device instances
because the swap->ops and swap tiering that come next need per-device
operations. A single boot-time vswap can't express that, and it breaks the
per-device conventions the rest of swap already follows. When I tested it,
there is no way to disable it at runtime, so it can't even be A/B-tested
against regular swap in the same boot. So I think a boot-time vswap is an
independent issue which deserves a separate patch posting with a convincing
justification later.

So concretely: xswap base first (runtime file-less device + VM_SPARSE
cluster foundation + sysfs create/destroy), then the use-case layer, where
your writeback work, etc is very welcome. The base should be the one that
doesn't need to be redone; by both our measurements, that is the flat-cluster
substrate.

Thanks
Baoquan

> 
> One thing I do want to be clear about: I'm glad other people care about
> this problem. Chris' ghost swapfile and Baoquan's xswap are both going
> after the same set of problems, and that's a good sign. It means this is
> real and shared, not something only Meta runs into.
> 
> What's been harder is the shape of the engagement. Alternatives keep
> getting posted and pushed that don't cover all the requirements, while this
> series sits without review. I don't think I'm owed anyone's interest in the
> problems I care about. But I do think working code, with benchmarks and
> production exposure behind it, deserves a fair hearing next to in-progress
> proposals.
> 
> So what I'm asking for: I'd like us to converge rather than keep two series
> in flight. My preference is that we land vswap first, then build Baoquan's
> sparse array on top of it as an optimization. That gets the feature in, and
> by then we'd have the A/B data to show whether the sparse array actually
> beats the xarray.
> 
> If you think that's the wrong order, I'd genuinely like to understand why -
> after 17 months and 10 revisions I still don't have a clear picture of the
> objection.
> 
> [1] https://lore.kernel.org/all/20250407234223.1059191-1-nphamcs@gmail.com/
> [2] https://lore.kernel.org/all/20250429233848.3093350-1-nphamcs@gmail.com/
> [3] https://lore.kernel.org/all/20260208215839.87595-1-nphamcs@gmail.com/
> [4] https://lore.kernel.org/all/20260318222953.441758-1-nphamcs@gmail.com/
> [5] https://lore.kernel.org/all/20260320192735.748051-1-nphamcs@gmail.com/
> [6] https://lore.kernel.org/all/20260505153854.1612033-1-nphamcs@gmail.com/
> [7] https://lore.kernel.org/all/20260528212955.1912856-1-nphamcs@gmail.com/
> [8] https://lore.kernel.org/all/20260612193738.2183968-1-nphamcs@gmail.com/
> [9] https://lore.kernel.org/all/20260806184254.3790858-1-nphamcs@gmail.com/
> [10] https://lore.kernel.org/all/20260825153238.2695446-1-nphamcs@gmail.com/
> [11] https://lwn.net/Articles/1016136/
> [12] https://lore.kernel.org/all/CAMgjq7AQNGK-a=AOgvn4-V+zGO21QMbMTVbrYSW_R2oDSLoC+A@mail.gmail.com/
> [13] https://lore.kernel.org/all/CACePvbVXQWgcPD-bgK7iDba4NFLo2tT89ZbLOa03maJU4er4ag@mail.gmail.com/
> [14] https://lore.kernel.org/all/aZyFxKGXc8J6PIij@cmpxchg.org/
> [15] https://lore.kernel.org/all/20260827094509.1016740-1-hebaoquan@kylinos.cn/
> [16] https://lore.kernel.org/lkml/CAKEwX=Pe+qMZd2xhnU-PAGQtgXkp56c-JwYCbt2Lux9htgB67Q@mail.gmail.com/
> [17] https://lore.kernel.org/all/20251121114011.GA71307@cmpxchg.org/
> [18] https://lore.kernel.org/all/anYIboHEUZb4fhHv@cmpxchg.org/
> [19] https://lore.kernel.org/all/CAKEwX=P4syV38jAVCWq198r2OHXXc=xA-fx1dk6+qYef6yzxWQ@mail.gmail.com/

Re: Path forward for Virtualized Swap?

[David Hildenbrand (Arm)]

On 9/8/26 18:45, Nhat Pham wrote:
> On Mon, Sep 7, 2026 at 4:30 AM David Hildenbrand (Arm) <david@kernel.org> wrote:
>>
>> On 9/4/26 23:14, Nhat Pham wrote:
>>
>> Hi,
>>
>>> To Kairui, Johannes, Yosry, Chris, David, Andrew, and other (z)swap folks -
>>> I need some help here.
>>
>> I'll mostly ignore swap-related stuff because (a) -EBUSY and (b) we have quite
>> some swap experts around :)
>>
>> Reading below, I do wonder if the swap subsystem needs some more coordination
>> and collaboration?
>>
>> For example, maybe having bi-weekly/monthly meetings to get everybody aligned
>> and uptodate could be something reasonable?
>>
>> This approach has proven fairly helpful in areas where (1) multiple companies
>> are involved (2) many concurrent development is happening, and (3) people have
>> different priorities and requirements.
>>
>> Is there already something like that to organize development? Should we get
>> started with that?
> 
> Hi David!

Hi!

> 
> Thank you for your suggestions :) More collaboration and communication
> is definitely needed here. Conference is a good place for this, but
> it's not something we can do very frequently, and many folks could not
> attend (visa issues, etc.).

Right, and conferences usually don't have the capacity to squeeze too many
swap-related topics in, so it would have to be side discussions or BOFs.

> 
> A bi-weekly/monthly series of meetings sound very appealing. Swap
> contributors are very spread out (China, NorAm, Europe, etc.), so
> coordination will be tricky, but let us figure out something.

Yes, that's an issue. I'd suggest swap maintainers take care of that if they
agree. I could try as well, but it's rather tricky right now on my end ...

As an alternative, we could also use on/two/ ... slots in the bi-weekly MM
alignment session to talk about swap. (I'd be happy to learn more about all this
as well)

-- 
Cheers,

David

Re: Path forward for Virtualized Swap?

[Nhat Pham]

On Thu, Sep 10, 2026 at 3:57 AM David Hildenbrand (Arm)
<david@kernel.org> wrote:
>
> On 9/8/26 18:45, Nhat Pham wrote:
> > On Mon, Sep 7, 2026 at 4:30 AM David Hildenbrand (Arm) <david@kernel.org> wrote:
> >>
> >> On 9/4/26 23:14, Nhat Pham wrote:
> >>
> >> Hi,
> >>
> >>> To Kairui, Johannes, Yosry, Chris, David, Andrew, and other (z)swap folks -
> >>> I need some help here.
> >>
> >> I'll mostly ignore swap-related stuff because (a) -EBUSY and (b) we have quite
> >> some swap experts around :)
> >>
> >> Reading below, I do wonder if the swap subsystem needs some more coordination
> >> and collaboration?
> >>
> >> For example, maybe having bi-weekly/monthly meetings to get everybody aligned
> >> and uptodate could be something reasonable?
> >>
> >> This approach has proven fairly helpful in areas where (1) multiple companies
> >> are involved (2) many concurrent development is happening, and (3) people have
> >> different priorities and requirements.
> >>
> >> Is there already something like that to organize development? Should we get
> >> started with that?
> >
> > Hi David!
>
> Hi!
>
> >
> > Thank you for your suggestions :) More collaboration and communication
> > is definitely needed here. Conference is a good place for this, but
> > it's not something we can do very frequently, and many folks could not
> > attend (visa issues, etc.).
>
> Right, and conferences usually don't have the capacity to squeeze too many
> swap-related topics in, so it would have to be side discussions or BOFs.
>
> >
> > A bi-weekly/monthly series of meetings sound very appealing. Swap
> > contributors are very spread out (China, NorAm, Europe, etc.), so
> > coordination will be tricky, but let us figure out something.
>
> Yes, that's an issue. I'd suggest swap maintainers take care of that if they
> agree. I could try as well, but it's rather tricky right now on my end ...
>
> As an alternative, we could also use on/two/ ... slots in the bi-weekly MM
> alignment session to talk about swap. (I'd be happy to learn more about all this
> as well)

That's awesome! Thanks for the offer David - I think MM alignment is at 9AM
PDT Wednesday, usually right? That works for me of course, but Baoquan
and Kairui (and other swap folks) - would this time be convenient for
you?


>
> --
> Cheers,
>
> David

Re: Path forward for Virtualized Swap?

[Shakeel Butt]

On Thu, Sep 10, 2026 at 03:09:59PM +0800, Baoquan He wrote:
> Hi Nhat,
> 
> On 09/04/26 at 02:14pm, Nhat Pham wrote:
> .....snip...

[...]

> With VM_SPARSE, xswap's cluster access is exactly the plain-array line the
> rest of swap already uses:
> 
>     return &si->cluster_info[offset / SWAPFILE_CLUSTER];
> 
> no branch, no RCU discipline, no tear-down state machine, and no NULL
> return. So VM_SPARSE doesn't add complexity to close a gap; it lets the
> cluster layer stay as simple as it already is, which is precisely the
> part later work (writeback, rmap lookup, memcg charging, THP) has to sit
> on.
> 
> I'm not going to claim xswap wins on throughput. I measured it:
> on a 64G/64-thread swapout, xswap, vswap and plain swap+zswap are all
> within ~2-3% of each other, effectively identical. 

So the claim is VM_SPARSE is simpler than xarray based approach. I feel like
we are discussing implementation details before deciding the design and
architecture. So, instead of VM_SPARSE vs xarray, let's discuss and decide the
need for dynamic growth. Why we want dynamic growth upfront or can it be added
later? Once we decide that then it will be very easy to pick an implementation
that would take us there.

Re: Path forward for Virtualized Swap?

[Johannes Weiner]

[This reply was not LLM-generated.]

On Thu, Sep 10, 2026 at 03:09:59PM +0800, Baoquan He wrote:
> Hi Nhat,
> 
> On 09/04/26 at 02:14pm, Nhat Pham wrote:
> .....snip...
> > Now, on xswap. Baoquan's working on a series [15] that covers some of the
> > same ground, and the VM_SPARSE cluster_info idea in it is genuinely good.
> > I've been reviewing that lineage since July [16] and I'd like whatever
> > lands to end up with the best parts of both. From my perspective the
> > differences are:
> > 
> > 1. Userspace knobs. xswap asks the admin for a size (a percent of RAM) plus
> >    a per-device limit to tune afterwards. I'm not aware of any use case
> >    that needs those, and I don't think users have a good way to answer the
> >    question anyway - sizing swap for compressed memory depends on memory
> >    size, workload, and compression ratio all at once. That's precisely the
> >    provisioning problem vswap exists to remove. The kernel should be as
> >    transparent and dynamic as possible here, and not add knobs unless
> >    there's a use case for them.
> > 
> > 2. Writeback support. Writeback is core functionality for zswap, not an
> >    add-on, and a design needs to account for it from the start. This came
> >    up before, in the discussion around Chris' ghost swapfile RFC [17]: for
> >    a solution here to be acceptable, it has to work with the primary
> >    usecase and support disk writeback. Without it, whatever zswap won't
> >    take (incompressible pages especially) has nowhere to go, and cold
> >    compressed data can never leave RAM.
> > 
> > 3. Cgroup charging behavior. vswap/xswap shouldn't be charged against the
> >    swap usage counter. It's fundamentally a different resource from
> >    physical swapfile space, and memory.swap.* should read 0 when nothing is
> >    on disk [18]. I made the longer argument for this in [19].
> > 
> > 4. Data structure (xarray vs sparse vmalloc array). Even with xarray, vswap
> >    is already on par with or beating baseline. I like the sparse array
> >    idea, but why are we landing an optimization before the feature itself,
> >    without any A/B data showing the difference matters?
> 
> 
> Thanks for laying this out, and for the honest push to converge. Let me
> be equally direct about the ordering: I think the xswap base should land
> first, and the things vswap demonstrates -  writeback, rmap lookup, the
> charging semantics, later THP -- should be built on top of it. Because
> it is the foundation that keeps the swap core simpler, and the first thing
> to merge should be the one that doesn't have to be redone.
> 
> The VM_SPARSE array is not an optimization to bolt on later; it is a
> structural choice, and the code reflects it. In vswap, the cluster
> metadata lives in a dynamically-allocated xarray.
> 
> struct swap_cluster_info_dynamic {
>         struct swap_cluster_info ci;
>         unsigned int index;             /* for cluster_index() */
>         struct rcu_head rcu;
>         atomic_long_t *virtual_table;   /* Backing pointers for vswap slots */
> };
> 
> To support dynamic growth and shrink, vswap stores its cluster metadata
> in an xarray, and that forces two things the plain swap_cluster_info[]
> array never needed:
> 
> 1. Every cluster has to carry an extra index and an rcu_head —
> 24 bytes per cluster — purely so the xarray can locate it and free
> it safely.
> 2. To keep that bookkeeping from leaking into the normal-swap code, the
> cluster had to be wrapped in a container, swap_cluster_info_dynamic,
> so the xarray holds a pointer to the wrapper instead of an inline
> array element.
> 
> So in vswap, every cluster access in the shared hot path has to answer
> "is this a vswap device?" and take a separate branch:
> 
> - swap_is_vswap() is checked in 36 places across page_io.c, swapfile.c,
>   zswap.c and swap.h;
> - __swap_offset_to_cluster() branches into xa_load() for vswap vs the
>   flat array otherwise, and the xarray path can return NULL (a cluster
>   can be torn down);
> - __swap_cluster_lock() branches into __vswap_cluster_lock(), which
>   wraps every access in rcu_read_lock() and a CLUSTER_FLAG_DEAD check,
>   plus kfree_rcu()/container_of()/rcu_head plumbing for node lifetime.

Well to state the obvious: the reason it does all that is to make the
compression space transparent to the user.

The user can answer a simple boolean question: whether they want
compression or not. And it will work on tiny machines, on humongous
machines, and everything in between. That's a simple policy question
with a clear answer.

What you're doing, asking the user for a static size, is much more
difficult and has usability issues.

You're comparing implementations that don't accomplish the same thing.

The problem we're trying to solve is implementing a clean compression
space abstraction. I'm arguing that vswap does, and xswap does not.

While they're both using parts of the swap device code to implement a
compression space, xswap actually PRESENTS IT TO THE USER as a swap
device, and then makes optimizations BASED ON BAKED IN LIMITATIONS.

But a conventional, statically sized swap device is a bad abstraction
for the compression space. Here is why:

In conventional swap space, one memory page translates to one swap
page. Compression space doesn't act this way: a memory page can
consume anything between a few bytes to a full page in compression
space. It depends on memory contents and compression algorithm. So
right off the bat, this is a hard question to answer at the host level
which could run all kinds of workloads.

In conventional swap space, the resource consumed is a different
one. You're offloading memory by consuming disk space. This eats into
the space available to the filesystem, which is totally unrelated.
Asking the user for this tradeoff is a legitimate policy question.

Compression space is not a separate resource. It's page tables,
backing pages, and swap descriptors. It's just MEMORY. There isn't a
size tradeoff, because moving pages from memory space into compression
space DOES NOT CONSUME A NEW RESOURCE. It's still just memory. All you
need for containment already exists: rlimits, OOM killer, cgroup
memory controls.

By making this a user-visible virtual swap device, you're sending
users down the wrong path. You're asking them to set a new limit on a
resource that's already limited by other means. You're framing the
question as conventional swap which behaves completely differently.

If you ask them "how much swap space", they WILL reference this to
available RAM capacity. Maybe half of ram, maybe twice the RAM.

But when compression space is referenced to RAM, it's trivial to fill
it up with zeroed pages or easily compressible data LONG BEFORE the
process or container would hit any of its MEMORY limits.

This creates an artificial resource shortages. It forces a competition
where there shouldn't be one. And then you need new controls to manage
a competition that doesn't have to exist.

Like I said before, including compression space (which is memory) in
memory.swap.* (which is for disk space) is not going to be acceptable
from the cgroup side. We can talk about that if you want.

But asking the user questions they shouldn't have to answer, or
already answered elsewhere, is weak interface design. Allowing, let
alone encouraging, answers that create a whole new host of
organizational issues is outright bad interface design.

So if you want to compare implementations, you first have to actually
implement the same thing:

Stop asking user "how large". Let compression space expand towards
existing memory limits, such that it doesn't create an awkward and
artificial new resource competition.

Then we can compare implementations.

If the optimizations still apply under those constraints, great.

Until then, there is little point in discussing differences that, by
your own admission, have little to no impact on real world performance.

Re: Path forward for Virtualized Swap?

[Nhat Pham]

On Thu, Sep 10, 2026 at 12:10 AM Baoquan He <baoquan.he@linux.dev> wrote:
>
> Hi Nhat,

Hi Baoquan,

Thanks for the detailed explanations. I'm glad it's all laid out - I
think this clarifies a lot of the confusions both of us have about
each other's perspectives so far. I suspect we're talking past each
other a bit (hence the effort to organize some form of VC, or
preferably a room where we can whiteboard things).

It's a very dense conversation, so I hope I don't miss any of your points.

>
> On 09/04/26 at 02:14pm, Nhat Pham wrote:
> .....snip...
> > Now, on xswap. Baoquan's working on a series [15] that covers some of the
> > same ground, and the VM_SPARSE cluster_info idea in it is genuinely good.
> > I've been reviewing that lineage since July [16] and I'd like whatever
> > lands to end up with the best parts of both. From my perspective the
> > differences are:
> >
> > 1. Userspace knobs. xswap asks the admin for a size (a percent of RAM) plus
> >    a per-device limit to tune afterwards. I'm not aware of any use case
> >    that needs those, and I don't think users have a good way to answer the
> >    question anyway - sizing swap for compressed memory depends on memory
> >    size, workload, and compression ratio all at once. That's precisely the
> >    provisioning problem vswap exists to remove. The kernel should be as
> >    transparent and dynamic as possible here, and not add knobs unless
> >    there's a use case for them.
> >
> > 2. Writeback support. Writeback is core functionality for zswap, not an
> >    add-on, and a design needs to account for it from the start. This came
> >    up before, in the discussion around Chris' ghost swapfile RFC [17]: for
> >    a solution here to be acceptable, it has to work with the primary
> >    usecase and support disk writeback. Without it, whatever zswap won't
> >    take (incompressible pages especially) has nowhere to go, and cold
> >    compressed data can never leave RAM.
> >
> > 3. Cgroup charging behavior. vswap/xswap shouldn't be charged against the
> >    swap usage counter. It's fundamentally a different resource from
> >    physical swapfile space, and memory.swap.* should read 0 when nothing is
> >    on disk [18]. I made the longer argument for this in [19].
> >
> > 4. Data structure (xarray vs sparse vmalloc array). Even with xarray, vswap
> >    is already on par with or beating baseline. I like the sparse array
> >    idea, but why are we landing an optimization before the feature itself,
> >    without any A/B data showing the difference matters?
>
>
> Thanks for laying this out, and for the honest push to converge. Let me
> be equally direct about the ordering: I think the xswap base should land
> first, and the things vswap demonstrates -  writeback, rmap lookup, the
> charging semantics, later THP -- should be built on top of it. Because
> it is the foundation that keeps the swap core simpler, and the first thing
> to merge should be the one that doesn't have to be redone.
>
> The VM_SPARSE array is not an optimization to bolt on later; it is a
> structural choice, and the code reflects it. In vswap, the cluster
> metadata lives in a dynamically-allocated xarray.
>
> struct swap_cluster_info_dynamic {
>         struct swap_cluster_info ci;
>         unsigned int index;             /* for cluster_index() */
>         struct rcu_head rcu;
>         atomic_long_t *virtual_table;   /* Backing pointers for vswap slots */
> };
>
> To support dynamic growth and shrink, vswap stores its cluster metadata
> in an xarray, and that forces two things the plain swap_cluster_info[]
> array never needed:
>
> 1. Every cluster has to carry an extra index and an rcu_head —
> 24 bytes per cluster — purely so the xarray can locate it and free
> it safely.

I will flip it the other way around too.

With this approach, you induce 8 bytes of overhead even on physical
cluster that has no business storing vswap related information. This
overhead exists as long as you build with CONFIG_XSWAP, even on
cgroups that don't use it.

That's sort of why I went with the new cluster struct embedding the
existing struct, rather than adding the vtable/xs_table to the
existing struct in v3 and v4. I mean it's small, but why do it if it's
avoidable?

> 2. To keep that bookkeeping from leaking into the normal-swap code, the
> cluster had to be wrapped in a container, swap_cluster_info_dynamic,
> so the xarray holds a pointer to the wrapper instead of an inline
> array element.
>
> So in vswap, every cluster access in the shared hot path has to answer
> "is this a vswap device?" and take a separate branch:
>
> - swap_is_vswap() is checked in 36 places across page_io.c, swapfile.c,
>   zswap.c and swap.h;

I think if you extend the vmalloc approach to cover the writeback use
case, you'll find equally many places where you want to check for
vswap.

It's simply cannot be avoided - if anything, for correctness. Some of
it will go away as we evolve the code (for e.g, if vswap becomes the
sole way we can use zswap).

> - __swap_offset_to_cluster() branches into xa_load() for vswap vs the
>   flat array otherwise, and the xarray path can return NULL (a cluster
>   can be torn down);

The xa_load() I have no arguments for. It's just data-structure
requirements, but it should go away were you to move to vmalloc array
down the line.

But the NULL-check is not an xarray property. It's a fundamental
requirement for true dynamicity - i.e if the cluster can go away under
us, then we have to null check wherever we do not have a surefire
reference, no? For e.g, in do_swap_page(), before adding the page into
swap cache, you cannot be certain that the swap entry is staying
around...

> - __swap_cluster_lock() branches into __vswap_cluster_lock(), which
>   wraps every access in rcu_read_lock() and a CLUSTER_FLAG_DEAD check,
>   plus kfree_rcu()/container_of()/rcu_head plumbing for node lifetime.
>
> With VM_SPARSE, xswap's cluster access is exactly the plain-array line the
> rest of swap already uses:
>
>     return &si->cluster_info[offset / SWAPFILE_CLUSTER];
>
> no branch, no RCU discipline, no tear-down state machine, and no NULL
> return. So VM_SPARSE doesn't add complexity to close a gap; it lets the
> cluster layer stay as simple as it already is, which is precisely the
> part later work (writeback, rmap lookup, memcg charging, THP) has to sit
> on.

Except you have to write multiple patches to handle dynamic growth
(and a limited version of the shrinking).

I don't think it's a fair characterization to say "vmalloc is simpler
(if you just avoid these multiple patches I wrote to custom roll an
extensible array)".

>
> I'm not going to claim xswap wins on throughput. I measured it:
> on a 64G/64-thread swapout, xswap, vswap and plain swap+zswap are all
> within ~2-3% of each other, effectively identical. Because the cost is
> dominated by zswap compression, not the cluster table. So the ordering
> question is not "which is faster" but "which structure should the use-case
> layer be built on". If vswap is chosen, the xarray-based table is an
> intermediate form. Your own ablative study already showed the flat array
> wins, and VM_SPARSE is exactly that flat array plus lazy mapping.

I need to clarify this point in particular. What my experiments on the
synthetic/semi-synthetic benchmarks show is

1. Baseline (old code) / vswap / baseline + flatarray (i.e replacing
zswap xarray with a flat array) performs within noises in most
benchmarks, on most metrics

2. vswap and baseline + flatarray both outperforms baseline on free
time of usemem, by the same margin.

Basically, what I wanted to show that a huge part of the wins come
from zswap's xarray being very inefficiently used in free path (for
e.g during process exit). Vswap also uses xarray, but in a more
optimized way in this path (one single tree walk + one flat array
walk).

So I'm not quite sure your characterization of "flat array wins" here
is quite fair. It wins... compared to status quo, not vswap.

So IOW, vmalloc array approach does NOT outperform xarray approach, in
both of our testing.

>
> On the metadata side I want to be precise, because it is easy to overstate:
> the per-cluster cost that xswap saves is the xarray-induced index + rcu_head
> (20 bytes/cluster or 24bytes for alignment), small.

Thanks for verifying my analysis. I understand that this overhead was
a long-time concern for folks - Chris pushed me a lot to minimize
this. Took me awhile to realize I could get around it.

>
> On writeback: agreed it is required in the end. But it is a consumer of
> the foundation, not a reason to pick a different one. xswap is deliberately
> the base;
> - writeback
> - rmap lookup
> - THP support
> - memcg accounting
> All these can land on top of the xswap base rather than be stranded on a table
> we later replace. As we have discussed and I have been mentioning in each
> cover-letter, I didn't touch these core changes, glad to see your work built
> on top of it.
>
> On the interface, xswap v2 drops the percent knob entirely (your point about
> "why not just max out" is taken): create now takes only an optional
> priority, a device starts at full RAM, which is free because the VM_SPARSE
> area is mapped lazily, with an optional per-device size limit for admins
> who want a ceiling. More importantly, xswap keeps per-device instances
> because the swap->ops and swap tiering that come next need per-device
> operations. A single boot-time vswap can't express that, and it breaks the

Why do we need multiple devices to support per-device operations.

I have decided not to opt it in to swapops yet (vswap swapops),
because it just doesn't really buy anything quite yet.  But vswap is
just a normal swap device, no? It can have its own swap device
operations etc.

> per-device conventions the rest of swap already follows. When I tested it,
> there is no way to disable it at runtime, so it can't even be A/B-tested
> against regular swap in the same boot. So I think a boot-time vswap is an
> independent issue which deserves a separate patch posting with a convincing
> justification later.


I think it's the other way around. We need justifications for why we
should expose virtual swap device to the swap tiering abstraction.

I think we should leave virtual swap device out of this interface (i.e
being transparent to the user), and add it only IF both are true:

1. There are some cgroups where we want vswap, and other that we don't.

AND

2. This is not something the kernel can decide based on the
information it already has.

1 seems reasonable, but for the use case I can think of, I can almost
always come up with a heuristics to decide whether vswap can be
bypassed (for e.g, if a cgroup only allows for physical swap devices,
it seems like there's no good reason to use vswap at the moment -
you're just unnecessarily paying indirection cost). So we don't need
extra userspace input here.

Of course, this might change in the future. But I struggle to see how
vswap cannot be extended to fit in the swap tiering model? It's just a
normal swap device - you can export some special identifier for it,
and user can echo <that identifier> to the cgroup's swap tiering file.

>
> So concretely: xswap base first (runtime file-less device + VM_SPARSE
> cluster foundation + sysfs create/destroy), then the use-case layer, where
> your writeback work, etc is very welcome. The base should be the one that
> doesn't need to be redone; by both our measurements, that is the flat-cluster
> substrate.

It's not quite as simple as that.

I think the other concerns is that can xswap be extended to support
vswap's full scope. I will note that as of the v1 of your patch
series, even the dynamic growth portion itself of xswap is still a bit
buggy. Maybe v2 will fix this, but that would be multiple more
versions to even unblock a subset of the use cases. Vswap meanwhile
has more efforts poured into it, including production workload
exposure. You're asking us to invalidate all of this effort, which
will set us back quite a bit of time.

Re: Path forward for Virtualized Swap?

[David Hildenbrand (Arm)]

On 9/10/26 18:22, Nhat Pham wrote:
> On Thu, Sep 10, 2026 at 3:57 AM David Hildenbrand (Arm)
> <david@kernel.org> wrote:
>>
>> On 9/8/26 18:45, Nhat Pham wrote:
>>>
>>> Hi David!
>>
>> Hi!
>>
>>>
>>> Thank you for your suggestions :) More collaboration and communication
>>> is definitely needed here. Conference is a good place for this, but
>>> it's not something we can do very frequently, and many folks could not
>>> attend (visa issues, etc.).
>>
>> Right, and conferences usually don't have the capacity to squeeze too many
>> swap-related topics in, so it would have to be side discussions or BOFs.
>>
>>>
>>> A bi-weekly/monthly series of meetings sound very appealing. Swap
>>> contributors are very spread out (China, NorAm, Europe, etc.), so
>>> coordination will be tricky, but let us figure out something.
>>
>> Yes, that's an issue. I'd suggest swap maintainers take care of that if they
>> agree. I could try as well, but it's rather tricky right now on my end ...
>>
>> As an alternative, we could also use on/two/ ... slots in the bi-weekly MM
>> alignment session to talk about swap. (I'd be happy to learn more about all this
>> as well)
> 
> That's awesome! Thanks for the offer David - I think MM alignment is at 9AM
> PDT Wednesday, usually right? 

Yes, exactly. David R. hosts it!

-- 
Cheers,

David

Re: Path forward for Virtualized Swap?

[Nhat Pham]

Hi Kairui,

To quantify how much we are locking in to the xarray design if we are to go
with it, I tried hacking the vmalloc array to replace the xarray, on top of
my v4 code.

I have not implemented the shrink side yet, and I have not switched from
"swap_cluster_info_dynamic" to an embedded table in the old struct
swap_cluster_info yet, but these are straightforward to extend.

I have done a simple stress test (and also fixed another issue that I
pointed out in my review on Baoquan's code in the process). Nothing has
crashed yet :)

Anyway, it's just a prototype so no need to look too closely, but as you
can see, xarray does not lock us into anything. The xarray's interaction
with swap code is well-abstracted and transparent enough that I don't think
this will be a problem :) We should land xarray version first.

---
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 8df74bd2b1e1..0464e9b55413 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -169,7 +169,9 @@ static inline void mm_account_reclaimed_pages(unsigned long pages)
 #ifdef __KERNEL__

 struct address_space;
+struct swap_cluster_info_dynamic;
 struct sysinfo;
+struct vm_struct;
 struct zone;

 /*
@@ -277,7 +279,11 @@ struct swap_info_struct {
 	struct list_head discard_clusters; /* discard clusters list */
 	struct plist_node avail_list;   /* entry in swap_avail_head */
 	const struct swap_ops *ops;
-	struct xarray cluster_info_pool; /* Xarray for vswap dynamic cluster info */
+	struct vm_struct *cluster_info_area; /* Vswap cluster array reservation */
+	struct swap_cluster_info_dynamic *vswap_cluster_info;
+					/* Vswap cluster array, mapped on demand */
+	unsigned int nr_mapped_clusters; /* Mapped prefix of the array */
+	struct mutex cluster_grow_lock;	/* Serialize growth of the array */
 };

 static inline bool swap_is_vswap(struct swap_info_struct *si)
diff --git a/mm/swap.h b/mm/swap.h
index 10a7be3c4341..91a641a70426 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -68,8 +68,6 @@ struct swap_cluster_info {

 struct swap_cluster_info_dynamic {
 	struct swap_cluster_info ci;
-	unsigned int index;		/* for cluster_index() */
-	struct rcu_head rcu;
 	atomic_long_t *virtual_table;	/* Backing pointers for vswap slots */
 };

@@ -83,7 +81,6 @@ enum swap_cluster_flags {
 	CLUSTER_FLAG_USABLE = CLUSTER_FLAG_FRAG,
 	CLUSTER_FLAG_FULL,
 	CLUSTER_FLAG_DISCARD,
-	CLUSTER_FLAG_DEAD,	/* Vswap dynamic cluster pending kfree_rcu */
 	CLUSTER_FLAG_MAX,
 };

@@ -148,17 +145,6 @@ static inline struct swap_info_struct *__swap_entry_to_info(swp_entry_t entry)
 	return __swap_type_to_info(swp_type(entry));
 }

-/**
- * __swap_offset_to_cluster - look up the cluster holding a swap offset
- * @si: the swap device
- * @offset: the swap entry offset
- *
- * Context: A vswap cluster is freed by kfree_rcu(). Callers must hold the
- * RCU read lock, or know the cluster is pinned by an in-use entry.
- *
- * Return: the cluster, or NULL if @si is a vswap device with no cluster
- * allocated at @offset.
- */
 static inline struct swap_cluster_info *__swap_offset_to_cluster(
 		struct swap_info_struct *si, pgoff_t offset)
 {
@@ -168,10 +154,9 @@ static inline struct swap_cluster_info *__swap_offset_to_cluster(
 	VM_WARN_ON_ONCE(offset >= roundup(si->max, SWAPFILE_CLUSTER));

 	if (swap_is_vswap(si)) {
-		struct swap_cluster_info_dynamic *ci_dyn;
-
-		ci_dyn = xa_load(&si->cluster_info_pool, cluster_idx);
-		return ci_dyn ? &ci_dyn->ci : NULL;
+		VM_WARN_ON_ONCE(cluster_idx >=
+				READ_ONCE(si->nr_mapped_clusters));
+		return &si->vswap_cluster_info[cluster_idx].ci;
 	}

 	return &si->cluster_info[cluster_idx];
@@ -183,32 +168,6 @@ static inline struct swap_cluster_info *__swap_entry_to_cluster(swp_entry_t entr
 					swp_offset(entry));
 }

-static inline struct swap_cluster_info *__vswap_cluster_lock(
-		struct swap_info_struct *si, unsigned long offset, bool irq)
-{
-	struct swap_cluster_info *ci;
-
-	rcu_read_lock();
-	ci = __swap_offset_to_cluster(si, offset);
-	if (ci) {
-		if (irq)
-			spin_lock_irq(&ci->lock);
-		else
-			spin_lock(&ci->lock);
-
-		/* The cluster can be torn down while we wait for the lock. */
-		if (ci->flags == CLUSTER_FLAG_DEAD) {
-			if (irq)
-				spin_unlock_irq(&ci->lock);
-			else
-				spin_unlock(&ci->lock);
-			ci = NULL;
-		}
-	}
-	rcu_read_unlock();
-	return ci;
-}
-
 static __always_inline struct swap_cluster_info *__swap_cluster_lock(
 		struct swap_info_struct *si, unsigned long offset, bool irq)
 {
@@ -226,9 +185,6 @@ static __always_inline struct swap_cluster_info *__swap_cluster_lock(
 	VM_WARN_ON_ONCE(!in_task());
 	VM_WARN_ON_ONCE(percpu_ref_is_zero(&si->users)); /* race with swapoff */

-	if (swap_is_vswap(si))
-		return __vswap_cluster_lock(si, offset, irq);
-
 	ci = __swap_offset_to_cluster(si, offset);
 	if (irq)
 		spin_lock_irq(&ci->lock);
@@ -244,8 +200,7 @@ static __always_inline struct swap_cluster_info *__swap_cluster_lock(
  *
  * Context: The caller must ensure the offset is in the valid range and
  * protect the swap device with reference count or locks.
- * Return: the locked cluster, or NULL if it is gone. Only a vswap device
- * can return NULL, as its clusters are allocated and freed on demand.
+ * Return: The locked cluster.
  */
 static inline struct swap_cluster_info *swap_cluster_lock(
 		struct swap_info_struct *si, unsigned long offset)
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 479814d19f50..645edcb29b2e 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -451,12 +451,9 @@ static struct folio *__swap_cache_alloc(swp_entry_t targ_entry, gfp_t gfp,
 	entry.val = round_down(targ_entry.val, nr_pages);

 	/* Check if the slot and range are available, skip allocation if not */
-	err = -ENOENT;
 	ci = swap_cluster_lock(si, offset);
-	if (ci) {
-		err = __swap_cache_add_check(ci, targ_entry, nr_pages, NULL, NULL);
-		swap_cluster_unlock(ci);
-	}
+	err = __swap_cache_add_check(ci, targ_entry, nr_pages, NULL, NULL);
+	swap_cluster_unlock(ci);
 	if (unlikely(err))
 		return ERR_PTR(err);

@@ -477,13 +474,10 @@ static struct folio *__swap_cache_alloc(swp_entry_t targ_entry, gfp_t gfp,
 		return ERR_PTR(-ENOMEM);

 	/* Double check the range is still not in conflict */
-	err = -ENOENT;
 	ci = swap_cluster_lock(si, offset);
-	if (ci)
-		err = __swap_cache_add_check(ci, targ_entry, nr_pages, &shadow, &memcg_id);
+	err = __swap_cache_add_check(ci, targ_entry, nr_pages, &shadow, &memcg_id);
 	if (unlikely(err)) {
-		if (ci)
-			swap_cluster_unlock(ci);
+		swap_cluster_unlock(ci);
 		folio_put(folio);
 		return ERR_PTR(err);
 	}
@@ -495,7 +489,6 @@ static struct folio *__swap_cache_alloc(swp_entry_t targ_entry, gfp_t gfp,

 	if (mem_cgroup_swapin_charge_folio(folio, memcg_id,
 					   vmf ? vmf->vma->vm_mm : NULL, gfp)) {
-		/* The folio pins the cluster */
 		ci = swap_cluster_lock(si, offset);
 		__swap_cache_do_del_folio(ci, folio, entry, shadow);
 		swap_cluster_unlock(ci);
diff --git a/mm/swap_table.h b/mm/swap_table.h
index 034da3546ef0..3bddea7dbe33 100644
--- a/mm/swap_table.h
+++ b/mm/swap_table.h
@@ -257,8 +257,6 @@ static inline unsigned long swap_table_get(struct swap_cluster_info *ci,
 	unsigned long swp_tb;

 	VM_WARN_ON_ONCE(off >= SWAPFILE_CLUSTER);
-	if (!ci)
-		return SWP_TB_NULL;

 	rcu_read_lock();
 	table = rcu_dereference(ci->table);
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 96c67dab238e..944eccfd8632 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -134,6 +134,14 @@ static DEFINE_PER_CPU(struct percpu_swap_cluster, percpu_swap_cluster) = {

 static atomic_long_t vswap_alloc_reject = ATOMIC_LONG_INIT(0);

+/* Virtual swap address space, reserved at init and mapped on demand. */
+#define VSWAP_MAX_SIZE		(8UL * SZ_1T)
+
+/* Clusters added per growth of the vswap cluster array, one page worth. */
+#define VSWAP_GROW_CLUSTERS						\
+	max_t(unsigned long,						\
+	      PAGE_SIZE / sizeof(struct swap_cluster_info_dynamic), 16)
+
 struct percpu_vswap_cluster {
 	unsigned long offset[SWAP_NR_ORDERS];
 	local_lock_t lock;
@@ -144,6 +152,34 @@ static DEFINE_PER_CPU(struct percpu_vswap_cluster, percpu_vswap_cluster) = {
 	.lock = INIT_LOCAL_LOCK(),
 };

+/*
+ * Vswap allocates from its own device with a separate percpu cluster cache,
+ * so the allocator has two local locks to pick from.
+ */
+static void swap_percpu_cluster_lock(struct swap_info_struct *si)
+{
+	if (swap_is_vswap(si))
+		local_lock(&percpu_vswap_cluster.lock);
+	else
+		local_lock(&percpu_swap_cluster.lock);
+}
+
+static void swap_percpu_cluster_unlock(struct swap_info_struct *si)
+{
+	if (swap_is_vswap(si))
+		local_unlock(&percpu_vswap_cluster.lock);
+	else
+		local_unlock(&percpu_swap_cluster.lock);
+}
+
+static void swap_percpu_cluster_assert_held(struct swap_info_struct *si)
+{
+	if (swap_is_vswap(si))
+		lockdep_assert_held(&this_cpu_ptr(&percpu_vswap_cluster)->lock);
+	else
+		lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)->lock);
+}
+
 static bool vswap_alloc(struct folio *folio);
 static void vswap_mark_cache_only(struct swap_cluster_info *ci,
 				  unsigned int ci_off);
@@ -424,7 +460,8 @@ static inline unsigned int cluster_index(struct swap_info_struct *si,
 					 struct swap_cluster_info *ci)
 {
 	if (swap_is_vswap(si))
-		return container_of(ci, struct swap_cluster_info_dynamic, ci)->index;
+		return container_of(ci, struct swap_cluster_info_dynamic, ci) -
+		       si->vswap_cluster_info;
 	return ci - si->cluster_info;
 }

@@ -442,10 +479,14 @@ static void swap_cluster_free_table_folio_rcu_cb(struct rcu_head *head)
 	folio_put(folio);
 }

-static void swap_cluster_free_table(struct swap_cluster_info *ci)
+static void swap_cluster_free_table(struct swap_info_struct *si,
+				    struct swap_cluster_info *ci)
 {
 	struct swap_table *table;

+	if (swap_is_vswap(si))
+		vswap_cluster_free_vtable(ci);
+
 #ifdef CONFIG_MEMCG
 	kfree(ci->memcg_table);
 	ci->memcg_table = NULL;
@@ -505,7 +546,7 @@ static int swap_cluster_alloc_table(struct swap_info_struct *si,
 		VM_WARN_ON_ONCE(ci->memcg_table);
 		ci->memcg_table = kzalloc_obj(*ci->memcg_table, gfp);
 		if (!ci->memcg_table) {
-			swap_cluster_free_table(ci);
+			swap_cluster_free_table(si, ci);
 			return -ENOMEM;
 		}
 	}
@@ -515,10 +556,16 @@ static int swap_cluster_alloc_table(struct swap_info_struct *si,
 	VM_WARN_ON_ONCE(ci->zero_bitmap);
 	ci->zero_bitmap = bitmap_zalloc(SWAPFILE_CLUSTER, gfp);
 	if (!ci->zero_bitmap) {
-		swap_cluster_free_table(ci);
+		swap_cluster_free_table(si, ci);
 		return -ENOMEM;
 	}
 #endif
+
+	/* The virtual table shares the swap table's lifetime. */
+	if (swap_is_vswap(si) && vswap_cluster_alloc_vtable(ci, gfp)) {
+		swap_cluster_free_table(si, ci);
+		return -ENOMEM;
+	}
 	return 0;
 }

@@ -564,10 +611,8 @@ swap_cluster_populate(struct swap_info_struct *si,
 	/*
 	 * Only cluster isolation from the allocator does table allocation.
 	 * Swap allocator uses percpu clusters and holds the local lock.
-	 * vswap clusters are destroyed rather than freed to si->free_clusters.
 	 */
-	VM_WARN_ON_ONCE(swap_is_vswap(si));
-	lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)->lock);
+	swap_percpu_cluster_assert_held(si);
 	if (!(si->flags & SWP_SOLIDSTATE))
 		lockdep_assert_held(&si->global_cluster_lock);
 	lockdep_assert_held(&ci->lock);
@@ -584,7 +629,7 @@ swap_cluster_populate(struct swap_info_struct *si,
 	spin_unlock(&ci->lock);
 	if (!(si->flags & SWP_SOLIDSTATE))
 		spin_unlock(&si->global_cluster_lock);
-	local_unlock(&percpu_swap_cluster.lock);
+	swap_percpu_cluster_unlock(si);

 	ret = swap_cluster_alloc_table(si, ci, __GFP_HIGH | __GFP_NOMEMALLOC |
 					       GFP_KERNEL);
@@ -597,7 +642,7 @@ swap_cluster_populate(struct swap_info_struct *si,
 	 * could happen with ignoring the percpu cluster is fragmentation,
 	 * which is acceptable since this fallback and race is rare.
 	 */
-	local_lock(&percpu_swap_cluster.lock);
+	swap_percpu_cluster_lock(si);
 	if (!(si->flags & SWP_SOLIDSTATE))
 		spin_lock(&si->global_cluster_lock);
 	spin_lock(&ci->lock);
@@ -645,20 +690,7 @@ static void swap_cluster_schedule_discard(struct swap_info_struct *si,
 static void __free_cluster(struct swap_info_struct *si, struct swap_cluster_info *ci)
 {
 	swap_cluster_assert_empty(ci, 0, SWAPFILE_CLUSTER, false);
-	swap_cluster_free_table(ci);
-
-	if (swap_is_vswap(si)) {
-		struct swap_cluster_info_dynamic *ci_dyn;
-
-		/* vswap clusters are destroyed, not returned to free_clusters. */
-		ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
-		xa_erase(&si->cluster_info_pool, ci_dyn->index);
-		move_cluster(si, ci, NULL, CLUSTER_FLAG_DEAD);
-		vswap_cluster_free_vtable(ci);
-		kfree_rcu(ci_dyn, rcu);
-		return;
-	}
-
+	swap_cluster_free_table(si, ci);
 	move_cluster(si, ci, &si->free_clusters, CLUSTER_FLAG_FREE);
 	ci->order = 0;
 }
@@ -1195,49 +1227,142 @@ static unsigned long alloc_swap_scan_list(struct swap_info_struct *si,
 	return found;
 }

-static unsigned long vswap_alloc_cluster(struct swap_info_struct *si,
-					 struct folio *folio)
+/*
+ * Reserve address space for the vswap cluster array. Nothing is mapped yet,
+ * so this costs address space only, plus an eighth of it in shadow under
+ * CONFIG_KASAN_VMALLOC.
+ */
+static int vswap_reserve_cluster_array(struct swap_info_struct *si,
+				       unsigned long maxpages)
+{
+	unsigned long nr_clusters = DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER);
+
+	mutex_init(&si->cluster_grow_lock);
+	si->cluster_info_area = get_vm_area(nr_clusters *
+					    sizeof(*si->vswap_cluster_info),
+					    VM_SPARSE);
+	if (!si->cluster_info_area)
+		return -ENOMEM;
+
+	si->vswap_cluster_info = si->cluster_info_area->addr;
+	return 0;
+}
+
+static void vswap_free_cluster_array(struct swap_info_struct *si)
+{
+	unsigned long addr, end;
+	struct page *page;
+
+	if (!si->cluster_info_area)
+		return;
+
+	end = round_up((unsigned long)&si->vswap_cluster_info[si->nr_mapped_clusters],
+		       PAGE_SIZE);
+	for (addr = (unsigned long)si->vswap_cluster_info; addr < end;
+	     addr += PAGE_SIZE) {
+		page = vmalloc_to_page((void *)addr);
+		vm_area_unmap_pages(si->cluster_info_area, addr,
+				    addr + PAGE_SIZE);
+		__free_page(page);
+	}
+
+	free_vm_area(si->cluster_info_area);
+	si->cluster_info_area = NULL;
+	si->vswap_cluster_info = NULL;
+	si->nr_mapped_clusters = 0;
+}
+
+static bool vswap_can_grow(struct swap_info_struct *si)
+{
+	return READ_ONCE(si->nr_mapped_clusters) <
+	       DIV_ROUND_UP(si->max, SWAPFILE_CLUSTER);
+}
+
+/*
+ * Map one more page of the vswap cluster array and hand the clusters it
+ * covers to the allocator. The caller must not hold the percpu cluster
+ * lock: vm_area_map_pages() might sleep.
+ *
+ * The mapped prefix only ever grows, so the pages already backing clusters
+ * [0, si->nr_mapped_clusters) are exactly those below the page boundary
+ * above the last one. A grow whose clusters all fall inside an already
+ * mapped page maps nothing.
+ */
+static int vswap_grow_clusters(struct swap_info_struct *si)
 {
-	struct swap_cluster_info_dynamic *ci_dyn;
 	struct swap_cluster_info *ci;
-	unsigned long offset;
+	unsigned int noreclaim_flags;
+	unsigned long start, end;
+	struct page *page;
+	unsigned int i, first, nr;
+	int err = -ENOSPC;

+	BUILD_BUG_ON(VSWAP_GROW_CLUSTERS *
+		     sizeof(struct swap_cluster_info_dynamic) > PAGE_SIZE);
 	VM_WARN_ON(!swap_is_vswap(si));

-	ci_dyn = kzalloc_obj(*ci_dyn, GFP_ATOMIC);
-	if (!ci_dyn)
-		return SWAP_ENTRY_INVALID;
+	/* Rechecked under the mutex, this only keeps a full device cheap. */
+	if (!vswap_can_grow(si))
+		return -ENOSPC;

-	spin_lock_init(&ci_dyn->ci.lock);
-	INIT_LIST_HEAD(&ci_dyn->ci.list);
+	/* Outside the mutex, so this one may still reclaim. */
+	page = alloc_page(__GFP_HIGH | __GFP_NOMEMALLOC | GFP_KERNEL |
+			  __GFP_ZERO);

-	if (swap_cluster_alloc_table(si, &ci_dyn->ci, GFP_ATOMIC)) {
-		kfree(ci_dyn);
-		return SWAP_ENTRY_INVALID;
-	}
+	mutex_lock(&si->cluster_grow_lock);
+	first = si->nr_mapped_clusters;
+	nr = min_t(unsigned int, VSWAP_GROW_CLUSTERS,
+		   DIV_ROUND_UP(si->max, SWAPFILE_CLUSTER) - first);
+	if (!nr)
+		goto out;

-	if (vswap_cluster_alloc_vtable(ci_dyn, GFP_ATOMIC)) {
-		swap_cluster_free_table(&ci_dyn->ci);
-		kfree(ci_dyn);
-		return SWAP_ENTRY_INVALID;
-	}
+	start = round_up((unsigned long)&si->vswap_cluster_info[first],
+			 PAGE_SIZE);
+	end = round_up((unsigned long)&si->vswap_cluster_info[first + nr],
+		       PAGE_SIZE);

-	/* Lock before publishing: xa_alloc makes the cluster findable by offset. */
-	ci = &ci_dyn->ci;
-	spin_lock(&ci->lock);
+	if (start != end) {
+		err = -ENOMEM;
+		if (!page)
+			goto out;
+		/*
+		 * vm_area_map_pages() allocates page tables with
+		 * GFP_PGTABLE_KERNEL, so they carry __GFP_DIRECT_RECLAIM.
+		 * A non-reclaim caller of folio_alloc_swap() would otherwise
+		 * recurse back here and deadlock on the mutex it already
+		 * holds. Callers already under PF_MEMALLOC do not need this,
+		 * swapon does. It grants the page tables reserve access, at
+		 * most three pages per grow.
+		 */
+		noreclaim_flags = memalloc_noreclaim_save();
+		err = vm_area_map_pages(si->cluster_info_area, start, end,
+					&page);
+		memalloc_noreclaim_restore(noreclaim_flags);
+		if (err)
+			goto out;
+		page = NULL;
+	}

-	if (xa_alloc(&si->cluster_info_pool, &ci_dyn->index, ci_dyn,
-		     XA_LIMIT(1, DIV_ROUND_UP(si->max, SWAPFILE_CLUSTER) - 1),
-		     GFP_ATOMIC)) {
+	/*
+	 * Publish the new clusters before they become reachable by offset.
+	 * A zeroed page leaves them off-list with CLUSTER_FLAG_NONE, which
+	 * is what move_cluster() expects.
+	 */
+	WRITE_ONCE(si->nr_mapped_clusters, first + nr);
+	for (i = first; i < first + nr; i++) {
+		ci = &si->vswap_cluster_info[i].ci;
+		spin_lock_init(&ci->lock);
+		INIT_LIST_HEAD(&ci->list);
+		spin_lock(&ci->lock);
+		move_cluster(si, ci, &si->free_clusters, CLUSTER_FLAG_FREE);
 		spin_unlock(&ci->lock);
-		swap_cluster_free_table(&ci_dyn->ci);
-		vswap_cluster_free_vtable(&ci_dyn->ci);
-		kfree(ci_dyn);
-		return SWAP_ENTRY_INVALID;
 	}
-
-	offset = cluster_offset(si, ci);
-	return alloc_swap_scan_cluster(si, ci, folio, offset, NULL);
+	err = 0;
+out:
+	mutex_unlock(&si->cluster_grow_lock);
+	if (page)
+		__free_page(page);
+	return err;
 }

 static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
@@ -1264,8 +1389,6 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
 				nr_reclaim = __try_to_reclaim_swap(si, offset,
 								   TTRS_ANYWAY);
 				ci = swap_cluster_lock(si, offset);
-				if (!ci)
-					goto next;
 				if (nr_reclaim) {
 					offset += abs(nr_reclaim);
 					continue;
@@ -1277,8 +1400,6 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
 				nr_reclaim = try_to_reclaim_vswap_backing(si, offset,
 									  vswap_entry);
 				ci = swap_cluster_lock(si, offset);
-				if (!ci)
-					goto next;
 				if (nr_reclaim) {
 					offset += abs(nr_reclaim);
 					continue;
@@ -1292,7 +1413,6 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
 			relocate_cluster(si, ci);

 		swap_cluster_unlock(ci);
-next:
 		if (to_scan <= 0)
 			break;

@@ -1370,10 +1490,19 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
 			goto done;
 	}

-	if (swap_is_vswap(si)) {
-		found = vswap_alloc_cluster(si, folio);
-		if (found)
-			goto done;
+	/*
+	 * Grow the vswap cluster array and let the free list scan below pick
+	 * up the new clusters. Growth sleeps, so drop the percpu cluster lock
+	 * across it; the scan does not care which CPU it lands back on. The
+	 * list_empty() test is racy either way: a stale empty costs one page,
+	 * a stale non-empty skips the grow and leaves the caller to the
+	 * fragment and stealing scans below.
+	 */
+	if (swap_is_vswap(si) && list_empty(&si->free_clusters) &&
+	    vswap_can_grow(si)) {
+		local_unlock(&percpu_vswap_cluster.lock);
+		vswap_grow_clusters(si);
+		local_lock(&percpu_vswap_cluster.lock);
 	}

 	if (!(si->flags & SWP_PAGE_DISCARD)) {
@@ -1625,11 +1754,11 @@ static swp_entry_t swap_alloc_fast(struct folio *folio)
 		return (swp_entry_t){};

 	ci = swap_cluster_lock(si, offset);
-	if (ci && cluster_is_usable(ci, order)) {
+	if (cluster_is_usable(ci, order)) {
 		if (cluster_is_empty(ci))
 			offset = cluster_offset(si, ci);
 		found = alloc_swap_scan_cluster(si, ci, folio, offset, NULL);
-	} else if (ci) {
+	} else {
 		swap_cluster_unlock(ci);
 	}

@@ -1755,7 +1884,6 @@ int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp)
 	if (!si)
 		return 0;

-	/* The source PTE pins the entry, so its cluster is alive. */
 	ci = __swap_offset_to_cluster(si, offset);
 	ret = swap_extend_table_alloc(si, ci, swp_cluster_offset(entry), gfp);

@@ -2009,12 +2137,12 @@ static bool vswap_alloc(struct folio *folio)

 	if (offset != SWAP_ENTRY_INVALID) {
 		ci = swap_cluster_lock(vswap_si, offset);
-		if (ci && cluster_is_usable(ci, order)) {
+		if (cluster_is_usable(ci, order)) {
 			if (cluster_is_empty(ci))
 				offset = cluster_offset(vswap_si, ci);
 			alloc_swap_scan_cluster(vswap_si, ci, folio, offset,
 						NULL);
-		} else if (ci) {
+		} else {
 			swap_cluster_unlock(ci);
 		}
 	}
@@ -2755,7 +2883,6 @@ static bool folio_maybe_swapped(struct folio *folio)
 	VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
 	VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);

-	/* Folio is locked and in swap cache, so ci->count > 0: cluster is alive. */
 	ci = __swap_entry_to_cluster(entry);
 	ci_off = swp_cluster_offset(entry);
 	ci_end = ci_off + folio_nr_pages(folio);
@@ -3861,25 +3988,22 @@ static void free_swap_cluster_info(struct swap_info_struct *si,
 				   struct swap_cluster_info *cluster_info,
 				   unsigned long maxpages)
 {
-	struct swap_cluster_info_dynamic *ci_dyn;
 	struct swap_cluster_info *ci;
-	unsigned long idx;
 	int i, nr_clusters = DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER);

 	if (swap_is_vswap(si)) {
-		xa_for_each(&si->cluster_info_pool, idx, ci_dyn) {
-			ci = &ci_dyn->ci;
+		nr_clusters = si->nr_mapped_clusters;
+		for (i = 0; i < nr_clusters; i++) {
+			ci = &si->vswap_cluster_info[i].ci;
 			spin_lock(&ci->lock);
 			if (cluster_table_is_alloced(ci)) {
 				swap_cluster_assert_empty(ci, 0,
 							  SWAPFILE_CLUSTER, true);
-				swap_cluster_free_table(ci);
+				swap_cluster_free_table(si, ci);
 			}
 			spin_unlock(&ci->lock);
-			vswap_cluster_free_vtable(ci);
-			kfree(ci_dyn);
 		}
-		xa_destroy(&si->cluster_info_pool);
+		vswap_free_cluster_array(si);
 		return;
 	}

@@ -3891,7 +4015,7 @@ static void free_swap_cluster_info(struct swap_info_struct *si,
 		spin_lock(&ci->lock);
 		if (cluster_table_is_alloced(ci)) {
 			swap_cluster_assert_empty(ci, 0, SWAPFILE_CLUSTER, true);
-			swap_cluster_free_table(ci);
+			swap_cluster_free_table(si, ci);
 		}
 		spin_unlock(&ci->lock);
 	}
@@ -4373,39 +4497,16 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
 {
 	unsigned long nr_clusters = DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER);
 	struct swap_cluster_info *cluster_info = NULL;
-	struct swap_cluster_info_dynamic *ci_dyn = NULL;
+	struct swap_cluster_info *ci;
 	int err = -ENOMEM;
 	unsigned long i;

-	/* A vswap device uses an xarray pool instead of a static array. */
+	/* A vswap device grows its cluster array on demand. */
 	if (swap_is_vswap(si)) {
 		nr_clusters = 0;
-		xa_init_flags(&si->cluster_info_pool, XA_FLAGS_ALLOC);
-
-		/*
-		 * Pre-allocate cluster 0 and mark slot 0 (header page)
-		 * as bad so the allocator never hands out page offset 0.
-		 */
-		ci_dyn = kzalloc_obj(*ci_dyn, GFP_KERNEL);
-		if (!ci_dyn)
-			goto err;
-		spin_lock_init(&ci_dyn->ci.lock);
-		INIT_LIST_HEAD(&ci_dyn->ci.list);
-
-		err = xa_insert(&si->cluster_info_pool, 0, ci_dyn, GFP_KERNEL);
-		if (err) {
-			kfree(ci_dyn);
-			goto err;
-		}
-
-		err = swap_cluster_setup_bad_slot(si, &ci_dyn->ci, 0, false);
+		err = vswap_reserve_cluster_array(si, maxpages);
 		if (err)
 			goto err;
-
-		err = vswap_cluster_alloc_vtable(ci_dyn, GFP_KERNEL);
-		if (err)
-			goto err;
-
 		goto setup_cluster_info;
 	}

@@ -4463,7 +4564,7 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
 	}

 	for (i = 0; i < nr_clusters; i++) {
-		struct swap_cluster_info *ci = &cluster_info[i];
+		ci = &cluster_info[i];

 		if (ci->count) {
 			ci->flags = CLUSTER_FLAG_NONFULL;
@@ -4476,8 +4577,23 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,

 	/* Slot 0 is bad, so cluster 0 never empties. The rest of it is usable. */
 	if (swap_is_vswap(si)) {
-		ci_dyn->ci.flags = CLUSTER_FLAG_NONFULL;
-		list_add_tail(&ci_dyn->ci.list, &si->nonfull_clusters[0]);
+		err = vswap_grow_clusters(si);
+		if (err)
+			goto err;
+
+		ci = &si->vswap_cluster_info->ci;
+		spin_lock(&ci->lock);
+		move_cluster(si, ci, NULL, CLUSTER_FLAG_NONE);
+		spin_unlock(&ci->lock);
+
+		err = swap_cluster_setup_bad_slot(si, ci, 0, false);
+		if (err)
+			goto err;
+
+		spin_lock(&ci->lock);
+		move_cluster(si, ci, &si->nonfull_clusters[0],
+			     CLUSTER_FLAG_NONFULL);
+		spin_unlock(&ci->lock);
 	}

 	si->cluster_info = cluster_info;
@@ -4886,12 +5002,8 @@ static int __init vswap_init(void)
 		return 0;
 	}

-	/*
-	 * One u32 xarray ID per cluster, so the device cannot be larger
-	 * than UINT_MAX clusters.
-	 */
-	maxpages = min(swapfile_maximum_size,
-		       (unsigned long)UINT_MAX * SWAPFILE_CLUSTER);
+	/* One fixed reservation for the cluster array, so one fixed size. */
+	maxpages = min(swapfile_maximum_size, VSWAP_MAX_SIZE >> PAGE_SHIFT);
 	/*
 	 * SWP_WRITEOK enables slot allocation. SWP_SOLIDSTATE selects
 	 * per-CPU cluster allocation; vswap has no si->global_cluster.
diff --git a/mm/vswap.h b/mm/vswap.h
index c66fa34e2e60..547e4453cf11 100644
--- a/mm/vswap.h
+++ b/mm/vswap.h
@@ -175,7 +175,7 @@ static inline void __vtable_set(struct swap_cluster_info_dynamic *ci_dyn,
  * @entry: the virtual swap entry
  * @voff: out param, receives @entry's slot offset within the cluster
  *
- * Return: the locked vswap cluster, or NULL if @entry has no live cluster.
+ * Return: the locked vswap cluster.
  */
 static inline struct swap_cluster_info_dynamic *
 vswap_lock_cluster(swp_entry_t entry, unsigned int *voff)
@@ -183,8 +183,6 @@ vswap_lock_cluster(swp_entry_t entry, unsigned int *voff)
 	struct swap_cluster_info *ci;

 	ci = swap_cluster_lock(__swap_entry_to_info(entry), swp_offset(entry));
-	if (!ci)
-		return NULL;
 	*voff = swp_cluster_offset(entry);
 	return container_of(ci, struct swap_cluster_info_dynamic, ci);
 }
@@ -204,9 +202,6 @@ static inline swp_entry_t vswap_to_phys(swp_entry_t entry)
 	unsigned long vt;

 	ci_dyn = vswap_lock_cluster(entry, &voff);
-	if (!ci_dyn)
-		return (swp_entry_t){};
-
 	vt = __vtable_get(ci_dyn, voff);
 	swap_cluster_unlock(&ci_dyn->ci);

@@ -255,8 +250,6 @@ static inline struct zswap_entry *vswap_zswap_load(swp_entry_t entry)
 	unsigned long vt;

 	ci_dyn = vswap_lock_cluster(entry, &voff);
-	if (!ci_dyn)
-		return NULL;
 	vt = __vtable_get(ci_dyn, voff);
 	swap_cluster_unlock(&ci_dyn->ci);

@@ -336,11 +329,6 @@ static inline int vswap_check_backing(swp_entry_t entry, int nr,
 	int ret;

 	ci_dyn = vswap_lock_cluster(entry, &voff);
-	if (!ci_dyn) {
-		if (typep)
-			*typep = VSWAP_NONE;
-		return 0;
-	}
 	ret = __vswap_check_backing(ci_dyn, voff, nr, typep);
 	swap_cluster_unlock(&ci_dyn->ci);
 	return ret;
@@ -365,9 +353,12 @@ static inline bool folio_phys_swap_backed(struct folio *folio)
 		type == VSWAP_SWAPFILE);
 }

-static inline int vswap_cluster_alloc_vtable(struct swap_cluster_info_dynamic *ci_dyn,
+static inline int vswap_cluster_alloc_vtable(struct swap_cluster_info *ci,
 					     gfp_t gfp)
 {
+	struct swap_cluster_info_dynamic *ci_dyn;
+
+	ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
 	ci_dyn->virtual_table = kcalloc(SWAPFILE_CLUSTER,
 					sizeof(*ci_dyn->virtual_table), gfp);
 	return ci_dyn->virtual_table ? 0 : -ENOMEM;

Re: Path forward for Virtualized Swap?

[Baoquan He]

On 09/10/26 at 01:03pm, Johannes Weiner wrote:
> [This reply was not LLM-generated.]
> 
> On Thu, Sep 10, 2026 at 03:09:59PM +0800, Baoquan He wrote:
> > Hi Nhat,
> > 
> > On 09/04/26 at 02:14pm, Nhat Pham wrote:
> > .....snip...
> > > Now, on xswap. Baoquan's working on a series [15] that covers some of the
> > > same ground, and the VM_SPARSE cluster_info idea in it is genuinely good.
> > > I've been reviewing that lineage since July [16] and I'd like whatever
> > > lands to end up with the best parts of both. From my perspective the
> > > differences are:
> > > 
> > > 1. Userspace knobs. xswap asks the admin for a size (a percent of RAM) plus
> > >    a per-device limit to tune afterwards. I'm not aware of any use case
> > >    that needs those, and I don't think users have a good way to answer the
> > >    question anyway - sizing swap for compressed memory depends on memory
> > >    size, workload, and compression ratio all at once. That's precisely the
> > >    provisioning problem vswap exists to remove. The kernel should be as
> > >    transparent and dynamic as possible here, and not add knobs unless
> > >    there's a use case for them.
> > > 
> > > 2. Writeback support. Writeback is core functionality for zswap, not an
> > >    add-on, and a design needs to account for it from the start. This came
> > >    up before, in the discussion around Chris' ghost swapfile RFC [17]: for
> > >    a solution here to be acceptable, it has to work with the primary
> > >    usecase and support disk writeback. Without it, whatever zswap won't
> > >    take (incompressible pages especially) has nowhere to go, and cold
> > >    compressed data can never leave RAM.
> > > 
> > > 3. Cgroup charging behavior. vswap/xswap shouldn't be charged against the
> > >    swap usage counter. It's fundamentally a different resource from
> > >    physical swapfile space, and memory.swap.* should read 0 when nothing is
> > >    on disk [18]. I made the longer argument for this in [19].
> > > 
> > > 4. Data structure (xarray vs sparse vmalloc array). Even with xarray, vswap
> > >    is already on par with or beating baseline. I like the sparse array
> > >    idea, but why are we landing an optimization before the feature itself,
> > >    without any A/B data showing the difference matters?
> > 
> > 
> > Thanks for laying this out, and for the honest push to converge. Let me
> > be equally direct about the ordering: I think the xswap base should land
> > first, and the things vswap demonstrates -  writeback, rmap lookup, the
> > charging semantics, later THP -- should be built on top of it. Because
> > it is the foundation that keeps the swap core simpler, and the first thing
> > to merge should be the one that doesn't have to be redone.
> > 
> > The VM_SPARSE array is not an optimization to bolt on later; it is a
> > structural choice, and the code reflects it. In vswap, the cluster
> > metadata lives in a dynamically-allocated xarray.
> > 
> > struct swap_cluster_info_dynamic {
> >         struct swap_cluster_info ci;
> >         unsigned int index;             /* for cluster_index() */
> >         struct rcu_head rcu;
> >         atomic_long_t *virtual_table;   /* Backing pointers for vswap slots */
> > };
> > 
> > To support dynamic growth and shrink, vswap stores its cluster metadata
> > in an xarray, and that forces two things the plain swap_cluster_info[]
> > array never needed:
> > 
> > 1. Every cluster has to carry an extra index and an rcu_head —
> > 24 bytes per cluster — purely so the xarray can locate it and free
> > it safely.
> > 2. To keep that bookkeeping from leaking into the normal-swap code, the
> > cluster had to be wrapped in a container, swap_cluster_info_dynamic,
> > so the xarray holds a pointer to the wrapper instead of an inline
> > array element.
> > 
> > So in vswap, every cluster access in the shared hot path has to answer
> > "is this a vswap device?" and take a separate branch:
> > 
> > - swap_is_vswap() is checked in 36 places across page_io.c, swapfile.c,
> >   zswap.c and swap.h;
> > - __swap_offset_to_cluster() branches into xa_load() for vswap vs the
> >   flat array otherwise, and the xarray path can return NULL (a cluster
> >   can be torn down);
> > - __swap_cluster_lock() branches into __vswap_cluster_lock(), which
> >   wraps every access in rcu_read_lock() and a CLUSTER_FLAG_DEAD check,
> >   plus kfree_rcu()/container_of()/rcu_head plumbing for node lifetime.
> 
> Well to state the obvious: the reason it does all that is to make the
> compression space transparent to the user.
> 
> The user can answer a simple boolean question: whether they want
> compression or not. And it will work on tiny machines, on humongous
> machines, and everything in between. That's a simple policy question
> with a clear answer.
> 
> What you're doing, asking the user for a static size, is much more
> difficult and has usability issues.
> 
> You're comparing implementations that don't accomplish the same thing.
> 
> The problem we're trying to solve is implementing a clean compression
> space abstraction. I'm arguing that vswap does, and xswap does not.
> 
> While they're both using parts of the swap device code to implement a
> compression space, xswap actually PRESENTS IT TO THE USER as a swap
> device, and then makes optimizations BASED ON BAKED IN LIMITATIONS.
> 
> But a conventional, statically sized swap device is a bad abstraction
> for the compression space. Here is why:
> 
> In conventional swap space, one memory page translates to one swap
> page. Compression space doesn't act this way: a memory page can
> consume anything between a few bytes to a full page in compression
> space. It depends on memory contents and compression algorithm. So
> right off the bat, this is a hard question to answer at the host level
> which could run all kinds of workloads.
> 
> In conventional swap space, the resource consumed is a different
> one. You're offloading memory by consuming disk space. This eats into
> the space available to the filesystem, which is totally unrelated.
> Asking the user for this tradeoff is a legitimate policy question.
> 
> Compression space is not a separate resource. It's page tables,
> backing pages, and swap descriptors. It's just MEMORY. There isn't a
> size tradeoff, because moving pages from memory space into compression
> space DOES NOT CONSUME A NEW RESOURCE. It's still just memory. All you
> need for containment already exists: rlimits, OOM killer, cgroup
> memory controls.
> 
> By making this a user-visible virtual swap device, you're sending
> users down the wrong path. You're asking them to set a new limit on a
> resource that's already limited by other means. You're framing the
> question as conventional swap which behaves completely differently.
> 
> If you ask them "how much swap space", they WILL reference this to
> available RAM capacity. Maybe half of ram, maybe twice the RAM.
> 
> But when compression space is referenced to RAM, it's trivial to fill
> it up with zeroed pages or easily compressible data LONG BEFORE the
> process or container would hit any of its MEMORY limits.
> 
> This creates an artificial resource shortages. It forces a competition
> where there shouldn't be one. And then you need new controls to manage
> a competition that doesn't have to exist.
> 
> Like I said before, including compression space (which is memory) in
> memory.swap.* (which is for disk space) is not going to be acceptable
> from the cgroup side. We can talk about that if you want.
> 
> But asking the user questions they shouldn't have to answer, or
> already answered elsewhere, is weak interface design. Allowing, let
> alone encouraging, answers that create a whole new host of
> organizational issues is outright bad interface design.
> 
> So if you want to compare implementations, you first have to actually
> implement the same thing:
> 
> Stop asking user "how large". Let compression space expand towards
> existing memory limits, such that it doesn't create an awkward and
> artificial new resource competition.
> 
> Then we can compare implementations.
> 
> If the optimizations still apply under those constraints, great.
> 
> Until then, there is little point in discussing differences that, by
> your own admission, have little to no impact on real world performance.

Thanks for your sharing with deliberate thought.

Agreed, and I want to be clear that the size knob is my implementation
choice, not something the design needs.

Now in v2, xswap's create() already takes no size, only an optional
priority: the device's address space is set by the kernel to the machine's
memory and cluster_info is mapped lazily, so nothing is allocated up front.
The only knob left is an optional per-device ceiling. If we really want to
remove it, that's quite easy thing, we can just remove the runtime
growth ceiling and the shrink machinery that serves it.

When I asked why shrink is needed, Nhat told on system, memory pressure
could reach a peak, than later may not reach it again for a long time. I
don't like the continuous automatic growing/shrinking, I think it
doesn't make much sense just for saving that memory serving struct
swap_cluster_info. But it's not bad to provide a mechanism for
admin/users to tune it.

But as I said, xswap/vswap both claim to solve the problem of zswap
physical disk slot and swap slot coupling, and meantime extend
functionality to make it more flexible than zswap/zram. While Nhat's
vswap is boot-time per-device swap. And Nhat's own description of it
in this thread is "vswap is just a normal swap device, no?". If I didn't
apply Nhat's code and test I couldn't realize it. I executed swapon but
can't see any output. I was shocked.

I really appreciated your patient and detailed sharing, while it takes
you so long words to explain it. IMHO, it deserves a separate patch
posting to justify it so that anyone can know why it is.

Thanks
Baoquan

Re: Path forward for Virtualized Swap?

[Baoquan He]

On 09/10/26 at 09:39am, Shakeel Butt wrote:
> On Thu, Sep 10, 2026 at 03:09:59PM +0800, Baoquan He wrote:
> > Hi Nhat,
> > 
> > On 09/04/26 at 02:14pm, Nhat Pham wrote:
> > .....snip...
> 
> [...]
> 
> > With VM_SPARSE, xswap's cluster access is exactly the plain-array line the
> > rest of swap already uses:
> > 
> >     return &si->cluster_info[offset / SWAPFILE_CLUSTER];
> > 
> > no branch, no RCU discipline, no tear-down state machine, and no NULL
> > return. So VM_SPARSE doesn't add complexity to close a gap; it lets the
> > cluster layer stay as simple as it already is, which is precisely the
> > part later work (writeback, rmap lookup, memcg charging, THP) has to sit
> > on.
> > 
> > I'm not going to claim xswap wins on throughput. I measured it:
> > on a 64G/64-thread swapout, xswap, vswap and plain swap+zswap are all
> > within ~2-3% of each other, effectively identical. 
> 
> So the claim is VM_SPARSE is simpler than xarray based approach. I feel like
> we are discussing implementation details before deciding the design and
> architecture. So, instead of VM_SPARSE vs xarray, let's discuss and decide the
> need for dynamic growth. Why we want dynamic growth upfront or can it be added
> later? Once we decide that then it will be very easy to pick an implementation
> that would take us there.

Hi Shakeel,

Thank you for joining the discussion and for taking the time to comment.

Agreed on requirement first - but this one was already decided, and not by me. In
the July ghost swapfile thread Nhat rejected exactly the shape of "grow only, can
be added later":

    "Except for my virtual swap design, which does support dynamic growth AND
     shrinking of capacity on demand ;) If it cannot grow (and furthermore, if it
     requires userspace operation to trigger swapfile growth), why do we need this
     at all? Might as well create a new swapfile with swapon?"

To me what it converged on was "dynamic growth and shrink, no writeback yet".
So automatic growth *and* shrink is the requirement, and the simpler alternative
was already on the table.

And I keep mentioning it in the cover-letter of each version of my posting. I
only did the foundtation via lazy vmalloc. And Nhat will do the core
part including writabck, rmap lookup, memcg accounting, zero page fill,
etc.

What is still genuinely open, and I would like us to settle, is how large the
device's address space should be, because the metadata scales with it:

    address space = machine memory (128G)  ->  65536 clusters  ->  ~4MB table
    address space = worst case, like (8PB)->  ~4M clusters    ->  ~256MB table

Once that is fixed, choosing the lazy representation is a normal implementation
question - and it is the same question Johannes is asking from the other side.

As you can see, xswap make swap_cluster_info[] kept as is. Accessing it
is simple and easy, and the xswap code is not intrusive into normal swap
code, and won't add complexity to normal swap. Even though Nhat doesn't
admit it, saying if writeback is done on top of xswap, the code is messy
too. In fact, it's not. I have done a draft writeback part, I will post
for checking, there won't be a lot of struct swap_cluster_info_dynamic , 
swap_is_vswap() flying around.

Thanks
Baoquan

Re: Path forward for Virtualized Swap?

[Kairui Song]

On Wed, Sep 9, 2026 at 12:36 AM Nhat Pham <nphamcs@gmail.com> wrote:
>
> On Sun, Sep 6, 2026 at 10:52 PM Kairui Song <ryncsn@gmail.com> wrote:
>
> Hi Kairui,
>
> Thank you for the kind words and encouragement :) It means a lot
> coming from you.
>
> My main concern so far has been - it seems like every time I satisfied
> one of your guys' requirements, another pre-requisite is added to the
> list. I mean the performance regression and memory overhead concerns
> make a lot of sense, especially when vswap is non-optional. But now it
> just seems like we're delaying real use case for the fabled perfect
> swap system. It gets very exhausting trying to keep expanding the
> scope of this.

Hi Nhat

I also wish we could do things at a faster pace in upstream :), that's
a challenge for all of us...

> >
> > I asked Baoquan the same thing, see the "Why not make both args
> > optional" part:
> >
> > https://lore.kernel.org/linux-mm/apaw_XmqnEcY8sv7@MiWiFi-R3L-srv/
> >
> > Where I've ended up is that unbounded growth is a real concern. On a
> > host with no memcg limit (root cgroup, and most desktop and embedded
> > setups), an unlimited pool means usage can keep growing, with no
> > admin visible ceiling at all. I'm not attached to xswap's percent of RAM
> > knob specifically, but I do think some kind of bound makes sense.
>
> Technically speaking, virtual swap *is* bounded, i.e 8PB (xarray
> induced a limit to the number of clusters it can allocate). But more
> importantly, it's effectively capped by the backend it can take:
> zswap, swap cache, disk swap, etc.

Right... See below.

> > Agreed completely, and I think this one is settled: whatever lands
> > has to support disk writeback in the end. Baoquan mentioned that
> > what he sent is just a foundation though. I know talking is cheap
> > but still, could it be combined with the writeback support from the
> > other side? Or am I missing anything here?
>
> I'm just very nervous about any design that doesn't quite take that
> into account and handwave it away into a follow-up :)

In fact, checking the code, it seems it won't be hard to just glue the
vswap writeback to the xswap design? Maybe that's not the right way,
but I'm just saying it seems it won't be a problem, we need to think
twice about this.

> > > 3. Cgroup charging behavior. vswap/xswap shouldn't be charged against the
> > >    swap usage counter. It's fundamentally a different resource from
> > >    physical swapfile space, and memory.swap.* should read 0 when nothing is
> > >    on disk [18]. I made the longer argument for this in [19].
> >
> > I personally don't have a strong opinion on this. But I think this
> > needs an explicit decision from the group rather than being
> > inherited from whichever series merges first. Skipping the charge
> > when a device has a special flag isn't hard to do if the code
> > abstraction is done well, so this is more of a design decision
> > than a technical problem.
>
> I'm fine with discussing about this. I just want to point out that
> this is not something I snuck in :) This has been the behavior since
> probably the first or second RFC of the first design though, and in
> most of them, it is a front-and-centre patch of its own, not hushed
> away in one big patch. Folks have had 1.5 years to comment on it (and
> still not too late - nothing is merged yet!)
>
> I'd love to hear opinions from other zswap users regarding this :)

Right, I still have no strong opinion on this yet, up to you guys :)

>
> >
> > > 4. Data structure (xarray vs sparse vmalloc array). Even with xarray, vswap
> > >    is already on par with or beating baseline. I like the sparse array
> > >    idea, but why are we landing an optimization before the feature itself,
> > >    without any A/B data showing the difference matters?
> >
> > This is where I'd push back a little. To be clear, my concern isn't the
> > size of the win, it's that VM_SPARSE isn't purely an optimization. It
> > changes the cluster/metadata layout and avoids RCU conventions,
> > could help avoid some future sleep while atomic limitations and
> > it's tangled with the initial size issue too. So bolting it on afterwards
> > could mean rewriting the parts of vswap that touch it and could be
> > costly some conflict in ideas. That's why I'd rather we agree on
> > it now, not that it has to land first.
>
> I don't think the current code differs significantly from the existing
> model, no?
>
> For the most part, RCU played quite nicely with the current swap code.
> Practically, it's just:
>
> 1. If you have a reference to the swap entry, you can assume entry and
> the cluster owning it is safe.
>
> 2. If you don't, then do not assume it's safe once you exit the rcu
> read section.
>
> What conflicts specifically do you have in mind here :)

Yeah you're right. I'm just a bit worried if that RCU period will be
extended, and it's always better to make the convention as simple as
possible. I'm already trying to get rid of the local lock, in that
swap queue series, that is not so painful and still a bit troublesome
for a few new use cases.

> > Using VM_SPARSE does outsmart my earlier xarray suggestion :) There
> > is an interesting early benchmark from Baoquan, VM_SPARSE wasn't there
> > yet, but with it we can use clusters natively in the same way. It shows
> > about 20% latency cut and 10% throughput gain, which seems a major
> > improvement, and that's serial performance, not even mentioning
> > scalability yet:
> > https://lore.kernel.org/linux-mm/20260707073215.72183-1-baoquan.he@linux.dev/
>
> I think a lot of these wins come from the fact that the zswap's xarray
> specifically is *really* bad in certain paths. For instance, in the
> free path (swap_range_free() -> zswap_invalidate()), there is no
> batching behavior, so we walk the tree from scratch for each entry
> rather than using an xas iterator for the entire batch. That goes away
> in the current vswap model, because the xarray manages cluster rather
> than entries, so you just do one tree walk + one flat array walk.

Right but with vswap replacing zswap, isn't the performance is still
basically unchanged according to your test (or just a little bit
faster)? My test still doesn't show a measurable improvement.

> zswap and swapfile coupled, but only remove the xarray and replace it
> with the flat array in swapfile's cluster struct. I observe the same
> win, basically (especially on the usemem benchmark).

Yeah, merging the zswap xarray into cluster should buy a lot of
performance gain, the only concern is the cluster it self is in a
xarray tree so if your vswap space gets huge, could that somehow
become an issue? Zswap used a split xarray, and swap also used to have
a 64M partition for the swap cache xarray, which is gone after the
swap table.

...

> > We were also discussing the limitation of the global cluster cache in
> > that thread, so I paused my other work for a while and posted the swap
> > queue series first. That one is orthogonal on purpose, and helps both
> > swap tier and vswap, it reduces the overhead enough to cancel out
> > the layering cost due to missing caching for either:
> > https://lore.kernel.org/linux-mm/20260829-swap-pcp-priq-v2-resend-0-68d3d925578c@gmail.com/
> > (this is V2, V1 is earlier - thanks to Lian for keeping this going!)
>
> Ah that! I sort of tabled that as a nice-idea-but-still-optimization :P
>
> I'll spend some time thinking about the design and reviewing it. In
> the long run, it will get things more simple, but another thing that
> should not be a blocker, correct?

Right, definitely not a blocker. I think the swap queue could be
merged after tiering and other stuff, as an optimization, just to
demonstrate that we don't need another layer of cache; we can just use
the devices directly without any cache above them. It will also
deprecate plist; with the only plist rotate user is gone, we can get
rid of a large chunk of common plist code.

...

> > The two interfaces aren't equivalent, xswap has its limitations, but a
> > couple of xswap's properties look genuinely better to me. Runtime
> > on/off rather than a boot parameter is better, more flexibility and
>
> I would also like to point out - I *do* have a runtime parameter in an
> older version of vswap (v3). There is also not a lot of engagement
> from folks in that version - no one expressed any interest in that
> runtime knob in particular. Only Johannes made the case that a boot
> parameter is superior, so I switched over. I think his reasoning makes
> sense at the time FWIW, and it also removes the need for CONFIG_VSWAP
> (which in turns remove a bunch of useless stubs - saving 100 LoCs), so
> I figure folks will prefer less code and less userspace decisions :)
>
> v3: https://lore.kernel.org/all/20260806184254.3790858-1-nphamcs@gmail.com/

Ah, thanks for the reminder.

> Regardless - do you have a specific use case in mind where a runtime
> knob would be useful?

A boot time bool will cause real trouble for us, at least. We have
many VMs with fixed boot images and fixed cmdlines. However, the VM
itself is used by different customers, some are ZRAM only users, and
quite a lot don't want any kind of compression (e.g. their data is
dense compression is just waste of CPU) and prefer plain swap. So I do
hope we can have a runtime tunable instead of modifying the cmdline
and rebooting or rebuilding the base image for different cases, and we
have already seen that a runtime toggle isn't hard at all.

> > doesn't conflict with what vswap is trying to do. Some other of these
> > differences involve design decisions rather than just surface knobs,
> > and those are the parts we're stuck with once they ship.
> >
> > So a few current interface difference of vswap and xswap I've noticed are:
> >
> >  - default size: somehow also tangled with VM_SPARSE, which
> > needs an initial virtual value. It's super cheap if not zero cost
> > (the only cost is occupying some vmalloc area, which is fine
> > I guess?). But we still have to pick a number, could be large
> > enough to satisfy everyone?
>
> 8PB? ;)

Right I saw that at V1 :).

I'm not sure if this is the reason that is in conflict with VM_SPARSE
in your mind? Techically, at least for 64 bit, we can just apply the
same limit to VM_SPARSE, for 64 bit machines we have a lot of virtual
space to consume and that should be fine (VM_SPARSE is just the
cluster map, which is much smaller compared to swap space).

And another thing is do we really need that much in any case? Will
things like, e.g. 64 times the physical memory (max pfn) be enough? I
know Baoquan suggest just use the physical memory size. So that could
be tunable, it's really easy to tune, just a number.

For things like an extremely high amount of zero folio swap, I think
remapping the zero folio it self is definately a better idea instead
of wasting a lot of SWAP entries and metadata in the long term

> >  - boot vs runtime toggle: I believe a runtime toggle is better. It
> > can still be tuned on boot via cmdline, but it offers the extra
> > ability to turn it on without modifying a base VM boot image or
> > kernel.
> >  - growth / limits: as talked above.
> >  - charging semantics
> >  - there could be other longterm design decisions on top of these as
> > we talked in above.
> >
> > Once these settle, I think the picture will be clean. Your writeback
> > logic and swap table integration are solid, while Baoquan's VM_SPARSE,
> > previous preparation with swap ops give us a structural foundation. A
> > clean long-term architecture means getting the core structure right,
> > let's try not treating this as a race between series.
>
> I've reached out to Baoquan as well, but I figured there's value in
> both discussions in big and small groups, and especially since there's
> a chance some of us might not be coming to LPC. Let's see what we come
> up with.

I'll be there this year if nothing goes wrong :), I hope we can come
up with something.

Re: Path forward for Virtualized Swap?

[Kairui Song]

On Fri, Sep 11, 2026 at 12:23 AM Nhat Pham <nphamcs@gmail.com> wrote:
>
> On Thu, Sep 10, 2026 at 3:57 AM David Hildenbrand (Arm)
> <david@kernel.org> wrote:
> >
> > On 9/8/26 18:45, Nhat Pham wrote:
> > > On Mon, Sep 7, 2026 at 4:30 AM David Hildenbrand (Arm) <david@kernel.org> wrote:
> > >>
> > >> On 9/4/26 23:14, Nhat Pham wrote:
> > >>
> > >> Hi,
> > >>
> > >>> To Kairui, Johannes, Yosry, Chris, David, Andrew, and other (z)swap folks -
> > >>> I need some help here.
> > >>
> > >> I'll mostly ignore swap-related stuff because (a) -EBUSY and (b) we have quite
> > >> some swap experts around :)
> > >>
> > >> Reading below, I do wonder if the swap subsystem needs some more coordination
> > >> and collaboration?
> > >>
> > >> For example, maybe having bi-weekly/monthly meetings to get everybody aligned
> > >> and uptodate could be something reasonable?
> > >>
> > >> This approach has proven fairly helpful in areas where (1) multiple companies
> > >> are involved (2) many concurrent development is happening, and (3) people have
> > >> different priorities and requirements.
> > >>
> > >> Is there already something like that to organize development? Should we get
> > >> started with that?
> > >
> > > Hi David!
> >
> > Hi!
> >
> > >
> > > Thank you for your suggestions :) More collaboration and communication
> > > is definitely needed here. Conference is a good place for this, but
> > > it's not something we can do very frequently, and many folks could not
> > > attend (visa issues, etc.).
> >
> > Right, and conferences usually don't have the capacity to squeeze too many
> > swap-related topics in, so it would have to be side discussions or BOFs.
> >
> > >
> > > A bi-weekly/monthly series of meetings sound very appealing. Swap
> > > contributors are very spread out (China, NorAm, Europe, etc.), so
> > > coordination will be tricky, but let us figure out something.
> >
> > Yes, that's an issue. I'd suggest swap maintainers take care of that if they
> > agree. I could try as well, but it's rather tricky right now on my end ...
> >
> > As an alternative, we could also use on/two/ ... slots in the bi-weekly MM
> > alignment session to talk about swap. (I'd be happy to learn more about all this
> > as well)
>
> That's awesome! Thanks for the offer David - I think MM alignment is at 9AM
> PDT Wednesday, usually right? That works for me of course, but Baoquan
> and Kairui (and other swap folks) - would this time be convenient for
> you?
>

Thanks for setting this up!

Is 9AM PDT fixed, or can it be adjusted? 9AM PDT is midnight for the
UTC+8 folks.

The time is still fine for me, though, I tend to stay up quite late
anyway ;) so that time works fine on my end if the slot is fixed.

Is there a calendar invite or something, please send me one, I'm happy
to do a swap sync discussion there.

Re: Path forward for Virtualized Swap?

[Johannes Weiner]

On Fri, Sep 11, 2026 at 08:27:00PM +0800, Baoquan He wrote:
> On 09/10/26 at 01:03pm, Johannes Weiner wrote:
> > [This reply was not LLM-generated.]
> > 
> > On Thu, Sep 10, 2026 at 03:09:59PM +0800, Baoquan He wrote:
> > > Hi Nhat,
> > > 
> > > On 09/04/26 at 02:14pm, Nhat Pham wrote:
> > > .....snip...
> > > > Now, on xswap. Baoquan's working on a series [15] that covers some of the
> > > > same ground, and the VM_SPARSE cluster_info idea in it is genuinely good.
> > > > I've been reviewing that lineage since July [16] and I'd like whatever
> > > > lands to end up with the best parts of both. From my perspective the
> > > > differences are:
> > > > 
> > > > 1. Userspace knobs. xswap asks the admin for a size (a percent of RAM) plus
> > > >    a per-device limit to tune afterwards. I'm not aware of any use case
> > > >    that needs those, and I don't think users have a good way to answer the
> > > >    question anyway - sizing swap for compressed memory depends on memory
> > > >    size, workload, and compression ratio all at once. That's precisely the
> > > >    provisioning problem vswap exists to remove. The kernel should be as
> > > >    transparent and dynamic as possible here, and not add knobs unless
> > > >    there's a use case for them.
> > > > 
> > > > 2. Writeback support. Writeback is core functionality for zswap, not an
> > > >    add-on, and a design needs to account for it from the start. This came
> > > >    up before, in the discussion around Chris' ghost swapfile RFC [17]: for
> > > >    a solution here to be acceptable, it has to work with the primary
> > > >    usecase and support disk writeback. Without it, whatever zswap won't
> > > >    take (incompressible pages especially) has nowhere to go, and cold
> > > >    compressed data can never leave RAM.
> > > > 
> > > > 3. Cgroup charging behavior. vswap/xswap shouldn't be charged against the
> > > >    swap usage counter. It's fundamentally a different resource from
> > > >    physical swapfile space, and memory.swap.* should read 0 when nothing is
> > > >    on disk [18]. I made the longer argument for this in [19].
> > > > 
> > > > 4. Data structure (xarray vs sparse vmalloc array). Even with xarray, vswap
> > > >    is already on par with or beating baseline. I like the sparse array
> > > >    idea, but why are we landing an optimization before the feature itself,
> > > >    without any A/B data showing the difference matters?
> > > 
> > > 
> > > Thanks for laying this out, and for the honest push to converge. Let me
> > > be equally direct about the ordering: I think the xswap base should land
> > > first, and the things vswap demonstrates -  writeback, rmap lookup, the
> > > charging semantics, later THP -- should be built on top of it. Because
> > > it is the foundation that keeps the swap core simpler, and the first thing
> > > to merge should be the one that doesn't have to be redone.
> > > 
> > > The VM_SPARSE array is not an optimization to bolt on later; it is a
> > > structural choice, and the code reflects it. In vswap, the cluster
> > > metadata lives in a dynamically-allocated xarray.
> > > 
> > > struct swap_cluster_info_dynamic {
> > >         struct swap_cluster_info ci;
> > >         unsigned int index;             /* for cluster_index() */
> > >         struct rcu_head rcu;
> > >         atomic_long_t *virtual_table;   /* Backing pointers for vswap slots */
> > > };
> > > 
> > > To support dynamic growth and shrink, vswap stores its cluster metadata
> > > in an xarray, and that forces two things the plain swap_cluster_info[]
> > > array never needed:
> > > 
> > > 1. Every cluster has to carry an extra index and an rcu_head —
> > > 24 bytes per cluster — purely so the xarray can locate it and free
> > > it safely.
> > > 2. To keep that bookkeeping from leaking into the normal-swap code, the
> > > cluster had to be wrapped in a container, swap_cluster_info_dynamic,
> > > so the xarray holds a pointer to the wrapper instead of an inline
> > > array element.
> > > 
> > > So in vswap, every cluster access in the shared hot path has to answer
> > > "is this a vswap device?" and take a separate branch:
> > > 
> > > - swap_is_vswap() is checked in 36 places across page_io.c, swapfile.c,
> > >   zswap.c and swap.h;
> > > - __swap_offset_to_cluster() branches into xa_load() for vswap vs the
> > >   flat array otherwise, and the xarray path can return NULL (a cluster
> > >   can be torn down);
> > > - __swap_cluster_lock() branches into __vswap_cluster_lock(), which
> > >   wraps every access in rcu_read_lock() and a CLUSTER_FLAG_DEAD check,
> > >   plus kfree_rcu()/container_of()/rcu_head plumbing for node lifetime.
> > 
> > Well to state the obvious: the reason it does all that is to make the
> > compression space transparent to the user.
> > 
> > The user can answer a simple boolean question: whether they want
> > compression or not. And it will work on tiny machines, on humongous
> > machines, and everything in between. That's a simple policy question
> > with a clear answer.
> > 
> > What you're doing, asking the user for a static size, is much more
> > difficult and has usability issues.
> > 
> > You're comparing implementations that don't accomplish the same thing.
> > 
> > The problem we're trying to solve is implementing a clean compression
> > space abstraction. I'm arguing that vswap does, and xswap does not.
> > 
> > While they're both using parts of the swap device code to implement a
> > compression space, xswap actually PRESENTS IT TO THE USER as a swap
> > device, and then makes optimizations BASED ON BAKED IN LIMITATIONS.
> > 
> > But a conventional, statically sized swap device is a bad abstraction
> > for the compression space. Here is why:
> > 
> > In conventional swap space, one memory page translates to one swap
> > page. Compression space doesn't act this way: a memory page can
> > consume anything between a few bytes to a full page in compression
> > space. It depends on memory contents and compression algorithm. So
> > right off the bat, this is a hard question to answer at the host level
> > which could run all kinds of workloads.
> > 
> > In conventional swap space, the resource consumed is a different
> > one. You're offloading memory by consuming disk space. This eats into
> > the space available to the filesystem, which is totally unrelated.
> > Asking the user for this tradeoff is a legitimate policy question.
> > 
> > Compression space is not a separate resource. It's page tables,
> > backing pages, and swap descriptors. It's just MEMORY. There isn't a
> > size tradeoff, because moving pages from memory space into compression
> > space DOES NOT CONSUME A NEW RESOURCE. It's still just memory. All you
> > need for containment already exists: rlimits, OOM killer, cgroup
> > memory controls.
> > 
> > By making this a user-visible virtual swap device, you're sending
> > users down the wrong path. You're asking them to set a new limit on a
> > resource that's already limited by other means. You're framing the
> > question as conventional swap which behaves completely differently.
> > 
> > If you ask them "how much swap space", they WILL reference this to
> > available RAM capacity. Maybe half of ram, maybe twice the RAM.
> > 
> > But when compression space is referenced to RAM, it's trivial to fill
> > it up with zeroed pages or easily compressible data LONG BEFORE the
> > process or container would hit any of its MEMORY limits.
> > 
> > This creates an artificial resource shortages. It forces a competition
> > where there shouldn't be one. And then you need new controls to manage
> > a competition that doesn't have to exist.
> > 
> > Like I said before, including compression space (which is memory) in
> > memory.swap.* (which is for disk space) is not going to be acceptable
> > from the cgroup side. We can talk about that if you want.
> > 
> > But asking the user questions they shouldn't have to answer, or
> > already answered elsewhere, is weak interface design. Allowing, let
> > alone encouraging, answers that create a whole new host of
> > organizational issues is outright bad interface design.
> > 
> > So if you want to compare implementations, you first have to actually
> > implement the same thing:
> > 
> > Stop asking user "how large". Let compression space expand towards
> > existing memory limits, such that it doesn't create an awkward and
> > artificial new resource competition.
> > 
> > Then we can compare implementations.
> > 
> > If the optimizations still apply under those constraints, great.
> > 
> > Until then, there is little point in discussing differences that, by
> > your own admission, have little to no impact on real world performance.
> 
> Thanks for your sharing with deliberate thought.
> 
> Agreed, and I want to be clear that the size knob is my implementation
> choice, not something the design needs.

+1

> Now in v2, xswap's create() already takes no size, only an optional
> priority: the device's address space is set by the kernel to the machine's
> memory

I've tried to explain above why this is a problem ;)

To be sure, the space cannot be completely unlimited. There are
internal restrictions, such as the refcount_t issue Nhat surfaced in
testing.

But I want to re-iterate the need to separate implementation choices
from what is visible to the user.

The address space should not be limited by anything but these hard
(and temporary) implementation limits. They can be fixed later as
machines evolve. But they shouldn't be anywhere near where real setups
can run out of compression space before they run out of memory.

Whether we want to vmap a petabyte+ of address space on boot, I'm a
bit skeptical... That sounds like an inherent point of tension between
large and small machines again. Dynamically extending the cluster
space as needed seems much more robust to me.

> and cluster_info is mapped lazily, so nothing is allocated up front.
> The only knob left is an optional per-device ceiling. If we really want to
> remove it, that's quite easy thing, we can just remove the runtime
> growth ceiling and the shrink machinery that serves it.

+1

> When I asked why shrink is needed, Nhat told on system, memory pressure
> could reach a peak, than later may not reach it again for a long time. I
> don't like the continuous automatic growing/shrinking, I think it
> doesn't make much sense just for saving that memory serving struct
> swap_cluster_info. But it's not bad to provide a mechanism for
> admin/users to tune it.

This also sounds like punting an implementation problem to userspace
again. If a workload peaks into compression space, the management
overhead is expected. But if they clear out, that memory should come
back predictably and automatically.

Requiring userspace to make garbage collection choices is not great.

I assume there is a limit to predictability either way because the
clusters may well fragment and remain sparsely populated. But IMO at
least the clusters that do clear out should free their memory quickly.

> But as I said, xswap/vswap both claim to solve the problem of zswap
> physical disk slot and swap slot coupling, and meantime extend
> functionality to make it more flexible than zswap/zram. While Nhat's
> vswap is boot-time per-device swap. And Nhat's own description of it
> in this thread is "vswap is just a normal swap device, no?". If I didn't
> apply Nhat's code and test I couldn't realize it. I executed swapon but
> can't see any output. I was shocked.

That's the implementation vs user-visible abstraction I was talking
about. This is why I think we should stop talking about swapfiles and
frame the discussion around how a compression space should act from a
user POV.

Compression space should not look like a conventional swap device to
users. Even if it shares an implementation.

Wanting visibility is fine. We have RSS, we have counts for zswap, we
should probably have counts for swapped zerofilled pages. Having
numbers for aggregate virtual swap entries is fine too. The problem is
showing some sort of hard upper bound, or showing it side to side with
conventional swapfiles that actually sit in storage.

> I really appreciated your patient and detailed sharing, while it takes
> you so long words to explain it. IMHO, it deserves a separate patch
> posting to justify it so that anyone can know why it is.

Thanks for the discussion!

Re: Path forward for Virtualized Swap?

[Shakeel Butt]

On Fri, Sep 11, 2026 at 09:06:29PM +0800, Baoquan He wrote:
> On 09/10/26 at 09:39am, Shakeel Butt wrote:
> > On Thu, Sep 10, 2026 at 03:09:59PM +0800, Baoquan He wrote:
> > > Hi Nhat,
> > > 
> > > On 09/04/26 at 02:14pm, Nhat Pham wrote:
> > > .....snip...
> > 
> > [...]
> > 
> > > With VM_SPARSE, xswap's cluster access is exactly the plain-array line the
> > > rest of swap already uses:
> > > 
> > >     return &si->cluster_info[offset / SWAPFILE_CLUSTER];
> > > 
> > > no branch, no RCU discipline, no tear-down state machine, and no NULL
> > > return. So VM_SPARSE doesn't add complexity to close a gap; it lets the
> > > cluster layer stay as simple as it already is, which is precisely the
> > > part later work (writeback, rmap lookup, memcg charging, THP) has to sit
> > > on.
> > > 
> > > I'm not going to claim xswap wins on throughput. I measured it:
> > > on a 64G/64-thread swapout, xswap, vswap and plain swap+zswap are all
> > > within ~2-3% of each other, effectively identical. 
> > 
> > So the claim is VM_SPARSE is simpler than xarray based approach. I feel like
> > we are discussing implementation details before deciding the design and
> > architecture. So, instead of VM_SPARSE vs xarray, let's discuss and decide the
> > need for dynamic growth. Why we want dynamic growth upfront or can it be added
> > later? Once we decide that then it will be very easy to pick an implementation
> > that would take us there.
> 
> Hi Shakeel,
> 
> Thank you for joining the discussion and for taking the time to comment.

Hi Baoquan,

I am mainly trying to facilitate the discussion but your use of LLM is causing
more confusion. LLM use is fine but please at least re-read before sending that
the sentences flow and makes sense.

> 
> Agreed on requirement first - but this one was already decided, and not by me. In
> the July ghost swapfile thread Nhat rejected exactly the shape of "grow only, can
> be added later":
> 
>     "Except for my virtual swap design, which does support dynamic growth AND
>      shrinking of capacity on demand ;) If it cannot grow (and furthermore, if it
>      requires userspace operation to trigger swapfile growth), why do we need this
>      at all? Might as well create a new swapfile with swapon?"
> 
> To me what it converged on was "dynamic growth and shrink, no writeback yet".

I am not getting how out of context above paragraph shows the conclusion about
dynamic growth/shrink and *no writeback*.

> So automatic growth *and* shrink is the requirement, and the simpler alternative
> was already on the table.
> 
> And I keep mentioning it in the cover-letter of each version of my posting. I
> only did the foundtation via lazy vmalloc. And Nhat will do the core
> part including writabck, rmap lookup, memcg accounting, zero page fill,
> etc.

I don't see any evidense of this decision. Actually this whole email thread
shows that there is no such decision.

> 
> What is still genuinely open, and I would like us to settle, is how large the
> device's address space should be, because the metadata scales with it:

With dynamic growth/shrink, is this really a blocker?

Anyways, I will let Nhat and others discuss the technical details (unless I am
asked for it). My main reason to join the conversation is to converge the
discussion to a decision and resolution.

Re: Path forward for Virtualized Swap?

[David Hildenbrand (Arm)]

On 9/11/26 18:20, Kairui Song wrote:
> On Fri, Sep 11, 2026 at 12:23 AM Nhat Pham <nphamcs@gmail.com> wrote:
>>
>> On Thu, Sep 10, 2026 at 3:57 AM David Hildenbrand (Arm)
>> <david@kernel.org> wrote:
>>>
>>>
>>> Hi!
>>>
>>>
>>> Right, and conferences usually don't have the capacity to squeeze too many
>>> swap-related topics in, so it would have to be side discussions or BOFs.
>>>
>>>
>>> Yes, that's an issue. I'd suggest swap maintainers take care of that if they
>>> agree. I could try as well, but it's rather tricky right now on my end ...
>>>
>>> As an alternative, we could also use on/two/ ... slots in the bi-weekly MM
>>> alignment session to talk about swap. (I'd be happy to learn more about all this
>>> as well)
>>
>> That's awesome! Thanks for the offer David - I think MM alignment is at 9AM
>> PDT Wednesday, usually right? That works for me of course, but Baoquan
>> and Kairui (and other swap folks) - would this time be convenient for
>> you?
>>
> 
> Thanks for setting this up!
> 
> Is 9AM PDT fixed, or can it be adjusted? 9AM PDT is midnight for the
> UTC+8 folks.
> 
> The time is still fine for me, though, I tend to stay up quite late
> anyway ;) so that time works fine on my end if the slot is fixed.
> 
> Is there a calendar invite or something, please send me one, I'm happy
> to do a swap sync discussion there.

Let's get David R. involved.

@David, we'd like to use a meeting slot for a high-level discussion on the next
steps / future for swap. When would the next be available?

-- 
Cheers,

David

Re: Path forward for Virtualized Swap?

[Nhat Pham]

On Fri, Sep 11, 2026 at 9:10 AM Kairui Song <ryncsn@gmail.com> wrote:
>
>> >
> > >
> > > > 4. Data structure (xarray vs sparse vmalloc array). Even with xarray, vswap
> > > >    is already on par with or beating baseline. I like the sparse array
> > > >    idea, but why are we landing an optimization before the feature itself,
> > > >    without any A/B data showing the difference matters?
> > >
> > > This is where I'd push back a little. To be clear, my concern isn't the
> > > size of the win, it's that VM_SPARSE isn't purely an optimization. It
> > > changes the cluster/metadata layout and avoids RCU conventions,
> > > could help avoid some future sleep while atomic limitations and
> > > it's tangled with the initial size issue too. So bolting it on afterwards
> > > could mean rewriting the parts of vswap that touch it and could be
> > > costly some conflict in ideas. That's why I'd rather we agree on
> > > it now, not that it has to land first.
> >
> > I don't think the current code differs significantly from the existing
> > model, no?
> >
> > For the most part, RCU played quite nicely with the current swap code.
> > Practically, it's just:
> >
> > 1. If you have a reference to the swap entry, you can assume entry and
> > the cluster owning it is safe.
> >
> > 2. If you don't, then do not assume it's safe once you exit the rcu
> > read section.
> >
> > What conflicts specifically do you have in mind here :)
>
> Yeah you're right. I'm just a bit worried if that RCU period will be
> extended, and it's always better to make the convention as simple as
> possible. I'm already trying to get rid of the local lock, in that
> swap queue series, that is not so painful and still a bit troublesome
> for a few new use cases.

Completely understood. Your inquiry made me give it a double take - I
took Baoquan's code and hacked it on top of my vswap patch series, and
it seems reasonably simple :)

https://lore.kernel.org/all/20260910232704.3364879-1-nphamcs@gmail.com/

Seems to slot in quite nicely. I mean, we're converging to the *same*
architecture, just different data structure. We can switch down the
line - it just would require a lot more testing, benchmarking,
evaluating, etc. to adopt this.

> >
> > I think a lot of these wins come from the fact that the zswap's xarray
> > specifically is *really* bad in certain paths. For instance, in the
> > free path (swap_range_free() -> zswap_invalidate()), there is no
> > batching behavior, so we walk the tree from scratch for each entry
> > rather than using an xas iterator for the entire batch. That goes away
> > in the current vswap model, because the xarray manages cluster rather
> > than entries, so you just do one tree walk + one flat array walk.
>
> Right but with vswap replacing zswap, isn't the performance is still
> basically unchanged according to your test (or just a little bit
> faster)? My test still doesn't show a measurable improvement.

Yeah in my test it also improves in some cases but not others. But
FWIW, the xarray-less approach does not outperform the xarray-version
yet :) I tested with the old coupled design, only replacing xarray
with a per-cluster flat array - so basically Baoquan's design but
without the vmalloc overhead. It performs basically on par with the
xarray (where it wins, it matches xarray's wins, etc.).

And I tried my best to keep the old code intact (all the
swap_is_vswap() that Baoquan disliked was for that too haha). That
make sure physical-swapfile-only users, and those who still disable
vswap at boottime, has no overhead.

>
> > zswap and swapfile coupled, but only remove the xarray and replace it
> > with the flat array in swapfile's cluster struct. I observe the same
> > win, basically (especially on the usemem benchmark).
>
> Yeah, merging the zswap xarray into cluster should buy a lot of
> performance gain, the only concern is the cluster it self is in a
> xarray tree so if your vswap space gets huge, could that somehow
> become an issue? Zswap used a split xarray, and swap also used to have
> a 64M partition for the swap cache xarray, which is gone after the
> swap table.

Hmm not in my testing so far, both in production and synth-benchs, and
we have some pretty big swap users.

As long as the xarray does not lock us into any model, we can always
optimize further IMHO. There are several axes of scaling - another one
is raising the size of the cluster.

>
> ...
>
> > > We were also discussing the limitation of the global cluster cache in
> > > that thread, so I paused my other work for a while and posted the swap
> > > queue series first. That one is orthogonal on purpose, and helps both
> > > swap tier and vswap, it reduces the overhead enough to cancel out
> > > the layering cost due to missing caching for either:
> > > https://lore.kernel.org/linux-mm/20260829-swap-pcp-priq-v2-resend-0-68d3d925578c@gmail.com/
> > > (this is V2, V1 is earlier - thanks to Lian for keeping this going!)
> >
> > Ah that! I sort of tabled that as a nice-idea-but-still-optimization :P
> >
> > I'll spend some time thinking about the design and reviewing it. In
> > the long run, it will get things more simple, but another thing that
> > should not be a blocker, correct?
>
> Right, definitely not a blocker. I think the swap queue could be
> merged after tiering and other stuff, as an optimization, just to
> demonstrate that we don't need another layer of cache; we can just use
> the devices directly without any cache above them. It will also
> deprecate plist; with the only plist rotate user is gone, we can get
> rid of a large chunk of common plist code.

Look forward to seeing more of this! It was a personal request from
me, so I appreciate that you look into this.

>
> ...
>
> > > The two interfaces aren't equivalent, xswap has its limitations, but a
> > > couple of xswap's properties look genuinely better to me. Runtime
> > > on/off rather than a boot parameter is better, more flexibility and
> >
> > I would also like to point out - I *do* have a runtime parameter in an
> > older version of vswap (v3). There is also not a lot of engagement
> > from folks in that version - no one expressed any interest in that
> > runtime knob in particular. Only Johannes made the case that a boot
> > parameter is superior, so I switched over. I think his reasoning makes
> > sense at the time FWIW, and it also removes the need for CONFIG_VSWAP
> > (which in turns remove a bunch of useless stubs - saving 100 LoCs), so
> > I figure folks will prefer less code and less userspace decisions :)
> >
> > v3: https://lore.kernel.org/all/20260806184254.3790858-1-nphamcs@gmail.com/
>
> Ah, thanks for the reminder.
>
> > Regardless - do you have a specific use case in mind where a runtime
> > knob would be useful?
>
> A boot time bool will cause real trouble for us, at least. We have
> many VMs with fixed boot images and fixed cmdlines. However, the VM
> itself is used by different customers, some are ZRAM only users, and
> quite a lot don't want any kind of compression (e.g. their data is
> dense compression is just waste of CPU) and prefer plain swap. So I do
> hope we can have a runtime tunable instead of modifying the cmdline
> and rebooting or rebuilding the base image for different cases, and we
> have already seen that a runtime toggle isn't hard at all.

Ah yeah I see. But before moving back to runtime parameter, do you
think we can come up with a heuristics for this that have does not
require userspace input? In v4, I'm doing this at folio_alloc_swap()
time:

a. If the cgroup enables zswap, then always go vswap.

b. If zswap is not available for the cgroup, then bypass vswap, and go
straight for physical swap entries. My intuition is that we are not
buying anything for these zram users, as we need a physical swap slot
anyway. Thanks to your design, both can co-exist in the system now -
that's something I actually did not envision possible in the original
version of vswap. Pretty cool eh :)

Would something like this fix it for you? ZRAM users will not get
vswap indirection overhead at all, because it would bypass vswap :)
Down the line we can revisit this decision - for e.g, if there is a
use case for vswap-on-top-of-swapfile. There might be other interface
that makes more sense.

>
> > > doesn't conflict with what vswap is trying to do. Some other of these
> > > differences involve design decisions rather than just surface knobs,
> > > and those are the parts we're stuck with once they ship.
> > >
> > > So a few current interface difference of vswap and xswap I've noticed are:
> > >
> > >  - default size: somehow also tangled with VM_SPARSE, which
> > > needs an initial virtual value. It's super cheap if not zero cost
> > > (the only cost is occupying some vmalloc area, which is fine
> > > I guess?). But we still have to pick a number, could be large
> > > enough to satisfy everyone?
> >
> > 8PB? ;)
>
> Right I saw that at V1 :).
>
> I'm not sure if this is the reason that is in conflict with VM_SPARSE
> in your mind? Techically, at least for 64 bit, we can just apply the
> same limit to VM_SPARSE, for 64 bit machines we have a lot of virtual
> space to consume and that should be fine (VM_SPARSE is just the
> cluster map, which is much smaller compared to swap space).
>
> And another thing is do we really need that much in any case? Will
> things like, e.g. 64 times the physical memory (max pfn) be enough? I
> know Baoquan suggest just use the physical memory size. So that could
> be tunable, it's really easy to tune, just a number.

Potentially, but we have many users at Meta. There's a huge diversity
of machine types, workingset size, access patterns (both frequency and
file:anon split), compressibility, etc.

Tuning even just one number, but for a combinatorial explosions of
these factors, cost a lot of time and effort. This is a per-host
property, which often means big hosts with multiple services
co-located cannot decide alone.

And even if sysadmin lets users decide this value, it's very likely
they'll get that wrong (improper tuning or just copy the config from
another place), which in turns cost engineering hours to go chase that
down.

So I always lean towards as plug-and-play as possible. And as I said
earlier, it's easier to add knobs than to remove them (or ask users to
stop using them).

>
> For things like an extremely high amount of zero folio swap, I think
> remapping the zero folio it self is definately a better idea instead
> of wasting a lot of SWAP entries and metadata in the long term
>

I don't disagree here.

We tried doing that in the past, but it was slightly buggy. It might
be solvable, we just don't have the manpower to deal with this yet :(


> > >  - boot vs runtime toggle: I believe a runtime toggle is better. It
> > > can still be tuned on boot via cmdline, but it offers the extra
> > > ability to turn it on without modifying a base VM boot image or
> > > kernel.
> > >  - growth / limits: as talked above.
> > >  - charging semantics
> > >  - there could be other longterm design decisions on top of these as
> > > we talked in above.
> > >
> > > Once these settle, I think the picture will be clean. Your writeback
> > > logic and swap table integration are solid, while Baoquan's VM_SPARSE,
> > > previous preparation with swap ops give us a structural foundation. A
> > > clean long-term architecture means getting the core structure right,
> > > let's try not treating this as a race between series.
> >
> > I've reached out to Baoquan as well, but I figured there's value in
> > both discussions in big and small groups, and especially since there's
> > a chance some of us might not be coming to LPC. Let's see what we come
> > up with.
>
> I'll be there this year if nothing goes wrong :), I hope we can come
> up with something.

I'm excited to see you and the rest of the swap cabal too :)

Re: Path forward for Virtualized Swap?

[Kairui Song]

On Sat, Sep 12, 2026 at 12:57 AM Nhat Pham <nphamcs@gmail.com> wrote:
>
> On Fri, Sep 11, 2026 at 9:10 AM Kairui Song <ryncsn@gmail.com> wrote:
> >
> > A boot time bool will cause real trouble for us, at least. We have
> > many VMs with fixed boot images and fixed cmdlines. However, the VM
> > itself is used by different customers, some are ZRAM only users, and
> > quite a lot don't want any kind of compression (e.g. their data is
> > dense compression is just waste of CPU) and prefer plain swap. So I do
> > hope we can have a runtime tunable instead of modifying the cmdline
> > and rebooting or rebuilding the base image for different cases, and we
> > have already seen that a runtime toggle isn't hard at all.
>
> Ah yeah I see. But before moving back to runtime parameter, do you
> think we can come up with a heuristics for this that have does not
> require userspace input? In v4, I'm doing this at folio_alloc_swap()
> time:
>
> a. If the cgroup enables zswap, then always go vswap.
>
> b. If zswap is not available for the cgroup, then bypass vswap, and go
> straight for physical swap entries. My intuition is that we are not
> buying anything for these zram users, as we need a physical swap slot
> anyway. Thanks to your design, both can co-exist in the system now -
> that's something I actually did not envision possible in the original
> version of vswap. Pretty cool eh :)

Thanks for the test and idea! Yeah, I always hoped we could do this
because there are some very promising use cases for raw and plain
swap, not just ZRAM.

> Would something like this fix it for you? ZRAM users will not get
> vswap indirection overhead at all, because it would bypass vswap :)
> Down the line we can revisit this decision - for e.g, if there is a
> use case for vswap-on-top-of-swapfile. There might be other interface
> that makes more sense.

Hmm, this interface looks confusing though. you mean tangles vswap
with zswap through the cgroup's zswap limit? I originally expected
this to be a problem solved by tiering, skipping certain tiers seems
much more intuitive. Maybe Youngjun have some idea here?

> > > > doesn't conflict with what vswap is trying to do. Some other of these
> > > > differences involve design decisions rather than just surface knobs,
> > > > and those are the parts we're stuck with once they ship.
> > > >
> > > > So a few current interface difference of vswap and xswap I've noticed are:
> > > >
> > > >  - default size: somehow also tangled with VM_SPARSE, which
> > > > needs an initial virtual value. It's super cheap if not zero cost
> > > > (the only cost is occupying some vmalloc area, which is fine
> > > > I guess?). But we still have to pick a number, could be large
> > > > enough to satisfy everyone?
> > >
> > > 8PB? ;)
> >
> > Right I saw that at V1 :).
> >
> > I'm not sure if this is the reason that is in conflict with VM_SPARSE
> > in your mind? Techically, at least for 64 bit, we can just apply the
> > same limit to VM_SPARSE, for 64 bit machines we have a lot of virtual
> > space to consume and that should be fine (VM_SPARSE is just the
> > cluster map, which is much smaller compared to swap space).
> >
> > And another thing is do we really need that much in any case? Will
> > things like, e.g. 64 times the physical memory (max pfn) be enough? I
> > know Baoquan suggest just use the physical memory size. So that could
> > be tunable, it's really easy to tune, just a number.
>
> Potentially, but we have many users at Meta. There's a huge diversity
> of machine types, workingset size, access patterns (both frequency and
> file:anon split), compressibility, etc.

You can just set the number as 8PB? :)

Re: Path forward for Virtualized Swap?

[Nhat Pham]

On Fri, Sep 11, 2026 at 11:14 AM Kairui Song <ryncsn@gmail.com> wrote:
>
> > Would something like this fix it for you? ZRAM users will not get
> > vswap indirection overhead at all, because it would bypass vswap :)
> > Down the line we can revisit this decision - for e.g, if there is a
> > use case for vswap-on-top-of-swapfile. There might be other interface
> > that makes more sense.
>
> Hmm, this interface looks confusing though. you mean tangles vswap
> with zswap through the cgroup's zswap limit? I originally expected
> this to be a problem solved by tiering, skipping certain tiers seems
> much more intuitive. Maybe Youngjun have some idea here?

It wouldn't be confusing if this is something users don't have to
think about at all :) My point is that we don't have a use case where
we need userspace input on vswap-enablement on a per-cgroup basis yet,
so let's just keep it all transparent. From user perspective, they
enable zswap, and it just works - vswap is just an internal
implementation details.

I have discussed with Youngjun regarding vswap participation in swap
tiering interface in the first version of this new design. It's
technically achievable, but we decided to post-pone that for now until
a true use case comes about - trying to cut down as much code as
possible...

>
> >
> > Potentially, but we have many users at Meta. There's a huge diversity
> > of machine types, workingset size, access patterns (both frequency and
> > file:anon split), compressibility, etc.
>
> You can just set the number as 8PB? :)

Sure, but with the vmalloc-array approach, there is more metadata
overhead even if we have not allocated the backing page of the
clusters yet. This will be annoying on the smaller size machines
(O(dozen of GB)) to also pay the overhead of 8PB-swap space metadata
reservation.

With xarray yeah it's truly just a number limit. Put it as big as it is allowed.

The "allowed" part brings me to the next point - not sure if you have
seen my other thread, but I think we need to be even more careful with
this limit - I have found another weird interaction between memcg and
swap subsytem, specifically with the refcount of private id. I think
this is another argument for transparency - this limit is now also
capped on architectural (page size?) and implementational (refcount
type) details of the host and the kernel.

https://lore.kernel.org/all/CAKEwX=NYoMH8pTKeCvA=XHFmmNVMeDz3mxbhp5q6-PxSTJ5wOg@mail.gmail.com/

Seems a bit much to ask the userspace to know all of this. Better to just let:

a. the space grow on demand, automatically.

b. the space be limited by an implementation-induced cap.

all transparent to user (until we have a true use case for a userspace
sizing knob).

Re: Path forward for Virtualized Swap?

[Kairui Song]

On Sat, Sep 12, 2026 at 3:04 AM Nhat Pham <nphamcs@gmail.com> wrote:
>
> On Fri, Sep 11, 2026 at 11:14 AM Kairui Song <ryncsn@gmail.com> wrote:
> >
> > > Would something like this fix it for you? ZRAM users will not get
> > > vswap indirection overhead at all, because it would bypass vswap :)
> > > Down the line we can revisit this decision - for e.g, if there is a
> > > use case for vswap-on-top-of-swapfile. There might be other interface
> > > that makes more sense.
> >
> > Hmm, this interface looks confusing though. you mean tangles vswap
> > with zswap through the cgroup's zswap limit? I originally expected
> > this to be a problem solved by tiering, skipping certain tiers seems
> > much more intuitive. Maybe Youngjun have some idea here?
>
> It wouldn't be confusing if this is something users don't have to
> think about at all :) My point is that we don't have a use case where
> we need userspace input on vswap-enablement on a per-cgroup basis yet,

Ha, you really can't assume that for everyone :), we even have
workloads working at different compression levels, and the tiering
series shows different cgroups working on different devices, there has
been pathes enabling per-cgroup swap device a decade ago.

> > > Potentially, but we have many users at Meta. There's a huge diversity
> > > of machine types, workingset size, access patterns (both frequency and
> > > file:anon split), compressibility, etc.
> >
> > You can just set the number as 8PB? :)
>
> Sure, but with the vmalloc-array approach, there is more metadata
> overhead even if we have not allocated the backing page of the
> clusters yet. This will be annoying on the smaller size machines
> (O(dozen of GB)) to also pay the overhead of 8PB-swap space metadata
> reservation.

Not al all? With VM_SPARSE, you literally have zero overhead, unless
you really use these slots.

And even if you really use these slots, the overhead in both CPU and
memory is lower than Xarray, if I did the math right. I just ran a
quick test with Baoquan's Xswap, forcing it to start with 8PB on boot;
it was smooth as batter on a 1GB box and still worked fine after heavy
anon memory pressure.

See below (8PB xswap device with almost no metadata overhead, the only
overhead comes from zswap so I reduced the tree number but that can be
easily merged into clusters as we discussed before):

### Before XSWAP
# free -m
               total        used        free      shared  buff/cache   available
Mem:            1464         405         756           1         544        1058
Swap:              0           0           0

### Create a XSWAP device, I hardcoded the max size in code to 8PB,
### and use fewer ZSWAP trees:
# echo 0 > /sys/kernel/mm/xswap/create
[  140.289280] xswap: adding extendable swap type 0 (prio -1, init 733
clusters = 375295 pages, max 2199023255552 pages)
# free -m
               total        used        free      shared  buff/cache   available
Mem:            1464         407         755           1         545        1057
Swap:           1465           0        1465

### Lift the ceiling to 8PB:
# echo 4294967296 > /sys/kernel/debug/xswap/type0_cluster_limit
# free -m
               total        used        free      shared  buff/cache   available
Mem:            1464         403         756           1         547        1060
Swap:     8589934591           0  8589934591

### Run memhog:
# time memhog 12G
...
________________________________________________________
Executed in    4.36 secs    fish           external
   usr time    0.46 secs    0.00 millis    0.46 secs
   sys time    3.77 secs    3.80 millis    3.76 secs

### After memhog:
# free -m
               total        used        free      shared  buff/cache   available
Mem:            1464         336        1136           0         222        1127
Swap:     8589934591          31  8589934560

The only different from vswap now is that you just hides everything :)
(writeback is not hard to add)

> https://lore.kernel.org/all/CAKEwX=NYoMH8pTKeCvA=XHFmmNVMeDz3mxbhp5q6-PxSTJ5wOg@mail.gmail.com/

Right I saw that, but 64 bit of private id will make shadows really
awkward, not for SWAP but for file folios and file mappings, swap
table is actully fine since memcg_table is standaone, and we can
workaround the hieratical refault distance issue in other ways (e.g.
record the reclaim level instead of memcgid direct). I think 24 bit is
good for now? As I've never see anyone ran into issues
with even 16 bits in production.

Re: Path forward for Virtualized Swap?

[Kairui Song]

On Thu, Sep 10, 2026 at 12:42 AM Nhat Pham <nphamcs@gmail.com> wrote:
>
> On Tue, Sep 8, 2026 at 11:30 AM Johannes Weiner <hannes@cmpxchg.org> wrote:
> >
> > On Mon, Sep 07, 2026 at 01:51:31PM +0800, Kairui Song wrote:
> > > Where I've ended up is that unbounded growth is a real concern. On a
> > > host with no memcg limit (root cgroup, and most desktop and embedded
> > > setups), an unlimited pool means usage can keep growing, with no
> > > admin visible ceiling at all. I'm not attached to xswap's percent of RAM
> > > knob specifically, but I do think some kind of bound makes sense.
> >
> > Swap space is just process virtual address space, no?
> >
> > Swap entries already have one or more page table entries pointing to
> > them, which in turn are managed by trees of vm_area_structs. That
> > means rlimits apply, overcommit protection applies, and OOM killer
> > attribution works as well (oom_badness()).
>
> I tested this theory. I spinned up a process, and let it spam 0-filled
> memory + swap these pages out continually.
>
> As you predicted, oom-killer picked it up eventually. The host was
> (and is) intact otherwise :)

Good to know the kill path works. But I think the accounting side cuts the
other way? Won't that conversely underestimate the host's ability to
handle memory alloc? Not to mention a lot of application are
swap  space aware, some built in logics like e.g. with vm_enough_memory:

On a 1G machine with vswap enabled:
[    0.241906] vswap: created virtual swap device (2199023255040 pages)

a 2G sparse anonymous allocation fails:
[   46.044002] __vm_enough_memory: pid: 1129, comm: search_agent,
bytes: 2147483648 not enough memory for the allocation.

The default "Heuristic overcommit handling" policy is meant to handle
seriously wild allocation (as documented, and it's named as
OVERCOMMIT_GUESS).
totalram_pages() + total_swap_pages

Is the limit, the heuristic exists only to make "a seriously wild
allocation fail" (as documented). But on a vswap-only machine,
reasonable allocations fail. Common swap devices, zram, or xswap don't
have this problem. One could argue that the size there is just an
optimistic guess, the compression ratio is not controllable. But that
heuristic is a guess by design, and a plausible number serves it fine.
"Unlimited" seems break that.

One can switch to strict overcommit limit, but that is limited by a
user specified ratio or number; again, we need a concrete number. A
fuzzy "unlimited" seems odd.

Re: Path forward for Virtualized Swap?

[Johannes Weiner]

On Sat, Sep 12, 2026 at 05:00:42PM +0800, Kairui Song wrote:
> On Thu, Sep 10, 2026 at 12:42 AM Nhat Pham <nphamcs@gmail.com> wrote:
> >
> > On Tue, Sep 8, 2026 at 11:30 AM Johannes Weiner <hannes@cmpxchg.org> wrote:
> > >
> > > On Mon, Sep 07, 2026 at 01:51:31PM +0800, Kairui Song wrote:
> > > > Where I've ended up is that unbounded growth is a real concern. On a
> > > > host with no memcg limit (root cgroup, and most desktop and embedded
> > > > setups), an unlimited pool means usage can keep growing, with no
> > > > admin visible ceiling at all. I'm not attached to xswap's percent of RAM
> > > > knob specifically, but I do think some kind of bound makes sense.
> > >
> > > Swap space is just process virtual address space, no?
> > >
> > > Swap entries already have one or more page table entries pointing to
> > > them, which in turn are managed by trees of vm_area_structs. That
> > > means rlimits apply, overcommit protection applies, and OOM killer
> > > attribution works as well (oom_badness()).
> >
> > I tested this theory. I spinned up a process, and let it spam 0-filled
> > memory + swap these pages out continually.
> >
> > As you predicted, oom-killer picked it up eventually. The host was
> > (and is) intact otherwise :)
> 
> Good to know the kill path works. But I think the accounting side cuts the
> other way? Won't that conversely underestimate the host's ability to
> handle memory alloc? Not to mention a lot of application are
> swap  space aware, some built in logics like e.g. with vm_enough_memory:
> 
> On a 1G machine with vswap enabled:
> [    0.241906] vswap: created virtual swap device (2199023255040 pages)
> 
> a 2G sparse anonymous allocation fails:
> [   46.044002] __vm_enough_memory: pid: 1129, comm: search_agent,
> bytes: 2147483648 not enough memory for the allocation.
> 
> The default "Heuristic overcommit handling" policy is meant to handle
> seriously wild allocation (as documented, and it's named as
> OVERCOMMIT_GUESS).
> totalram_pages() + total_swap_pages
> 
> Is the limit, the heuristic exists only to make "a seriously wild
> allocation fail" (as documented). But on a vswap-only machine,
> reasonable allocations fail. Common swap devices, zram, or xswap don't
> have this problem. One could argue that the size there is just an
> optimistic guess, the compression ratio is not controllable. But that
> heuristic is a guess by design, and a plausible number serves it fine.
> "Unlimited" seems break that.

That cuts both ways, though. If the configured compression space is
large and compression ratio is poor, it lets through allocations that
can be considered "seriously wild" for this machine.

It's a filter. I wouldn't say that letting things through is evidence
that it's working.

Since compression space is backed by memory, it still makes sense to
me to reference this heuristic to RAM. Cut it off at 2xRAM or 3xRAM
tops if compression space is available.

That all being said, I've laid out implications and concerns around
limiting compression space in this thread, too. Collecting pros and
cons is fine, but if you bring up cons for the unlimited case, it's
fair to ask that you engage with cons brought up on the limited case,
too, so that we can weigh the tradeoffs.