Path forward for Virtualized Swap?

Path forward for Virtualized Swap?

[Nhat Pham]

To Kairui, Johannes, Yosry, Chris, David, Andrew, and other (z)swap folks -
I need some help here.

I'd like to ask for a decision, or failing that, the specific thing that
would get us to one. I'd much rather work against a concrete objection than
keep respinning into silence.

Quick recap: today a swapped-out page holds a disk slot even when the data
never reaches disk, like zswap or zero-filled pages. My patch series
(latest version at [10]) puts a virtual entry in the page table instead and
resolves the backing separately, so zswap stops reserving disk it never
uses and can work on machines with no storage swap space at all.

I've been working on this since April 2025, going through 10 revisions (6
on the original design ([1] through [6]) and 4 on the swap-table-based
rewrite ([7] through [10])). I also gave a talk on it at LPC [11].

Throughout the process, I've been very receptive to reviewers - including
parties whose use cases are very different from my own, like zram users and
the ghost-swapfile-style deployments that want zswap with no backing disk
at all - and the current design reflects that:

* Kairui asked that this be optional, not a replacement [12]. It is now:
  vswap is opt-in behind the vswap= cmdline parameter, and with vswap=off
  the existing swap paths aren't touched at all.

* Kairui also pointed out that if I kept shrinking the metadata I'd "end up
  reimplementing the swap table format" [12]. I took that feedback, and
  reworked the entire series on top of swap table.

* Per Chris' requests, I've figured out ways to minimize memory overhead
  [13]. On a zswap-only setup there's almost 0 overhead now, because the
  vtable is merged with zswap's xarray rather than sitting on top of it.
  The indirection is paid for by the tree we delete.

* Johannes pointed out that a physical cluster backing only vswap entries
  still had to allocate a memcg_table, which was an overhead regression
  against my older design on the zswap writeback case [14] (since a swap
  entry occupies multiple levels). I've also fixed this.

* I've spent a lot of time chasing regressions on setups we don't even run
  at Meta - zram in particular [6]. And if anyone's still worried after all
  that, vswap=off keeps the old swap code available.

Numbers are in the v4 cover letter [10]. In terms of performance, vswap is
on par or better than baseline on zswap across memhog, usemem and kernel
build, and vswap=off tracks baseline within noise. I've even included an
A/B experiment on a real production workload (Instagram/django): throughput
on par, latency slightly better, and the swapfile footprint for every zswap
user on the host goes away.

Now, on xswap. Baoquan's working on a series [15] that covers some of the
same ground, and the VM_SPARSE cluster_info idea in it is genuinely good.
I've been reviewing that lineage since July [16] and I'd like whatever
lands to end up with the best parts of both. From my perspective the
differences are:

1. Userspace knobs. xswap asks the admin for a size (a percent of RAM) plus
   a per-device limit to tune afterwards. I'm not aware of any use case
   that needs those, and I don't think users have a good way to answer the
   question anyway - sizing swap for compressed memory depends on memory
   size, workload, and compression ratio all at once. That's precisely the
   provisioning problem vswap exists to remove. The kernel should be as
   transparent and dynamic as possible here, and not add knobs unless
   there's a use case for them.

2. Writeback support. Writeback is core functionality for zswap, not an
   add-on, and a design needs to account for it from the start. This came
   up before, in the discussion around Chris' ghost swapfile RFC [17]: for
   a solution here to be acceptable, it has to work with the primary
   usecase and support disk writeback. Without it, whatever zswap won't
   take (incompressible pages especially) has nowhere to go, and cold
   compressed data can never leave RAM.

3. Cgroup charging behavior. vswap/xswap shouldn't be charged against the
   swap usage counter. It's fundamentally a different resource from
   physical swapfile space, and memory.swap.* should read 0 when nothing is
   on disk [18]. I made the longer argument for this in [19].

4. Data structure (xarray vs sparse vmalloc array). Even with xarray, vswap
   is already on par with or beating baseline. I like the sparse array
   idea, but why are we landing an optimization before the feature itself,
   without any A/B data showing the difference matters?

One thing I do want to be clear about: I'm glad other people care about
this problem. Chris' ghost swapfile and Baoquan's xswap are both going
after the same set of problems, and that's a good sign. It means this is
real and shared, not something only Meta runs into.

What's been harder is the shape of the engagement. Alternatives keep
getting posted and pushed that don't cover all the requirements, while this
series sits without review. I don't think I'm owed anyone's interest in the
problems I care about. But I do think working code, with benchmarks and
production exposure behind it, deserves a fair hearing next to in-progress
proposals.

So what I'm asking for: I'd like us to converge rather than keep two series
in flight. My preference is that we land vswap first, then build Baoquan's
sparse array on top of it as an optimization. That gets the feature in, and
by then we'd have the A/B data to show whether the sparse array actually
beats the xarray.

If you think that's the wrong order, I'd genuinely like to understand why -
after 17 months and 10 revisions I still don't have a clear picture of the
objection.

[1] https://lore.kernel.org/all/20250407234223.1059191-1-nphamcs@gmail.com/
[2] https://lore.kernel.org/all/20250429233848.3093350-1-nphamcs@gmail.com/
[3] https://lore.kernel.org/all/20260208215839.87595-1-nphamcs@gmail.com/
[4] https://lore.kernel.org/all/20260318222953.441758-1-nphamcs@gmail.com/
[5] https://lore.kernel.org/all/20260320192735.748051-1-nphamcs@gmail.com/
[6] https://lore.kernel.org/all/20260505153854.1612033-1-nphamcs@gmail.com/
[7] https://lore.kernel.org/all/20260528212955.1912856-1-nphamcs@gmail.com/
[8] https://lore.kernel.org/all/20260612193738.2183968-1-nphamcs@gmail.com/
[9] https://lore.kernel.org/all/20260806184254.3790858-1-nphamcs@gmail.com/
[10] https://lore.kernel.org/all/20260825153238.2695446-1-nphamcs@gmail.com/
[11] https://lwn.net/Articles/1016136/
[12] https://lore.kernel.org/all/CAMgjq7AQNGK-a=AOgvn4-V+zGO21QMbMTVbrYSW_R2oDSLoC+A@mail.gmail.com/
[13] https://lore.kernel.org/all/CACePvbVXQWgcPD-bgK7iDba4NFLo2tT89ZbLOa03maJU4er4ag@mail.gmail.com/
[14] https://lore.kernel.org/all/aZyFxKGXc8J6PIij@cmpxchg.org/
[15] https://lore.kernel.org/all/20260827094509.1016740-1-hebaoquan@kylinos.cn/
[16] https://lore.kernel.org/lkml/CAKEwX=Pe+qMZd2xhnU-PAGQtgXkp56c-JwYCbt2Lux9htgB67Q@mail.gmail.com/
[17] https://lore.kernel.org/all/20251121114011.GA71307@cmpxchg.org/
[18] https://lore.kernel.org/all/anYIboHEUZb4fhHv@cmpxchg.org/
[19] https://lore.kernel.org/all/CAKEwX=P4syV38jAVCWq198r2OHXXc=xA-fx1dk6+qYef6yzxWQ@mail.gmail.com/

Re: Path forward for Virtualized Swap?

[Kairui Song]

On Sat, Sep 5, 2026 at 5:14 AM Nhat Pham <nphamcs@gmail.com> wrote:
>
> To Kairui, Johannes, Yosry, Chris, David, Andrew, and other (z)swap folks -
> I need some help here.

Hello Nhat,

>
> I'd like to ask for a decision, or failing that, the specific thing that
> would get us to one. I'd much rather work against a concrete objection than
> keep respinning into silence.
>
> Quick recap: today a swapped-out page holds a disk slot even when the data
> never reaches disk, like zswap or zero-filled pages. My patch series
> (latest version at [10]) puts a virtual entry in the page table instead and
> resolves the backing separately, so zswap stops reserving disk it never
> uses and can work on machines with no storage swap space at all.
>
> I've been working on this since April 2025, going through 10 revisions (6
> on the original design ([1] through [6]) and 4 on the swap-table-based
> rewrite ([7] through [10])). I also gave a talk on it at LPC [11].

Thanks for the hard work! And I mean that. Upstream work at this scale
does take a long time and it isn't a reflection on the series: the swap
cache optimization idea also took me over 2 years and changed direction
multiple times and split multiple times before it landed as
the swap table series, and the readahead unification part is
still in flight after nearly 3 years. It's frustrating and I don't think
anyone enjoys it. I mentioned this once with a longer history:

https://lore.kernel.org/linux-mm/CAMgjq7CF9RgnZCAS-+Gv0LAvkzzHk4jiok+_6-KOFw-o+s8E_g@mail.gmail.com/

There is also a quote from LSF/MM/BPF saying it's for improvements to
"find their way into the mainline kernel and Linux distributions in the
next 24-48 months.", suffering souls everywhere :P

>
> Throughout the process, I've been very receptive to reviewers - including
> parties whose use cases are very different from my own, like zram users and
> the ghost-swapfile-style deployments that want zswap with no backing disk
> at all - and the current design reflects that:
>
> * Kairui asked that this be optional, not a replacement [12]. It is now:
>   vswap is opt-in behind the vswap= cmdline parameter, and with vswap=off
>   the existing swap paths aren't touched at all.
>
> * Kairui also pointed out that if I kept shrinking the metadata I'd "end up
>   reimplementing the swap table format" [12]. I took that feedback, and
>   reworked the entire series on top of swap table.
>
> * Per Chris' requests, I've figured out ways to minimize memory overhead
>   [13]. On a zswap-only setup there's almost 0 overhead now, because the
>   vtable is merged with zswap's xarray rather than sitting on top of it.
>   The indirection is paid for by the tree we delete.
>
> * Johannes pointed out that a physical cluster backing only vswap entries
>   still had to allocate a memcg_table, which was an overhead regression
>   against my older design on the zswap writeback case [14] (since a swap
>   entry occupies multiple levels). I've also fixed this.
>
> * I've spent a lot of time chasing regressions on setups we don't even run
>   at Meta - zram in particular [6]. And if anyone's still worried after all
>   that, vswap=off keeps the old swap code available.

Acknowledged, specifically reworking the whole series on top of swap
table, the memcg table fix, and chasing zram regressions you don't even
run were all real work on cases that don't benefit your fleet. Thank
you!

Roughly 80% of my own time also goes to things that don't run on my
fleet either. Desktop and Android (I do use them though), lowend and even
32bits. Upstream covers every usage case as the baseline and it's tiring,
I know what it costs. :)

> Numbers are in the v4 cover letter [10]. In terms of performance, vswap is
> on par or better than baseline on zswap across memhog, usemem and kernel
> build, and vswap=off tracks baseline within noise. I've even included an
> A/B experiment on a real production workload (Instagram/django): throughput
> on par, latency slightly better, and the swapfile footprint for every zswap
> user on the host goes away.
>
> Now, on xswap. Baoquan's working on a series [15] that covers some of the
> same ground, and the VM_SPARSE cluster_info idea in it is genuinely good.

My read is that xswap and vswap differ mostly in the interface, and that
both could sit on VM_SPARSE for cluster/metadata management.
Simpler and faster. There are some other issues though, see below.

> I've been reviewing that lineage since July [16] and I'd like whatever
> lands to end up with the best parts of both. From my perspective the
> differences are:
>
> 1. Userspace knobs. xswap asks the admin for a size (a percent of RAM) plus
>    a per-device limit to tune afterwards. I'm not aware of any use case
>    that needs those, and I don't think users have a good way to answer the
>    question anyway - sizing swap for compressed memory depends on memory
>    size, workload, and compression ratio all at once. That's precisely the
>    provisioning problem vswap exists to remove. The kernel should be as
>    transparent and dynamic as possible here, and not add knobs unless
>    there's a use case for them.

I asked Baoquan the same thing, see the "Why not make both args
optional" part:

https://lore.kernel.org/linux-mm/apaw_XmqnEcY8sv7@MiWiFi-R3L-srv/

Where I've ended up is that unbounded growth is a real concern. On a
host with no memcg limit (root cgroup, and most desktop and embedded
setups), an unlimited pool means usage can keep growing, with no
admin visible ceiling at all. I'm not attached to xswap's percent of RAM
knob specifically, but I do think some kind of bound makes sense.

> 2. Writeback support. Writeback is core functionality for zswap, not an
>    add-on, and a design needs to account for it from the start. This came
>    up before, in the discussion around Chris' ghost swapfile RFC [17]: for
>    a solution here to be acceptable, it has to work with the primary
>    usecase and support disk writeback. Without it, whatever zswap won't
>    take (incompressible pages especially) has nowhere to go, and cold
>    compressed data can never leave RAM.

Agreed completely, and I think this one is settled: whatever lands
has to support disk writeback in the end. Baoquan mentioned that
what he sent is just a foundation though. I know talking is cheap
but still, could it be combined with the writeback support from the
other side? Or am I missing anything here?

> 3. Cgroup charging behavior. vswap/xswap shouldn't be charged against the
>    swap usage counter. It's fundamentally a different resource from
>    physical swapfile space, and memory.swap.* should read 0 when nothing is
>    on disk [18]. I made the longer argument for this in [19].

I personally don't have a strong opinion on this. But I think this
needs an explicit decision from the group rather than being
inherited from whichever series merges first. Skipping the charge
when a device has a special flag isn't hard to do if the code
abstraction is done well, so this is more of a design decision
than a technical problem.

> 4. Data structure (xarray vs sparse vmalloc array). Even with xarray, vswap
>    is already on par with or beating baseline. I like the sparse array
>    idea, but why are we landing an optimization before the feature itself,
>    without any A/B data showing the difference matters?

This is where I'd push back a little. To be clear, my concern isn't the
size of the win, it's that VM_SPARSE isn't purely an optimization. It
changes the cluster/metadata layout and avoids RCU conventions,
could help avoid some future sleep while atomic limitations and
it's tangled with the initial size issue too. So bolting it on afterwards
could mean rewriting the parts of vswap that touch it and could be
costly some conflict in ideas. That's why I'd rather we agree on
it now, not that it has to land first.

Using VM_SPARSE does outsmart my earlier xarray suggestion :) There
is an interesting early benchmark from Baoquan, VM_SPARSE wasn't there
yet, but with it we can use clusters natively in the same way. It shows
about 20% latency cut and 10% throughput gain, which seems a major
improvement, and that's serial performance, not even mentioning
scalability yet:
https://lore.kernel.org/linux-mm/20260707073215.72183-1-baoquan.he@linux.dev/

And it matters more than most micro-optimizations: under heavy
pressure many workloads spend less than half of system time in
compression itself, the rest is mm/swap. Swap has had a terrible
reputation for good reasons, and even with zswap/zram users
have been scared of the performance.

After the recent refactor and optimization we already noticed many
users can now run much heavier workloads using compressed memory only
or fast storages, which is a real breakthrough. Every chunk of
overhead we remove there unblocks another class of user from using
compressed memory as a "secondary memory".

>
> One thing I do want to be clear about: I'm glad other people care about
> this problem. Chris' ghost swapfile and Baoquan's xswap are both going
> after the same set of problems, and that's a good sign. It means this is
> real and shared, not something only Meta runs into.
>
> What's been harder is the shape of the engagement. Alternatives keep
> getting posted and pushed that don't cover all the requirements, while this
> series sits without review. I don't think I'm owed anyone's interest in the
> problems I care about. But I do think working code, with benchmarks and
> production exposure behind it, deserves a fair hearing next to in-progress
> proposals.

I'm sorry it has felt that way, and I don't want to hand wave it. For
what it's worth, I did review it, and I did think it could be merged
with some adjustments:

https://lore.kernel.org/linux-mm/CAMgjq7D4XsAD4NGDL7FC2kaYAQAP8PDJdn4bpzGZwXYtjEpJ6w@mail.gmail.com/#t

We were also discussing the limitation of the global cluster cache in
that thread, so I paused my other work for a while and posted the swap
queue series first. That one is orthogonal on purpose, and helps both
swap tier and vswap, it reduces the overhead enough to cancel out
the layering cost due to missing caching for either:
https://lore.kernel.org/linux-mm/20260829-swap-pcp-priq-v2-resend-0-68d3d925578c@gmail.com/
(this is V2, V1 is earlier - thanks to Lian for keeping this going!)

Then xswap appeared with the VM_SPARSE idea and exploring it seems
necessary because getting the cluster foundation right affects
everything built above where it went. Not because the vswap series
wasn't worth reviewing.

> So what I'm asking for: I'd like us to converge rather than keep two series
> in flight. My preference is that we land vswap first, then build Baoquan's
> sparse array on top of it as an optimization. That gets the feature in, and
> by then we'd have the A/B data to show whether the sparse array actually
> beats the xarray.
>
> If you think that's the wrong order, I'd genuinely like to understand why -
> after 17 months and 10 revisions I still don't have a clear picture of the
> objection.

I fully agree we must converge. At the moment I don't think one can
can settle that alone as there are several of us reviewing both. I'll try
to be more concrete about what I think has to be agreed first.

The two interfaces aren't equivalent, xswap has its limitations, but a
couple of xswap's properties look genuinely better to me. Runtime
on/off rather than a boot parameter is better, more flexibility and
doesn't conflict with what vswap is trying to do. Some other of these
differences involve design decisions rather than just surface knobs,
and those are the parts we're stuck with once they ship.

So a few current interface difference of vswap and xswap I've noticed are:

 - default size: somehow also tangled with VM_SPARSE, which
needs an initial virtual value. It's super cheap if not zero cost
(the only cost is occupying some vmalloc area, which is fine
I guess?). But we still have to pick a number, could be large
enough to satisfy everyone?
 - boot vs runtime toggle: I believe a runtime toggle is better. It
can still be tuned on boot via cmdline, but it offers the extra
ability to turn it on without modifying a base VM boot image or
kernel.
 - growth / limits: as talked above.
 - charging semantics
 - there could be other longterm design decisions on top of these as
we talked in above.

Once these settle, I think the picture will be clean. Your writeback
logic and swap table integration are solid, while Baoquan's VM_SPARSE,
previous preparation with swap ops give us a structural foundation. A
clean long-term architecture means getting the core structure right,
let's try not treating this as a race between series.

Re: Path forward for Virtualized Swap?

[Nhat Pham]

On Sun, Sep 6, 2026 at 10:52 PM Kairui Song <ryncsn@gmail.com> wrote:

Hi Kairui,

Thank you for the kind words and encouragement :) It means a lot
coming from you.

My main concern so far has been - it seems like every time I satisfied
one of your guys' requirements, another pre-requisite is added to the
list. I mean the performance regression and memory overhead concerns
make a lot of sense, especially when vswap is non-optional. But now it
just seems like we're delaying real use case for the fabled perfect
swap system. It gets very exhausting trying to keep expanding the
scope of this.

>
> I asked Baoquan the same thing, see the "Why not make both args
> optional" part:
>
> https://lore.kernel.org/linux-mm/apaw_XmqnEcY8sv7@MiWiFi-R3L-srv/
>
> Where I've ended up is that unbounded growth is a real concern. On a
> host with no memcg limit (root cgroup, and most desktop and embedded
> setups), an unlimited pool means usage can keep growing, with no
> admin visible ceiling at all. I'm not attached to xswap's percent of RAM
> knob specifically, but I do think some kind of bound makes sense.

Technically speaking, virtual swap *is* bounded, i.e 8PB (xarray
induced a limit to the number of clusters it can allocate). But more
importantly, it's effectively capped by the backend it can take:
zswap, swap cache, disk swap, etc.

I'm not entirely certain this will be problem in practice...? Besides,
we can always add new knobs if needs arise (especially since the old
design still exists if unlimited swapping is somehow real), but if we
started out with some knobs that turns out to be unnecessary or
undesirable, removing them and/or educating users to be careful with
using them is harder, in my opinion and experience deploying zswap
across the fleet in the past 4 years.

>
> > 2. Writeback support. Writeback is core functionality for zswap, not an
> >    add-on, and a design needs to account for it from the start. This came
> >    up before, in the discussion around Chris' ghost swapfile RFC [17]: for
> >    a solution here to be acceptable, it has to work with the primary
> >    usecase and support disk writeback. Without it, whatever zswap won't
> >    take (incompressible pages especially) has nowhere to go, and cold
> >    compressed data can never leave RAM.
>
> Agreed completely, and I think this one is settled: whatever lands
> has to support disk writeback in the end. Baoquan mentioned that
> what he sent is just a foundation though. I know talking is cheap
> but still, could it be combined with the writeback support from the
> other side? Or am I missing anything here?

I'm just very nervous about any design that doesn't quite take that
into account and handwave it away into a follow-up :)

>
> > 3. Cgroup charging behavior. vswap/xswap shouldn't be charged against the
> >    swap usage counter. It's fundamentally a different resource from
> >    physical swapfile space, and memory.swap.* should read 0 when nothing is
> >    on disk [18]. I made the longer argument for this in [19].
>
> I personally don't have a strong opinion on this. But I think this
> needs an explicit decision from the group rather than being
> inherited from whichever series merges first. Skipping the charge
> when a device has a special flag isn't hard to do if the code
> abstraction is done well, so this is more of a design decision
> than a technical problem.

I'm fine with discussing about this. I just want to point out that
this is not something I snuck in :) This has been the behavior since
probably the first or second RFC of the first design though, and in
most of them, it is a front-and-centre patch of its own, not hushed
away in one big patch. Folks have had 1.5 years to comment on it (and
still not too late - nothing is merged yet!)

I'd love to hear opinions from other zswap users regarding this :)

>
> > 4. Data structure (xarray vs sparse vmalloc array). Even with xarray, vswap
> >    is already on par with or beating baseline. I like the sparse array
> >    idea, but why are we landing an optimization before the feature itself,
> >    without any A/B data showing the difference matters?
>
> This is where I'd push back a little. To be clear, my concern isn't the
> size of the win, it's that VM_SPARSE isn't purely an optimization. It
> changes the cluster/metadata layout and avoids RCU conventions,
> could help avoid some future sleep while atomic limitations and
> it's tangled with the initial size issue too. So bolting it on afterwards
> could mean rewriting the parts of vswap that touch it and could be
> costly some conflict in ideas. That's why I'd rather we agree on
> it now, not that it has to land first.

I don't think the current code differs significantly from the existing
model, no?

For the most part, RCU played quite nicely with the current swap code.
Practically, it's just:

1. If you have a reference to the swap entry, you can assume entry and
the cluster owning it is safe.

2. If you don't, then do not assume it's safe once you exit the rcu
read section.

What conflicts specifically do you have in mind here :)

>
> Using VM_SPARSE does outsmart my earlier xarray suggestion :) There
> is an interesting early benchmark from Baoquan, VM_SPARSE wasn't there
> yet, but with it we can use clusters natively in the same way. It shows
> about 20% latency cut and 10% throughput gain, which seems a major
> improvement, and that's serial performance, not even mentioning
> scalability yet:
> https://lore.kernel.org/linux-mm/20260707073215.72183-1-baoquan.he@linux.dev/

I think a lot of these wins come from the fact that the zswap's xarray
specifically is *really* bad in certain paths. For instance, in the
free path (swap_range_free() -> zswap_invalidate()), there is no
batching behavior, so we walk the tree from scratch for each entry
rather than using an xas iterator for the entire batch. That goes away
in the current vswap model, because the xarray manages cluster rather
than entries, so you just do one tree walk + one flat array walk.

You can see some performance numbers in my cover letter - it's
promising on both micro benchmarks, and a production workload that I
throw vswap on :) I've also done an ablative study, where I still let
zswap and swapfile coupled, but only remove the xarray and replace it
with the flat array in swapfile's cluster struct. I observe the same
win, basically (especially on the usemem benchmark).

>
> And it matters more than most micro-optimizations: under heavy
> pressure many workloads spend less than half of system time in
> compression itself, the rest is mm/swap. Swap has had a terrible
> reputation for good reasons, and even with zswap/zram users
> have been scared of the performance.
>
> After the recent refactor and optimization we already noticed many
> users can now run much heavier workloads using compressed memory only
> or fast storages, which is a real breakthrough. Every chunk of
> overhead we remove there unblocks another class of user from using
> compressed memory as a "secondary memory".

I don't necessarily disagree with any of the points you made here.
FWIW, I like the vmalloc array idea too. I have actually put in work
to review Baoquan's patch series, because I'm hopeful that it will be
superior in the end (we're using this infra too so it helps us too :))

I just want to re-iterate that:

1. Vswap is already decently performant.

2. We're not removing the old code, so no one will regress with vswap
(they can just disable it for now). Optimization can be done on top of
it after we land the use cases (which also unblock zswap usage for
many parties who are capped by the storage limitations).

>
> >
> > One thing I do want to be clear about: I'm glad other people care about
> > this problem. Chris' ghost swapfile and Baoquan's xswap are both going
> > after the same set of problems, and that's a good sign. It means this is
> > real and shared, not something only Meta runs into.
> >
> > What's been harder is the shape of the engagement. Alternatives keep
> > getting posted and pushed that don't cover all the requirements, while this
> > series sits without review. I don't think I'm owed anyone's interest in the
> > problems I care about. But I do think working code, with benchmarks and
> > production exposure behind it, deserves a fair hearing next to in-progress
> > proposals.
>
> I'm sorry it has felt that way, and I don't want to hand wave it. For
> what it's worth, I did review it, and I did think it could be merged
> with some adjustments:
>
> https://lore.kernel.org/linux-mm/CAMgjq7D4XsAD4NGDL7FC2kaYAQAP8PDJdn4bpzGZwXYtjEpJ6w@mail.gmail.com/#t
>
> We were also discussing the limitation of the global cluster cache in
> that thread, so I paused my other work for a while and posted the swap
> queue series first. That one is orthogonal on purpose, and helps both
> swap tier and vswap, it reduces the overhead enough to cancel out
> the layering cost due to missing caching for either:
> https://lore.kernel.org/linux-mm/20260829-swap-pcp-priq-v2-resend-0-68d3d925578c@gmail.com/
> (this is V2, V1 is earlier - thanks to Lian for keeping this going!)

Ah that! I sort of tabled that as a nice-idea-but-still-optimization :P

I'll spend some time thinking about the design and reviewing it. In
the long run, it will get things more simple, but another thing that
should not be a blocker, correct?

>
> Then xswap appeared with the VM_SPARSE idea and exploring it seems
> necessary because getting the cluster foundation right affects
> everything built above where it went. Not because the vswap series
> wasn't worth reviewing.
>
> > So what I'm asking for: I'd like us to converge rather than keep two series
> > in flight. My preference is that we land vswap first, then build Baoquan's
> > sparse array on top of it as an optimization. That gets the feature in, and
> > by then we'd have the A/B data to show whether the sparse array actually
> > beats the xarray.
> >
> > If you think that's the wrong order, I'd genuinely like to understand why -
> > after 17 months and 10 revisions I still don't have a clear picture of the
> > objection.
>
> I fully agree we must converge. At the moment I don't think one can
> can settle that alone as there are several of us reviewing both. I'll try
> to be more concrete about what I think has to be agreed first.
>
> The two interfaces aren't equivalent, xswap has its limitations, but a
> couple of xswap's properties look genuinely better to me. Runtime
> on/off rather than a boot parameter is better, more flexibility and

I would also like to point out - I *do* have a runtime parameter in an
older version of vswap (v3). There is also not a lot of engagement
from folks in that version - no one expressed any interest in that
runtime knob in particular. Only Johannes made the case that a boot
parameter is superior, so I switched over. I think his reasoning makes
sense at the time FWIW, and it also removes the need for CONFIG_VSWAP
(which in turns remove a bunch of useless stubs - saving 100 LoCs), so
I figure folks will prefer less code and less userspace decisions :)

v3: https://lore.kernel.org/all/20260806184254.3790858-1-nphamcs@gmail.com/

Regardless - do you have a specific use case in mind where a runtime
knob would be useful?

> doesn't conflict with what vswap is trying to do. Some other of these
> differences involve design decisions rather than just surface knobs,
> and those are the parts we're stuck with once they ship.
>
> So a few current interface difference of vswap and xswap I've noticed are:
>
>  - default size: somehow also tangled with VM_SPARSE, which
> needs an initial virtual value. It's super cheap if not zero cost
> (the only cost is occupying some vmalloc area, which is fine
> I guess?). But we still have to pick a number, could be large
> enough to satisfy everyone?

8PB? ;)

I think choosing a meaningful number that covers from O(dozen of GB)
to O(terrabytes) scale machine, with different workload shapes, access
patterns, memory compressibility, etc. is an unsolvable problem. If
it's cheap, why don't we just max out?

>  - boot vs runtime toggle: I believe a runtime toggle is better. It
> can still be tuned on boot via cmdline, but it offers the extra
> ability to turn it on without modifying a base VM boot image or
> kernel.
>  - growth / limits: as talked above.
>  - charging semantics
>  - there could be other longterm design decisions on top of these as
> we talked in above.
>
> Once these settle, I think the picture will be clean. Your writeback
> logic and swap table integration are solid, while Baoquan's VM_SPARSE,
> previous preparation with swap ops give us a structural foundation. A
> clean long-term architecture means getting the core structure right,
> let's try not treating this as a race between series.

I've reached out to Baoquan as well, but I figured there's value in
both discussions in big and small groups, and especially since there's
a chance some of us might not be coming to LPC. Let's see what we come
up with.

Re: Path forward for Virtualized Swap?

[Johannes Weiner]

On Mon, Sep 07, 2026 at 01:51:31PM +0800, Kairui Song wrote:
> Where I've ended up is that unbounded growth is a real concern. On a
> host with no memcg limit (root cgroup, and most desktop and embedded
> setups), an unlimited pool means usage can keep growing, with no
> admin visible ceiling at all. I'm not attached to xswap's percent of RAM
> knob specifically, but I do think some kind of bound makes sense.

Swap space is just process virtual address space, no?

Swap entries already have one or more page table entries pointing to
them, which in turn are managed by trees of vm_area_structs. That
means rlimits apply, overcommit protection applies, and OOM killer
attribution works as well (oom_badness()).

Shmem has its own defaults and limit interface on address space.

I'm not quite seeing how the swap space needs an additional limit. How
could users break things in unique new ways without it?

It would be good to spell out that vector before discussing numbers :)

Re: Path forward for Virtualized Swap?

[Nhat Pham]

On Tue, Sep 8, 2026 at 11:30 AM Johannes Weiner <hannes@cmpxchg.org> wrote:
>
> On Mon, Sep 07, 2026 at 01:51:31PM +0800, Kairui Song wrote:
> > Where I've ended up is that unbounded growth is a real concern. On a
> > host with no memcg limit (root cgroup, and most desktop and embedded
> > setups), an unlimited pool means usage can keep growing, with no
> > admin visible ceiling at all. I'm not attached to xswap's percent of RAM
> > knob specifically, but I do think some kind of bound makes sense.
>
> Swap space is just process virtual address space, no?
>
> Swap entries already have one or more page table entries pointing to
> them, which in turn are managed by trees of vm_area_structs. That
> means rlimits apply, overcommit protection applies, and OOM killer
> attribution works as well (oom_badness()).

I tested this theory. I spinned up a process, and let it spam 0-filled
memory + swap these pages out continually.

As you predicted, oom-killer picked it up eventually. The host was
(and is) intact otherwise :)

Re: Path forward for Virtualized Swap?

[Nhat Pham]

On Wed, Sep 9, 2026 at 9:41 AM Nhat Pham <nphamcs@gmail.com> wrote:
>
> On Tue, Sep 8, 2026 at 11:30 AM Johannes Weiner <hannes@cmpxchg.org> wrote:
> >
> > On Mon, Sep 07, 2026 at 01:51:31PM +0800, Kairui Song wrote:
> > > Where I've ended up is that unbounded growth is a real concern. On a
> > > host with no memcg limit (root cgroup, and most desktop and embedded
> > > setups), an unlimited pool means usage can keep growing, with no
> > > admin visible ceiling at all. I'm not attached to xswap's percent of RAM
> > > knob specifically, but I do think some kind of bound makes sense.
> >
> > Swap space is just process virtual address space, no?
> >
> > Swap entries already have one or more page table entries pointing to
> > them, which in turn are managed by trees of vm_area_structs. That
> > means rlimits apply, overcommit protection applies, and OOM killer
> > attribution works as well (oom_badness()).
>
> I tested this theory. I spinned up a process, and let it spam 0-filled
> memory + swap these pages out continually.
>
> As you predicted, oom-killer picked it up eventually. The host was
> (and is) intact otherwise :)

Oh digging through dmesg - I did find something quite interesting:

[12462.749663] ------------[ cut here ]------------
[12462.749670] refcount_t: saturated; leaking memory.
[12462.749817] ------------[ cut here ]------------
[12462.759267] WARNING: lib/refcount.c:19 at
refcount_warn_saturate+0x1b/0x70, CPU#17: zeroleak/76075
[12462.759276] refcount_t: underflow; use-after-free.
[12462.777208] Modules linked in: sch_fq(E) tcp_diag(E) inet_diag(E)
act_gact(E) cls_bpf(E)
[12462.786790] WARNING: lib/refcount.c:28 at
refcount_warn_saturate+0x42/0x70, CPU#25: dmesg/76074

Seems like it managed to oversaturate mem_cgroup_private_id:

struct mem_cgroup_private_id {
    int id;
    refcount_t ref;
};

We should either fix mem_cgroup_private_id to use 64-bit refererence
count here, or hardcap the swap usage to below the saturation point of
refcount_t ref. I'm leaning towards the former in the long run.

This is sort of an argument for handling all of this transparently
though - expecting users to know what the cap should be based on a
myriad of architectural and arcane implementation details (which can
change arbitrarily in the future) is very unreasonable.

Re: Path forward for Virtualized Swap?

[David Hildenbrand (Arm)]

On 9/4/26 23:14, Nhat Pham wrote:

Hi,

> To Kairui, Johannes, Yosry, Chris, David, Andrew, and other (z)swap folks -
> I need some help here.

I'll mostly ignore swap-related stuff because (a) -EBUSY and (b) we have quite
some swap experts around :)

Reading below, I do wonder if the swap subsystem needs some more coordination
and collaboration?

For example, maybe having bi-weekly/monthly meetings to get everybody aligned
and uptodate could be something reasonable?

This approach has proven fairly helpful in areas where (1) multiple companies
are involved (2) many concurrent development is happening, and (3) people have
different priorities and requirements.

Is there already something like that to organize development? Should we get
started with that?

-- 
Cheers,

David

Re: Path forward for Virtualized Swap?

[Nhat Pham]

On Mon, Sep 7, 2026 at 4:30 AM David Hildenbrand (Arm) <david@kernel.org> wrote:
>
> On 9/4/26 23:14, Nhat Pham wrote:
>
> Hi,
>
> > To Kairui, Johannes, Yosry, Chris, David, Andrew, and other (z)swap folks -
> > I need some help here.
>
> I'll mostly ignore swap-related stuff because (a) -EBUSY and (b) we have quite
> some swap experts around :)
>
> Reading below, I do wonder if the swap subsystem needs some more coordination
> and collaboration?
>
> For example, maybe having bi-weekly/monthly meetings to get everybody aligned
> and uptodate could be something reasonable?
>
> This approach has proven fairly helpful in areas where (1) multiple companies
> are involved (2) many concurrent development is happening, and (3) people have
> different priorities and requirements.
>
> Is there already something like that to organize development? Should we get
> started with that?

Hi David!

Thank you for your suggestions :) More collaboration and communication
is definitely needed here. Conference is a good place for this, but
it's not something we can do very frequently, and many folks could not
attend (visa issues, etc.).

A bi-weekly/monthly series of meetings sound very appealing. Swap
contributors are very spread out (China, NorAm, Europe, etc.), so
coordination will be tricky, but let us figure out something.

Re: Path forward for Virtualized Swap?

[David Hildenbrand (Arm)]

On 9/8/26 18:45, Nhat Pham wrote:
> On Mon, Sep 7, 2026 at 4:30 AM David Hildenbrand (Arm) <david@kernel.org> wrote:
>>
>> On 9/4/26 23:14, Nhat Pham wrote:
>>
>> Hi,
>>
>>> To Kairui, Johannes, Yosry, Chris, David, Andrew, and other (z)swap folks -
>>> I need some help here.
>>
>> I'll mostly ignore swap-related stuff because (a) -EBUSY and (b) we have quite
>> some swap experts around :)
>>
>> Reading below, I do wonder if the swap subsystem needs some more coordination
>> and collaboration?
>>
>> For example, maybe having bi-weekly/monthly meetings to get everybody aligned
>> and uptodate could be something reasonable?
>>
>> This approach has proven fairly helpful in areas where (1) multiple companies
>> are involved (2) many concurrent development is happening, and (3) people have
>> different priorities and requirements.
>>
>> Is there already something like that to organize development? Should we get
>> started with that?
> 
> Hi David!

Hi!

> 
> Thank you for your suggestions :) More collaboration and communication
> is definitely needed here. Conference is a good place for this, but
> it's not something we can do very frequently, and many folks could not
> attend (visa issues, etc.).

Right, and conferences usually don't have the capacity to squeeze too many
swap-related topics in, so it would have to be side discussions or BOFs.

> 
> A bi-weekly/monthly series of meetings sound very appealing. Swap
> contributors are very spread out (China, NorAm, Europe, etc.), so
> coordination will be tricky, but let us figure out something.

Yes, that's an issue. I'd suggest swap maintainers take care of that if they
agree. I could try as well, but it's rather tricky right now on my end ...

As an alternative, we could also use on/two/ ... slots in the bi-weekly MM
alignment session to talk about swap. (I'd be happy to learn more about all this
as well)

-- 
Cheers,

David

Re: Path forward for Virtualized Swap?

[Baoquan He]

Hi Nhat,

On 09/04/26 at 02:14pm, Nhat Pham wrote:
.....snip...
> Now, on xswap. Baoquan's working on a series [15] that covers some of the
> same ground, and the VM_SPARSE cluster_info idea in it is genuinely good.
> I've been reviewing that lineage since July [16] and I'd like whatever
> lands to end up with the best parts of both. From my perspective the
> differences are:
> 
> 1. Userspace knobs. xswap asks the admin for a size (a percent of RAM) plus
>    a per-device limit to tune afterwards. I'm not aware of any use case
>    that needs those, and I don't think users have a good way to answer the
>    question anyway - sizing swap for compressed memory depends on memory
>    size, workload, and compression ratio all at once. That's precisely the
>    provisioning problem vswap exists to remove. The kernel should be as
>    transparent and dynamic as possible here, and not add knobs unless
>    there's a use case for them.
> 
> 2. Writeback support. Writeback is core functionality for zswap, not an
>    add-on, and a design needs to account for it from the start. This came
>    up before, in the discussion around Chris' ghost swapfile RFC [17]: for
>    a solution here to be acceptable, it has to work with the primary
>    usecase and support disk writeback. Without it, whatever zswap won't
>    take (incompressible pages especially) has nowhere to go, and cold
>    compressed data can never leave RAM.
> 
> 3. Cgroup charging behavior. vswap/xswap shouldn't be charged against the
>    swap usage counter. It's fundamentally a different resource from
>    physical swapfile space, and memory.swap.* should read 0 when nothing is
>    on disk [18]. I made the longer argument for this in [19].
> 
> 4. Data structure (xarray vs sparse vmalloc array). Even with xarray, vswap
>    is already on par with or beating baseline. I like the sparse array
>    idea, but why are we landing an optimization before the feature itself,
>    without any A/B data showing the difference matters?


Thanks for laying this out, and for the honest push to converge. Let me
be equally direct about the ordering: I think the xswap base should land
first, and the things vswap demonstrates -  writeback, rmap lookup, the
charging semantics, later THP -- should be built on top of it. Because
it is the foundation that keeps the swap core simpler, and the first thing
to merge should be the one that doesn't have to be redone.

The VM_SPARSE array is not an optimization to bolt on later; it is a
structural choice, and the code reflects it. In vswap, the cluster
metadata lives in a dynamically-allocated xarray.

struct swap_cluster_info_dynamic {
        struct swap_cluster_info ci;
        unsigned int index;             /* for cluster_index() */
        struct rcu_head rcu;
        atomic_long_t *virtual_table;   /* Backing pointers for vswap slots */
};

To support dynamic growth and shrink, vswap stores its cluster metadata
in an xarray, and that forces two things the plain swap_cluster_info[]
array never needed:

1. Every cluster has to carry an extra index and an rcu_head —
24 bytes per cluster — purely so the xarray can locate it and free
it safely.
2. To keep that bookkeeping from leaking into the normal-swap code, the
cluster had to be wrapped in a container, swap_cluster_info_dynamic,
so the xarray holds a pointer to the wrapper instead of an inline
array element.

So in vswap, every cluster access in the shared hot path has to answer
"is this a vswap device?" and take a separate branch:

- swap_is_vswap() is checked in 36 places across page_io.c, swapfile.c,
  zswap.c and swap.h;
- __swap_offset_to_cluster() branches into xa_load() for vswap vs the
  flat array otherwise, and the xarray path can return NULL (a cluster
  can be torn down);
- __swap_cluster_lock() branches into __vswap_cluster_lock(), which
  wraps every access in rcu_read_lock() and a CLUSTER_FLAG_DEAD check,
  plus kfree_rcu()/container_of()/rcu_head plumbing for node lifetime.

With VM_SPARSE, xswap's cluster access is exactly the plain-array line the
rest of swap already uses:

    return &si->cluster_info[offset / SWAPFILE_CLUSTER];

no branch, no RCU discipline, no tear-down state machine, and no NULL
return. So VM_SPARSE doesn't add complexity to close a gap; it lets the
cluster layer stay as simple as it already is, which is precisely the
part later work (writeback, rmap lookup, memcg charging, THP) has to sit
on.

I'm not going to claim xswap wins on throughput. I measured it:
on a 64G/64-thread swapout, xswap, vswap and plain swap+zswap are all
within ~2-3% of each other, effectively identical. Because the cost is
dominated by zswap compression, not the cluster table. So the ordering
question is not "which is faster" but "which structure should the use-case
layer be built on". If vswap is chosen, the xarray-based table is an
intermediate form. Your own ablative study already showed the flat array
wins, and VM_SPARSE is exactly that flat array plus lazy mapping.
 
On the metadata side I want to be precise, because it is easy to overstate:
the per-cluster cost that xswap saves is the xarray-induced index + rcu_head
(20 bytes/cluster or 24bytes for alignment), small. 

On writeback: agreed it is required in the end. But it is a consumer of
the foundation, not a reason to pick a different one. xswap is deliberately
the base; 
- writeback
- rmap lookup
- THP support
- memcg accounting
All these can land on top of the xswap base rather than be stranded on a table
we later replace. As we have discussed and I have been mentioning in each
cover-letter, I didn't touch these core changes, glad to see your work built
on top of it.

On the interface, xswap v2 drops the percent knob entirely (your point about
"why not just max out" is taken): create now takes only an optional
priority, a device starts at full RAM, which is free because the VM_SPARSE
area is mapped lazily, with an optional per-device size limit for admins
who want a ceiling. More importantly, xswap keeps per-device instances
because the swap->ops and swap tiering that come next need per-device
operations. A single boot-time vswap can't express that, and it breaks the
per-device conventions the rest of swap already follows. When I tested it,
there is no way to disable it at runtime, so it can't even be A/B-tested
against regular swap in the same boot. So I think a boot-time vswap is an
independent issue which deserves a separate patch posting with a convincing
justification later.

So concretely: xswap base first (runtime file-less device + VM_SPARSE
cluster foundation + sysfs create/destroy), then the use-case layer, where
your writeback work, etc is very welcome. The base should be the one that
doesn't need to be redone; by both our measurements, that is the flat-cluster
substrate.

Thanks
Baoquan

> 
> One thing I do want to be clear about: I'm glad other people care about
> this problem. Chris' ghost swapfile and Baoquan's xswap are both going
> after the same set of problems, and that's a good sign. It means this is
> real and shared, not something only Meta runs into.
> 
> What's been harder is the shape of the engagement. Alternatives keep
> getting posted and pushed that don't cover all the requirements, while this
> series sits without review. I don't think I'm owed anyone's interest in the
> problems I care about. But I do think working code, with benchmarks and
> production exposure behind it, deserves a fair hearing next to in-progress
> proposals.
> 
> So what I'm asking for: I'd like us to converge rather than keep two series
> in flight. My preference is that we land vswap first, then build Baoquan's
> sparse array on top of it as an optimization. That gets the feature in, and
> by then we'd have the A/B data to show whether the sparse array actually
> beats the xarray.
> 
> If you think that's the wrong order, I'd genuinely like to understand why -
> after 17 months and 10 revisions I still don't have a clear picture of the
> objection.
> 
> [1] https://lore.kernel.org/all/20250407234223.1059191-1-nphamcs@gmail.com/
> [2] https://lore.kernel.org/all/20250429233848.3093350-1-nphamcs@gmail.com/
> [3] https://lore.kernel.org/all/20260208215839.87595-1-nphamcs@gmail.com/
> [4] https://lore.kernel.org/all/20260318222953.441758-1-nphamcs@gmail.com/
> [5] https://lore.kernel.org/all/20260320192735.748051-1-nphamcs@gmail.com/
> [6] https://lore.kernel.org/all/20260505153854.1612033-1-nphamcs@gmail.com/
> [7] https://lore.kernel.org/all/20260528212955.1912856-1-nphamcs@gmail.com/
> [8] https://lore.kernel.org/all/20260612193738.2183968-1-nphamcs@gmail.com/
> [9] https://lore.kernel.org/all/20260806184254.3790858-1-nphamcs@gmail.com/
> [10] https://lore.kernel.org/all/20260825153238.2695446-1-nphamcs@gmail.com/
> [11] https://lwn.net/Articles/1016136/
> [12] https://lore.kernel.org/all/CAMgjq7AQNGK-a=AOgvn4-V+zGO21QMbMTVbrYSW_R2oDSLoC+A@mail.gmail.com/
> [13] https://lore.kernel.org/all/CACePvbVXQWgcPD-bgK7iDba4NFLo2tT89ZbLOa03maJU4er4ag@mail.gmail.com/
> [14] https://lore.kernel.org/all/aZyFxKGXc8J6PIij@cmpxchg.org/
> [15] https://lore.kernel.org/all/20260827094509.1016740-1-hebaoquan@kylinos.cn/
> [16] https://lore.kernel.org/lkml/CAKEwX=Pe+qMZd2xhnU-PAGQtgXkp56c-JwYCbt2Lux9htgB67Q@mail.gmail.com/
> [17] https://lore.kernel.org/all/20251121114011.GA71307@cmpxchg.org/
> [18] https://lore.kernel.org/all/anYIboHEUZb4fhHv@cmpxchg.org/
> [19] https://lore.kernel.org/all/CAKEwX=P4syV38jAVCWq198r2OHXXc=xA-fx1dk6+qYef6yzxWQ@mail.gmail.com/