* Re: [PATCH RFC] memcg: add per-cgroup dirty page controls (dirty_ratio, dirty_min)
From: Jan Kara @ 2026-05-06 14:21 UTC (permalink / raw)
To: haghdoost
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, Andrew Morton, Matthew Wilcox (Oracle), Jan Kara,
cgroups, linux-mm, linux-kernel, linux-fsdevel, Kshitij Doshi
In-Reply-To: <20260501-rfc-memcg-dirty-v1-v1-1-9a8c80036ec1@uber.com>
On Fri 01-05-26 22:28:38, Alireza Haghdoost via B4 Relay wrote:
> From: Alireza Haghdoost <haghdoost@uber.com>
>
> Add two cgroup v2 memory-controller knobs that bring
> balance_dirty_pages() throttling into per-cgroup scope so one noisy
> writer cannot stall peers sharing the same host:
>
> memory.dirty_ratio Per-cgroup dirty-page ceiling, 0..100 percent of
> the cgroup's dirtyable memory. 0 (default) leaves
> the cgroup subject to the global threshold only.
>
> memory.dirty_min Guaranteed dirty-page floor, byte value (default 0).
>
> The two knobs compose: dirty_ratio bounds how much dirty memory a
> cgroup may accrue, dirty_min guarantees a floor below which it is
> never throttled.
>
> Motivation, design trade-offs, cost analysis, validation data, and
> open questions are in the cover letter.
>
> Co-developed-by: Kshitij Doshi <kshitijd@uber.com>
> Signed-off-by: Kshitij Doshi <kshitijd@uber.com>
> Signed-off-by: Alireza Haghdoost <haghdoost@uber.com>
> Assisted-by: Cursor:claude-sonnet-4.5
> ---
Things like motivation actually belong to the changelog itself, measured
results how the patch helps as well. On the other hand stuff like history
is largely irrelevant here, frankly I don't have a bandwidth to carefully
read the huge amount of text LLM has generated below so please try to make
it more concise for next time.
> This RFC adds two cgroup v2 memory-controller knobs that give operators
> per-cgroup control over dirty-page throttling in balance_dirty_pages():
> memory.dirty_ratio (per-cgroup ceiling) and memory.dirty_min (guaranteed
> floor). A third knob, memory.dirty_weight, is forthcoming in a follow-up
> once we have validated the application site (see "Follow-ups" below).
> We are posting this as an RFC, as a single squashed patch, to get design
> feedback before splitting the prototype into a per-logical-change series.
>
> Motivation
> ==========
>
> balance_dirty_pages() (BDP) is a global throttle. It sleeps writers
> once the host-wide dirty count crosses a single threshold. On a
> container host that threshold is shared across cgroups. A cgroup
> that dirties pages faster than storage can drain them pushes the
> count over the limit. Every writer on the host then parks in
> io_schedule_timeout() -- including cgroups that have not dirtied a
> single page of their own.
>
> cgroup v2 already has per-memcg dirty accounting, but that accounting
> does not translate into per-memcg dirty throttling.
Not quite true. We do have per-memcg writeback workers and we do have
per-memcg dirty limits (inferred from global dirty limit tunings) that are
enforced...
> We see this in production: a buffered write-heavy container generates
> multi-second stalls for co-located latency-sensitive workloads.
> Moreover, dirty-page accumulation from a single noisy neighbor is a
> recurring contributor to mount-responsiveness degradation on shared hosts.
... and I quite don't see how a multisecond stalls you are describing would
happen. There is something I must be missing. The throttling works as
follows: Until we cross global freerun limit (that is (background_limit +
dirty_limit) / 2) nobody is throttled. Once we cross it, memcg dirty limits
start to be taken into account. If we are below freerun in the memcg, the
task dirtying folios from that memcg shouldn't be throttled at all, once we
get above freerun we throttle by maximum of throttling delay decided from
global and memcg situation so then long delays can start happening but it
would mean the "innocent" task's memcg had to get at least over the freerun
limit.
So can you perhaps share more details about the configuration where you
observe these delays to innocent tasks due to another task dirtying a lot of
memory? How many page cache in total and dirty pages are there in each
memcg (both for the aggressive dirtier and wrongly delayed task)? Is the
delayed task really throttled in balance_dirty_pages()?
Honza
> Prior work
> ==========
>
> Per-memcg dirty-page limits have been proposed before. Andrea Righi
> posted an initial RFC in February 2010 [1]; Greg Thelen continued the
> work through v9 in August 2011 [2]. That series added per-memcg dirty
> counters and hooked them into balance_dirty_pages(), but it bolted
> per-cgroup limits onto the global writeback path without making
> writeback itself cgroup-aware. Without cgroup-aware flusher threads,
> a cgroup exceeding its limit triggered writeback of inodes from any
> cgroup, giving poor isolation. The series was not merged.
>
> Konstantin Khlebnikov posted "[PATCHSET RFC 0/6] memcg: inode-based
> dirty-set controller" in January 2015 [4], which proposed
> memory.dirty_ratio (the same interface name this series uses) via an
> inode-tagged, filtered-writeback approach. Tejun Heo reviewed it
> and rejected it as a "dead end" that duplicated lower-layer policy
> without solving the underlying isolation problem; this rejection
> directly motivated Tejun's native cgwb rework described below.
>
> Tejun Heo's 48-patch cgroup-writeback rework, merged in Linux 4.2
> (commit e4bc13adfd01, "Merge branch 'for-4.2/writeback'"), took the
> different approach of restructuring writeback to be natively
> cgroup-aware: per-memcg wb_domain (commit 841710aa6e4a), per-memcg
> NR_FILE_DIRTY / NR_WRITEBACK accounting, and cgroup-aware flusher
> threads [3]. That work deliberately deferred user-facing policy knobs.
> This series adds the policy surface that consumes Tejun's
> infrastructure. The dirty_min reservation concept is, to our
> knowledge, new.
>
> A November 2023 LKML thread by Chengming Zhou [5] independently
> identified the identical throttling regression on cgroup v2 (a 5 GB
> container constantly throttled because memory.max * dirty_ratio yields
> too small a threshold for bursty workloads). Jan Kara participated and
> endorsed a bytes-based per-memcg dirty limit; no patches followed that
> discussion, confirming the gap this series fills.
>
> [1] https://lwn.net/Articles/408349/
> [2] https://lore.kernel.org/lkml/1313597705-6093-1-git-send-email-gthelen@google.com/
> [3] https://lwn.net/Articles/648292/
> [4] https://lore.kernel.org/all/20150115180242.10450.92.stgit@buzz/
> [5] https://lore.kernel.org/all/109029e0-1772-4102-a2a8-ab9076462454@linux.dev/
>
> Proposed interface
> ==================
>
> Two new cgroup v2 files under the memory controller, absent on the root
> cgroup (CFTYPE_NOT_ON_ROOT):
>
> memory.dirty_ratio
> Integer [0, 100]. Per-cgroup dirty-page ceiling as a percentage of
> the cgroup's dirtyable memory (mdtc->avail: file cache plus
> reclaimable slack), the same base the global vm_dirty_ratio scales
> against. 0 (the default) disables the per-cgroup ceiling and leaves
> the cgroup subject to the global threshold only. A non-zero value
> that is stricter than vm_dirty_ratio overrides the global ratio for
> this cgroup via min(mdtc->thresh, cg_thresh); because both sides
> scale off the same base, the knob can never widen the cgroup past
> the global ceiling. A memory.dirty_bytes companion for byte-precise
> caps (mirroring vm_dirty_bytes) is noted under "Follow-ups" below.
> The prototype reads the value for the immediate memcg only;
> hierarchical enforcement (clamping against ancestors, like
> memory.max) is not implemented yet. We would like guidance on
> whether this is required for v1 or can follow in a subsequent
> series.
>
> memory.dirty_min
> Byte value (K/M/G suffixes accepted), default 0 (no reservation).
> Guaranteed dirty-page floor: while cgroup_dirty < dirty_min,
> throttling is bypassed (goto free_running). Lets a latency-sensitive
> cgroup buffer a small write burst even under global dirty pressure.
>
> dirty_min is an admission guarantee, so we have to prevent it from
> breaking the global dirty invariant. Two aspects:
>
> - Global cap. The sum of dirty_min reservations across all cgroups
> must not exceed a fraction of the global dirty threshold (our
> working number is 80%), so the system always retains some shared
> capacity. The prototype does not enforce this cap yet; we expect
> to either reject at write() time or clamp on read in a cheap
> precomputed effective_dirty_min. We would appreciate feedback on
> which approach the cgroup maintainers prefer.
>
> - Per-cgroup cap. A cgroup should not be able to reserve more dirty
> capacity than it can hold. Our tentative rule is
> effective_dirty_min = min(dirty_min, memory.max - memory.min),
> evaluated at enforcement time so it tracks live memory.max changes,
> rather than rejected at write. This is similar to how memory.low
> composes with memory.max.
>
> Neither cap is implemented in the prototype; both would land before
> a non-RFC posting.
>
> The two knobs compose: dirty_ratio bounds how much dirty memory a
> cgroup may accrue, dirty_min guarantees a floor below which it is never
> throttled.
>
> Test setup and results
> ======================
>
> To show the problem and the fix, we built a single reproducer that runs
> on an unmodified stock kernel and then on the patched kernel, using the
> same setup for both.
>
> Setup: QEMU guest with a virtio-blk disk throttled to 256 KB/s
> (bps_wr=262144). Two sibling cgroups, no io.weight; both share disk
> bandwidth equally. dirty_bytes=32MB, dirty_background_bytes=16MB
> (freerun = 24 MB). Files pre-allocated with fallocate before dirty
> pressure. Two phases per run: (1) victim alone (baseline), (2) noisy
> fills global dirty to the 32 MB cap, then victim runs contended for
> 30 s.
>
> - noisy: single fio job, unlimited write rate, fills global dirty pool.
> - victim: single fio job, rate-limited to 500 KB/s (128 IOPS target),
> 4 KiB sequential write().
>
> The high freerun (24 MB) ensures victim's solo dirty accumulation
> (244 KB/s x 30 s = 7.3 MB) stays below the threshold. BDP does not fire
> during the solo phase on either kernel, giving identical baselines.
>
> Stock kernel results (the problem):
>
> solo (no contention) contended inflation
> victim IOPS 125 5.1 24.5x worse
> victim p99 0.6 ms 152 ms 253x worse
>
> The contended p99 sits at fio's percentile bucket nearest MAX_PAUSE =
> HZ/5 = 200 ms (mm/page-writeback.c:49), the hard kernel ceiling on BDP
> sleep. The victim has near-zero dirty pages of its own but is forced to
> sleep because balance_dirty_pages() sees gdtc->dirty = NR_FILE_DIRTY +
> NR_WRITEBACK above the freerun threshold. Most of noisy's pages are
> queued in NR_WRITEBACK waiting for the throttled disk.
> memory.events.local on both cgroups shows max 0 throughout the run;
> this is not memory pressure inside either cgroup.
>
> Patched kernel results (the fix), with victim/memory.dirty_min = 16 MB:
>
> solo (no contention) contended inflation
> victim IOPS 125 125 1.0x
> victim p99 0.9 ms 0.7 ms 1.0x
>
> The patched kernel checks cgroup_dirty < dirty_min (4096 pages) before
> computing any sleep. Because the rate-limited victim's resident dirty
> set stays well below the reservation, the check fires on every write()
> -> goto free_running -> write() returns in ~7 us. The victim is fully
> protected.
>
> The figures above are the per-kernel medians of N=5 iterations and
> reflect a deterministic outcome on every iter: stock had cont_iops in
> 4.4..6.1 (retention 0.035..0.049) on all five iters, patched had
> cont_iops = 125.0 (retention = 1.000) on all five iters. The 5/5 stock
> iters all hit BDP's throttled regime; the 5/5 patched iters all
> bypassed it via the dirty_min check.
>
> Implementation
> ==============
>
> The patch touches five files:
>
> - include/linux/memcontrol.h: two new fields on struct mem_cgroup
> (dirty_ratio, dirty_min).
> - include/linux/writeback.h: a per-pass cg_dirty_cap field on
> struct dirty_throttle_control used to publish the memcg clamp to
> BDP's setpoint and rate-limit math.
> - include/trace/events/writeback.h: cg_dirty_cap added to the
> balance_dirty_pages tracepoint so operators can distinguish the
> memcg clamp from the global dirty_limit at runtime.
> - mm/memcontrol.c: registers the two cgroup v2 files with input
> validation.
> - mm/page-writeback.c: the throttling changes.
>
> The key changes in page-writeback.c:
>
> - The dirty_ratio clamp lives inside domain_dirty_limits(), keyed
> on wb->memcg_css (the inode owner's memcg). Every consumer of
> the memcg dtc -- writer throttle, flusher kworker,
> cgwb_calc_thresh -- sees the same clamped thresh and bg_thresh.
> The clamp uses mult_frac() so a small memcg does not collapse
> to a zero ceiling.
>
> - The dirty_min bypass lives in balance_dirty_pages() and is
> writer-keyed: the writing task's memcg is looked up under RCU,
> and when its dirty+in-flight backlog is below dirty_min the
> loop jumps to free_running, bypassing both the global and the
> per-memcg BDP gates. dirty_min is an admission guarantee for
> the writer's own cgroup, not for inode owners.
>
> - When the clamp engages, mdtc->dirty is replaced with the
> memcg-wide NR_FILE_DIRTY + NR_WRITEBACK sum so freerun /
> setpoint / rate-limit smoothing see the real backlog and pages
> migrating from NR_FILE_DIRTY into NR_WRITEBACK cannot silently
> widen the cap.
>
> Fast-path cost when neither knob is set: one rcu_read_lock/unlock
> pair plus a READ_ONCE(dirty_min) per balance_dirty_pages()
> iteration, and one READ_ONCE(dirty_ratio) per domain_dirty_limits()
> call. The memcg counter reads are gated on "knob armed" and do
> not fire on the default path. We have not measured the added
> cost yet, but we expect it to be in the noise of existing BDP
> bookkeeping. A tight pwrite() microbenchmark will confirm this
> before a non-RFC posting.
>
> Scope
> =====
>
> This series affects the writer-side throttle (balance_dirty_pages())
> only. It does not partition the flusher-side writeback queue. A
> cgroup's fsync() can still block behind inodes from other cgroups in
> writeback_sb_inodes(). We document this limit explicitly and expect
> writeback-queue partitioning to be a separate, larger effort.
>
> Interaction with block-layer throttles
> ======================================
>
> The two knobs are orthogonal to io.max / io.cost. balance_dirty_pages()
> runs before the bio reaches the block layer, so dirty_min simply allows
> a cgroup to keep accepting write() syscalls up to its reservation; the
> actual I/O is still subject to whatever block throttle is in effect. In
> the reproducer above, the disk-level bandwidth limit (256 KB/s) is
> applied at the QEMU virtio-blk layer, and the protected victim dirties
> roughly equal to the rate it can drain, so the block throttle is
> exercised on both kernels. We have not yet tested interaction with
> guest-side io.max settings; this is on the list before a non-RFC
> posting.
>
> Questions for maintainers
> =========================
>
> 1. Is writer-throttle-only scope (no flusher/writeback-queue work)
> acceptable for the first series?
> 2. Does dirty_ratio belong on struct mem_cgroup (as the prototype
> has it) or on the memcg's wb_domain? Routing it through wb_domain
> would let us reuse __wb_calc_thresh() and keep all threshold
> policy in one place; a possible split is dirty_ratio on wb_domain
> (it is threshold policy), dirty_min on the memcg (throttle
> bypass). We can go either way.
> 3. For dirty_min safety caps (per-cgroup and global sum), which
> approach do you prefer: reject at write time, or clamp on read in
> an effective_dirty_min?
> 4. Is hierarchical enforcement of dirty_ratio (clamp against
> ancestors) required for v1, or can it follow in a subsequent
> series?
>
> What's missing before a non-RFC posting
> =======================================
>
> - Split the monolithic prototype into a proper series (one patch per
> concept + Documentation + selftest).
> - Documentation/admin-guide/cgroup-v2.rst entries for both knobs.
> - tools/testing/selftests/cgroup/ test for interface surface and
> noisy-neighbor protection.
> - Implement the per-cgroup and global dirty_min safety caps described
> in the memory.dirty_min bullet.
> - Fast-path microbenchmark: confirm zero measurable regression for
> cgroups that have neither knob set.
> - Larger-N validation on real hardware (the current N=5 data is from
> a QEMU guest on a throttled virtio-blk).
>
> Follow-ups (out of scope for this series)
> =========================================
>
> - memory.dirty_weight: a priority weight knob that scales the BDP
> pause length, planned as a separate series. The prototype validated
> the interface surface but the application site (post-pause scaling
> vs. folding into pos_ratio / dirty_ratelimit) needs to be settled
> before we ship it. Happy to discuss in advance of that posting.
> - memory.dirty_bytes: a byte-value companion to memory.dirty_ratio,
> mirroring the global vm_dirty_bytes. For operators who want a
> byte-predictable per-cgroup dirty cap rather than a ratio of the
> cgroup's dirtyable memory. We have not prototyped this yet; we
> are listing it so reviewers know it is on the roadmap, since
> the ratio-only interface omits that use case.
> - Writeback-queue partitioning: flusher-side fairness across
> cgroups, as noted in Scope above.
>
> Looking forward to feedback.
>
> Thanks,
> Alireza Haghdoost <haghdoost@uber.com>
> Kshitij Doshi <kshitijd@uber.com>
> ---
> include/linux/memcontrol.h | 10 +++
> include/linux/writeback.h | 4 +
> include/trace/events/writeback.h | 5 +-
> mm/memcontrol.c | 62 ++++++++++++++
> mm/page-writeback.c | 179 ++++++++++++++++++++++++++++++++++++---
> 5 files changed, 249 insertions(+), 11 deletions(-)
>
> diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
> index dc3fa687759b..45ca949a4c68 100644
> --- a/include/linux/memcontrol.h
> +++ b/include/linux/memcontrol.h
> @@ -323,6 +323,16 @@ struct mem_cgroup {
> spinlock_t event_list_lock;
> #endif /* CONFIG_MEMCG_V1 */
>
> + /* Per-memcg dirty-page controls (memory.dirty_ratio, memory.dirty_min) */
> + /*
> + * dirty_ratio: [0, 100] percent of dirtyable memory (mdtc->avail),
> + * matching the global vm_dirty_ratio base; 0 inherits the global
> + * threshold.
> + * dirty_min: dirty-page reservation, in pages; 0 disables the bypass.
> + */
> + unsigned int dirty_ratio;
> + unsigned long dirty_min;
> +
> struct mem_cgroup_per_node *nodeinfo[];
> };
>
> diff --git a/include/linux/writeback.h b/include/linux/writeback.h
> index 62552a2ce5b9..e37632f728be 100644
> --- a/include/linux/writeback.h
> +++ b/include/linux/writeback.h
> @@ -318,6 +318,10 @@ struct dirty_throttle_control {
> unsigned long thresh; /* dirty threshold */
> unsigned long bg_thresh; /* dirty background threshold */
> unsigned long limit; /* hard dirty limit */
> + unsigned long cg_dirty_cap; /* per-memcg dirty_ratio clamp for
> + * this pass, or PAGE_COUNTER_MAX
> + * when no memcg clamp applies
> + */
>
> unsigned long wb_dirty; /* per-wb counterparts */
> unsigned long wb_thresh;
> diff --git a/include/trace/events/writeback.h b/include/trace/events/writeback.h
> index bdac0d685a98..0bf86b3c903c 100644
> --- a/include/trace/events/writeback.h
> +++ b/include/trace/events/writeback.h
> @@ -672,6 +672,7 @@ TRACE_EVENT(balance_dirty_pages,
> __array( char, bdi, 32)
> __field(u64, cgroup_ino)
> __field(unsigned long, limit)
> + __field(unsigned long, cg_dirty_cap)
> __field(unsigned long, setpoint)
> __field(unsigned long, dirty)
> __field(unsigned long, wb_setpoint)
> @@ -691,6 +692,7 @@ TRACE_EVENT(balance_dirty_pages,
> strscpy_pad(__entry->bdi, bdi_dev_name(wb->bdi), 32);
>
> __entry->limit = dtc->limit;
> + __entry->cg_dirty_cap = dtc->cg_dirty_cap;
> __entry->setpoint = (dtc->limit + freerun) / 2;
> __entry->dirty = dtc->dirty;
> __entry->wb_setpoint = __entry->setpoint *
> @@ -710,13 +712,14 @@ TRACE_EVENT(balance_dirty_pages,
>
>
> TP_printk("bdi %s: "
> - "limit=%lu setpoint=%lu dirty=%lu "
> + "limit=%lu cg_dirty_cap=%lu setpoint=%lu dirty=%lu "
> "wb_setpoint=%lu wb_dirty=%lu "
> "dirty_ratelimit=%lu task_ratelimit=%lu "
> "dirtied=%u dirtied_pause=%u "
> "paused=%lu pause=%ld period=%lu think=%ld cgroup_ino=%llu",
> __entry->bdi,
> __entry->limit,
> + __entry->cg_dirty_cap,
> __entry->setpoint,
> __entry->dirty,
> __entry->wb_setpoint,
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index c3d98ab41f1f..c43fe4f394eb 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -4748,6 +4748,56 @@ static ssize_t memory_max_write(struct kernfs_open_file *of,
> return nbytes;
> }
>
> +static int memory_dirty_ratio_show(struct seq_file *m, void *v)
> +{
> + struct mem_cgroup *memcg = mem_cgroup_from_seq(m);
> +
> + seq_printf(m, "%u\n", READ_ONCE(memcg->dirty_ratio));
> + return 0;
> +}
> +
> +static ssize_t memory_dirty_ratio_write(struct kernfs_open_file *of,
> + char *buf, size_t nbytes, loff_t off)
> +{
> + struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
> + unsigned int ratio;
> + int err;
> +
> + err = kstrtouint(strstrip(buf), 0, &ratio);
> + if (err)
> + return err;
> +
> + if (ratio > 100)
> + return -EINVAL;
> +
> + WRITE_ONCE(memcg->dirty_ratio, ratio);
> + return nbytes;
> +}
> +
> +static int memory_dirty_min_show(struct seq_file *m, void *v)
> +{
> + struct mem_cgroup *memcg = mem_cgroup_from_seq(m);
> +
> + /* seq_puts_memcg_tunable automatically multiplies by PAGE_SIZE for the user */
> + return seq_puts_memcg_tunable(m, READ_ONCE(memcg->dirty_min));
> +}
> +
> +static ssize_t memory_dirty_min_write(struct kernfs_open_file *of,
> + char *buf, size_t nbytes, loff_t off)
> +{
> + struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
> + unsigned long dirty_min;
> + int err;
> +
> + /* page_counter_memparse converts strings like "512M" into a page count */
> + err = page_counter_memparse(strstrip(buf), "max", &dirty_min);
> + if (err)
> + return err;
> +
> + WRITE_ONCE(memcg->dirty_min, dirty_min);
> + return nbytes;
> +}
> +
> /*
> * Note: don't forget to update the 'samples/cgroup/memcg_event_listener'
> * if any new events become available.
> @@ -4950,6 +5000,18 @@ static struct cftype memory_files[] = {
> .flags = CFTYPE_NS_DELEGATABLE,
> .write = memory_reclaim,
> },
> + {
> + .name = "dirty_ratio",
> + .flags = CFTYPE_NOT_ON_ROOT,
> + .seq_show = memory_dirty_ratio_show,
> + .write = memory_dirty_ratio_write,
> + },
> + {
> + .name = "dirty_min",
> + .flags = CFTYPE_NOT_ON_ROOT,
> + .seq_show = memory_dirty_min_show,
> + .write = memory_dirty_min_write,
> + },
> { } /* terminate */
> };
>
> diff --git a/mm/page-writeback.c b/mm/page-writeback.c
> index 88cd53d4ba09..2847b2c1e59a 100644
> --- a/mm/page-writeback.c
> +++ b/mm/page-writeback.c
> @@ -124,14 +124,17 @@ struct wb_domain global_wb_domain;
>
> #define GDTC_INIT(__wb) .wb = (__wb), \
> .dom = &global_wb_domain, \
> - .wb_completions = &(__wb)->completions
> + .wb_completions = &(__wb)->completions, \
> + .cg_dirty_cap = PAGE_COUNTER_MAX
>
> -#define GDTC_INIT_NO_WB .dom = &global_wb_domain
> +#define GDTC_INIT_NO_WB .dom = &global_wb_domain, \
> + .cg_dirty_cap = PAGE_COUNTER_MAX
>
> #define MDTC_INIT(__wb, __gdtc) .wb = (__wb), \
> .dom = mem_cgroup_wb_domain(__wb), \
> .wb_completions = &(__wb)->memcg_completions, \
> - .gdtc = __gdtc
> + .gdtc = __gdtc, \
> + .cg_dirty_cap = PAGE_COUNTER_MAX
>
> static bool mdtc_valid(struct dirty_throttle_control *dtc)
> {
> @@ -183,8 +186,9 @@ static void wb_min_max_ratio(struct bdi_writeback *wb,
> #else /* CONFIG_CGROUP_WRITEBACK */
>
> #define GDTC_INIT(__wb) .wb = (__wb), \
> - .wb_completions = &(__wb)->completions
> -#define GDTC_INIT_NO_WB
> + .wb_completions = &(__wb)->completions, \
> + .cg_dirty_cap = PAGE_COUNTER_MAX
> +#define GDTC_INIT_NO_WB .cg_dirty_cap = PAGE_COUNTER_MAX
> #define MDTC_INIT(__wb, __gdtc)
>
> static bool mdtc_valid(struct dirty_throttle_control *dtc)
> @@ -392,6 +396,58 @@ static void domain_dirty_limits(struct dirty_throttle_control *dtc)
> bg_thresh += bg_thresh / 4 + global_wb_domain.dirty_limit / 32;
> thresh += thresh / 4 + global_wb_domain.dirty_limit / 32;
> }
> +
> + /*
> + * Apply the per-memcg dirty_ratio clamp on mdtc (gdtc != NULL
> + * iff @dtc is a memcg dtc). dirty_ratio is scaled against
> + * the memcg's own dirtyable memory (@available_memory), matching
> + * the semantics of vm_dirty_ratio so the two knobs share a base
> + * and compose via a plain min() on thresh. The clamp is keyed
> + * on wb->memcg_css (the inode-owner's memcg) rather than on
> + * current's memcg, so balance_dirty_pages(), wb_over_bg_thresh()
> + * (flusher kworker context), and cgwb_calc_thresh() all see the
> + * same clamped value.
> + *
> + * Published on dtc->cg_dirty_cap as well so hard_dirty_limit()
> + * callers in balance_dirty_pages() can ignore the slower
> + * dom->dirty_limit smoothing when deriving setpoint/
> + * rate-limit from the clamped ceiling.
> + *
> + * Clamp is applied after the rt/dl boost: dirty_ratio is a
> + * strict override, not widened by priority. bg_thresh is
> + * scaled by the same factor we apply to thresh so the
> + * user-configured bg/thresh ratio survives clamping instead
> + * of snapping to thresh/2 via the bg_thresh >= thresh guard
> + * below. mult_frac() preserves precision for small memcgs
> + * where a plain "(avail / 100) * ratio" would collapse to 0.
> + */
> + if (gdtc) {
> + struct mem_cgroup *memcg =
> + mem_cgroup_from_css(dtc->wb->memcg_css);
> + unsigned int cg_ratio = memcg ?
> + READ_ONCE(memcg->dirty_ratio) : 0;
> +
> + /*
> + * dtc is reused across balance_dirty_pages() iterations,
> + * so reset the published clamp every call -- an admin
> + * clearing memory.dirty_ratio mid-flight must take effect
> + * on the next pass.
> + */
> + dtc->cg_dirty_cap = PAGE_COUNTER_MAX;
> +
> + if (cg_ratio) {
> + unsigned long cg_thresh = mult_frac(available_memory,
> + cg_ratio, 100);
> +
> + if (cg_thresh < thresh) {
> + bg_thresh = mult_frac(bg_thresh, cg_thresh,
> + thresh);
> + thresh = cg_thresh;
> + dtc->cg_dirty_cap = cg_thresh;
> + }
> + }
> + }
> +
> /*
> * Dirty throttling logic assumes the limits in page units fit into
> * 32-bits. This gives 16TB dirty limits max which is hopefully enough.
> @@ -1065,7 +1121,9 @@ static void wb_position_ratio(struct dirty_throttle_control *dtc)
> struct bdi_writeback *wb = dtc->wb;
> unsigned long write_bw = READ_ONCE(wb->avg_write_bandwidth);
> unsigned long freerun = dirty_freerun_ceiling(dtc->thresh, dtc->bg_thresh);
> - unsigned long limit = dtc->limit = hard_dirty_limit(dtc_dom(dtc), dtc->thresh);
> + unsigned long limit = dtc->limit = min(hard_dirty_limit(dtc_dom(dtc),
> + dtc->thresh),
> + dtc->cg_dirty_cap);
> unsigned long wb_thresh = dtc->wb_thresh;
> unsigned long x_intercept;
> unsigned long setpoint; /* dirty pages' target balance point */
> @@ -1334,7 +1392,8 @@ static void wb_update_dirty_ratelimit(struct dirty_throttle_control *dtc,
> struct bdi_writeback *wb = dtc->wb;
> unsigned long dirty = dtc->dirty;
> unsigned long freerun = dirty_freerun_ceiling(dtc->thresh, dtc->bg_thresh);
> - unsigned long limit = hard_dirty_limit(dtc_dom(dtc), dtc->thresh);
> + unsigned long limit = min(hard_dirty_limit(dtc_dom(dtc), dtc->thresh),
> + dtc->cg_dirty_cap);
> unsigned long setpoint = (freerun + limit) / 2;
> unsigned long write_bw = wb->avg_write_bandwidth;
> unsigned long dirty_ratelimit = wb->dirty_ratelimit;
> @@ -1822,22 +1881,122 @@ static int balance_dirty_pages(struct bdi_writeback *wb,
> int ret = 0;
>
> for (;;) {
> + unsigned long cg_dirty_min = 0;
> + unsigned long cg_dirty_pages = 0;
> unsigned long now = jiffies;
>
> nr_dirty = global_node_page_state(NR_FILE_DIRTY);
>
> balance_domain_limits(gdtc, strictlimit);
> +
> + /*
> + * Under RCU, snapshot the current memcg's memory.dirty_min
> + * reservation. When it is non-zero, also snapshot the
> + * memcg-wide dirty backlog. These feed the per-writer
> + * dirty_min bypass below; the dirty_ratio clamp itself
> + * is applied inside domain_dirty_limits() keyed on
> + * wb->memcg_css so balance_dirty_pages(),
> + * wb_over_bg_thresh() (flusher kworker context), and
> + * cgwb_calc_thresh() all see a consistent clamped
> + * threshold.
> + *
> + * rcu_read_lock() is held only for the __rcu dereference
> + * of current->cgroups; the memcg pointer does not escape
> + * the critical section. The counter read matches
> + * domain_dirty_avail(mdtc, true) so the bypass compares
> + * the same dirty+in-flight backlog the global path uses.
> + */
> + rcu_read_lock();
> + {
> + struct mem_cgroup *memcg =
> + mem_cgroup_from_task(current);
> +
> + if (memcg) {
> + cg_dirty_min = READ_ONCE(memcg->dirty_min);
> + if (cg_dirty_min)
> + cg_dirty_pages =
> + memcg_page_state(memcg,
> + NR_FILE_DIRTY) +
> + memcg_page_state(memcg,
> + NR_WRITEBACK);
> + }
> + }
> + rcu_read_unlock();
> +
> if (mdtc) {
> /*
> - * If @wb belongs to !root memcg, repeat the same
> - * basic calculations for the memcg domain.
> + * For !root memcg, repeat the same three-step
> + * sequence as balance_domain_limits(gdtc):
> + * avail -> limits -> freerun. We inline it here
> + * so we can insert the mdtc->dirty override
> + * between step 2 (domain_dirty_limits, which
> + * publishes the per-memcg dirty_ratio clamp on
> + * cg_dirty_cap) and step 3 (domain_dirty_freerun,
> + * which consumes mdtc->dirty along with
> + * thresh/bg_thresh).
> + */
> + domain_dirty_avail(mdtc, true);
> + domain_dirty_limits(mdtc);
> +
> + /*
> + * When the dirty_ratio clamp engaged, replace the
> + * per-wb dirty count from mem_cgroup_wb_stats()
> + * with the memcg-wide NR_FILE_DIRTY + NR_WRITEBACK
> + * sum so freerun, the setpoint, and the rate-limit
> + * smoothing see the true memcg backlog instead of
> + * the subset that has migrated to this cgwb (cgwb
> + * migration is lazy and can lag by many seconds),
> + * and so a burst of buffered writes cannot silently
> + * bypass the clamp by shifting pages from
> + * NR_FILE_DIRTY into NR_WRITEBACK.
> + *
> + * Keyed on wb->memcg_css to match the clamp itself.
> + * The cgwb holds a css reference, so the memcg
> + * pointer is stable without additional locking.
> + *
> + * Caveat: memcg_page_state() aggregates across ALL
> + * backing devices owned by this memcg, while mdtc
> + * is scoped to one wb. A writer to a fast BDI may
> + * observe backlog accumulated on slow BDIs in the
> + * same memcg and throttle more than strictly needed.
> + * Accepted for v1; the alternative (summing per-wb
> + * dirty across the memcg's cgwbs) walks the cgwb
> + * list under RCU on a hot path.
> */
> - balance_domain_limits(mdtc, strictlimit);
> + if (mdtc->cg_dirty_cap != PAGE_COUNTER_MAX) {
> + struct mem_cgroup *wb_memcg =
> + mem_cgroup_from_css(mdtc->wb->memcg_css);
> +
> + if (wb_memcg)
> + mdtc->dirty =
> + memcg_page_state(wb_memcg,
> + NR_FILE_DIRTY) +
> + memcg_page_state(wb_memcg,
> + NR_WRITEBACK);
> + }
> +
> + domain_dirty_freerun(mdtc, strictlimit);
> }
>
> if (nr_dirty > gdtc->bg_thresh && !writeback_in_progress(wb))
> wb_start_background_writeback(wb);
>
> + /*
> + * dirty_min bypass: when the current memcg's dirty+in-flight
> + * backlog is below its memory.dirty_min reservation, let the
> + * writer proceed without throttling. This check must live
> + * outside the if (mdtc) block because a writer's file may not
> + * yet have been migrated to a cgwb; without cgwb, mdtc is NULL
> + * and the per-memcg block above is skipped entirely.
> + *
> + * cg_dirty_min and cg_dirty_pages come from the per-iteration
> + * snapshot taken above under rcu_read_lock; both are stored
> + * in pages (page_counter_memparse converts bytes -> pages for
> + * dirty_min), so no unit conversion is needed.
> + */
> + if (cg_dirty_min && cg_dirty_pages < cg_dirty_min)
> + goto free_running;
> +
> /*
> * If memcg domain is in effect, @dirty should be under
> * both global and memcg freerun ceilings.
>
> ---
> base-commit: 254f49634ee16a731174d2ae34bc50bd5f45e731
> change-id: 20260501-rfc-memcg-dirty-v1-ed4644c3fa8a
>
> Best regards,
> --
> Alireza Haghdoost <haghdoost@uber.com>
>
>
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply
* Re: [PATCH 0/2] cgroup/dmem: introduce a peak file
From: Thadeu Lima de Souza Cascardo @ 2026-05-06 14:18 UTC (permalink / raw)
To: Michal Koutný
Cc: Tejun Heo, Johannes Weiner, Michal Hocko, Roman Gushchin,
Shakeel Butt, Muchun Song, Andrew Morton, Jonathan Corbet,
Shuah Khan, Maarten Lankhorst, Maxime Ripard, Natalie Vock,
Tvrtko Ursulin, cgroups, linux-kernel, linux-mm, linux-doc,
dri-devel, kernel-dev
In-Reply-To: <aftB-cc5EhDXxCGA@localhost.localdomain>
On Wed, May 06, 2026 at 03:53:59PM +0200, Michal Koutný wrote:
> Hello Thadeu.
>
> On Wed, May 06, 2026 at 08:58:23AM -0300, Thadeu Lima de Souza Cascardo <cascardo@igalia.com> wrote:
> > Just like we have memory.peak, introduce a dmem.peak, which uses the
> > page_counter support for that.
> >
> > It can be written to in order to reset the peak, but different from
> > memory.peak, which expects any write, dmem.peak expects the region name to
> > be written to it. That region peak is the one that is reset.
> >
> > That requires ofp_peak to carry a pointer to the pool that was reset.
>
> (It'd be nicer to have generic data in that generic structure, at least
> some void *priv. But see below.)
>
I used void *, at first, but as the only current use is for the pool and as
mixing different uses may lead to misuse, I thought it would be safer to
use the type directly. This has been pointed out before for other members
of cgroup_file_ctx. See [1].
> > Writing a different region name will reset the different region and make
> > the original region peak get back to its non-reset value.
>
> I'm slightly confused by this fds x pool matricity when there's only
> a single slot in cgroup_file_ctx::cgroup_of_peak.
>
> The intended use case is that users should maintain one fd per pool and
> not mix it up?
> This stanza would better fit to cgroup-v2.rst proper than the commit
> message. Or make it simpler and start with non-resettable peak file
> (like memory.peak had started too) and see how it fares. WDYT?
>
That is also due to the limitation that each file descriptor has a single
linked list under cgroup_file_ctx::cgroup_of_peak. To allow for all the
pools to be reset at once, we would need one list per file descriptor.
But, on the other hand, as you pointed out, this leads to the flexibility
of being able to reset only one pool, while leaving the others as is.
Whereas, if one needs to reset all pools, they can use one file descriptor
per region.
I started with a non-resettable peak file, but as memory.peak can be reset,
I added that feature too. If we want to merge a non-resettable support
ealier and need to take longer to discuss how to work on the resettable
support given the above, I can resubmit. But I guess we can see if we can
reach an agreement sonner rather than later.
Thanks.
Cascardo.
>
> Thanks,
> Michal
[1] https://lore.kernel.org/all/CAHk-=wgiYkECT=hZRKj8ZwfBPw2Uz=gpOGBGd4ny0KYhSsjC0w@mail.gmail.com/
^ permalink raw reply
* Re: [PATCH 0/2] cgroup/dmem: introduce a peak file
From: Michal Koutný @ 2026-05-06 13:53 UTC (permalink / raw)
To: Thadeu Lima de Souza Cascardo
Cc: Tejun Heo, Johannes Weiner, Michal Hocko, Roman Gushchin,
Shakeel Butt, Muchun Song, Andrew Morton, Jonathan Corbet,
Shuah Khan, Maarten Lankhorst, Maxime Ripard, Natalie Vock,
Tvrtko Ursulin, cgroups, linux-kernel, linux-mm, linux-doc,
dri-devel, kernel-dev
In-Reply-To: <20260506-dmem_peak-v1-0-8d803eb3449c@igalia.com>
[-- Attachment #1: Type: text/plain, Size: 1205 bytes --]
Hello Thadeu.
On Wed, May 06, 2026 at 08:58:23AM -0300, Thadeu Lima de Souza Cascardo <cascardo@igalia.com> wrote:
> Just like we have memory.peak, introduce a dmem.peak, which uses the
> page_counter support for that.
>
> It can be written to in order to reset the peak, but different from
> memory.peak, which expects any write, dmem.peak expects the region name to
> be written to it. That region peak is the one that is reset.
>
> That requires ofp_peak to carry a pointer to the pool that was reset.
(It'd be nicer to have generic data in that generic structure, at least
some void *priv. But see below.)
> Writing a different region name will reset the different region and make
> the original region peak get back to its non-reset value.
I'm slightly confused by this fds x pool matricity when there's only
a single slot in cgroup_file_ctx::cgroup_of_peak.
The intended use case is that users should maintain one fd per pool and
not mix it up?
This stanza would better fit to cgroup-v2.rst proper than the commit
message. Or make it simpler and start with non-resettable peak file
(like memory.peak had started too) and see how it fares. WDYT?
Thanks,
Michal
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 265 bytes --]
^ permalink raw reply
* Re: [PATCH v3 01/12] mm, swap: simplify swap cache allocation helper
From: Chris Li @ 2026-05-06 13:51 UTC (permalink / raw)
To: kasong
Cc: linux-mm, Andrew Morton, David Hildenbrand, Zi Yan, Baolin Wang,
Barry Song, Hugh Dickins, Kemeng Shi, Nhat Pham, Baoquan He,
Johannes Weiner, Youngjun Park, Chengming Zhou, Roman Gushchin,
Shakeel Butt, Muchun Song, Qi Zheng, linux-kernel, cgroups,
Yosry Ahmed, Lorenzo Stoakes, Dev Jain, Lance Yang, Michal Hocko,
Michal Hocko, Suren Baghdasaryan, Axel Rasmussen
In-Reply-To: <20260421-swap-table-p4-v3-1-2f23759a76bc@tencent.com>
On Tue, Apr 21, 2026 at 8:16 AM Kairui Song via B4 Relay
<devnull+kasong.tencent.com@kernel.org> wrote:
>
> From: Kairui Song <kasong@tencent.com>
>
> Instead of trying to return the existing folio if the entry is already
> cached, simply return an error code if the allocation fails and drop the
Nitpick: Spell out which function changes the return type here. It is
__swap_cache_prepare_and_add()
> output argument. And introduce proper wrappers that handle the
Nitpick: Spell out the helper function. It is swap_cache_read_folio().
> allocation failure in different ways.
>
> For async swapin and readahead, the caller only wants to ensure that a
> swap-in read is issued when the allocation succeeded. And for zswap swap
> out, the caller will abort if the allocation failed because the entry is
> gone or cached already.
Should you add no functional change expected?
>
> Signed-off-by: Kairui Song <kasong@tencent.com>
Very nice clean ups. I like it. Here are some nitpicks; feel free to
ignore them.
Acked-by: Chris Li <chrisl@kerel.org>
> ---
> mm/swap.h | 3 +-
> mm/swap_state.c | 180 +++++++++++++++++++++++++++++---------------------------
> mm/zswap.c | 23 +++-----
> 3 files changed, 103 insertions(+), 103 deletions(-)
>
> diff --git a/mm/swap.h b/mm/swap.h
> index a77016f2423b..ad8b17a93758 100644
> --- a/mm/swap.h
> +++ b/mm/swap.h
> @@ -281,8 +281,7 @@ struct folio *swap_cache_get_folio(swp_entry_t entry);
> void *swap_cache_get_shadow(swp_entry_t entry);
> void swap_cache_del_folio(struct folio *folio);
> struct folio *swap_cache_alloc_folio(swp_entry_t entry, gfp_t gfp_flags,
> - struct mempolicy *mpol, pgoff_t ilx,
> - bool *alloced);
> + struct mempolicy *mpol, pgoff_t ilx);
> /* Below helpers require the caller to lock and pass in the swap cluster. */
> void __swap_cache_add_folio(struct swap_cluster_info *ci,
> struct folio *folio, swp_entry_t entry);
> diff --git a/mm/swap_state.c b/mm/swap_state.c
> index 1415a5c54a43..204a9499d50c 100644
> --- a/mm/swap_state.c
> +++ b/mm/swap_state.c
> @@ -459,54 +459,38 @@ void swap_update_readahead(struct folio *folio, struct vm_area_struct *vma,
> * All swap slots covered by the folio must have a non-zero swap count.
> *
> * Context: Caller must protect the swap device with reference count or locks.
> - * Return: Returns the folio being added on success. Returns the existing folio
> - * if @entry is already cached. Returns NULL if raced with swapin or swapoff.
> + * Return: 0 if success, error code if failed.
> */
> -static struct folio *__swap_cache_prepare_and_add(swp_entry_t entry,
> - struct folio *folio,
> - gfp_t gfp, bool charged)
> +static int __swap_cache_prepare_and_add(swp_entry_t entry,
> + struct folio *folio,
> + gfp_t gfp, bool charged)
> {
> - struct folio *swapcache = NULL;
> void *shadow;
> int ret;
>
> __folio_set_locked(folio);
> __folio_set_swapbacked(folio);
>
> - if (!charged && mem_cgroup_swapin_charge_folio(folio, NULL, gfp, entry))
> + if (!charged && mem_cgroup_swapin_charge_folio(folio, NULL, gfp, entry)) {
> + ret = -ENOMEM;
> goto failed;
> -
> - for (;;) {
> - ret = swap_cache_add_folio(folio, entry, &shadow);
> - if (!ret)
> - break;
> -
> - /*
> - * Large order allocation needs special handling on
> - * race: if a smaller folio exists in cache, swapin needs
> - * to fallback to order 0, and doing a swap cache lookup
> - * might return a folio that is irrelevant to the faulting
> - * entry because @entry is aligned down. Just return NULL.
> - */
> - if (ret != -EEXIST || folio_test_large(folio))
> - goto failed;
> -
> - swapcache = swap_cache_get_folio(entry);
> - if (swapcache)
> - goto failed;
> }
>
> + ret = swap_cache_add_folio(folio, entry, &shadow);
> + if (ret)
> + goto failed;
> +
> memcg1_swapin(entry, folio_nr_pages(folio));
> if (shadow)
> workingset_refault(folio, shadow);
>
> /* Caller will initiate read into locked folio */
> folio_add_lru(folio);
> - return folio;
> + return 0;
>
> failed:
> folio_unlock(folio);
> - return swapcache;
> + return ret;
> }
>
> /**
> @@ -515,7 +499,6 @@ static struct folio *__swap_cache_prepare_and_add(swp_entry_t entry,
> * @gfp_mask: memory allocation flags
> * @mpol: NUMA memory allocation policy to be applied
> * @ilx: NUMA interleave index, for use only when MPOL_INTERLEAVE
> - * @new_page_allocated: sets true if allocation happened, false otherwise
> *
> * Allocate a folio in the swap cache for one swap slot, typically before
> * doing IO (e.g. swap in or zswap writeback). The swap slot indicated by
> @@ -523,18 +506,40 @@ static struct folio *__swap_cache_prepare_and_add(swp_entry_t entry,
> * Currently only supports order 0.
> *
> * Context: Caller must protect the swap device with reference count or locks.
> - * Return: Returns the existing folio if @entry is cached already. Returns
> - * NULL if failed due to -ENOMEM or @entry have a swap count < 1.
> + * Return: Returns the folio if allocation succeeded and folio is added to
> + * swap cache. Returns error code if allocation failed due to race.
> */
> struct folio *swap_cache_alloc_folio(swp_entry_t entry, gfp_t gfp_mask,
> - struct mempolicy *mpol, pgoff_t ilx,
> - bool *new_page_allocated)
> + struct mempolicy *mpol, pgoff_t ilx)
> +{
> + int ret;
Nitpick: Suggest renaming it to "err" to make it obvious that it is an
int type for the error code. Because this function previously returned
a folio pointer, I have to remind myself that it is an int type not a
folio.
> + struct folio *folio;
> +
> + /* Allocate a new folio to be added into the swap cache. */
> + folio = folio_alloc_mpol(gfp_mask, 0, mpol, ilx, numa_node_id());
> + if (!folio)
> + return ERR_PTR(-ENOMEM);
> +
> + /*
> + * Try to add the new folio to the swap cache. It returns
> + * -EEXIST if the entry is already cached.
> + */
> + ret = __swap_cache_prepare_and_add(entry, folio, gfp_mask, false);
> + if (ret) {
> + folio_put(folio);
> + return ERR_PTR(ret);
> + }
> +
> + return folio;
> +}
> +
> +static struct folio *swap_cache_read_folio(swp_entry_t entry, gfp_t gfp,
> + struct mempolicy *mpol, pgoff_t ilx,
> + struct swap_iocb **plug, bool readahead)
> {
> struct swap_info_struct *si = __swap_entry_to_info(entry);
> struct folio *folio;
> - struct folio *result = NULL;
>
> - *new_page_allocated = false;
> /* Check the swap cache again for readahead path. */
> folio = swap_cache_get_folio(entry);
> if (folio)
> @@ -544,17 +549,24 @@ struct folio *swap_cache_alloc_folio(swp_entry_t entry, gfp_t gfp_mask,
> if (!swap_entry_swapped(si, entry))
> return NULL;
>
> - /* Allocate a new folio to be added into the swap cache. */
> - folio = folio_alloc_mpol(gfp_mask, 0, mpol, ilx, numa_node_id());
> - if (!folio)
> + do {
> + folio = swap_cache_get_folio(entry);
> + if (folio)
> + return folio;
> +
> + folio = swap_cache_alloc_folio(entry, gfp, mpol, ilx);
> + } while (IS_ERR(folio) && PTR_ERR(folio) == -EEXIST);
Nitpick: IS_ERR() only checks that the pointer is in the error code
range. If the pointer is -EEXIST, it will always be in the error code
range. I think the "IS_ERR(folio)" test can be dropped.
Chris
^ permalink raw reply
* [PATCH] blk-cgroup: fix blkg->online set on radix_tree_insert failure
From: Tao Cui @ 2026-05-06 13:11 UTC (permalink / raw)
To: tj, josef, axboe, cgroups; +Cc: Tao Cui
In blkg_create(), blkg->online was set to true unconditionally outside
the radix_tree_insert success check. When the insertion fails, the blkg
is not in the radix tree nor the hash/list, yet it is incorrectly marked
online. This state inconsistency could cause future code paths that
depend on blkg->online to behave incorrectly.
Move blkg->online = true inside the if (likely(!ret)) block so that it
is only set when the blkg is fully initialized and inserted.
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
---
block/blk-cgroup.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index 554c87bb4a86..539301f4f645 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -431,8 +431,8 @@ static struct blkcg_gq *blkg_create(struct blkcg *blkcg, struct gendisk *disk,
blkg->pd[i]->online = true;
}
}
+ blkg->online = true;
}
- blkg->online = true;
spin_unlock(&blkcg->lock);
if (!ret)
--
2.43.0
^ permalink raw reply related
* [PATCH v2] selftests: cgroup: Add basic tests for rdma controller
From: Tao Cui @ 2026-05-06 12:43 UTC (permalink / raw)
To: cuitao; +Cc: cgroups, hannes, mkoutny, tj
In-Reply-To: <f374cebb-f8c1-4d7e-8771-aac018cbd9fd@kylinos.cn>
This patch adds (and wires in) new test program for verifying rdma
controller behavior -- checking that rdma.current correctly tracks
resource allocation/deallocation and that rdma.max limits are enforced.
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
---
Changes in v2:
- Replace rdma.max interface-only tests with tests that verify actual
rdma.current behavior and limit enforcement
- Use libibverbs (conditionally compiled) to allocate real IB resources
- Skip gracefully when no RDMA device is present
---
tools/testing/selftests/cgroup/Makefile | 9 +
tools/testing/selftests/cgroup/test_rdma.c | 327 +++++++++++++++++++++
2 files changed, 336 insertions(+)
create mode 100644 tools/testing/selftests/cgroup/test_rdma.c
diff --git a/tools/testing/selftests/cgroup/Makefile b/tools/testing/selftests/cgroup/Makefile
index e01584c2189a..df6abf633bab 100644
--- a/tools/testing/selftests/cgroup/Makefile
+++ b/tools/testing/selftests/cgroup/Makefile
@@ -16,6 +16,7 @@ TEST_GEN_PROGS += test_kill
TEST_GEN_PROGS += test_kmem
TEST_GEN_PROGS += test_memcontrol
TEST_GEN_PROGS += test_pids
+TEST_GEN_PROGS += test_rdma
TEST_GEN_PROGS += test_zswap
LOCAL_HDRS += $(selfdir)/clone3/clone3_selftests.h $(selfdir)/pidfd/pidfd.h
@@ -32,4 +33,12 @@ $(OUTPUT)/test_kill: $(LIBCGROUP_O)
$(OUTPUT)/test_kmem: $(LIBCGROUP_O)
$(OUTPUT)/test_memcontrol: $(LIBCGROUP_O)
$(OUTPUT)/test_pids: $(LIBCGROUP_O)
+$(OUTPUT)/test_rdma: $(LIBCGROUP_O)
$(OUTPUT)/test_zswap: $(LIBCGROUP_O)
+
+# Conditionally enable rdma.current/limit tests when libibverbs is available
+IBVERBS_AVAILABLE := $(shell pkg-config --exists libibverbs 2>/dev/null && echo y)
+ifeq ($(IBVERBS_AVAILABLE),y)
+$(OUTPUT)/test_rdma: CFLAGS += -DHAVE_LIBIBVERBS
+$(OUTPUT)/test_rdma: LDLIBS += -libverbs
+endif
diff --git a/tools/testing/selftests/cgroup/test_rdma.c b/tools/testing/selftests/cgroup/test_rdma.c
new file mode 100644
index 000000000000..5c2719ed7e62
--- /dev/null
+++ b/tools/testing/selftests/cgroup/test_rdma.c
@@ -0,0 +1,327 @@
+// SPDX-License-Identifier: GPL-2.0
+#define _GNU_SOURCE
+
+#include <errno.h>
+#include <linux/limits.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+#include "kselftest.h"
+#include "cgroup_util.h"
+
+#ifdef HAVE_LIBIBVERBS
+#include <infiniband/verbs.h>
+
+static char *rdmacg_get_first_device(const char *cgroup)
+{
+ char buf[PAGE_SIZE];
+ char *space;
+
+ if (cg_read(cgroup, "rdma.max", buf, sizeof(buf)))
+ return NULL;
+
+ if (buf[0] == '\0')
+ return NULL;
+
+ space = strchr(buf, ' ');
+ if (!space)
+ return NULL;
+
+ return strndup(buf, space - buf);
+}
+
+static long rdmacg_get_current_value(const char *cgroup, const char *device,
+ const char *resource)
+{
+ char buf[PAGE_SIZE];
+ char pattern[256];
+ char *p;
+
+ if (cg_read(cgroup, "rdma.current", buf, sizeof(buf)))
+ return -1;
+
+ snprintf(pattern, sizeof(pattern), "%s ", device);
+ p = strstr(buf, pattern);
+ if (!p)
+ return -1;
+
+ snprintf(pattern, sizeof(pattern), "%s=", resource);
+ p = strstr(p, pattern);
+ if (!p)
+ return -1;
+ p += strlen(pattern);
+
+ return strtol(p, NULL, 10);
+}
+
+static struct ibv_context *rdmacg_open_device(const char *device_name)
+{
+ struct ibv_device **dev_list;
+ struct ibv_context *ctx = NULL;
+ int i;
+
+ dev_list = ibv_get_device_list(NULL);
+ if (!dev_list)
+ return NULL;
+
+ for (i = 0; dev_list[i]; i++) {
+ if (!strcmp(ibv_get_device_name(dev_list[i]), device_name)) {
+ ctx = ibv_open_device(dev_list[i]);
+ break;
+ }
+ }
+ ibv_free_device_list(dev_list);
+ return ctx;
+}
+
+static int rdmacg_current_fn(const char *cgroup, void *arg)
+{
+ const char *device_name = (const char *)arg;
+ struct ibv_context *ctx = NULL;
+ struct ibv_pd *pd = NULL;
+ long val;
+ int ret = EXIT_FAILURE;
+
+ ctx = rdmacg_open_device(device_name);
+ if (!ctx)
+ return EXIT_FAILURE;
+
+ val = rdmacg_get_current_value(cgroup, device_name, "hca_handle");
+ if (val != 1) {
+ ksft_print_msg("hca_handle should be 1 after open, got %ld\n", val);
+ goto cleanup;
+ }
+ val = rdmacg_get_current_value(cgroup, device_name, "hca_object");
+ if (val != 0) {
+ ksft_print_msg("hca_object should be 0 before alloc, got %ld\n", val);
+ goto cleanup;
+ }
+
+ pd = ibv_alloc_pd(ctx);
+ if (!pd) {
+ ksft_print_msg("ibv_alloc_pd failed: %s\n", strerror(errno));
+ goto cleanup;
+ }
+ val = rdmacg_get_current_value(cgroup, device_name, "hca_object");
+ if (val != 1) {
+ ksft_print_msg("hca_object should be 1 after alloc_pd, got %ld\n", val);
+ goto cleanup;
+ }
+
+ /* After ibv_dealloc_pd: hca_object should be 0 */
+ ibv_dealloc_pd(pd);
+ pd = NULL;
+ val = rdmacg_get_current_value(cgroup, device_name, "hca_object");
+ if (val != 0) {
+ ksft_print_msg("hca_object should be 0 after dealloc_pd, got %ld\n", val);
+ goto cleanup;
+ }
+
+ /* After ibv_close_device: hca_handle should be 0 */
+ ibv_close_device(ctx);
+ ctx = NULL;
+ val = rdmacg_get_current_value(cgroup, device_name, "hca_handle");
+ if (val != 0) {
+ ksft_print_msg("hca_handle should be 0 after close, got %ld\n", val);
+ goto cleanup;
+ }
+
+ ret = EXIT_SUCCESS;
+
+cleanup:
+ if (pd)
+ ibv_dealloc_pd(pd);
+ if (ctx)
+ ibv_close_device(ctx);
+ return ret;
+}
+
+/*
+ * Test: rdma.current responds to actual IB resource allocation and deallocation.
+ */
+static int test_rdmacg_current_response(const char *root)
+{
+ int ret = KSFT_FAIL;
+ char *cg;
+ char *device = NULL;
+
+ cg = cg_name(root, "rdmacg_test_1");
+ if (!cg)
+ return KSFT_FAIL;
+
+ if (cg_create(cg))
+ goto cleanup;
+
+ device = rdmacg_get_first_device(cg);
+ if (!device) {
+ ret = KSFT_SKIP;
+ goto cleanup;
+ }
+
+ if (!cg_run(cg, rdmacg_current_fn, device))
+ ret = KSFT_PASS;
+
+cleanup:
+ free(device);
+ cg_destroy(cg);
+ free(cg);
+ return ret;
+}
+
+static int rdmacg_limit_fn(const char *cgroup, void *arg)
+{
+ const char *device_name = (const char *)arg;
+ struct ibv_context *ctx = NULL;
+ struct ibv_pd *pd1 = NULL, *pd2 = NULL;
+ int ret = EXIT_FAILURE;
+
+ ctx = rdmacg_open_device(device_name);
+ if (!ctx)
+ return EXIT_FAILURE;
+
+ /* First PD allocation should succeed (within hca_object=1 limit) */
+ pd1 = ibv_alloc_pd(ctx);
+ if (!pd1) {
+ ksft_print_msg("first ibv_alloc_pd failed: %s\n", strerror(errno));
+ goto cleanup;
+ }
+
+ /* Second PD allocation should fail (exceeds hca_object=1 limit) */
+ pd2 = ibv_alloc_pd(ctx);
+ if (pd2) {
+ ksft_print_msg("second ibv_alloc_pd should have failed\n");
+ goto cleanup;
+ }
+
+ /* Free first PD, then try again -- should succeed */
+ ibv_dealloc_pd(pd1);
+ pd1 = NULL;
+
+ pd1 = ibv_alloc_pd(ctx);
+ if (!pd1) {
+ ksft_print_msg("ibv_alloc_pd after free failed: %s\n", strerror(errno));
+ goto cleanup;
+ }
+
+ ret = EXIT_SUCCESS;
+
+cleanup:
+ if (pd1)
+ ibv_dealloc_pd(pd1);
+ if (pd2)
+ ibv_dealloc_pd(pd2);
+ if (ctx)
+ ibv_close_device(ctx);
+ return ret;
+}
+
+/*
+ * Test: rdma.max limits are enforced -- exceeding hca_object limit causes
+ * allocation failure.
+ */
+static int test_rdmacg_limit_enforcement(const char *root)
+{
+ int ret = KSFT_FAIL;
+ char *cg;
+ char *device = NULL;
+ char buf[256];
+
+ cg = cg_name(root, "rdmacg_test_2");
+ if (!cg)
+ return KSFT_FAIL;
+
+ if (cg_create(cg))
+ goto cleanup;
+
+ device = rdmacg_get_first_device(cg);
+ if (!device) {
+ ret = KSFT_SKIP;
+ goto cleanup;
+ }
+
+ snprintf(buf, sizeof(buf), "%s hca_handle=max hca_object=1", device);
+ if (cg_write(cg, "rdma.max", buf)) {
+ ksft_print_msg("failed to set hca_object=1 limit\n");
+ goto cleanup;
+ }
+
+ if (!cg_run(cg, rdmacg_limit_fn, device))
+ ret = KSFT_PASS;
+
+cleanup:
+ free(device);
+ cg_destroy(cg);
+ free(cg);
+ return ret;
+}
+
+#define T(x) { x, #x }
+struct rdmacg_test {
+ int (*fn)(const char *root);
+ const char *name;
+} tests[] = {
+ T(test_rdmacg_current_response),
+ T(test_rdmacg_limit_enforcement),
+};
+#undef T
+
+int main(int argc, char **argv)
+{
+ char root[PATH_MAX];
+ char orig_subtree[PAGE_SIZE] = {0};
+ bool rdma_was_enabled = false;
+
+ ksft_print_header();
+ ksft_set_plan(ARRAY_SIZE(tests));
+
+ if (cg_find_unified_root(root, sizeof(root), NULL))
+ ksft_exit_skip("cgroup v2 isn't mounted\n");
+
+ if (cg_read_strstr(root, "cgroup.controllers", "rdma"))
+ ksft_exit_skip("rdma controller isn't available\n");
+
+ /* Save original subtree_control so we can restore it later */
+ if (cg_read(root, "cgroup.subtree_control", orig_subtree,
+ sizeof(orig_subtree)))
+ orig_subtree[0] = '\0';
+
+ rdma_was_enabled = (strstr(orig_subtree, "rdma") != NULL);
+
+ /* Enable rdma controller if not already enabled */
+ if (!rdma_was_enabled) {
+ if (cg_write(root, "cgroup.subtree_control", "+rdma"))
+ ksft_exit_skip("Failed to enable rdma controller\n");
+ }
+
+ for (int i = 0; i < ARRAY_SIZE(tests); i++) {
+ switch (tests[i].fn(root)) {
+ case KSFT_PASS:
+ ksft_test_result_pass("%s\n", tests[i].name);
+ break;
+ case KSFT_SKIP:
+ ksft_test_result_skip("%s\n", tests[i].name);
+ break;
+ default:
+ ksft_test_result_fail("%s\n", tests[i].name);
+ break;
+ }
+ }
+
+ /* Restore original subtree_control state */
+ if (!rdma_was_enabled)
+ cg_write(root, "cgroup.subtree_control", "-rdma");
+
+ ksft_finished();
+}
+
+#else /* !HAVE_LIBIBVERBS */
+
+int main(int argc, char **argv)
+{
+ ksft_print_header();
+ ksft_exit_skip("test requires libibverbs\n");
+}
+
+#endif /* HAVE_LIBIBVERBS */
--
2.43.0
^ permalink raw reply related
* Re: [PATCH v2 1/2] selftests/cgroup: Fix hardcoded page size in test_percpu_basic
From: Sayali Patil @ 2026-05-06 12:10 UTC (permalink / raw)
To: Li Wang, akpm, hannes, mhocko, roman.gushchin, shakeel.butt,
muchun.song, tj, mkoutny, shuah
Cc: cgroups, linux-mm, linux-kselftest, linux-kernel, Waiman Long,
Christoph Lameter, Shakeel Butt, Vlastimil Babka
In-Reply-To: <20260501022058.18024-2-li.wang@linux.dev>
On 01/05/26 07:50, Li Wang wrote:
> MAX_VMSTAT_ERROR uses a hardcoded page size of 4096, which assumes
> 4K pages. This causes test_percpu_basic to fail on systems where
> the kernel is configured with a larger page size, such as aarch64
> systems using 16K or 64K pages, where the maximum permissible
> discrepancy between memory.current and percpu charges is
> proportionally larger.
>
> Replace the hardcoded 4096 with sysconf(_SC_PAGESIZE) to correctly
> derive the page size at runtime regardless of the underlying
> architecture or kernel configuration.
>
> Signed-off-by: Li Wang <li.wang@linux.dev>
> Cc: Waiman Long <longman@redhat.com>
> Cc: Christoph Lameter <cl@linux.com>
> Cc: Johannes Weiner <hannes@cmpxchg.org>
> Cc: Michal Hocko <mhocko@kernel.org>
> Cc: Shakeel Butt <shakeelb@google.com>
> Cc: Tejun Heo <tj@kernel.org>
> Cc: Vlastimil Babka <vbabka@suse.cz>
> Acked-by: Waiman Long <longman@redhat.com>
> ---
> tools/testing/selftests/cgroup/test_kmem.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tools/testing/selftests/cgroup/test_kmem.c b/tools/testing/selftests/cgroup/test_kmem.c
> index eeabd34bf08..249d7911306 100644
> --- a/tools/testing/selftests/cgroup/test_kmem.c
> +++ b/tools/testing/selftests/cgroup/test_kmem.c
> @@ -24,7 +24,7 @@
> * the maximum discrepancy between charge and vmstat entries is number
> * of cpus multiplied by 64 pages.
> */
> -#define MAX_VMSTAT_ERROR (4096 * 64 * get_nprocs())
> +#define MAX_VMSTAT_ERROR (sysconf(_SC_PAGESIZE) * 64 * get_nprocs())
>
> #define KMEM_DEAD_WAIT_RETRIES 80
>
Reviewed-by: Sayali Patil <sayalip@linux.ibm.com>
Thanks,
Sayali
^ permalink raw reply
* [PATCH 2/2] cgroup/dmem: introduce a peak file
From: Thadeu Lima de Souza Cascardo @ 2026-05-06 11:58 UTC (permalink / raw)
To: Tejun Heo, Johannes Weiner, Michal Koutný, Michal Hocko,
Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton,
Jonathan Corbet, Shuah Khan, Maarten Lankhorst, Maxime Ripard,
Natalie Vock, Tvrtko Ursulin
Cc: cgroups, linux-kernel, linux-mm, linux-doc, dri-devel,
Thadeu Lima de Souza Cascardo, kernel-dev
In-Reply-To: <20260506-dmem_peak-v1-0-8d803eb3449c@igalia.com>
Just like we have memory.peak, introduce a dmem.peak, which uses the
page_counter support for that.
It can be written to in order to reset the peak, but different from
memory.peak, which expects any write, dmem.peak expects the region name to
be written to it. That region peak is the one that is reset.
That requires ofp_peak to carry a pointer to the pool that was reset.
Writing a different region name will reset the different region and make
the original region peak get back to its non-reset value.
Signed-off-by: Thadeu Lima de Souza Cascardo <cascardo@igalia.com>
---
Documentation/admin-guide/cgroup-v2.rst | 10 +++
include/linux/cgroup-defs.h | 1 +
kernel/cgroup/dmem.c | 132 ++++++++++++++++++++++++++++++--
3 files changed, 137 insertions(+), 6 deletions(-)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 6efd0095ed99..3ba7ab3a36b3 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -2808,6 +2808,16 @@ DMEM Interface Files
The semantics are the same as for the memory cgroup controller, and are
calculated in the same way.
+ dmem.peak
+ A readwrite nested-keyed file that exists on non-root cgroups.
+
+ The max memory usage recorded for the cgroup and its descendants since
+ either the creation of the cgroup or the most recent reset for that FD.
+
+ A write of a region name to this file resets it to the current memory
+ usage for subsequent reads through the same file descriptor for that
+ region.
+
dmem.capacity
A read-only file that describes maximum region capacity.
It only exists on the root cgroup. Not all memory can be
diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h
index a85044cb0553..b536054bd916 100644
--- a/include/linux/cgroup-defs.h
+++ b/include/linux/cgroup-defs.h
@@ -874,6 +874,7 @@ extern bool cgroup_enable_per_threadgroup_rwsem;
struct cgroup_of_peak {
unsigned long value;
struct list_head list;
+ struct dmem_cgroup_pool_state *pool;
};
void of_peak_reset(struct cgroup_of_peak *ofp, struct page_counter *pc,
diff --git a/kernel/cgroup/dmem.c b/kernel/cgroup/dmem.c
index 1ab1fb47f271..afa380c9839b 100644
--- a/kernel/cgroup/dmem.c
+++ b/kernel/cgroup/dmem.c
@@ -57,6 +57,9 @@ struct dmemcg_state {
struct cgroup_subsys_state css;
struct list_head pools;
+
+ /** @peaks_lock: Protects access to the pools' peaks lists */
+ spinlock_t peaks_lock;
};
struct dmem_cgroup_pool_state {
@@ -72,6 +75,10 @@ struct dmem_cgroup_pool_state {
struct rcu_head rcu;
struct page_counter cnt;
+
+ /* Protected by the dmemcg_state peaks_lock */
+ struct list_head peaks;
+
struct dmem_cgroup_pool_state *parent;
refcount_t ref;
@@ -162,26 +169,45 @@ set_resource_max(struct dmem_cgroup_pool_state *pool, u64 val)
page_counter_set_max(&pool->cnt, val);
}
-static u64 get_resource_low(struct dmem_cgroup_pool_state *pool)
+static u64 get_resource_low(struct seq_file *sf, struct dmem_cgroup_pool_state *pool)
{
return pool ? READ_ONCE(pool->cnt.low) : 0;
}
-static u64 get_resource_min(struct dmem_cgroup_pool_state *pool)
+static u64 get_resource_min(struct seq_file *sf, struct dmem_cgroup_pool_state *pool)
{
return pool ? READ_ONCE(pool->cnt.min) : 0;
}
-static u64 get_resource_max(struct dmem_cgroup_pool_state *pool)
+static u64 get_resource_max(struct seq_file *sf, struct dmem_cgroup_pool_state *pool)
{
return pool ? READ_ONCE(pool->cnt.max) : PAGE_COUNTER_MAX;
}
-static u64 get_resource_current(struct dmem_cgroup_pool_state *pool)
+static u64 get_resource_current(struct seq_file *sf, struct dmem_cgroup_pool_state *pool)
{
return pool ? page_counter_read(&pool->cnt) : 0;
}
+static u64 get_resource_peak(struct seq_file *sf, struct dmem_cgroup_pool_state *pool)
+{
+ struct cgroup_of_peak *ofp = of_peak(sf->private);
+ u64 fd_peak, peak;
+ struct dmem_cgroup_pool_state *of_pool;
+
+ if (!pool)
+ return 0;
+
+ of_pool = READ_ONCE(ofp->pool);
+
+ fd_peak = READ_ONCE(ofp->value);
+ if (of_pool != pool || fd_peak == OFP_PEAK_UNSET)
+ peak = pool->cnt.watermark;
+ else
+ peak = max(fd_peak, READ_ONCE(pool->cnt.local_watermark));
+ return peak;
+}
+
static void reset_all_resource_limits(struct dmem_cgroup_pool_state *rpool)
{
set_resource_min(rpool, 0);
@@ -227,6 +253,7 @@ dmemcs_alloc(struct cgroup_subsys_state *parent_css)
return ERR_PTR(-ENOMEM);
INIT_LIST_HEAD(&dmemcs->pools);
+ spin_lock_init(&dmemcs->peaks_lock);
return &dmemcs->css;
}
@@ -377,6 +404,7 @@ alloc_pool_single(struct dmemcg_state *dmemcs, struct dmem_cgroup_region *region
ppool ? &ppool->cnt : NULL, true);
reset_all_resource_limits(pool);
refcount_set(&pool->ref, 1);
+ INIT_LIST_HEAD(&pool->peaks);
kref_get(®ion->ref);
if (ppool && !pool->parent) {
pool->parent = ppool;
@@ -784,7 +812,7 @@ static ssize_t dmemcg_limit_write(struct kernfs_open_file *of,
}
static int dmemcg_limit_show(struct seq_file *sf, void *v,
- u64 (*fn)(struct dmem_cgroup_pool_state *))
+ u64 (*fn)(struct seq_file *, struct dmem_cgroup_pool_state *))
{
struct dmemcg_state *dmemcs = css_to_dmemcs(seq_css(sf));
struct dmem_cgroup_region *region;
@@ -796,7 +824,7 @@ static int dmemcg_limit_show(struct seq_file *sf, void *v,
seq_puts(sf, region->name);
- val = fn(pool);
+ val = fn(sf, pool);
if (val < PAGE_COUNTER_MAX)
seq_printf(sf, " %lld\n", val);
else
@@ -807,6 +835,90 @@ static int dmemcg_limit_show(struct seq_file *sf, void *v,
return 0;
}
+static int dmem_cgroup_region_peak_open(struct kernfs_open_file *of)
+{
+ struct cgroup_of_peak *ofp = of_peak(of);
+
+ ofp->value = OFP_PEAK_UNSET;
+
+ return 0;
+}
+
+static void dmem_cgroup_region_peak_remove(struct cgroup_of_peak *ofp)
+{
+ struct dmem_cgroup_pool_state *pool;
+ struct dmemcg_state *dmemcs;
+
+ pool = xchg(&ofp->pool, NULL);
+ if (!pool)
+ return;
+
+ dmemcs = pool->cs;
+
+ spin_lock(&dmemcs->peaks_lock);
+ list_del(&ofp->list);
+ spin_unlock(&dmemcs->peaks_lock);
+
+ WRITE_ONCE(ofp->value, OFP_PEAK_UNSET);
+
+ dmemcg_pool_put(pool);
+}
+
+static void dmem_cgroup_region_peak_release(struct kernfs_open_file *of)
+{
+ struct cgroup_of_peak *ofp = of_peak(of);
+
+ if (ofp->value == OFP_PEAK_UNSET) {
+ /* fast path (no writes on this fd) */
+ return;
+ }
+
+ dmem_cgroup_region_peak_remove(ofp);
+}
+
+static ssize_t dmem_cgroup_region_peak_write(struct kernfs_open_file *of,
+ char *buf, size_t nbytes, loff_t off)
+{
+ struct dmemcg_state *dmemcs = css_to_dmemcs(of_css(of));
+ struct cgroup_of_peak *ofp = of_peak(of);
+ struct dmem_cgroup_pool_state *pool = NULL;
+ struct dmem_cgroup_region *region;
+ int err = 0;
+
+ buf = strstrip(buf);
+ if (!buf[0])
+ return -EINVAL;
+
+ rcu_read_lock();
+ region = dmemcg_get_region_by_name(buf);
+ rcu_read_unlock();
+
+ if (!region)
+ return -EINVAL;
+
+ pool = get_cg_pool_unlocked(dmemcs, region);
+ if (IS_ERR(pool)) {
+ err = PTR_ERR(pool);
+ goto out_put;
+ }
+
+ dmem_cgroup_region_peak_remove(ofp);
+
+ xchg(&ofp->pool, pool);
+ spin_lock(&dmemcs->peaks_lock);
+ of_peak_reset(ofp, &pool->cnt, &pool->peaks);
+ spin_unlock(&dmemcs->peaks_lock);
+
+out_put:
+ kref_put(®ion->ref, dmemcg_free_region);
+ return err ?: nbytes;
+}
+
+static int dmem_cgroup_region_peak_show(struct seq_file *sf, void *v)
+{
+ return dmemcg_limit_show(sf, v, get_resource_peak);
+}
+
static int dmem_cgroup_region_current_show(struct seq_file *sf, void *v)
{
return dmemcg_limit_show(sf, v, get_resource_current);
@@ -855,6 +967,14 @@ static struct cftype files[] = {
.name = "current",
.seq_show = dmem_cgroup_region_current_show,
},
+ {
+ .name = "peak",
+ .open = dmem_cgroup_region_peak_open,
+ .release = dmem_cgroup_region_peak_release,
+ .write = dmem_cgroup_region_peak_write,
+ .seq_show = dmem_cgroup_region_peak_show,
+ .flags = CFTYPE_NOT_ON_ROOT,
+ },
{
.name = "min",
.write = dmem_cgroup_region_min_write,
--
2.47.3
^ permalink raw reply related
* [PATCH 1/2] mm/page_counter: decouple peak_reset from peak_write
From: Thadeu Lima de Souza Cascardo @ 2026-05-06 11:58 UTC (permalink / raw)
To: Tejun Heo, Johannes Weiner, Michal Koutný, Michal Hocko,
Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton,
Jonathan Corbet, Shuah Khan, Maarten Lankhorst, Maxime Ripard,
Natalie Vock, Tvrtko Ursulin
Cc: cgroups, linux-kernel, linux-mm, linux-doc, dri-devel,
Thadeu Lima de Souza Cascardo, kernel-dev
In-Reply-To: <20260506-dmem_peak-v1-0-8d803eb3449c@igalia.com>
Create a new function of_peak_reset that resets the page_counter peak for a
given writer. This should allow it to be reused by other cgroups.
Signed-off-by: Thadeu Lima de Souza Cascardo <cascardo@igalia.com>
---
include/linux/cgroup-defs.h | 6 ++++++
kernel/cgroup/cgroup.c | 32 ++++++++++++++++++++++++++++++++
mm/memcontrol.c | 42 ++++++++----------------------------------
3 files changed, 46 insertions(+), 34 deletions(-)
diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h
index f42563739d2e..a85044cb0553 100644
--- a/include/linux/cgroup-defs.h
+++ b/include/linux/cgroup-defs.h
@@ -22,6 +22,7 @@
#include <linux/workqueue.h>
#include <linux/bpf-cgroup-defs.h>
#include <linux/psi_types.h>
+#include <linux/page_counter.h>
#ifdef CONFIG_CGROUPS
@@ -868,11 +869,16 @@ struct cgroup_subsys {
extern struct percpu_rw_semaphore cgroup_threadgroup_rwsem;
extern bool cgroup_enable_per_threadgroup_rwsem;
+#define OFP_PEAK_UNSET (((-1UL)))
+
struct cgroup_of_peak {
unsigned long value;
struct list_head list;
};
+void of_peak_reset(struct cgroup_of_peak *ofp, struct page_counter *pc,
+ struct list_head *watchers);
+
/**
* cgroup_threadgroup_change_begin - threadgroup exclusion for cgroups
* @tsk: target task
diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c
index 45c0b1ed687a..9b98a5cccf0e 100644
--- a/kernel/cgroup/cgroup.c
+++ b/kernel/cgroup/cgroup.c
@@ -1981,6 +1981,38 @@ struct cgroup_of_peak *of_peak(struct kernfs_open_file *of)
return &ctx->peak;
}
+/**
+ * of_peak_reset - reset peak
+ * @ofp: open file context
+ * @pc: counter
+ * @watchers: list of other open file contexts
+ *
+ * This function updates all contexts in @watchers to the new usage of @pc.
+ * If @ofp is not in the list yet, that is, if its value is
+ * %OFP_PEAK_UNSET, it is added to @watchers list.
+ *
+ * A lock must be used to protect @watchers.
+ */
+void of_peak_reset(struct cgroup_of_peak *ofp, struct page_counter *pc,
+ struct list_head *watchers)
+{
+ unsigned long usage;
+ struct cgroup_of_peak *peer_ctx;
+
+ usage = page_counter_read(pc);
+ WRITE_ONCE(pc->local_watermark, usage);
+
+ list_for_each_entry(peer_ctx, watchers, list)
+ if (usage > peer_ctx->value)
+ WRITE_ONCE(peer_ctx->value, usage);
+
+ /* initial write, register watcher */
+ if (ofp->value == OFP_PEAK_UNSET)
+ list_add(&ofp->list, watchers);
+
+ WRITE_ONCE(ofp->value, usage);
+}
+
static void apply_cgroup_root_flags(unsigned int root_flags)
{
if (current->nsproxy->cgroup_ns == &init_cgroup_ns) {
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index c03d4787d466..8754927070d3 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -4517,8 +4517,6 @@ static u64 memory_current_read(struct cgroup_subsys_state *css,
return (u64)page_counter_read(&memcg->memory) * PAGE_SIZE;
}
-#define OFP_PEAK_UNSET (((-1UL)))
-
static int peak_show(struct seq_file *sf, void *v, struct page_counter *pc)
{
struct cgroup_of_peak *ofp = of_peak(sf->private);
@@ -4563,45 +4561,18 @@ static void peak_release(struct kernfs_open_file *of)
spin_unlock(&memcg->peaks_lock);
}
-static ssize_t peak_write(struct kernfs_open_file *of, char *buf, size_t nbytes,
- loff_t off, struct page_counter *pc,
- struct list_head *watchers)
+static ssize_t memory_peak_write(struct kernfs_open_file *of, char *buf,
+ size_t nbytes, loff_t off)
{
- unsigned long usage;
- struct cgroup_of_peak *peer_ctx;
struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
struct cgroup_of_peak *ofp = of_peak(of);
spin_lock(&memcg->peaks_lock);
-
- usage = page_counter_read(pc);
- WRITE_ONCE(pc->local_watermark, usage);
-
- list_for_each_entry(peer_ctx, watchers, list)
- if (usage > peer_ctx->value)
- WRITE_ONCE(peer_ctx->value, usage);
-
- /* initial write, register watcher */
- if (ofp->value == OFP_PEAK_UNSET)
- list_add(&ofp->list, watchers);
-
- WRITE_ONCE(ofp->value, usage);
+ of_peak_reset(ofp, &memcg->memory, &memcg->memory_peaks);
spin_unlock(&memcg->peaks_lock);
-
return nbytes;
}
-static ssize_t memory_peak_write(struct kernfs_open_file *of, char *buf,
- size_t nbytes, loff_t off)
-{
- struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
-
- return peak_write(of, buf, nbytes, off, &memcg->memory,
- &memcg->memory_peaks);
-}
-
-#undef OFP_PEAK_UNSET
-
static int memory_min_show(struct seq_file *m, void *v)
{
return seq_puts_memcg_tunable(m,
@@ -5611,9 +5582,12 @@ static ssize_t swap_peak_write(struct kernfs_open_file *of, char *buf,
size_t nbytes, loff_t off)
{
struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
+ struct cgroup_of_peak *ofp = of_peak(of);
- return peak_write(of, buf, nbytes, off, &memcg->swap,
- &memcg->swap_peaks);
+ spin_lock(&memcg->peaks_lock);
+ of_peak_reset(ofp, &memcg->swap, &memcg->swap_peaks);
+ spin_unlock(&memcg->peaks_lock);
+ return nbytes;
}
static int swap_high_show(struct seq_file *m, void *v)
--
2.47.3
^ permalink raw reply related
* [PATCH 0/2] cgroup/dmem: introduce a peak file
From: Thadeu Lima de Souza Cascardo @ 2026-05-06 11:58 UTC (permalink / raw)
To: Tejun Heo, Johannes Weiner, Michal Koutný, Michal Hocko,
Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton,
Jonathan Corbet, Shuah Khan, Maarten Lankhorst, Maxime Ripard,
Natalie Vock, Tvrtko Ursulin
Cc: cgroups, linux-kernel, linux-mm, linux-doc, dri-devel,
Thadeu Lima de Souza Cascardo, kernel-dev
Just like we have memory.peak, introduce a dmem.peak, which uses the
page_counter support for that.
It can be written to in order to reset the peak, but different from
memory.peak, which expects any write, dmem.peak expects the region name to
be written to it. That region peak is the one that is reset.
That requires ofp_peak to carry a pointer to the pool that was reset.
Writing a different region name will reset the different region and make
the original region peak get back to its non-reset value.
While at it, we reuse a helper from memcontrol, which we moved to
kernel/cgroup/cgroup.c.
Signed-off-by: Thadeu Lima de Souza Cascardo <cascardo@igalia.com>
---
Thadeu Lima de Souza Cascardo (2):
mm/page_counter: decouple peak_reset from peak_write
cgroup/dmem: introduce a peak file
Documentation/admin-guide/cgroup-v2.rst | 10 +++
include/linux/cgroup-defs.h | 7 ++
kernel/cgroup/cgroup.c | 32 ++++++++
kernel/cgroup/dmem.c | 132 ++++++++++++++++++++++++++++++--
mm/memcontrol.c | 42 ++--------
5 files changed, 183 insertions(+), 40 deletions(-)
---
base-commit: 7fd2df204f342fc17d1a0bfcd474b24232fb0f32
change-id: 20260409-dmem_peak-3abc1be95072
Best regards,
--
Thadeu Lima de Souza Cascardo <cascardo@igalia.com>
^ permalink raw reply
* Re: [PATCH] selftests: cgroup: Add basic tests for rdma controller
From: Tao Cui @ 2026-05-06 8:35 UTC (permalink / raw)
To: Michal Koutný; +Cc: tj, hannes, cgroups
In-Reply-To: <afOT2cX2WOs0U05S@localhost.localdomain>
Hello Michal,
Thank you for the review.
On Thu, Apr 30, 2026 at 04:43:10PM +0800, Michal Koutný wrote:
> IIUC, these are tests for proper parsing of the limits but not so useful
> wrt RDMA controller (test_rdmacg_max_read has apparently little use).
You are right. The initial test cases were modeled after the pids
controller selftests -- since both pids and rdma controllers track
non-reclaimable resources, I focused on the limit interface (rdma.max)
parsing and validation, similar to how test_pids.c verifies pids.max
defaults and limit enforcement via forking.
However, as you pointed out, this approach does not cover the essential
behavior of the rdma controller itself. I agree that testing whether
rdma.current actually responds to IB resource allocations and whether
rdma.max limits are properly enforced is more valuable.
> I see that you try to work with a first found device -- if that's
> available, it'd be good to have a test that checks whether respective
> rdma.current-s respond to object allocations.
> As I am looking at test_hugetlb_memcg.c that does only simple
> testing of the .current would be sufficient, not sure how difficult
> would be to implement a test for actual limit enforcement (but would be
> nice too).
I have reworked the tests in the next version to include:
- test_rdmacg_current_response: verifies that rdma.current correctly
tracks hca_handle and hca_object as IB resources (ibv_open_device,
ibv_alloc_pd, ibv_dealloc_pd, ibv_close_device) are allocated and
freed, similar to how test_hugetlb_memcg.c validates memory.current
against hugetlb mmap/munmap operations.
- test_rdmacg_limit_enforcement: verifies that exceeding the
hca_object limit causes allocation failure (EAGAIN/ENOMEM) and that
freeing a resource allows subsequent allocations to succeed.
These tests require libibverbs and will be skipped if no RDMA device
is available or the library is not present.
I will send the updated patch soon.
Thanks,
Tao Cui
^ permalink raw reply
* Re: [GIT PULL] cgroup: Fixes for v7.1-rc2
From: pr-tracker-bot @ 2026-05-05 23:55 UTC (permalink / raw)
To: Tejun Heo
Cc: Linus Torvalds, linux-kernel, cgroups, Johannes Weiner,
Michal Koutný, Waiman Long
In-Reply-To: <f5258b93b0a513c90b4129177a2eb50d@kernel.org>
The pull request you sent on Tue, 05 May 2026 12:05:39 -1000:
> git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git tags/cgroup-for-7.1-rc2-fixes
has been merged into torvalds/linux.git:
https://git.kernel.org/torvalds/c/11f00074f72a977274c64c100372764eb04e6a3f
Thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/prtracker.html
^ permalink raw reply
* [tj-cgroup:cgroup-drain-for-7.2] BUILD SUCCESS 3867a2d19f2be9560c0ba280e912dcf23b775e2a
From: kernel test robot @ 2026-05-05 22:46 UTC (permalink / raw)
To: Tejun Heo; +Cc: cgroups
tree/branch: https://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git cgroup-drain-for-7.2
branch HEAD: 3867a2d19f2be9560c0ba280e912dcf23b775e2a cgroup: Defer kill_css_finish() in cgroup_apply_control_disable()
elapsed time: 1218m
configs tested: 357
configs skipped: 42
The following configs have been built successfully.
More configs may be tested in the coming days.
tested configs:
alpha allnoconfig gcc-15.2.0
alpha allyesconfig gcc-15.2.0
alpha defconfig gcc-15.2.0
arc allmodconfig clang-16
arc allnoconfig gcc-15.2.0
arc allyesconfig clang-23
arc defconfig gcc-15.2.0
arc randconfig-001-20260505 gcc-8.5.0
arc randconfig-001-20260506 gcc-12.5.0
arc randconfig-002-20260505 gcc-8.5.0
arc randconfig-002-20260506 gcc-12.5.0
arm allnoconfig gcc-15.2.0
arm allyesconfig clang-16
arm defconfig gcc-15.2.0
arm exynos_defconfig clang-23
arm mmp2_defconfig gcc-15.2.0
arm randconfig-001-20260505 gcc-8.5.0
arm randconfig-001-20260506 gcc-12.5.0
arm randconfig-002-20260505 gcc-8.5.0
arm randconfig-002-20260506 gcc-12.5.0
arm randconfig-003-20260505 gcc-8.5.0
arm randconfig-003-20260506 gcc-12.5.0
arm randconfig-004-20260505 gcc-8.5.0
arm randconfig-004-20260506 gcc-12.5.0
arm64 allmodconfig clang-23
arm64 allnoconfig gcc-15.2.0
arm64 defconfig gcc-15.2.0
arm64 randconfig-001 gcc-14.3.0
arm64 randconfig-001-20260505 gcc-14.3.0
arm64 randconfig-001-20260506 gcc-15.2.0
arm64 randconfig-002 gcc-14.3.0
arm64 randconfig-002-20260505 gcc-14.3.0
arm64 randconfig-002-20260506 gcc-15.2.0
arm64 randconfig-003 gcc-14.3.0
arm64 randconfig-003-20260505 gcc-14.3.0
arm64 randconfig-003-20260506 gcc-15.2.0
arm64 randconfig-004 gcc-14.3.0
arm64 randconfig-004-20260505 gcc-14.3.0
arm64 randconfig-004-20260506 gcc-15.2.0
csky alldefconfig gcc-15.2.0
csky allmodconfig gcc-15.2.0
csky allnoconfig gcc-15.2.0
csky defconfig gcc-15.2.0
csky randconfig-001 gcc-14.3.0
csky randconfig-001-20260505 gcc-14.3.0
csky randconfig-001-20260506 gcc-15.2.0
csky randconfig-002 gcc-14.3.0
csky randconfig-002-20260505 gcc-14.3.0
csky randconfig-002-20260506 gcc-15.2.0
hexagon allmodconfig gcc-15.2.0
hexagon allnoconfig gcc-15.2.0
hexagon defconfig gcc-15.2.0
hexagon randconfig-001 gcc-11.5.0
hexagon randconfig-001-20260505 clang-23
hexagon randconfig-001-20260505 gcc-11.5.0
hexagon randconfig-001-20260506 clang-23
hexagon randconfig-001-20260506 gcc-11.5.0
hexagon randconfig-002 gcc-11.5.0
hexagon randconfig-002-20260505 clang-23
hexagon randconfig-002-20260505 gcc-11.5.0
hexagon randconfig-002-20260506 clang-23
hexagon randconfig-002-20260506 gcc-11.5.0
i386 allmodconfig clang-20
i386 allnoconfig gcc-15.2.0
i386 allyesconfig clang-20
i386 buildonly-randconfig-001 clang-20
i386 buildonly-randconfig-001 gcc-14
i386 buildonly-randconfig-001-20260505 gcc-14
i386 buildonly-randconfig-001-20260506 clang-20
i386 buildonly-randconfig-002 clang-20
i386 buildonly-randconfig-002 gcc-14
i386 buildonly-randconfig-002-20260505 gcc-14
i386 buildonly-randconfig-002-20260506 clang-20
i386 buildonly-randconfig-003 clang-20
i386 buildonly-randconfig-003 gcc-14
i386 buildonly-randconfig-003-20260505 gcc-14
i386 buildonly-randconfig-003-20260506 clang-20
i386 buildonly-randconfig-004 clang-20
i386 buildonly-randconfig-004 gcc-14
i386 buildonly-randconfig-004-20260505 gcc-14
i386 buildonly-randconfig-004-20260506 clang-20
i386 buildonly-randconfig-005 clang-20
i386 buildonly-randconfig-005 gcc-14
i386 buildonly-randconfig-005-20260505 gcc-14
i386 buildonly-randconfig-005-20260506 clang-20
i386 buildonly-randconfig-006 clang-20
i386 buildonly-randconfig-006 gcc-14
i386 buildonly-randconfig-006-20260505 gcc-14
i386 buildonly-randconfig-006-20260506 clang-20
i386 defconfig gcc-15.2.0
i386 randconfig-001 clang-20
i386 randconfig-001 gcc-14
i386 randconfig-001-20260505 clang-20
i386 randconfig-001-20260505 gcc-14
i386 randconfig-001-20260506 gcc-14
i386 randconfig-002 clang-20
i386 randconfig-002 gcc-14
i386 randconfig-002-20260505 clang-20
i386 randconfig-002-20260505 gcc-14
i386 randconfig-002-20260506 gcc-14
i386 randconfig-003 clang-20
i386 randconfig-003 gcc-14
i386 randconfig-003-20260505 clang-20
i386 randconfig-003-20260506 gcc-14
i386 randconfig-004 clang-20
i386 randconfig-004 gcc-14
i386 randconfig-004-20260505 clang-20
i386 randconfig-004-20260505 gcc-14
i386 randconfig-004-20260506 gcc-14
i386 randconfig-005 clang-20
i386 randconfig-005 gcc-14
i386 randconfig-005-20260505 clang-20
i386 randconfig-005-20260506 gcc-14
i386 randconfig-006 clang-20
i386 randconfig-006 gcc-14
i386 randconfig-006-20260505 clang-20
i386 randconfig-006-20260506 gcc-14
i386 randconfig-007 clang-20
i386 randconfig-007 gcc-14
i386 randconfig-007-20260505 clang-20
i386 randconfig-007-20260505 gcc-14
i386 randconfig-007-20260506 gcc-14
i386 randconfig-011 clang-20
i386 randconfig-011-20260505 clang-20
i386 randconfig-011-20260506 clang-20
i386 randconfig-012 clang-20
i386 randconfig-012-20260505 clang-20
i386 randconfig-012-20260506 clang-20
i386 randconfig-013 clang-20
i386 randconfig-013-20260505 clang-20
i386 randconfig-013-20260506 clang-20
i386 randconfig-014 clang-20
i386 randconfig-014-20260505 clang-20
i386 randconfig-014-20260506 clang-20
i386 randconfig-015 clang-20
i386 randconfig-015-20260505 clang-20
i386 randconfig-015-20260506 clang-20
i386 randconfig-016 clang-20
i386 randconfig-016-20260505 clang-20
i386 randconfig-016-20260506 clang-20
i386 randconfig-017 clang-20
i386 randconfig-017-20260505 clang-20
i386 randconfig-017-20260506 clang-20
loongarch allmodconfig clang-23
loongarch allnoconfig gcc-15.2.0
loongarch defconfig clang-19
loongarch randconfig-001 gcc-11.5.0
loongarch randconfig-001-20260505 clang-23
loongarch randconfig-001-20260505 gcc-11.5.0
loongarch randconfig-001-20260506 clang-23
loongarch randconfig-001-20260506 gcc-11.5.0
loongarch randconfig-002 gcc-11.5.0
loongarch randconfig-002-20260505 clang-23
loongarch randconfig-002-20260505 gcc-11.5.0
loongarch randconfig-002-20260506 clang-23
loongarch randconfig-002-20260506 gcc-11.5.0
m68k allmodconfig gcc-15.2.0
m68k allnoconfig gcc-15.2.0
m68k allyesconfig clang-16
m68k defconfig clang-19
microblaze allnoconfig gcc-15.2.0
microblaze allyesconfig gcc-15.2.0
microblaze defconfig clang-19
mips allmodconfig gcc-15.2.0
mips allnoconfig gcc-15.2.0
mips allyesconfig gcc-15.2.0
nios2 alldefconfig gcc-11.5.0
nios2 allmodconfig clang-23
nios2 allnoconfig clang-23
nios2 defconfig clang-19
nios2 randconfig-001 gcc-11.5.0
nios2 randconfig-001-20260505 clang-23
nios2 randconfig-001-20260505 gcc-11.5.0
nios2 randconfig-001-20260506 clang-23
nios2 randconfig-001-20260506 gcc-11.5.0
nios2 randconfig-001-20260506 gcc-8.5.0
nios2 randconfig-002 gcc-11.5.0
nios2 randconfig-002-20260505 clang-23
nios2 randconfig-002-20260505 gcc-11.5.0
nios2 randconfig-002-20260506 clang-23
nios2 randconfig-002-20260506 gcc-11.5.0
nios2 randconfig-002-20260506 gcc-8.5.0
openrisc allmodconfig clang-23
openrisc allnoconfig clang-23
openrisc de0_nano_multicore_defconfig gcc-15.2.0
openrisc defconfig gcc-15.2.0
parisc allmodconfig gcc-15.2.0
parisc allnoconfig clang-23
parisc allyesconfig clang-19
parisc defconfig gcc-15.2.0
parisc generic-32bit_defconfig gcc-15.2.0
parisc randconfig-001 gcc-14.3.0
parisc randconfig-001-20260505 gcc-14.3.0
parisc randconfig-001-20260506 gcc-13.4.0
parisc randconfig-002 gcc-14.3.0
parisc randconfig-002-20260505 gcc-14.3.0
parisc randconfig-002-20260506 gcc-13.4.0
parisc64 defconfig clang-19
powerpc allmodconfig gcc-15.2.0
powerpc allnoconfig clang-23
powerpc mpc8315_rdb_defconfig clang-23
powerpc randconfig-001 gcc-14.3.0
powerpc randconfig-001-20260505 gcc-14.3.0
powerpc randconfig-001-20260506 gcc-13.4.0
powerpc randconfig-002 gcc-14.3.0
powerpc randconfig-002-20260505 gcc-14.3.0
powerpc randconfig-002-20260506 gcc-13.4.0
powerpc tqm8541_defconfig clang-23
powerpc64 randconfig-001 gcc-14.3.0
powerpc64 randconfig-001-20260505 gcc-14.3.0
powerpc64 randconfig-001-20260506 gcc-13.4.0
powerpc64 randconfig-002 gcc-14.3.0
powerpc64 randconfig-002-20260505 gcc-14.3.0
powerpc64 randconfig-002-20260506 gcc-13.4.0
riscv allmodconfig clang-23
riscv allnoconfig clang-23
riscv allyesconfig clang-16
riscv defconfig gcc-15.2.0
riscv randconfig-001-20260505 gcc-10.5.0
riscv randconfig-001-20260506 gcc-8.5.0
riscv randconfig-002-20260506 gcc-8.5.0
s390 allmodconfig clang-19
s390 allnoconfig clang-23
s390 allyesconfig gcc-15.2.0
s390 defconfig gcc-15.2.0
s390 randconfig-001-20260505 gcc-10.5.0
s390 randconfig-001-20260506 gcc-8.5.0
s390 randconfig-002-20260505 gcc-10.5.0
s390 randconfig-002-20260506 gcc-8.5.0
sh allmodconfig gcc-15.2.0
sh allnoconfig clang-23
sh allyesconfig clang-19
sh defconfig gcc-14
sh randconfig-001-20260505 gcc-10.5.0
sh randconfig-001-20260506 gcc-8.5.0
sh randconfig-002-20260505 gcc-10.5.0
sh randconfig-002-20260506 gcc-8.5.0
sh urquell_defconfig gcc-15.2.0
sparc allnoconfig clang-23
sparc defconfig gcc-15.2.0
sparc randconfig-001 gcc-15.2.0
sparc randconfig-001-20260505 gcc-15.2.0
sparc randconfig-001-20260506 gcc-11.5.0
sparc randconfig-002 gcc-15.2.0
sparc randconfig-002-20260505 gcc-15.2.0
sparc randconfig-002-20260506 gcc-11.5.0
sparc64 allmodconfig clang-23
sparc64 defconfig gcc-14
sparc64 randconfig-001 gcc-15.2.0
sparc64 randconfig-001-20260505 gcc-15.2.0
sparc64 randconfig-001-20260506 gcc-11.5.0
sparc64 randconfig-002 gcc-15.2.0
sparc64 randconfig-002-20260505 gcc-15.2.0
sparc64 randconfig-002-20260506 gcc-11.5.0
um allmodconfig clang-19
um allnoconfig clang-23
um allyesconfig gcc-15.2.0
um defconfig gcc-14
um i386_defconfig gcc-14
um randconfig-001 gcc-15.2.0
um randconfig-001-20260505 gcc-15.2.0
um randconfig-001-20260506 gcc-11.5.0
um randconfig-002 gcc-15.2.0
um randconfig-002-20260505 gcc-15.2.0
um randconfig-002-20260506 gcc-11.5.0
um x86_64_defconfig gcc-14
x86_64 allmodconfig clang-20
x86_64 allnoconfig clang-23
x86_64 allyesconfig clang-20
x86_64 buildonly-randconfig-001-20260505 clang-20
x86_64 buildonly-randconfig-001-20260506 clang-20
x86_64 buildonly-randconfig-002-20260505 clang-20
x86_64 buildonly-randconfig-002-20260506 clang-20
x86_64 buildonly-randconfig-002-20260506 gcc-14
x86_64 buildonly-randconfig-003-20260505 clang-20
x86_64 buildonly-randconfig-003-20260505 gcc-14
x86_64 buildonly-randconfig-003-20260506 clang-20
x86_64 buildonly-randconfig-004-20260505 clang-20
x86_64 buildonly-randconfig-004-20260505 gcc-14
x86_64 buildonly-randconfig-004-20260506 clang-20
x86_64 buildonly-randconfig-005-20260505 clang-20
x86_64 buildonly-randconfig-005-20260505 gcc-14
x86_64 buildonly-randconfig-005-20260506 clang-20
x86_64 buildonly-randconfig-005-20260506 gcc-14
x86_64 buildonly-randconfig-006-20260505 clang-20
x86_64 buildonly-randconfig-006-20260505 gcc-14
x86_64 buildonly-randconfig-006-20260506 clang-20
x86_64 defconfig gcc-14
x86_64 kexec clang-20
x86_64 randconfig-001-20260505 clang-20
x86_64 randconfig-001-20260506 clang-20
x86_64 randconfig-002-20260505 clang-20
x86_64 randconfig-002-20260506 clang-20
x86_64 randconfig-003-20260505 clang-20
x86_64 randconfig-003-20260506 clang-20
x86_64 randconfig-004-20260505 clang-20
x86_64 randconfig-004-20260506 clang-20
x86_64 randconfig-005-20260505 clang-20
x86_64 randconfig-005-20260506 clang-20
x86_64 randconfig-006-20260505 clang-20
x86_64 randconfig-006-20260506 clang-20
x86_64 randconfig-011 clang-20
x86_64 randconfig-011-20260505 clang-20
x86_64 randconfig-011-20260506 clang-20
x86_64 randconfig-012 clang-20
x86_64 randconfig-012-20260505 clang-20
x86_64 randconfig-012-20260506 clang-20
x86_64 randconfig-013 clang-20
x86_64 randconfig-013-20260505 clang-20
x86_64 randconfig-013-20260506 clang-20
x86_64 randconfig-014 clang-20
x86_64 randconfig-014-20260505 clang-20
x86_64 randconfig-014-20260506 clang-20
x86_64 randconfig-015 clang-20
x86_64 randconfig-015-20260505 clang-20
x86_64 randconfig-015-20260506 clang-20
x86_64 randconfig-015-20260506 gcc-12
x86_64 randconfig-016 clang-20
x86_64 randconfig-016-20260505 clang-20
x86_64 randconfig-016-20260506 clang-20
x86_64 randconfig-016-20260506 gcc-12
x86_64 randconfig-071 clang-20
x86_64 randconfig-071-20260505 clang-20
x86_64 randconfig-071-20260505 gcc-14
x86_64 randconfig-071-20260506 clang-20
x86_64 randconfig-072 clang-20
x86_64 randconfig-072-20260505 clang-20
x86_64 randconfig-072-20260506 clang-20
x86_64 randconfig-073 clang-20
x86_64 randconfig-073-20260505 clang-20
x86_64 randconfig-073-20260505 gcc-14
x86_64 randconfig-073-20260506 clang-20
x86_64 randconfig-074 clang-20
x86_64 randconfig-074-20260505 clang-20
x86_64 randconfig-074-20260505 gcc-14
x86_64 randconfig-074-20260506 clang-20
x86_64 randconfig-075 clang-20
x86_64 randconfig-075-20260505 clang-20
x86_64 randconfig-075-20260506 clang-20
x86_64 randconfig-076 clang-20
x86_64 randconfig-076-20260505 clang-20
x86_64 randconfig-076-20260506 clang-20
x86_64 rhel-9.4 clang-20
x86_64 rhel-9.4-bpf gcc-14
x86_64 rhel-9.4-func clang-20
x86_64 rhel-9.4-kselftests clang-20
x86_64 rhel-9.4-kunit gcc-14
x86_64 rhel-9.4-ltp gcc-14
x86_64 rhel-9.4-rust clang-20
xtensa allnoconfig clang-23
xtensa allyesconfig clang-23
xtensa randconfig-001 gcc-15.2.0
xtensa randconfig-001-20260505 gcc-15.2.0
xtensa randconfig-001-20260506 gcc-11.5.0
xtensa randconfig-002 gcc-15.2.0
xtensa randconfig-002-20260505 gcc-15.2.0
xtensa randconfig-002-20260506 gcc-11.5.0
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply
* Re: [LSF/MM/BPF TOPIC][RFC PATCH v4 00/27] Private Memory Nodes (w/ Compressed RAM)
From: Yiannis Nikolakopoulos @ 2026-05-05 22:21 UTC (permalink / raw)
To: Gregory Price
Cc: lsf-pc, linux-kernel, linux-cxl, cgroups, linux-mm,
linux-trace-kernel, damon, kernel-team, gregkh, rafael, dakr,
dave, jonathan.cameron, dave.jiang, alison.schofield,
vishal.l.verma, Ira Weiny, dan.j.williams, longman, akpm, david,
lorenzo.stoakes, Liam.Howlett, vbabka, rppt, Suren Baghdasaryan,
Michal Hocko, osalvador, ziy, matthew.brost, joshua.hahnjy,
rakie.kim, byungchul, ying.huang, apopple, axelrasmussen, yuanchu,
weixugc, yury.norov, linux, mhiramat, mathieu.desnoyers, tj,
hannes, mkoutny, jackmanb, sj, baolin.wang, npache, ryan.roberts,
dev.jain, baohua, lance.yang, muchun.song, xu.xin16,
chengming.zhou, jannh, linmiaohe, nao.horiguchi, pfalcato,
David Rientjes, shakeel.butt, riel, harry.yoo, cl, roman.gushchin,
chrisl, kasong, shikemeng, nphamcs, bhe, zhengqi.arch,
terry.bowman, Yiannis Nikolakopoulos
In-Reply-To: <20260222084842.1824063-1-gourry@gourry.net>
> On 22 Feb 2026, at 09:48, Gregory Price <gourry@gourry.net> wrote:
>
> Topic type: MM
>
> Presenter: Gregory Price <gourry@gourry.net>
>
> This series introduces N_MEMORY_PRIVATE, a NUMA node state for memory
> managed by the buddy allocator but excluded from normal allocations.
>
> I present it with an end-to-end Compressed RAM service (mm/cram.c)
> that would otherwise not be possible (or would be considerably more
> difficult, be device-specific, and add to the ZONE_DEVICE boondoggle).
>
>
> TL;DR
> ===
>
> N_MEMORY_PRIVATE is all about isolating NUMA nodes and then punching
> explicit holes in that isolation to do useful things we couldn't do
> before without re-implementing entire portions of mm/ in a driver.
>
>
> /* This is my memory. There are many like it, but this one is mine. */
> rc = add_private_memory_driver_managed(nid, start, size, name, flags,
> online_type, private_context);
>
> page = alloc_pages_node(nid, __GFP_PRIVATE, 0);
>
> /* Ok but I want to do something useful with it */
> static const struct node_private_ops ops = {
> .migrate_to = my_migrate_to,
> .folio_migrate = my_folio_migrate,
> .flags = NP_OPS_MIGRATION | NP_OPS_MEMPOLICY,
> };
> node_private_set_ops(nid, &ops);
>
> /* And now I can use mempolicy with my memory */
> buf = mmap(...);
> mbind(buf, len, mode, private_node, ...);
> buf[0] = 0xdeadbeef; /* Faults onto private node */
>
> /* And to be clear, no one else gets my memory */
> buf2 = malloc(4096); /* Standard allocation */
> buf2[0] = 0xdeadbeef; /* Can never land on private node */
>
> /* But i can choose to migrate it to the private node */
> move_pages(0, 1, &buf, &private_node, NULL, ...);
>
> /* And more fun things like this */
>
>
> Patchwork
> ===
> A fully working branch based on cxl/next can be found here:
> https://github.com/gourryinverse/linux/tree/private_compression
>
> A QEMU device which can inject high/low interrupts can be found here:
> https://github.com/gourryinverse/qemu/tree/compressed_cxl_clean
>
> The additional patches on these branches are CXL and DAX driver
> housecleaning only tangentially relevant to this RFC, so i've
> omitted them for the sake of trying to keep it somewhat clean
> here. Those patches should (hopefully) be going upstream anyway.
>
> Patches 1-22: Core Private Node Infrastructure
>
> Patch 1: Introduce N_MEMORY_PRIVATE scaffolding
> Patch 2: Introduce __GFP_PRIVATE
> Patch 3: Apply allocation isolation mechanisms
> Patch 4: Add N_MEMORY nodes to private fallback lists
> Patches 5-9: Filter operations not yet supported
> Patch 10: free_folio callback
> Patch 11: split_folio callback
> Patches 12-20: mm/ service opt-ins:
> Migration, Mempolicy, Demotion, Write Protect,
> Reclaim, OOM, NUMA Balancing, Compaction,
> LongTerm Pinning
> Patch 21: memory_failure callback
> Patch 22: Memory hotplug plumbing for private nodes
>
> Patch 23: mm/cram -- Compressed RAM Management
>
> Patches 24-27: CXL Driver examples
> Sysram Regions with Private node support
> Basic Driver Example: (MIGRATION | MEMPOLICY)
> Compression Driver Example (Generic)
>
Hi,
As I think this is about to be discussed in the conference, I thought
to share some high level comments.
I have tested this for some time on a device with compression (after some
necessary fixes for CXL RCD to work, that Greg helped me with).
Overall, the isolation property that this provides is something I deem necessary
for this technology. Others are better placed to judge the MM plumbing
itself, but I wanted to say that this functionality is an important piece of the puzzle
from the device/use-case side.
For cram itself, as it is in this RFC, I think there is still performance and
value left on the table (as noted in the description), but I fully understand Gregory’s
premise in approaching it this way.
<snip>
>
> Future CRAM : Loosening the read-only constraint
> ===
>
> The read-only model is safe but conservative. For workloads where
> compressed pages are occasionally written, the promotion fault adds
> latency. A future optimization could allow a tunable fraction of
> compressed pages to be mapped writable, accepting some risk of
> write-driven decompression in exchange for lower overhead.
>
> The private node ops make this straightforward:
>
> - Adjust fixup_migration_pte to selectively skip
> write-protection.
> - Use the backpressure system to either revoke writable mappings,
> deny additional demotions, or evict when device pressure rises.
I have some quick hacks playing with these ideas but I haven’t had the time
to test it thoroughly and get to something robust yet. I saw in another thread
that there is a follow up cooking which looks interesting.
Thanks Greg for pushing this, and I’m happy to test more on HW in our lab.
Best,
/Yiannis
^ permalink raw reply
* [GIT PULL] cgroup: Fixes for v7.1-rc2
From: Tejun Heo @ 2026-05-05 22:05 UTC (permalink / raw)
To: Linus Torvalds
Cc: linux-kernel, cgroups, Johannes Weiner, Michal Koutný,
Waiman Long
Hello,
The following changes since commit 981cd338614c96070cf9854679014fd027c1fb1d:
docs: cgroup: fix typo 'protetion' -> 'protection' (2026-04-27 07:55:40 -1000)
are available in the Git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git tags/cgroup-for-7.1-rc2-fixes
for you to fetch changes up to d8769544bde51b0ac980d10f8fe9f9fed6c95995:
docs: cgroup-v1: Update charge-commit section (2026-05-04 11:02:12 -1000)
----------------------------------------------------------------
cgroup: Fixes for v7.1-rc2
- During v6.19, cgroup task unlink was moved from do_exit() to after the
final task switch to satisfy a controller invariant. That left the kernel
seeing tasks past exit_signals() longer than userspace expected, and
several v7.0 follow-ups tried to bridge the gap by making rmdir wait for
the kernel side. None held up. The latest is an A-A deadlock when rmdir
is invoked by the reaper of zombies whose pidns teardown the rmdir itself
is waiting on, which points at the synchronizing approach being
fundamentally wrong:
- Take a different approach: drop the wait, leave rmdir's user-visible
side returning as soon as cgroup.procs is empty, and defer the css
percpu_ref kill that drives ->css_offline() until the cgroup is fully
depopulated.
- Tagged for stable. Somewhat invasive but contained. The hope is that
fixing forward sticks. If not, the fallback is to revert the entire
chain and rework on the development branch.
- Doesn't plug a pre-existing analogous race in
cgroup_apply_control_disable() (controller disable via subtree_control).
Not a regression. The development branch will do the more invasive
restructuring needed for that.
- Documentation update for cgroup-v1 charge-commit section that still
referenced functions removed when the memcg hugetlb try-commit-cancel
protocol was retired.
----------------------------------------------------------------
T.J. Mercier (1):
docs: cgroup-v1: Update charge-commit section
Tejun Heo (1):
cgroup: Defer css percpu_ref kill on rmdir until cgroup is depopulated
Documentation/admin-guide/cgroup-v1/memcg_test.rst | 6 +-
include/linux/cgroup-defs.h | 4 +-
kernel/cgroup/cgroup.c | 250 ++++++++++-----------
3 files changed, 121 insertions(+), 139 deletions(-)
Thanks.
--
tejun
^ permalink raw reply
* Re: [RFC PATCH v5 20/29] sched/deadline: Allow deeper hierarchies of RT cgroups
From: Tejun Heo @ 2026-05-05 19:56 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Yuri Andriaccio, Ingo Molnar, Juri Lelli, Vincent Guittot,
Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
Valentin Schneider, linux-kernel, Luca Abeni, Yuri Andriaccio,
hannes, mkoutny, cgroups
In-Reply-To: <20260505151523.GF3102624@noisy.programming.kicks-ass.net>
Hello,
Some high level comments:
- Please align it with existing cgroup2 interface files. See cpu.max. This
can be e.g. cpu.rt.max without about the same semantics.
- cgroup2 enforces that internal cgroups w/ controllers enabled cannot have
threads in them. No need to enforce that separately.
- However, the cpu controller is a threaded controller which means that it
can have threaded sub-hierarchy where the no-internal-process rule doesn't
apply. This was created explicitly for cpu controller. The proposed change
blocks it effectively forcing cpu controller into regular domain
controller behavior subject to no-internal-process rule. Note these are
enforced at controller granularity and this means that users who use the
threaded mode will be forced to pick between the two.
- This has the same problem with cgroup1's rt cgroup sched support where
there is no way to have a permissive default configuration, which means
that users who don't really care about distributing rt shares
hierarchically would get blocked from running rt processes by default,
which basically forces distros to disable rt cgroup sched support. This is
not new but it'd be a shame to put in all the work and the end result is
that most people don't even have access to the feature.
Here's my suggestion if there is desire for this to become something most
people have easy access to:
- Don't make it impossible to use in conjunction with other resource control
mechanisms especially not CPU controller itself. Don't force people to
choose between threaded mode and rt control. Allow them to co-exist in a
reasonable manner.
- The same in the wider scope. Don't let it get in the way of people who
don't care about it. Compromising on interface / failure mode is better
than people not being able to use it in most cases.
Thanks.
--
tejun
^ permalink raw reply
* [PATCH v6 22/22] vswap: cache cluster lookup
From: Nhat Pham @ 2026-05-05 15:38 UTC (permalink / raw)
To: kasong
Cc: Liam.Howlett, akpm, apopple, axelrasmussen, baohua, baolin.wang,
bhe, byungchul, cgroups, chengming.zhou, chrisl, corbet, david,
dev.jain, gourry, hannes, hughd, jannh, joshua.hahnjy, lance.yang,
lenb, linux-doc, linux-kernel, linux-mm, linux-pm,
lorenzo.stoakes, matthew.brost, mhocko, muchun.song, npache,
nphamcs, pavel, peterx, peterz, pfalcato, rafael, rakie.kim,
roman.gushchin, rppt, ryan.roberts, shakeel.butt, shikemeng,
surenb, tglx, vbabka, weixugc, ying.huang, yosry.ahmed, yuanchu,
zhengqi.arch, ziy, kernel-team, riel, haowenchao22
In-Reply-To: <20260505153854.1612033-1-nphamcs@gmail.com>
Looking up the vswap cluster from the global xarray on every vswap
operation shows up in profiles for swap-heavy workloads. The xa_load
cost is paid even when consecutive operations target the same cluster,
which is the common case for swap out, swap in, and swap freeing on
process teardown paths.
Inspired by how the TLB caches virtual-to-physical address
translations, add a per-CPU cache for the most recently used vswap
cluster pointer so consecutive lookups on the same cluster can skip
the xarray traversal.
The cache holds a reference on the cluster. It is cleared on:
- a lookup that targets a different cluster,
- the cached cluster becoming empty at the end of a swap operation,
- the CPU going offline.
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
mm/vswap.c | 344 ++++++++++++++++++++++++++++++++++++++++++++---------
1 file changed, 286 insertions(+), 58 deletions(-)
diff --git a/mm/vswap.c b/mm/vswap.c
index f07e6d9ec1df..f87df0f9d186 100644
--- a/mm/vswap.c
+++ b/mm/vswap.c
@@ -216,6 +216,62 @@ static DEFINE_PER_CPU(struct percpu_vswap_cluster, percpu_vswap_cluster) = {
.lock = INIT_LOCAL_LOCK(),
};
+/*
+ * Per-CPU cache of the most recently used cluster during vswap_iter().
+ * This allows us to skip the xarray lookup if the same cluster is being
+ * iterated over again. We increment a reference to the cluster when it
+ * is cached here to keep the cluster alive.
+ */
+struct percpu_vswap_iter_cache {
+ struct vswap_cluster *cluster;
+ local_lock_t lock;
+};
+
+static DEFINE_PER_CPU(struct percpu_vswap_iter_cache, percpu_vswap_iter_cache) = {
+ .cluster = NULL,
+ .lock = INIT_LOCAL_LOCK(),
+};
+
+static inline struct vswap_cluster *read_cached_vswap_cluster(void)
+{
+ struct vswap_cluster *cached;
+
+ /*
+ * The returned pointer (if non-NULL) is only protected by RCU. The per-CPU
+ * cache holds a +1 refcount on its cached cluster, but another context may
+ * evict the cache between this read and the caller's use of the pointer:
+ * the refcount can drop to 0, vswap_cluster_free() may queue the cluster
+ * for kvfree_rcu(), and the per-CPU slot may even be replaced by a
+ * different cluster. Because the caller holds rcu_read_lock(), the
+ * kvfree_rcu() grace period cannot complete and the returned pointer
+ * remains valid memory throughout the RCU critical section. Pointer
+ * comparisons (e.g. inside try_to_clear_local_vswap_iter_cache()) stay
+ * meaningful for the same reason.
+ */
+ RCU_LOCKDEP_WARN(!rcu_read_lock_held(),
+ "must be called under rcu_read_lock()");
+
+ local_lock(&percpu_vswap_iter_cache.lock);
+ cached = this_cpu_read(percpu_vswap_iter_cache.cluster);
+ local_unlock(&percpu_vswap_iter_cache.lock);
+
+ return cached;
+}
+
+static inline struct vswap_cluster *read_cached_vswap_cluster_irq(void)
+{
+ struct vswap_cluster *cached;
+
+ RCU_LOCKDEP_WARN(!rcu_read_lock_held(),
+ "must be called under rcu_read_lock()");
+
+ local_lock_irq(&percpu_vswap_iter_cache.lock);
+ cached = this_cpu_read(percpu_vswap_iter_cache.cluster);
+ local_unlock_irq(&percpu_vswap_iter_cache.lock);
+
+ return cached;
+}
+
static atomic_t vswap_alloc_reject;
static atomic_t vswap_used;
@@ -261,28 +317,74 @@ static struct swp_desc *__vswap_iter(struct vswap_cluster *cluster, unsigned lon
return NULL;
}
+static void vswap_cluster_put(struct vswap_cluster *cluster, int nr);
+
+static void try_to_clear_local_vswap_iter_cache(struct vswap_cluster *cluster)
+{
+ lockdep_assert_held(&cluster->lock);
+ local_lock(&percpu_vswap_iter_cache.lock);
+ if (this_cpu_read(percpu_vswap_iter_cache.cluster) == cluster) {
+ this_cpu_write(percpu_vswap_iter_cache.cluster, NULL);
+ vswap_cluster_put(cluster, 1);
+ }
+ local_unlock(&percpu_vswap_iter_cache.lock);
+}
+
+static void try_to_cache_local_vswap_iter_cache(struct vswap_cluster *cluster)
+{
+ lockdep_assert_held(&cluster->lock);
+ local_lock(&percpu_vswap_iter_cache.lock);
+ if (!this_cpu_read(percpu_vswap_iter_cache.cluster)) {
+ this_cpu_write(percpu_vswap_iter_cache.cluster, cluster);
+ refcount_inc(&cluster->refcnt);
+ }
+ local_unlock(&percpu_vswap_iter_cache.lock);
+}
+
static struct swp_desc *vswap_iter(struct vswap_cluster **clusterp, unsigned long i)
{
unsigned long cluster_id = VSWAP_VAL_CLUSTER_IDX(i);
struct vswap_cluster *cluster = *clusterp;
+ struct vswap_cluster *cached = NULL;
struct swp_desc *desc = NULL;
unsigned long slot_index;
+ bool need_clear_cache = false;
if (!cluster || cluster_id != cluster->id) {
if (cluster)
spin_unlock(&cluster->lock);
- cluster = xa_load(&vswap_cluster_map, cluster_id);
+
+ cached = read_cached_vswap_cluster();
+
+ if (cached && cached->id == cluster_id) {
+ cluster = cached;
+ } else {
+ need_clear_cache = (cached != NULL);
+ cluster = xa_load(&vswap_cluster_map, cluster_id);
+ }
+
+ if (need_clear_cache) {
+ spin_lock(&cached->lock);
+ try_to_clear_local_vswap_iter_cache(cached);
+ spin_unlock(&cached->lock);
+ }
+
if (!cluster)
goto done;
VM_WARN_ON(cluster->id != cluster_id);
spin_lock(&cluster->lock);
+ } else {
+ cached = cluster;
}
slot_index = VSWAP_IDX_WITHIN_CLUSTER_VAL(i);
- if (test_bit(slot_index, cluster->bitmap))
+ if (test_bit(slot_index, cluster->bitmap)) {
desc = &cluster->descriptors[slot_index];
-
- if (!desc) {
+ if (cached != cluster)
+ try_to_cache_local_vswap_iter_cache(cluster);
+ } else {
+ if (!cluster->count)
+ try_to_clear_local_vswap_iter_cache(cluster);
spin_unlock(&cluster->lock);
cluster = NULL;
}
@@ -292,6 +394,17 @@ static struct swp_desc *vswap_iter(struct vswap_cluster **clusterp, unsigned lon
return desc;
}
+static void vswap_iter_break(struct vswap_cluster *cluster)
+{
+ if (!cluster)
+ return;
+
+ if (!cluster->count)
+ try_to_clear_local_vswap_iter_cache(cluster);
+
+ spin_unlock(&cluster->lock);
+}
+
static bool cluster_is_alloc_candidate(struct vswap_cluster *cluster)
{
return cluster->count + (1 << (cluster->order)) <= VSWAP_CLUSTER_SIZE;
@@ -412,7 +525,7 @@ int folio_alloc_swap(struct folio *folio)
cluster = kmem_cache_zalloc(vswap_cluster_cache, GFP_KERNEL);
if (cluster) {
cluster->descriptors = vzalloc(array_size(VSWAP_CLUSTER_SIZE,
- sizeof(struct swp_desc)));
+ sizeof(struct swp_desc)));
if (!cluster->descriptors) {
kmem_cache_free(vswap_cluster_cache, cluster);
cluster = NULL;
@@ -529,6 +642,13 @@ static void vswap_cluster_free(struct vswap_cluster *cluster)
call_rcu(&cluster->rcu, vswap_cluster_free_rcu);
}
+static void vswap_cluster_put(struct vswap_cluster *cluster, int nr)
+{
+ lockdep_assert_held(&cluster->lock);
+ if (refcount_sub_and_test(nr, &cluster->refcnt))
+ vswap_cluster_free(cluster);
+}
+
static inline void release_vswap_slot_nr(struct vswap_cluster *cluster,
unsigned long index, int nr)
{
@@ -819,7 +939,7 @@ swp_slot_t vswap_alloc_swap_slot(struct folio *folio)
desc->type = VSWAP_SWAPFILE;
desc->slot.val = slot.val + i;
}
- spin_unlock(&cluster->lock);
+ vswap_iter_break(cluster);
return slot;
}
@@ -854,7 +974,7 @@ swp_slot_t swp_entry_to_swp_slot(swp_entry_t entry)
slot.val = 0;
else
slot = desc->slot;
- spin_unlock(&cluster->lock);
+ vswap_iter_break(cluster);
rcu_read_unlock();
return slot;
@@ -946,8 +1066,7 @@ static bool vswap_free_nr_any_cache_only(swp_entry_t entry, int nr)
}
if (free_nr)
vswap_free_nr(cluster, free_start, free_nr);
- if (cluster)
- spin_unlock(&cluster->lock);
+ vswap_iter_break(cluster);
rcu_read_unlock();
return ret;
}
@@ -979,8 +1098,7 @@ static int swap_duplicate_nr(swp_entry_t entry, int nr)
desc->swap_count++;
}
done:
- if (cluster)
- spin_unlock(&cluster->lock);
+ vswap_iter_break(cluster);
rcu_read_unlock();
if (i && i < nr)
swap_free_nr(entry, i);
@@ -1008,7 +1126,7 @@ int swap_duplicate(swp_entry_t entry)
*/
bool folio_free_swap(struct folio *folio)
{
- struct vswap_cluster *cluster;
+ struct vswap_cluster *cached, *cluster;
swp_entry_t entry = folio->swap;
int i, nr = folio_nr_pages(folio);
unsigned long cluster_id;
@@ -1028,8 +1146,21 @@ bool folio_free_swap(struct folio *folio)
cluster_id = VSWAP_CLUSTER_IDX(entry);
rcu_read_lock();
- cluster = xa_load(&vswap_cluster_map, cluster_id);
+
+ cached = read_cached_vswap_cluster_irq();
+
+ if (!cached || cached->id != cluster_id)
+ cluster = xa_load(&vswap_cluster_map, cluster_id);
+ else
+ cluster = cached;
VM_WARN_ON(!cluster);
+
+ if (cached && cached != cluster) {
+ spin_lock_irq(&cached->lock);
+ try_to_clear_local_vswap_iter_cache(cached);
+ spin_unlock_irq(&cached->lock);
+ }
+
spin_lock_irq(&cluster->lock);
/* check if any PTE still points to the swap entries */
@@ -1050,6 +1181,8 @@ bool folio_free_swap(struct folio *folio)
}
vswap_free_nr(cluster, entry, nr);
+ if (!cluster->count)
+ try_to_clear_local_vswap_iter_cache(cluster);
spin_unlock_irq(&cluster->lock);
rcu_read_unlock();
@@ -1080,8 +1213,7 @@ int swp_swapcount(swp_entry_t entry)
rcu_read_lock();
desc = vswap_iter(&cluster, entry.val);
ret = desc ? desc->swap_count : 0;
- if (cluster)
- spin_unlock(&cluster->lock);
+ vswap_iter_break(cluster);
rcu_read_unlock();
return ret;
@@ -1142,7 +1274,7 @@ void swapcache_clear(swp_entry_t entry, int nr)
desc = vswap_iter(&cluster, entry.val);
VM_WARN_ON(!desc);
__swapcache_clear(cluster, entry, nr);
- spin_unlock(&cluster->lock);
+ vswap_iter_break(cluster);
rcu_read_unlock();
}
@@ -1174,8 +1306,7 @@ int swapcache_prepare(swp_entry_t entry, int nr)
desc->in_swapcache = true;
}
done:
- if (cluster)
- spin_unlock(&cluster->lock);
+ vswap_iter_break(cluster);
rcu_read_unlock();
if (i && i < nr)
swapcache_clear(entry, i);
@@ -1199,8 +1330,7 @@ bool is_swap_cached(swp_entry_t entry)
rcu_read_lock();
desc = vswap_iter(&cluster, entry.val);
cached = desc ? desc->in_swapcache : false;
- if (cluster)
- spin_unlock(&cluster->lock);
+ vswap_iter_break(cluster);
rcu_read_unlock();
return cached;
@@ -1231,8 +1361,7 @@ int non_swapcache_batch(swp_entry_t entry, int max_nr)
goto done;
}
done:
- if (cluster)
- spin_unlock(&cluster->lock);
+ vswap_iter_break(cluster);
rcu_read_unlock();
return i;
}
@@ -1268,7 +1397,7 @@ void vswap_store_folio(swp_entry_t entry, struct folio *folio)
desc->type = VSWAP_FOLIO;
desc->swap_cache = folio;
}
- spin_unlock(&cluster->lock);
+ vswap_iter_break(cluster);
rcu_read_unlock();
}
@@ -1349,7 +1478,7 @@ void swap_zeromap_folio_set(struct folio *folio)
VM_WARN_ON(!desc);
desc->type = VSWAP_ZERO;
}
- spin_unlock(&cluster->lock);
+ vswap_iter_break(cluster);
rcu_read_unlock();
count_vm_events(SWPOUT_ZERO, nr);
@@ -1419,8 +1548,7 @@ static int vswap_check_backing(swp_entry_t entry, enum swap_type *type, int nr,
/* Caller takes ownership of the lock and rcu_read_lock */
*clusterp = cluster;
} else {
- if (cluster)
- spin_unlock(&cluster->lock);
+ vswap_iter_break(cluster);
rcu_read_unlock();
}
if (type)
@@ -1498,8 +1626,7 @@ int swap_zeromap_batch(swp_entry_t entry, int max_nr, bool *is_zeromap)
goto done;
}
done:
- if (cluster)
- spin_unlock(&cluster->lock);
+ vswap_iter_break(cluster);
rcu_read_unlock();
if (i && is_zeromap)
*is_zeromap = is_zero;
@@ -1614,9 +1741,9 @@ void put_swap_entry(swp_entry_t entry, struct swap_info_struct *si)
rcu_read_lock();
cluster = xa_load(&vswap_cluster_map, VSWAP_CLUSTER_IDX(entry));
+ VM_WARN_ON(!cluster);
spin_lock(&cluster->lock);
- if (refcount_dec_and_test(&cluster->refcnt))
- vswap_cluster_free(cluster);
+ vswap_cluster_put(cluster, 1);
spin_unlock(&cluster->lock);
rcu_read_unlock();
}
@@ -1637,14 +1764,26 @@ static int vswap_cpu_dead(unsigned int cpu)
int order;
guard(rcu)();
+
+ /*
+ * No need to take local_lock for the dead CPU's per-CPU caches,
+ * as no local modifications can happen on a dead CPU.
+ */
+ cluster = per_cpu(percpu_vswap_iter_cache.cluster, cpu);
+ if (cluster) {
+ spin_lock(&cluster->lock);
+ per_cpu(percpu_vswap_iter_cache.cluster, cpu) = NULL;
+ vswap_cluster_put(cluster, 1);
+ spin_unlock(&cluster->lock);
+ }
+
for (order = 0; order < SWAP_NR_ORDERS; order++) {
cluster = per_cpu(percpu_vswap_cluster.clusters[order], cpu);
if (cluster) {
per_cpu(percpu_vswap_cluster.clusters[order], cpu) = NULL;
spin_lock(&cluster->lock);
cluster->cached = false;
- if (refcount_dec_and_test(&cluster->refcnt))
- vswap_cluster_free(cluster);
+ vswap_cluster_put(cluster, 1);
spin_unlock(&cluster->lock);
}
}
@@ -1660,13 +1799,26 @@ static int vswap_cpu_dead(unsigned int cpu)
*/
void swap_cache_lock(swp_entry_t entry)
{
- struct vswap_cluster *cluster;
+ struct vswap_cluster *cached, *cluster;
unsigned long cluster_id = VSWAP_CLUSTER_IDX(entry);
rcu_read_lock();
- cluster = xa_load(&vswap_cluster_map, cluster_id);
+ cached = read_cached_vswap_cluster();
+
+ if (!cached || cached->id != cluster_id)
+ cluster = xa_load(&vswap_cluster_map, cluster_id);
+ else
+ cluster = cached;
VM_WARN_ON(!cluster);
+
+ if (cached && cached != cluster) {
+ spin_lock(&cached->lock);
+ try_to_clear_local_vswap_iter_cache(cached);
+ spin_unlock(&cached->lock);
+ }
+
spin_lock(&cluster->lock);
+ try_to_cache_local_vswap_iter_cache(cluster);
rcu_read_unlock();
}
@@ -1678,12 +1830,20 @@ void swap_cache_lock(swp_entry_t entry)
*/
void swap_cache_unlock(swp_entry_t entry)
{
- struct vswap_cluster *cluster;
+ struct vswap_cluster *cached, *cluster;
unsigned long cluster_id = VSWAP_CLUSTER_IDX(entry);
rcu_read_lock();
- cluster = xa_load(&vswap_cluster_map, cluster_id);
+ cached = read_cached_vswap_cluster();
+
+ if (!cached || cached->id != cluster_id)
+ cluster = xa_load(&vswap_cluster_map, cluster_id);
+ else {
+ cluster = cached;
+ try_to_clear_local_vswap_iter_cache(cached);
+ }
VM_WARN_ON(!cluster);
+
spin_unlock(&cluster->lock);
rcu_read_unlock();
}
@@ -1696,13 +1856,27 @@ void swap_cache_unlock(swp_entry_t entry)
*/
void swap_cache_lock_irq(swp_entry_t entry)
{
- struct vswap_cluster *cluster;
+ struct vswap_cluster *cached, *cluster;
unsigned long cluster_id = VSWAP_CLUSTER_IDX(entry);
rcu_read_lock();
- cluster = xa_load(&vswap_cluster_map, cluster_id);
+
+ cached = read_cached_vswap_cluster_irq();
+
+ if (!cached || cached->id != cluster_id)
+ cluster = xa_load(&vswap_cluster_map, cluster_id);
+ else
+ cluster = cached;
VM_WARN_ON(!cluster);
+
+ if (cached && cached != cluster) {
+ spin_lock_irq(&cached->lock);
+ try_to_clear_local_vswap_iter_cache(cached);
+ spin_unlock_irq(&cached->lock);
+ }
+
spin_lock_irq(&cluster->lock);
+ try_to_cache_local_vswap_iter_cache(cluster);
rcu_read_unlock();
}
@@ -1714,11 +1888,18 @@ void swap_cache_lock_irq(swp_entry_t entry)
*/
void swap_cache_unlock_irq(swp_entry_t entry)
{
- struct vswap_cluster *cluster;
+ struct vswap_cluster *cached, *cluster;
unsigned long cluster_id = VSWAP_CLUSTER_IDX(entry);
rcu_read_lock();
- cluster = xa_load(&vswap_cluster_map, cluster_id);
+ cached = read_cached_vswap_cluster();
+
+ if (!cached || cached->id != cluster_id)
+ cluster = xa_load(&vswap_cluster_map, cluster_id);
+ else {
+ cluster = cached;
+ try_to_clear_local_vswap_iter_cache(cluster);
+ }
VM_WARN_ON(!cluster);
spin_unlock_irq(&cluster->lock);
rcu_read_unlock();
@@ -1746,24 +1927,24 @@ static struct folio *vswap_get_swap_cache(swp_entry_t entry, bool swap_cache_onl
if (!desc->in_swapcache ||
xa_is_value(desc->shadow) ||
(swap_cache_only && desc->swap_count)) {
- spin_unlock(&cluster->lock);
+ vswap_iter_break(cluster);
rcu_read_unlock();
return NULL;
}
folio = desc->swap_cache;
if (!folio) {
- spin_unlock(&cluster->lock);
+ vswap_iter_break(cluster);
rcu_read_unlock();
return NULL;
}
if (likely(folio_try_get(folio))) {
- spin_unlock(&cluster->lock);
+ vswap_iter_break(cluster);
rcu_read_unlock();
return folio;
}
- spin_unlock(&cluster->lock);
+ vswap_iter_break(cluster);
cluster = NULL;
rcu_read_unlock();
}
@@ -1812,7 +1993,7 @@ void *swap_cache_get_shadow(swp_entry_t entry)
}
shadow = desc->shadow;
- spin_unlock(&cluster->lock);
+ vswap_iter_break(cluster);
rcu_read_unlock();
if (xa_is_value(shadow))
@@ -1834,7 +2015,7 @@ void *swap_cache_get_shadow(swp_entry_t entry)
*/
void swap_cache_add_folio(struct folio *folio, swp_entry_t entry, void **shadowp)
{
- struct vswap_cluster *cluster;
+ struct vswap_cluster *cached, *cluster;
unsigned long nr_pages = folio_nr_pages(folio);
unsigned long cluster_id = VSWAP_CLUSTER_IDX(entry);
unsigned long i;
@@ -1850,9 +2031,23 @@ void swap_cache_add_folio(struct folio *folio, swp_entry_t entry, void **shadowp
folio->swap = entry;
rcu_read_lock();
- cluster = xa_load(&vswap_cluster_map, cluster_id);
+
+ cached = read_cached_vswap_cluster_irq();
+
+ if (!cached || cached->id != cluster_id)
+ cluster = xa_load(&vswap_cluster_map, cluster_id);
+ else
+ cluster = cached;
VM_WARN_ON(!cluster);
+
+ if (cached && cached != cluster) {
+ spin_lock_irq(&cached->lock);
+ try_to_clear_local_vswap_iter_cache(cached);
+ spin_unlock_irq(&cached->lock);
+ }
+
spin_lock_irq(&cluster->lock);
+ try_to_cache_local_vswap_iter_cache(cluster);
for (i = 0; i < nr_pages; i++) {
desc = __vswap_iter(cluster, entry.val + i);
@@ -1892,7 +2087,7 @@ void swap_cache_add_folio(struct folio *folio, swp_entry_t entry, void **shadowp
void __vswap_remove_mapping(struct folio *folio, swp_entry_t entry, void *shadow)
{
long nr_pages = folio_nr_pages(folio);
- struct vswap_cluster *cluster;
+ struct vswap_cluster *cached, *cluster;
unsigned long cluster_id = VSWAP_CLUSTER_IDX(entry);
struct swp_desc *desc;
int i;
@@ -1902,7 +2097,18 @@ void __vswap_remove_mapping(struct folio *folio, swp_entry_t entry, void *shadow
VM_WARN_ON_ONCE_FOLIO(folio_test_writeback(folio), folio);
rcu_read_lock();
- cluster = xa_load(&vswap_cluster_map, cluster_id);
+
+ /*
+ * The caller has already taken the cluster lock via
+ * swap_cache_lock_irq(), which consults and updates the per-CPU
+ * cluster cache. Read from the cache to avoid a redundant
+ * xa_load() on the reclaim hot path.
+ */
+ cached = read_cached_vswap_cluster();
+ if (cached && cached->id == cluster_id)
+ cluster = cached;
+ else
+ cluster = xa_load(&vswap_cluster_map, cluster_id);
VM_WARN_ON(!cluster);
/* Set shadow on all descriptors */
@@ -1914,6 +2120,8 @@ void __vswap_remove_mapping(struct folio *folio, swp_entry_t entry, void *shadow
memcg1_swapout(folio, entry);
__swapcache_clear(cluster, entry, nr_pages);
+ if (cached == cluster && !cluster->count)
+ try_to_clear_local_vswap_iter_cache(cluster);
spin_unlock_irq(&cluster->lock);
rcu_read_unlock();
@@ -1938,7 +2146,7 @@ void swap_cache_del_folio(struct folio *folio)
{
long nr_pages = folio_nr_pages(folio);
swp_entry_t entry = folio->swap;
- struct vswap_cluster *cluster;
+ struct vswap_cluster *cached, *cluster;
unsigned long cluster_id = VSWAP_CLUSTER_IDX(entry);
struct swp_desc *desc;
int i;
@@ -1948,9 +2156,23 @@ void swap_cache_del_folio(struct folio *folio)
VM_WARN_ON_ONCE_FOLIO(folio_test_writeback(folio), folio);
rcu_read_lock();
- cluster = xa_load(&vswap_cluster_map, cluster_id);
+
+ cached = read_cached_vswap_cluster_irq();
+
+ if (!cached || cached->id != cluster_id)
+ cluster = xa_load(&vswap_cluster_map, cluster_id);
+ else
+ cluster = cached;
VM_WARN_ON(!cluster);
+
+ if (cached && cached != cluster) {
+ spin_lock_irq(&cached->lock);
+ try_to_clear_local_vswap_iter_cache(cached);
+ spin_unlock_irq(&cached->lock);
+ }
+
spin_lock_irq(&cluster->lock);
+ try_to_cache_local_vswap_iter_cache(cluster);
for (i = 0; i < nr_pages; i++) {
desc = __vswap_iter(cluster, entry.val + i);
@@ -1958,6 +2180,7 @@ void swap_cache_del_folio(struct folio *folio)
desc->shadow = NULL;
}
+ try_to_clear_local_vswap_iter_cache(cluster);
spin_unlock_irq(&cluster->lock);
rcu_read_unlock();
@@ -1987,7 +2210,7 @@ void __swap_cache_replace_folio(struct folio *old, struct folio *new)
{
swp_entry_t entry = new->swap;
unsigned long nr_pages = folio_nr_pages(new);
- struct vswap_cluster *cluster;
+ struct vswap_cluster *cached, *cluster;
struct swp_desc *desc;
unsigned long cluster_id = VSWAP_CLUSTER_IDX(entry);
void *old_entry;
@@ -1998,7 +2221,13 @@ void __swap_cache_replace_folio(struct folio *old, struct folio *new)
VM_WARN_ON_ONCE(!entry.val);
rcu_read_lock();
- cluster = xa_load(&vswap_cluster_map, cluster_id);
+
+ cached = read_cached_vswap_cluster();
+
+ if (cached && cached->id == cluster_id)
+ cluster = cached;
+ else
+ cluster = xa_load(&vswap_cluster_map, cluster_id);
VM_WARN_ON(!cluster);
for (i = 0; i < nr_pages; i++) {
@@ -2031,7 +2260,7 @@ void zswap_entry_store(swp_entry_t swpentry, struct zswap_entry *entry)
release_backing(cluster, swpentry, 1, NULL);
desc->zswap_entry = entry;
desc->type = VSWAP_ZSWAP;
- spin_unlock(&cluster->lock);
+ vswap_iter_break(cluster);
rcu_read_unlock();
}
@@ -2059,7 +2288,7 @@ void *zswap_entry_load(swp_entry_t swpentry)
type = desc->type;
zswap_entry = desc->zswap_entry;
- spin_unlock(&cluster->lock);
+ vswap_iter_break(cluster);
rcu_read_unlock();
if (type != VSWAP_ZSWAP)
@@ -2130,7 +2359,7 @@ static unsigned short vswap_cgroup_record(swp_entry_t entry,
desc = vswap_iter(&cluster, entry.val);
VM_WARN_ON(!desc);
oldid = __vswap_cgroup_record(cluster, entry, memcgid, nr_ents);
- spin_unlock(&cluster->lock);
+ vswap_iter_break(cluster);
rcu_read_unlock();
return oldid;
@@ -2199,8 +2428,7 @@ unsigned short lookup_swap_cgroup_id(swp_entry_t entry)
rcu_read_lock();
desc = vswap_iter(&cluster, entry.val);
ret = desc ? desc->memcgid : 0;
- if (cluster)
- spin_unlock(&cluster->lock);
+ vswap_iter_break(cluster);
rcu_read_unlock();
return ret;
}
--
2.52.0
^ permalink raw reply related
* [PATCH v6 21/22] vswap: batch contiguous vswap free calls
From: Nhat Pham @ 2026-05-05 15:38 UTC (permalink / raw)
To: kasong
Cc: Liam.Howlett, akpm, apopple, axelrasmussen, baohua, baolin.wang,
bhe, byungchul, cgroups, chengming.zhou, chrisl, corbet, david,
dev.jain, gourry, hannes, hughd, jannh, joshua.hahnjy, lance.yang,
lenb, linux-doc, linux-kernel, linux-mm, linux-pm,
lorenzo.stoakes, matthew.brost, mhocko, muchun.song, npache,
nphamcs, pavel, peterx, peterz, pfalcato, rafael, rakie.kim,
roman.gushchin, rppt, ryan.roberts, shakeel.butt, shikemeng,
surenb, tglx, vbabka, weixugc, ying.huang, yosry.ahmed, yuanchu,
zhengqi.arch, ziy, kernel-team, riel, haowenchao22
In-Reply-To: <20260505153854.1612033-1-nphamcs@gmail.com>
In vswap_free(), we release and reacquire the cluster lock for every
single entry, even for non-disk-swap backends where the lock drop is
unnecessary. Batch consecutive free operations to avoid this overhead.
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
mm/vswap.c | 97 ++++++++++++++++++++++++++++++++++--------------------
1 file changed, 61 insertions(+), 36 deletions(-)
diff --git a/mm/vswap.c b/mm/vswap.c
index 3f86bbb3a5ea..f07e6d9ec1df 100644
--- a/mm/vswap.c
+++ b/mm/vswap.c
@@ -529,18 +529,18 @@ static void vswap_cluster_free(struct vswap_cluster *cluster)
call_rcu(&cluster->rcu, vswap_cluster_free_rcu);
}
-static inline void release_vswap_slot(struct vswap_cluster *cluster,
- unsigned long index)
+static inline void release_vswap_slot_nr(struct vswap_cluster *cluster,
+ unsigned long index, int nr)
{
unsigned long slot_index = VSWAP_IDX_WITHIN_CLUSTER_VAL(index);
lockdep_assert_held(&cluster->lock);
- cluster->count--;
+ cluster->count -= nr;
- bitmap_clear(cluster->bitmap, slot_index, 1);
+ bitmap_clear(cluster->bitmap, slot_index, nr);
/* we only free uncached empty clusters */
- if (refcount_dec_and_test(&cluster->refcnt))
+ if (refcount_sub_and_test(nr, &cluster->refcnt))
vswap_cluster_free(cluster);
else if (cluster->full && cluster_is_alloc_candidate(cluster)) {
cluster->full = false;
@@ -553,7 +553,7 @@ static inline void release_vswap_slot(struct vswap_cluster *cluster,
}
}
- atomic_dec(&vswap_used);
+ atomic_sub(nr, &vswap_used);
}
/*
@@ -585,7 +585,7 @@ static unsigned short swp_desc_memcgid(struct swp_desc *desc);
*
* 1. Callers ensure no concurrent modification of the swap entry's internal
* state can occur. This is guaranteed by one of the following:
- * - For vswap_free() callers: the swap entry's refcnt (swap count and
+ * - For vswap_free_nr() callers: the swap entry's refcnt (swap count and
* swapcache pin) is down to 0.
* - For vswap_store_folio(), swap_zeromap_folio_set(), and zswap_entry_store()
* callers: the folio is locked and in the swap cache.
@@ -706,26 +706,17 @@ static void __vswap_swap_cgroup_clear(struct vswap_cluster *cluster,
/*
* Entered with the cluster locked. The cluster lock is held throughout.
- *
- * This is safe, because:
- *
- * 1. The swap entry to be freed has refcnt (swap count and swapcache pin)
- * down to 0, so no one can change its internal state.
- *
- * 2. The swap entry to be freed still holds a refcnt to the cluster, keeping
- * the cluster itself valid.
- *
- * 3. swap_slot_free_nr() takes the physical swap cluster lock (ci->lock),
- * but the only vswap function called under ci->lock is vswap_rmap_set(),
- * which uses atomic ops and does not take cluster->lock. So there is no
- * ABBA deadlock risk.
*/
-static void vswap_free(struct vswap_cluster *cluster, struct swp_desc *desc,
- swp_entry_t entry)
+static void vswap_free_nr(struct vswap_cluster *cluster, swp_entry_t entry,
+ int nr)
{
- unsigned short id = swp_desc_memcgid(desc);
+ struct swp_desc *desc = __vswap_iter(cluster, entry.val);
+ unsigned short id;
struct mem_cgroup *memcg;
+ VM_WARN_ON(!desc);
+ id = swp_desc_memcgid(desc);
+
/*
* The swap_cgroup id reference taken at swapout time pins this
* memcg until swap_cgroup_clear() runs below, so we can resolve
@@ -733,11 +724,11 @@ static void vswap_free(struct vswap_cluster *cluster, struct swp_desc *desc,
*/
memcg = id ? mem_cgroup_from_id(id) : NULL;
- release_backing(cluster, entry, 1, memcg);
- __vswap_swap_cgroup_clear(cluster, entry, 1, memcg);
+ release_backing(cluster, entry, nr, memcg);
+ __vswap_swap_cgroup_clear(cluster, entry, nr, memcg);
- /* erase forward mapping and release the virtual slot for reallocation */
- release_vswap_slot(cluster, entry.val);
+ /* erase forward mapping and release the virtual slots for reallocation */
+ release_vswap_slot_nr(cluster, entry.val, nr);
}
@@ -908,10 +899,18 @@ static bool vswap_free_nr_any_cache_only(swp_entry_t entry, int nr)
struct vswap_cluster *cluster = NULL;
struct swp_desc *desc;
bool ret = false;
- int i;
+ swp_entry_t free_start;
+ unsigned short batch_memcgid = 0;
+ int i, free_nr = 0;
+ free_start.val = 0;
rcu_read_lock();
for (i = 0; i < nr; i++) {
+ /* flush pending free batch at cluster boundary */
+ if (free_nr && !VSWAP_IDX_WITHIN_CLUSTER_VAL(entry.val)) {
+ vswap_free_nr(cluster, free_start, free_nr);
+ free_nr = 0;
+ }
desc = vswap_iter(&cluster, entry.val);
VM_WARN_ON(!desc);
ret |= (desc->swap_count == 1 && desc->in_swapcache);
@@ -919,18 +918,34 @@ static bool vswap_free_nr_any_cache_only(swp_entry_t entry, int nr)
if (!desc->swap_count && !desc->in_swapcache) {
if (xa_is_value(desc->shadow))
desc->shadow = NULL;
- vswap_free(cluster, desc, entry);
- } else if (!desc->swap_count && desc->in_swapcache &&
- desc->type == VSWAP_SWAPFILE) {
+ /* flush at cgroup boundary */
+ if (free_nr &&
+ swp_desc_memcgid(desc) != batch_memcgid) {
+ vswap_free_nr(cluster, free_start, free_nr);
+ free_nr = 0;
+ }
+ if (!free_nr)
+ batch_memcgid = swp_desc_memcgid(desc);
+ if (!free_nr++)
+ free_start = entry;
+ } else {
+ if (free_nr) {
+ vswap_free_nr(cluster, free_start, free_nr);
+ free_nr = 0;
+ }
/*
* swap_count just dropped to 0, but still in swap
* cache. If backed by a physical swap slot, mark it
* so the physical swap allocator can check cheaply.
*/
- swap_rmap_mark_cache_only(desc->slot);
+ if (!desc->swap_count && desc->in_swapcache &&
+ desc->type == VSWAP_SWAPFILE)
+ swap_rmap_mark_cache_only(desc->slot);
}
entry.val++;
}
+ if (free_nr)
+ vswap_free_nr(cluster, free_start, free_nr);
if (cluster)
spin_unlock(&cluster->lock);
rcu_read_unlock();
@@ -1032,8 +1047,9 @@ bool folio_free_swap(struct folio *folio)
VM_WARN_ON_FOLIO(!desc || desc->swap_cache != folio, folio);
desc->swap_cache = NULL;
desc->in_swapcache = false;
- vswap_free(cluster, desc, (swp_entry_t){ entry.val + i });
}
+
+ vswap_free_nr(cluster, entry, nr);
spin_unlock_irq(&cluster->lock);
rcu_read_unlock();
@@ -1095,14 +1111,23 @@ static void __swapcache_clear(struct vswap_cluster *cluster,
swp_entry_t entry, int nr)
{
struct swp_desc *desc;
- int i;
+ swp_entry_t free_start;
+ int i, free_nr = 0;
+ free_start = entry;
for (i = 0; i < nr; i++) {
desc = __vswap_iter(cluster, entry.val + i);
desc->in_swapcache = false;
- if (!desc->swap_count)
- vswap_free(cluster, desc, (swp_entry_t){ entry.val + i });
+ if (!desc->swap_count) {
+ if (!free_nr++)
+ free_start.val = entry.val + i;
+ } else if (free_nr) {
+ vswap_free_nr(cluster, free_start, free_nr);
+ free_nr = 0;
+ }
}
+ if (free_nr)
+ vswap_free_nr(cluster, free_start, free_nr);
}
void swapcache_clear(swp_entry_t entry, int nr)
--
2.52.0
^ permalink raw reply related
* [PATCH v6 20/22] swapfile: replace the swap map with bitmaps
From: Nhat Pham @ 2026-05-05 15:38 UTC (permalink / raw)
To: kasong
Cc: Liam.Howlett, akpm, apopple, axelrasmussen, baohua, baolin.wang,
bhe, byungchul, cgroups, chengming.zhou, chrisl, corbet, david,
dev.jain, gourry, hannes, hughd, jannh, joshua.hahnjy, lance.yang,
lenb, linux-doc, linux-kernel, linux-mm, linux-pm,
lorenzo.stoakes, matthew.brost, mhocko, muchun.song, npache,
nphamcs, pavel, peterx, peterz, pfalcato, rafael, rakie.kim,
roman.gushchin, rppt, ryan.roberts, shakeel.butt, shikemeng,
surenb, tglx, vbabka, weixugc, ying.huang, yosry.ahmed, yuanchu,
zhengqi.arch, ziy, kernel-team, riel, haowenchao22
In-Reply-To: <20260505153854.1612033-1-nphamcs@gmail.com>
Now that we have moved the swap count state to virtual swap layer, each
swap map entry only has 3 possible states: free, allocated, and bad.
Replace the swap map with 2 bitmaps (one for allocated state and one for
bad state), saving 6 bits per swap entry.
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
include/linux/swap.h | 3 +-
mm/swapfile.c | 81 +++++++++++++++++++++++---------------------
2 files changed, 44 insertions(+), 40 deletions(-)
diff --git a/include/linux/swap.h b/include/linux/swap.h
index ad5f59c807c6..aac9971633f2 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -259,7 +259,8 @@ struct swap_info_struct {
struct plist_node list; /* entry in swap_active_head */
signed char type; /* strange name for an index */
unsigned int max; /* extent of the swap_map */
- unsigned char *swap_map; /* vmalloc'ed array of usage counts */
+ unsigned long *swap_map; /* bitmap for allocated state */
+ unsigned long *bad_map; /* bitmap for bad state */
struct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */
struct list_head free_clusters; /* free clusters list */
struct list_head full_clusters; /* full clusters list */
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 701bc80bc381..b5b126904d20 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -767,25 +767,19 @@ static bool cluster_reclaim_range(struct swap_info_struct *si,
struct swap_cluster_info *ci,
unsigned long start, unsigned long end)
{
- unsigned char *map = si->swap_map;
unsigned long offset = start;
int nr_reclaim;
spin_unlock(&ci->lock);
do {
- switch (READ_ONCE(map[offset])) {
- case 0:
+ if (!test_bit(offset, si->swap_map)) {
offset++;
- break;
- case SWAP_MAP_ALLOCATED:
+ } else {
nr_reclaim = __try_to_reclaim_swap(si, offset, TTRS_ANYWAY);
if (nr_reclaim > 0)
offset += nr_reclaim;
else
goto out;
- break;
- default:
- goto out;
}
} while (offset < end);
out:
@@ -794,11 +788,7 @@ static bool cluster_reclaim_range(struct swap_info_struct *si,
* Recheck the range no matter reclaim succeeded or not, the slot
* could have been be freed while we are not holding the lock.
*/
- for (offset = start; offset < end; offset++)
- if (READ_ONCE(map[offset]))
- return false;
-
- return true;
+ return find_next_bit(si->swap_map, end, start) >= end;
}
static bool cluster_scan_range(struct swap_info_struct *si,
@@ -807,15 +797,16 @@ static bool cluster_scan_range(struct swap_info_struct *si,
bool *need_reclaim)
{
unsigned long offset, end = start + nr_pages;
- unsigned char *map = si->swap_map;
- unsigned char count;
if (cluster_is_empty(ci))
return true;
for (offset = start; offset < end; offset++) {
- count = READ_ONCE(map[offset]);
- if (!count)
+ /* Bad slots cannot be used for allocation */
+ if (test_bit(offset, si->bad_map))
+ return false;
+
+ if (!test_bit(offset, si->swap_map))
continue;
if (swap_cache_only(si, offset)) {
@@ -848,7 +839,7 @@ static bool cluster_alloc_range(struct swap_info_struct *si, struct swap_cluster
if (cluster_is_empty(ci))
ci->order = order;
- memset(si->swap_map + start, usage, nr_pages);
+ bitmap_set(si->swap_map, start, nr_pages);
swap_range_alloc(si, nr_pages);
ci->count += nr_pages;
@@ -1414,7 +1405,7 @@ static struct swap_info_struct *_swap_info_get(swp_slot_t slot)
offset = swp_slot_offset(slot);
if (offset >= si->max)
goto bad_offset;
- if (data_race(!si->swap_map[swp_slot_offset(slot)]))
+ if (data_race(!test_bit(offset, si->swap_map)))
goto bad_free;
return si;
@@ -1528,8 +1519,7 @@ static void swap_slots_free(struct swap_info_struct *si,
swp_slot_t slot, unsigned int nr_pages)
{
unsigned long offset = swp_slot_offset(slot);
- unsigned char *map = si->swap_map + offset;
- unsigned char *map_end = map + nr_pages;
+ unsigned long end = offset + nr_pages;
/* It should never free entries across different clusters */
VM_BUG_ON(ci != __swap_offset_to_cluster(si, offset + nr_pages - 1));
@@ -1537,10 +1527,8 @@ static void swap_slots_free(struct swap_info_struct *si,
VM_BUG_ON(ci->count < nr_pages);
ci->count -= nr_pages;
- do {
- VM_BUG_ON(!swap_is_last_ref(*map));
- *map = 0;
- } while (++map < map_end);
+ VM_BUG_ON(find_next_zero_bit(si->swap_map, end, offset) < end);
+ bitmap_clear(si->swap_map, offset, nr_pages);
swap_range_free(si, offset, nr_pages);
@@ -1700,9 +1688,7 @@ unsigned int count_swap_pages(int type, int free)
static bool swap_slot_allocated(struct swap_info_struct *si,
unsigned long offset)
{
- unsigned char count = READ_ONCE(si->swap_map[offset]);
-
- return count && swap_count(count) != SWAP_MAP_BAD;
+ return test_bit(offset, si->swap_map);
}
/*
@@ -2023,7 +2009,7 @@ static int setup_swap_extents(struct swap_info_struct *sis, sector_t *span)
}
static void setup_swap_info(struct swap_info_struct *si, int prio,
- unsigned char *swap_map,
+ unsigned long *swap_map,
struct swap_cluster_info *cluster_info)
{
si->prio = prio;
@@ -2051,7 +2037,7 @@ static void _enable_swap_info(struct swap_info_struct *si)
}
static void enable_swap_info(struct swap_info_struct *si, int prio,
- unsigned char *swap_map,
+ unsigned long *swap_map,
struct swap_cluster_info *cluster_info)
{
spin_lock(&swap_lock);
@@ -2144,7 +2130,8 @@ static void flush_percpu_swap_cluster(struct swap_info_struct *si)
SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
{
struct swap_info_struct *p = NULL;
- unsigned char *swap_map;
+ unsigned long *swap_map;
+ unsigned long *bad_map;
struct swap_cluster_info *cluster_info;
struct file *swap_file, *victim;
struct address_space *mapping;
@@ -2239,6 +2226,8 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
p->swap_file = NULL;
swap_map = p->swap_map;
p->swap_map = NULL;
+ bad_map = p->bad_map;
+ p->bad_map = NULL;
maxpages = p->max;
cluster_info = p->cluster_info;
p->max = 0;
@@ -2249,7 +2238,8 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
mutex_unlock(&swapon_mutex);
kfree(p->global_cluster);
p->global_cluster = NULL;
- vfree(swap_map);
+ kvfree(swap_map);
+ kvfree(bad_map);
free_cluster_info(cluster_info, maxpages);
inode = mapping->host;
@@ -2597,18 +2587,20 @@ static unsigned long read_swap_header(struct swap_info_struct *si,
static int setup_swap_map(struct swap_info_struct *si,
union swap_header *swap_header,
- unsigned char *swap_map,
+ unsigned long *swap_map,
+ unsigned long *bad_map,
unsigned long maxpages)
{
unsigned long i;
- swap_map[0] = SWAP_MAP_BAD; /* omit header page */
+ set_bit(0, bad_map); /* omit header page */
+
for (i = 0; i < swap_header->info.nr_badpages; i++) {
unsigned int page_nr = swap_header->info.badpages[i];
if (page_nr == 0 || page_nr > swap_header->info.last_page)
return -EINVAL;
if (page_nr < maxpages) {
- swap_map[page_nr] = SWAP_MAP_BAD;
+ set_bit(page_nr, bad_map);
si->pages--;
}
}
@@ -2712,7 +2704,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
int nr_extents;
sector_t span;
unsigned long maxpages;
- unsigned char *swap_map = NULL;
+ unsigned long *swap_map = NULL, *bad_map = NULL;
struct swap_cluster_info *cluster_info = NULL;
struct folio *folio = NULL;
struct inode *inode = NULL;
@@ -2808,16 +2800,24 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
maxpages = si->max;
/* OK, set up the swap map and apply the bad block list */
- swap_map = vzalloc(maxpages);
+ swap_map = kvcalloc(BITS_TO_LONGS(maxpages), sizeof(long), GFP_KERNEL);
if (!swap_map) {
error = -ENOMEM;
goto bad_swap_unlock_inode;
}
- error = setup_swap_map(si, swap_header, swap_map, maxpages);
+ bad_map = kvcalloc(BITS_TO_LONGS(maxpages), sizeof(long), GFP_KERNEL);
+ if (!bad_map) {
+ error = -ENOMEM;
+ goto bad_swap_unlock_inode;
+ }
+
+ error = setup_swap_map(si, swap_header, swap_map, bad_map, maxpages);
if (error)
goto bad_swap_unlock_inode;
+ si->bad_map = bad_map;
+
if (si->bdev && bdev_stable_writes(si->bdev))
si->flags |= SWP_STABLE_WRITES;
@@ -2911,7 +2911,10 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
si->swap_file = NULL;
si->flags = 0;
spin_unlock(&swap_lock);
- vfree(swap_map);
+ if (swap_map)
+ kvfree(swap_map);
+ if (bad_map)
+ kvfree(bad_map);
if (cluster_info)
free_cluster_info(cluster_info, maxpages);
if (inced_nr_rotate_swap)
--
2.52.0
^ permalink raw reply related
* [PATCH v6 19/22] swap: simplify swapoff using virtual swap
From: Nhat Pham @ 2026-05-05 15:38 UTC (permalink / raw)
To: kasong
Cc: Liam.Howlett, akpm, apopple, axelrasmussen, baohua, baolin.wang,
bhe, byungchul, cgroups, chengming.zhou, chrisl, corbet, david,
dev.jain, gourry, hannes, hughd, jannh, joshua.hahnjy, lance.yang,
lenb, linux-doc, linux-kernel, linux-mm, linux-pm,
lorenzo.stoakes, matthew.brost, mhocko, muchun.song, npache,
nphamcs, pavel, peterx, peterz, pfalcato, rafael, rakie.kim,
roman.gushchin, rppt, ryan.roberts, shakeel.butt, shikemeng,
surenb, tglx, vbabka, weixugc, ying.huang, yosry.ahmed, yuanchu,
zhengqi.arch, ziy, kernel-team, riel, haowenchao22
In-Reply-To: <20260505153854.1612033-1-nphamcs@gmail.com>
This patch presents the second applications of virtual swap design -
simplifying and optimizing swapoff.
With virtual swap slots stored at page table entries and used as indices
to various swap-related data structures, we no longer have to perform a
page table walk in swapoff. Simply iterate through all the allocated
swap slots on the swapfile, find their corresponding virtual swap slots,
and fault them in.
This is significantly cleaner, as well as slightly more performant,
especially when there are a lot of unrelated VMAs (since the old swapoff
code would have to traverse through all of them).
In a simple benchmark, in which we swapoff a 32 GB swapfile that is 50%
full, and in which there is a process that maps a 128GB file into
memory:
Baseline:
sys: 11.48s
New Design:
sys: 9.96s
Disregarding the real time reduction (which is mostly due to more IO
asynchrony), the new design reduces the kernel CPU time by about 13%.
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
include/linux/shmem_fs.h | 9 +-
mm/filemap.c | 14 +-
mm/shmem.c | 196 +---------------
mm/swapfile.c | 474 +++++++++------------------------------
4 files changed, 127 insertions(+), 566 deletions(-)
diff --git a/include/linux/shmem_fs.h b/include/linux/shmem_fs.h
index e2069b3179c4..8f45bd65dc20 100644
--- a/include/linux/shmem_fs.h
+++ b/include/linux/shmem_fs.h
@@ -40,18 +40,14 @@ struct shmem_inode_info {
unsigned long alloced; /* data pages alloced to file */
unsigned long swapped; /* subtotal assigned to swap */
union {
- struct offset_ctx dir_offsets; /* stable directory offsets */
- struct {
- struct list_head shrinklist; /* shrinkable hpage inodes */
- struct list_head swaplist; /* chain of maybes on swap */
- };
+ struct offset_ctx dir_offsets; /* stable directory offsets */
+ struct list_head shrinklist; /* shrinkable hpage inodes */
};
struct timespec64 i_crtime; /* file creation time */
struct shared_policy policy; /* NUMA memory alloc policy */
struct simple_xattrs xattrs; /* list of xattrs */
pgoff_t fallocend; /* highest fallocate endindex */
unsigned int fsflags; /* for FS_IOC_[SG]ETFLAGS */
- atomic_t stop_eviction; /* hold when working on inode */
#ifdef CONFIG_TMPFS_QUOTA
struct dquot __rcu *i_dquot[MAXQUOTAS];
#endif
@@ -127,7 +123,6 @@ struct page *shmem_read_mapping_page_gfp(struct address_space *mapping,
int shmem_writeout(struct folio *folio, struct swap_iocb **plug,
struct list_head *folio_list);
void shmem_truncate_range(struct inode *inode, loff_t start, uoff_t end);
-int shmem_unuse(unsigned int type);
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
unsigned long shmem_allowable_huge_orders(struct inode *inode,
diff --git a/mm/filemap.c b/mm/filemap.c
index ebd75684cb0a..53aad273ea2f 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -4614,13 +4614,13 @@ static void filemap_cachestat(struct address_space *mapping,
/*
* Getting a swap entry from the shmem
- * inode means we beat
- * shmem_unuse(). rcu_read_lock()
- * ensures swapoff waits for us before
- * freeing the swapper space. However,
- * we can race with swapping and
- * invalidation, so there might not be
- * a shadow in the swapcache (yet).
+ * inode means we beat swapoff.
+ * rcu_read_lock() ensures swapoff waits
+ * for us before freeing the swapper
+ * space. However, we can race with
+ * swapping and invalidation, so there
+ * might not be a shadow in the swapcache
+ * (yet).
*/
shadow = swap_cache_get_shadow(swp);
if (!shadow)
diff --git a/mm/shmem.c b/mm/shmem.c
index 3a346cca114a..984e01ea88d3 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -290,9 +290,6 @@ bool vma_is_shmem(const struct vm_area_struct *vma)
return vma_is_anon_shmem(vma) || vma->vm_ops == &shmem_vm_ops;
}
-static LIST_HEAD(shmem_swaplist);
-static DEFINE_SPINLOCK(shmem_swaplist_lock);
-
#ifdef CONFIG_TMPFS_QUOTA
static int shmem_enable_quotas(struct super_block *sb,
@@ -1413,16 +1410,6 @@ static void shmem_evict_inode(struct inode *inode)
}
spin_unlock(&sbinfo->shrinklist_lock);
}
- while (!list_empty(&info->swaplist)) {
- /* Wait while shmem_unuse() is scanning this inode... */
- wait_var_event(&info->stop_eviction,
- !atomic_read(&info->stop_eviction));
- spin_lock(&shmem_swaplist_lock);
- /* ...but beware of the race if we peeked too early */
- if (!atomic_read(&info->stop_eviction))
- list_del_init(&info->swaplist);
- spin_unlock(&shmem_swaplist_lock);
- }
}
simple_xattrs_free(&info->xattrs, sbinfo->max_inodes ? &freed : NULL);
@@ -1435,153 +1422,6 @@ static void shmem_evict_inode(struct inode *inode)
#endif
}
-static unsigned int shmem_find_swap_entries(struct address_space *mapping,
- pgoff_t start, struct folio_batch *fbatch,
- pgoff_t *indices, unsigned int type)
-{
- XA_STATE(xas, &mapping->i_pages, start);
- struct folio *folio;
- swp_entry_t entry;
- swp_slot_t slot;
-
- rcu_read_lock();
- xas_for_each(&xas, folio, ULONG_MAX) {
- if (xas_retry(&xas, folio))
- continue;
-
- if (!xa_is_value(folio))
- continue;
-
- entry = radix_to_swp_entry(folio);
- slot = swp_entry_to_swp_slot(entry);
-
- /*
- * swapin error entries can be found in the mapping. But they're
- * deliberately ignored here as we've done everything we can do.
- */
- if (!slot.val || swp_slot_type(slot) != type)
- continue;
-
- indices[folio_batch_count(fbatch)] = xas.xa_index;
- if (!folio_batch_add(fbatch, folio))
- break;
-
- if (need_resched()) {
- xas_pause(&xas);
- cond_resched_rcu();
- }
- }
- rcu_read_unlock();
-
- return folio_batch_count(fbatch);
-}
-
-/*
- * Move the swapped pages for an inode to page cache. Returns the count
- * of pages swapped in, or the error in case of failure.
- */
-static int shmem_unuse_swap_entries(struct inode *inode,
- struct folio_batch *fbatch, pgoff_t *indices)
-{
- int i = 0;
- int ret = 0;
- int error = 0;
- struct address_space *mapping = inode->i_mapping;
-
- for (i = 0; i < folio_batch_count(fbatch); i++) {
- struct folio *folio = fbatch->folios[i];
-
- error = shmem_swapin_folio(inode, indices[i], &folio, SGP_CACHE,
- mapping_gfp_mask(mapping), NULL, NULL);
- if (error == 0) {
- folio_unlock(folio);
- folio_put(folio);
- ret++;
- }
- if (error == -ENOMEM)
- break;
- error = 0;
- }
- return error ? error : ret;
-}
-
-/*
- * If swap found in inode, free it and move page from swapcache to filecache.
- */
-static int shmem_unuse_inode(struct inode *inode, unsigned int type)
-{
- struct address_space *mapping = inode->i_mapping;
- pgoff_t start = 0;
- struct folio_batch fbatch;
- pgoff_t indices[PAGEVEC_SIZE];
- int ret = 0;
-
- do {
- folio_batch_init(&fbatch);
- if (!shmem_find_swap_entries(mapping, start, &fbatch,
- indices, type)) {
- ret = 0;
- break;
- }
-
- ret = shmem_unuse_swap_entries(inode, &fbatch, indices);
- if (ret < 0)
- break;
-
- start = indices[folio_batch_count(&fbatch) - 1];
- } while (true);
-
- return ret;
-}
-
-/*
- * Read all the shared memory data that resides in the swap
- * device 'type' back into memory, so the swap device can be
- * unused.
- */
-int shmem_unuse(unsigned int type)
-{
- struct shmem_inode_info *info, *next;
- int error = 0;
-
- if (list_empty(&shmem_swaplist))
- return 0;
-
- spin_lock(&shmem_swaplist_lock);
-start_over:
- list_for_each_entry_safe(info, next, &shmem_swaplist, swaplist) {
- if (!info->swapped) {
- list_del_init(&info->swaplist);
- continue;
- }
- /*
- * Drop the swaplist mutex while searching the inode for swap;
- * but before doing so, make sure shmem_evict_inode() will not
- * remove placeholder inode from swaplist, nor let it be freed
- * (igrab() would protect from unlink, but not from unmount).
- */
- atomic_inc(&info->stop_eviction);
- spin_unlock(&shmem_swaplist_lock);
-
- error = shmem_unuse_inode(&info->vfs_inode, type);
- cond_resched();
-
- spin_lock(&shmem_swaplist_lock);
- if (atomic_dec_and_test(&info->stop_eviction))
- wake_up_var(&info->stop_eviction);
- if (error)
- break;
- if (list_empty(&info->swaplist))
- goto start_over;
- next = list_next_entry(info, swaplist);
- if (!info->swapped)
- list_del_init(&info->swaplist);
- }
- spin_unlock(&shmem_swaplist_lock);
-
- return error;
-}
-
/**
* shmem_writeout - Write the folio to swap
* @folio: The folio to write
@@ -1668,24 +1508,9 @@ int shmem_writeout(struct folio *folio, struct swap_iocb **plug,
}
if (!folio_alloc_swap(folio)) {
- bool first_swapped = shmem_recalc_inode(inode, 0, nr_pages);
int error;
- /*
- * Add inode to shmem_unuse()'s list of swapped-out inodes,
- * if it's not already there. Do it now before the folio is
- * removed from page cache, when its pagelock no longer
- * protects the inode from eviction. And do it now, after
- * we've incremented swapped, because shmem_unuse() will
- * prune a !swapped inode from the swaplist.
- */
- if (first_swapped) {
- spin_lock(&shmem_swaplist_lock);
- if (list_empty(&info->swaplist))
- list_add(&info->swaplist, &shmem_swaplist);
- spin_unlock(&shmem_swaplist_lock);
- }
-
+ shmem_recalc_inode(inode, 0, nr_pages);
swap_shmem_alloc(folio->swap, nr_pages);
shmem_delete_from_page_cache(folio, swp_to_radix_entry(folio->swap));
@@ -2116,12 +1941,12 @@ static struct folio *shmem_swap_alloc_folio(struct inode *inode,
}
/*
- * When a page is moved from swapcache to shmem filecache (either by the
- * usual swapin of shmem_get_folio_gfp(), or by the less common swapoff of
- * shmem_unuse_inode()), it may have been read in earlier from swap, in
- * ignorance of the mapping it belongs to. If that mapping has special
- * constraints (like the gma500 GEM driver, which requires RAM below 4GB),
- * we may need to copy to a suitable page before moving to filecache.
+ * When a page is moved from swapcache to shmem filecache (by the usual
+ * swapin of shmem_get_folio_gfp()), it may have been read in earlier from
+ * swap, in ignorance of the mapping it belongs to. If that mapping has
+ * special constraints (like the gma500 GEM driver, which requires RAM
+ * below 4GB), we may need to copy to a suitable page before moving to
+ * filecache.
*
* In a future release, this may well be extended to respect cpuset and
* NUMA mempolicy, and applied also to anonymous pages in do_swap_page();
@@ -3106,7 +2931,6 @@ static struct inode *__shmem_get_inode(struct mnt_idmap *idmap,
info = SHMEM_I(inode);
memset(info, 0, (char *)inode - (char *)info);
spin_lock_init(&info->lock);
- atomic_set(&info->stop_eviction, 0);
info->seals = F_SEAL_SEAL;
info->flags = (flags & VM_NORESERVE) ? SHMEM_F_NORESERVE : 0;
info->i_crtime = inode_get_mtime(inode);
@@ -3115,7 +2939,6 @@ static struct inode *__shmem_get_inode(struct mnt_idmap *idmap,
if (info->fsflags)
shmem_set_inode_flags(inode, info->fsflags, NULL);
INIT_LIST_HEAD(&info->shrinklist);
- INIT_LIST_HEAD(&info->swaplist);
simple_xattrs_init(&info->xattrs);
cache_no_acl(inode);
if (sbinfo->noswap)
@@ -5785,11 +5608,6 @@ void __init shmem_init(void)
BUG_ON(IS_ERR(shm_mnt));
}
-int shmem_unuse(unsigned int type)
-{
- return 0;
-}
-
int shmem_lock(struct file *file, int lock, struct ucounts *ucounts)
{
return 0;
diff --git a/mm/swapfile.c b/mm/swapfile.c
index ce1254733e96..701bc80bc381 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -1697,300 +1697,12 @@ unsigned int count_swap_pages(int type, int free)
}
#endif /* CONFIG_HIBERNATION */
-static inline int pte_same_as_swp(pte_t pte, pte_t swp_pte)
+static bool swap_slot_allocated(struct swap_info_struct *si,
+ unsigned long offset)
{
- return pte_same(pte_swp_clear_flags(pte), swp_pte);
-}
-
-/*
- * No need to decide whether this PTE shares the swap entry with others,
- * just let do_wp_page work it out if a write is requested later - to
- * force COW, vm_page_prot omits write permission from any private vma.
- */
-static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd,
- unsigned long addr, swp_entry_t entry, struct folio *folio)
-{
- struct page *page;
- struct folio *swapcache;
- spinlock_t *ptl;
- pte_t *pte, new_pte, old_pte;
- bool hwpoisoned = false;
- int ret = 1;
-
- /*
- * If the folio is removed from swap cache by others, continue to
- * unuse other PTEs. try_to_unuse may try again if we missed this one.
- */
- if (!folio_matches_swap_entry(folio, entry))
- return 0;
-
- swapcache = folio;
- folio = ksm_might_need_to_copy(folio, vma, addr);
- if (unlikely(!folio))
- return -ENOMEM;
- else if (unlikely(folio == ERR_PTR(-EHWPOISON))) {
- hwpoisoned = true;
- folio = swapcache;
- }
-
- page = folio_file_page(folio, swp_offset(entry));
- if (PageHWPoison(page))
- hwpoisoned = true;
-
- pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
- if (unlikely(!pte || !pte_same_as_swp(ptep_get(pte),
- swp_entry_to_pte(entry)))) {
- ret = 0;
- goto out;
- }
-
- old_pte = ptep_get(pte);
-
- if (unlikely(hwpoisoned || !folio_test_uptodate(folio))) {
- swp_entry_t swp_entry;
-
- dec_mm_counter(vma->vm_mm, MM_SWAPENTS);
- if (hwpoisoned) {
- swp_entry = make_hwpoison_entry(page);
- } else {
- swp_entry = make_poisoned_swp_entry();
- }
- new_pte = swp_entry_to_pte(swp_entry);
- ret = 0;
- goto setpte;
- }
-
- /*
- * Some architectures may have to restore extra metadata to the page
- * when reading from swap. This metadata may be indexed by swap entry
- * so this must be called before swap_free().
- */
- arch_swap_restore(folio_swap(entry, folio), folio);
-
- dec_mm_counter(vma->vm_mm, MM_SWAPENTS);
- inc_mm_counter(vma->vm_mm, MM_ANONPAGES);
- folio_get(folio);
- if (folio == swapcache) {
- rmap_t rmap_flags = RMAP_NONE;
-
- /*
- * See do_swap_page(): writeback would be problematic.
- * However, we do a folio_wait_writeback() just before this
- * call and have the folio locked.
- */
- VM_BUG_ON_FOLIO(folio_test_writeback(folio), folio);
- if (pte_swp_exclusive(old_pte))
- rmap_flags |= RMAP_EXCLUSIVE;
- /*
- * We currently only expect small !anon folios, which are either
- * fully exclusive or fully shared. If we ever get large folios
- * here, we have to be careful.
- */
- if (!folio_test_anon(folio)) {
- VM_WARN_ON_ONCE(folio_test_large(folio));
- VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
- folio_add_new_anon_rmap(folio, vma, addr, rmap_flags);
- } else {
- folio_add_anon_rmap_pte(folio, page, vma, addr, rmap_flags);
- }
- } else { /* ksm created a completely new copy */
- folio_add_new_anon_rmap(folio, vma, addr, RMAP_EXCLUSIVE);
- folio_add_lru_vma(folio, vma);
- }
- new_pte = pte_mkold(mk_pte(page, vma->vm_page_prot));
- if (pte_swp_soft_dirty(old_pte))
- new_pte = pte_mksoft_dirty(new_pte);
- if (pte_swp_uffd_wp(old_pte))
- new_pte = pte_mkuffd_wp(new_pte);
-setpte:
- set_pte_at(vma->vm_mm, addr, pte, new_pte);
- swap_free(entry);
-out:
- if (pte)
- pte_unmap_unlock(pte, ptl);
- if (folio != swapcache) {
- folio_unlock(folio);
- folio_put(folio);
- }
- return ret;
-}
-
-static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
- unsigned long addr, unsigned long end,
- unsigned int type)
-{
- pte_t *pte = NULL;
- struct swap_info_struct *si;
-
- si = swap_info[type];
- do {
- struct folio *folio;
- unsigned long offset;
- unsigned char swp_count;
- softleaf_t entry;
- swp_slot_t slot;
- int ret;
- pte_t ptent;
-
- if (!pte++) {
- pte = pte_offset_map(pmd, addr);
- if (!pte)
- break;
- }
-
- ptent = ptep_get_lockless(pte);
- entry = softleaf_from_pte(ptent);
-
- if (!softleaf_is_swap(entry))
- continue;
-
- slot = swp_entry_to_swp_slot(entry);
- if (swp_slot_type(slot) != type)
- continue;
-
- offset = swp_slot_offset(slot);
- pte_unmap(pte);
- pte = NULL;
-
- folio = swap_cache_get_folio(entry);
- if (!folio) {
- struct vm_fault vmf = {
- .vma = vma,
- .address = addr,
- .real_address = addr,
- .pmd = pmd,
- };
-
- folio = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE,
- &vmf);
- }
- if (!folio) {
- swp_count = READ_ONCE(si->swap_map[offset]);
- if (swp_count == 0 || swp_count == SWAP_MAP_BAD)
- continue;
- return -ENOMEM;
- }
-
- folio_lock(folio);
- folio_wait_writeback(folio);
- ret = unuse_pte(vma, pmd, addr, entry, folio);
- if (ret < 0) {
- folio_unlock(folio);
- folio_put(folio);
- return ret;
- }
-
- folio_free_swap(folio);
- folio_unlock(folio);
- folio_put(folio);
- } while (addr += PAGE_SIZE, addr != end);
-
- if (pte)
- pte_unmap(pte);
- return 0;
-}
-
-static inline int unuse_pmd_range(struct vm_area_struct *vma, pud_t *pud,
- unsigned long addr, unsigned long end,
- unsigned int type)
-{
- pmd_t *pmd;
- unsigned long next;
- int ret;
-
- pmd = pmd_offset(pud, addr);
- do {
- cond_resched();
- next = pmd_addr_end(addr, end);
- ret = unuse_pte_range(vma, pmd, addr, next, type);
- if (ret)
- return ret;
- } while (pmd++, addr = next, addr != end);
- return 0;
-}
-
-static inline int unuse_pud_range(struct vm_area_struct *vma, p4d_t *p4d,
- unsigned long addr, unsigned long end,
- unsigned int type)
-{
- pud_t *pud;
- unsigned long next;
- int ret;
-
- pud = pud_offset(p4d, addr);
- do {
- next = pud_addr_end(addr, end);
- if (pud_none_or_clear_bad(pud))
- continue;
- ret = unuse_pmd_range(vma, pud, addr, next, type);
- if (ret)
- return ret;
- } while (pud++, addr = next, addr != end);
- return 0;
-}
-
-static inline int unuse_p4d_range(struct vm_area_struct *vma, pgd_t *pgd,
- unsigned long addr, unsigned long end,
- unsigned int type)
-{
- p4d_t *p4d;
- unsigned long next;
- int ret;
-
- p4d = p4d_offset(pgd, addr);
- do {
- next = p4d_addr_end(addr, end);
- if (p4d_none_or_clear_bad(p4d))
- continue;
- ret = unuse_pud_range(vma, p4d, addr, next, type);
- if (ret)
- return ret;
- } while (p4d++, addr = next, addr != end);
- return 0;
-}
-
-static int unuse_vma(struct vm_area_struct *vma, unsigned int type)
-{
- pgd_t *pgd;
- unsigned long addr, end, next;
- int ret;
-
- addr = vma->vm_start;
- end = vma->vm_end;
-
- pgd = pgd_offset(vma->vm_mm, addr);
- do {
- next = pgd_addr_end(addr, end);
- if (pgd_none_or_clear_bad(pgd))
- continue;
- ret = unuse_p4d_range(vma, pgd, addr, next, type);
- if (ret)
- return ret;
- } while (pgd++, addr = next, addr != end);
- return 0;
-}
+ unsigned char count = READ_ONCE(si->swap_map[offset]);
-static int unuse_mm(struct mm_struct *mm, unsigned int type)
-{
- struct vm_area_struct *vma;
- int ret = 0;
- VMA_ITERATOR(vmi, mm, 0);
-
- mmap_read_lock(mm);
- if (check_stable_address_space(mm))
- goto unlock;
- for_each_vma(vmi, vma) {
- if (vma->anon_vma && !is_vm_hugetlb_page(vma)) {
- ret = unuse_vma(vma, type);
- if (ret)
- break;
- }
-
- cond_resched();
- }
-unlock:
- mmap_read_unlock(mm);
- return ret;
+ return count && swap_count(count) != SWAP_MAP_BAD;
}
/*
@@ -2002,7 +1714,6 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si,
unsigned int prev)
{
unsigned int i;
- unsigned char count;
/*
* No need for swap_lock here: we're just looking
@@ -2011,8 +1722,7 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si,
* allocations from this area (while holding swap_lock).
*/
for (i = prev + 1; i < si->max; i++) {
- count = READ_ONCE(si->swap_map[i]);
- if (count && swap_count(count) != SWAP_MAP_BAD)
+ if (swap_slot_allocated(si, i))
break;
if ((i % LATENCY_LIMIT) == 0)
cond_resched();
@@ -2024,101 +1734,139 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si,
return i;
}
+#define for_each_allocated_offset(si, offset) \
+ while (swap_usage_in_pages(si) && \
+ !signal_pending(current) && \
+ (offset = find_next_to_unuse(si, offset)) != 0)
+
+static struct folio *pagein(swp_entry_t entry, struct swap_iocb **splug,
+ struct mempolicy *mpol)
+{
+ bool folio_was_allocated;
+ struct folio *folio = __read_swap_cache_async(entry, GFP_KERNEL, mpol,
+ NO_INTERLEAVE_INDEX, &folio_was_allocated, false);
+
+ if (folio_was_allocated)
+ swap_read_folio(folio, splug);
+ return folio;
+}
+
static int try_to_unuse(unsigned int type)
{
- struct mm_struct *prev_mm;
- struct mm_struct *mm;
- struct list_head *p;
- int retval = 0;
struct swap_info_struct *si = swap_info[type];
+ struct swap_iocb *splug = NULL;
+ struct mempolicy *mpol;
+ struct blk_plug plug;
+ unsigned long offset;
struct folio *folio;
swp_entry_t entry;
swp_slot_t slot;
- unsigned int i;
+ int ret = 0;
if (!swap_usage_in_pages(si))
goto success;
-retry:
- retval = shmem_unuse(type);
- if (retval)
- return retval;
-
- prev_mm = &init_mm;
- mmget(prev_mm);
-
- spin_lock(&mmlist_lock);
- p = &init_mm.mmlist;
- while (swap_usage_in_pages(si) &&
- !signal_pending(current) &&
- (p = p->next) != &init_mm.mmlist) {
+ mpol = get_task_policy(current);
+ blk_start_plug(&plug);
- mm = list_entry(p, struct mm_struct, mmlist);
- if (!mmget_not_zero(mm))
+ /* first round - submit the reads */
+ offset = 0;
+ for_each_allocated_offset(si, offset) {
+ slot = swp_slot(type, offset);
+ entry = swp_slot_to_swp_entry(slot);
+ if (!entry.val)
continue;
- spin_unlock(&mmlist_lock);
- mmput(prev_mm);
- prev_mm = mm;
- retval = unuse_mm(mm, type);
- if (retval) {
- mmput(prev_mm);
- return retval;
- }
- /*
- * Make sure that we aren't completely killing
- * interactive performance.
- */
- cond_resched();
- spin_lock(&mmlist_lock);
+ folio = pagein(entry, &splug, mpol);
+ if (folio)
+ folio_put(folio);
}
- spin_unlock(&mmlist_lock);
+ blk_finish_plug(&plug);
+ swap_read_unplug(splug);
+ splug = NULL;
+ lru_add_drain();
+
+ /* second round - updating the virtual swap slots' backing state */
+ offset = 0;
+ for_each_allocated_offset(si, offset) {
+ slot = swp_slot(type, offset);
+retry:
+ entry = swp_slot_to_swp_entry(slot);
+ if (!entry.val) {
+ if (!swap_slot_allocated(si, offset))
+ continue;
- mmput(prev_mm);
+ if (signal_pending(current)) {
+ ret = -EINTR;
+ goto out;
+ }
- i = 0;
- while (swap_usage_in_pages(si) &&
- !signal_pending(current) &&
- (i = find_next_to_unuse(si, i)) != 0) {
+ /* we might be racing with zswap writeback or disk swapout */
+ schedule_timeout_uninterruptible(1);
+ goto retry;
+ }
- slot = swp_slot(type, i);
- entry = swp_slot_to_swp_entry(slot);
- folio = swap_cache_get_folio(entry);
- if (!folio)
- continue;
+ /* try to allocate swap cache folio */
+ folio = pagein(entry, &splug, mpol);
+ if (!folio) {
+ if (!swp_slot_to_swp_entry(swp_slot(type, offset)).val)
+ continue;
+ ret = -ENOMEM;
+ pr_err("swapoff: unable to allocate swap cache folio for %lu\n",
+ entry.val);
+ goto out;
+ }
+
+ folio_lock(folio);
/*
- * It is conceivable that a racing task removed this folio from
- * swap cache just before we acquired the page lock. The folio
- * might even be back in swap cache on another swap area. But
- * that is okay, folio_free_swap() only removes stale folios.
+ * We need to check if the folio is still in swap cache, and is still
+ * backed by the physical swap slot we are trying to release.
+ *
+ * We can, for instance, race with zswap writeback, obtaining the
+ * temporary folio it allocated for decompression and writeback, which
+ * would be promptly deleted from swap cache. By the time we lock that
+ * folio, it might have already contained stale data.
+ *
+ * Concurrent swap operations might have also come in before we
+ * reobtain the folio's lock, deleting the folio from swap cache,
+ * invalidating the virtual swap slot, then swapping out the folio
+ * again to a different swap backends.
+ *
+ * In all of these cases, we must retry the physical -> virtual lookup.
*/
- folio_lock(folio);
+ if (!folio_matches_swap_slot(folio, entry, slot)) {
+ folio_unlock(folio);
+ folio_put(folio);
+ if (signal_pending(current)) {
+ ret = -EINTR;
+ goto out;
+ }
+ schedule_timeout_uninterruptible(1);
+ goto retry;
+ }
+
folio_wait_writeback(folio);
- folio_free_swap(folio);
+ vswap_store_folio(entry, folio);
+ folio_mark_dirty(folio);
folio_unlock(folio);
folio_put(folio);
}
- /*
- * Lets check again to see if there are still swap entries in the map.
- * If yes, we would need to do retry the unuse logic again.
- * Under global memory pressure, swap entries can be reinserted back
- * into process space after the mmlist loop above passes over them.
- *
- * Limit the number of retries? No: when mmget_not_zero()
- * above fails, that mm is likely to be freeing swap from
- * exit_mmap(), which proceeds at its own independent pace;
- * and even shmem_writeout() could have been preempted after
- * folio_alloc_swap(), temporarily hiding that swap. It's easy
- * and robust (though cpu-intensive) just to keep retrying.
- */
- if (swap_usage_in_pages(si)) {
- if (!signal_pending(current))
- goto retry;
- return -EINTR;
+ /* concurrent swappers might still be releasing physical swap slots... */
+ while (swap_usage_in_pages(si)) {
+ if (signal_pending(current)) {
+ ret = -EINTR;
+ goto out;
+ }
+ schedule_timeout_uninterruptible(1);
}
+out:
+ swap_read_unplug(splug);
+ if (ret)
+ return ret;
+
success:
/*
* Make sure that further cleanups after try_to_unuse() returns happen
--
2.52.0
^ permalink raw reply related
* [PATCH v6 18/22] memcg: swap: only charge physical swap slots
From: Nhat Pham @ 2026-05-05 15:38 UTC (permalink / raw)
To: kasong
Cc: Liam.Howlett, akpm, apopple, axelrasmussen, baohua, baolin.wang,
bhe, byungchul, cgroups, chengming.zhou, chrisl, corbet, david,
dev.jain, gourry, hannes, hughd, jannh, joshua.hahnjy, lance.yang,
lenb, linux-doc, linux-kernel, linux-mm, linux-pm,
lorenzo.stoakes, matthew.brost, mhocko, muchun.song, npache,
nphamcs, pavel, peterx, peterz, pfalcato, rafael, rakie.kim,
roman.gushchin, rppt, ryan.roberts, shakeel.butt, shikemeng,
surenb, tglx, vbabka, weixugc, ying.huang, yosry.ahmed, yuanchu,
zhengqi.arch, ziy, kernel-team, riel, haowenchao22
In-Reply-To: <20260505153854.1612033-1-nphamcs@gmail.com>
Now that zswap and the zero-filled swap page optimization no longer
takes up any physical swap space, we should not charge towards the swap
usage and limits of the memcg in these cases. We will only record the
memcg id on virtual swap slot allocation, and defer physical swap
charging (i.e towards memory.swap.current) until the virtual swap slot
is backed by an actual physical swap slot (on zswap store failure
fallback or zswap writeback).
Note that on cgroup v1, memsw is still charged regardless of backend,
so release_backing() uncharges memsw for the entire freed range.
Also convert the memcg swap charge/uncharge/clear APIs to take
struct mem_cgroup * directly, eliminating IDR lookups on the hot paths:
mem_cgroup_try_charge_swap(memcg, nr)
mem_cgroup_uncharge_swap(memcg, nr)
mem_cgroup_clear_swap(memcg, entry, nr)
In vswap_alloc_swap_slot(), the folio's memcg is compared against the
recorded swap memcg id; on match (the common case), zero IDR lookups.
In vswap_free(), the memcg is resolved once and threaded through
release_backing() and __vswap_swap_cgroup_clear(), reducing 2 IDR
lookups to 1. In memcg1_swapin(), one IDR lookup now feeds both
mem_cgroup_uncharge_swap() and mem_cgroup_clear_swap() (was 2).
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
include/linux/memcontrol.h | 5 +
include/linux/swap.h | 58 +++++----
include/linux/swap_cgroup.h | 6 -
mm/memcontrol-v1.c | 15 ++-
mm/memcontrol.c | 136 ++++++++++-----------
mm/vswap.c | 235 ++++++++++++++++++++++--------------
6 files changed, 269 insertions(+), 186 deletions(-)
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 0651865a4564..c1d63d03c9a6 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -827,6 +827,7 @@ static inline unsigned short mem_cgroup_id(struct mem_cgroup *memcg)
return memcg->id.id;
}
struct mem_cgroup *mem_cgroup_from_id(unsigned short id);
+void mem_cgroup_id_put_many(struct mem_cgroup *memcg, unsigned int n);
#ifdef CONFIG_SHRINKER_DEBUG
static inline unsigned long mem_cgroup_ino(struct mem_cgroup *memcg)
@@ -1289,6 +1290,10 @@ static inline struct mem_cgroup *mem_cgroup_from_id(unsigned short id)
return NULL;
}
+static inline void mem_cgroup_id_put_many(struct mem_cgroup *memcg, unsigned int n)
+{
+}
+
#ifdef CONFIG_SHRINKER_DEBUG
static inline unsigned long mem_cgroup_ino(struct mem_cgroup *memcg)
{
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 4d152fa811f2..ad5f59c807c6 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -674,49 +674,65 @@ static inline void folio_throttle_swaprate(struct folio *folio, gfp_t gfp)
#endif
#if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP)
-int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry);
-static inline int mem_cgroup_try_charge_swap(struct folio *folio,
- swp_entry_t entry)
+unsigned short __mem_cgroup_get_swap_memcgid(struct folio *folio);
+static inline unsigned short mem_cgroup_get_swap_memcgid(struct folio *folio)
{
- if (mem_cgroup_disabled())
- return 0;
- return __mem_cgroup_try_charge_swap(folio, entry);
+ if (!mem_cgroup_disabled())
+ return __mem_cgroup_get_swap_memcgid(folio);
+ return 0;
+}
+
+void __mem_cgroup_clear_swap(struct mem_cgroup *memcg, swp_entry_t entry,
+ unsigned int nr_pages);
+static inline void mem_cgroup_clear_swap(struct mem_cgroup *memcg,
+ swp_entry_t entry, unsigned int nr_pages)
+{
+ if (!mem_cgroup_disabled())
+ __mem_cgroup_clear_swap(memcg, entry, nr_pages);
}
-extern void __mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_pages);
-static inline void mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_pages)
+int __mem_cgroup_try_charge_swap(struct mem_cgroup *memcg,
+ unsigned int nr_pages);
+static inline int mem_cgroup_try_charge_swap(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
{
if (mem_cgroup_disabled())
- return;
- __mem_cgroup_uncharge_swap(entry, nr_pages);
+ return 0;
+ return __mem_cgroup_try_charge_swap(memcg, nr_pages);
}
-extern void __mem_cgroup_uncharge_swap_by_id(unsigned short id,
- unsigned int nr_pages);
-static inline void mem_cgroup_uncharge_swap_by_id(unsigned short id,
- unsigned int nr_pages)
+extern void __mem_cgroup_uncharge_swap(struct mem_cgroup *memcg,
+ unsigned int nr_pages);
+static inline void mem_cgroup_uncharge_swap(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
{
if (mem_cgroup_disabled())
return;
- __mem_cgroup_uncharge_swap_by_id(id, nr_pages);
+ __mem_cgroup_uncharge_swap(memcg, nr_pages);
}
extern long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg);
extern bool mem_cgroup_swap_full(struct folio *folio);
#else
-static inline int mem_cgroup_try_charge_swap(struct folio *folio,
- swp_entry_t entry)
+static inline unsigned short mem_cgroup_get_swap_memcgid(struct folio *folio)
{
return 0;
}
-static inline void mem_cgroup_uncharge_swap(swp_entry_t entry,
- unsigned int nr_pages)
+static inline void mem_cgroup_clear_swap(struct mem_cgroup *memcg,
+ swp_entry_t entry,
+ unsigned int nr_pages)
{
}
-static inline void mem_cgroup_uncharge_swap_by_id(unsigned short id,
- unsigned int nr_pages)
+static inline int mem_cgroup_try_charge_swap(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
+{
+ return 0;
+}
+
+static inline void mem_cgroup_uncharge_swap(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
{
}
diff --git a/include/linux/swap_cgroup.h b/include/linux/swap_cgroup.h
index 8e83856f801a..9f5b1dd9a2cf 100644
--- a/include/linux/swap_cgroup.h
+++ b/include/linux/swap_cgroup.h
@@ -6,7 +6,6 @@
#if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP)
-extern void swap_cgroup_record(struct folio *folio, unsigned short id, swp_entry_t ent);
extern void __swap_cgroup_record(struct folio *folio, unsigned short id, swp_entry_t ent);
extern unsigned short swap_cgroup_clear(swp_entry_t ent, unsigned int nr_ents);
extern unsigned short lookup_swap_cgroup_id(swp_entry_t ent);
@@ -14,11 +13,6 @@ extern int vswap_cgroup_batch(swp_entry_t entry, int max_nr);
#else
-static inline
-void swap_cgroup_record(struct folio *folio, unsigned short id, swp_entry_t ent)
-{
-}
-
static inline
void __swap_cgroup_record(struct folio *folio, unsigned short id, swp_entry_t ent)
{
diff --git a/mm/memcontrol-v1.c b/mm/memcontrol-v1.c
index 7b010e165e1b..2abee9b4b2a4 100644
--- a/mm/memcontrol-v1.c
+++ b/mm/memcontrol-v1.c
@@ -674,12 +674,25 @@ void memcg1_swapin(swp_entry_t entry, unsigned int nr_pages)
* page to memory here, and uncharge swap when the slot is freed.
*/
if (do_memsw_account()) {
+ unsigned short id = lookup_swap_cgroup_id(entry);
+ struct mem_cgroup *memcg;
+
+ rcu_read_lock();
+ memcg = id ? mem_cgroup_from_id(id) : NULL;
+ rcu_read_unlock();
+
/*
* The swap entry might not get freed for a long time,
* let's not wait for it. The page already received a
* memory+swap charge, drop the swap entry duplicate.
*/
- mem_cgroup_uncharge_swap(entry, nr_pages);
+ mem_cgroup_uncharge_swap(memcg, nr_pages);
+
+ /*
+ * Clear the cgroup association now to prevent double memsw
+ * uncharging when the backends are released later.
+ */
+ mem_cgroup_clear_swap(memcg, entry, nr_pages);
}
}
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 4ff24bd53bd7..1ff2b1002598 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -3597,7 +3597,7 @@ void __maybe_unused mem_cgroup_id_get_many(struct mem_cgroup *memcg,
refcount_add(n, &memcg->id.ref);
}
-static void mem_cgroup_id_put_many(struct mem_cgroup *memcg, unsigned int n)
+void mem_cgroup_id_put_many(struct mem_cgroup *memcg, unsigned int n)
{
if (refcount_sub_and_test(n, &memcg->id.ref)) {
mem_cgroup_id_remove(memcg);
@@ -5173,20 +5173,17 @@ int __init mem_cgroup_init(void)
#ifdef CONFIG_SWAP
/**
- * __mem_cgroup_try_charge_swap - try charging swap space for a folio
- * @folio: folio being added to swap
- * @entry: swap entry to charge
- *
- * Try to charge @folio's memcg for the swap space at @entry.
+ * __mem_cgroup_get_swap_memcgid - get the pinned memcg ID for swap accounting.
+ * @folio: folio being swapped out.
*
- * Returns 0 on success, -ENOMEM on failure.
+ * Return: the memcg ID with references pinned, or 0 if not applicable.
*/
-int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry)
+unsigned short __mem_cgroup_get_swap_memcgid(struct folio *folio)
{
unsigned int nr_pages = folio_nr_pages(folio);
- struct page_counter *counter;
struct mem_cgroup *memcg;
+ /* Recording will be done by memcg1_swapout(). */
if (do_memsw_account())
return 0;
@@ -5196,98 +5193,99 @@ int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry)
if (!memcg)
return 0;
- if (!entry.val) {
- memcg_memory_event(memcg, MEMCG_SWAP_FAIL);
- return 0;
- }
-
memcg = mem_cgroup_id_get_online(memcg);
+ if (nr_pages > 1)
+ mem_cgroup_id_get_many(memcg, nr_pages - 1);
+ return mem_cgroup_id(memcg);
+}
+
+/**
+ * __mem_cgroup_clear_swap - clear cgroup information of the swap entries.
+ * @memcg: the mem_cgroup recorded for @entry (may be NULL).
+ * @entry: the first swap entry in the range.
+ * @nr_pages: the number of pages in the range.
+ *
+ * The caller must guarantee that @memcg is still alive; normally this is
+ * because the swap_cgroup id reference taken at swapout time is still held.
+ * The reference is dropped here.
+ */
+void __mem_cgroup_clear_swap(struct mem_cgroup *memcg, swp_entry_t entry,
+ unsigned int nr_pages)
+{
+ swap_cgroup_clear(entry, nr_pages);
+ if (memcg)
+ mem_cgroup_id_put_many(memcg, nr_pages);
+}
+
+/**
+ * __mem_cgroup_try_charge_swap - try charging swap space for a folio
+ * @memcg: the mem_cgroup to charge (may be NULL)
+ * @nr_pages: number of swap pages to charge
+ *
+ * Try to charge @memcg for @nr_pages of swapfile slots.
+ *
+ * Returns 0 on success, -ENOMEM on failure.
+ */
+int __mem_cgroup_try_charge_swap(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
+{
+ struct page_counter *counter;
+
+ if (do_memsw_account())
+ return 0;
+ if (!memcg)
+ return 0;
if (!mem_cgroup_is_root(memcg) &&
!page_counter_try_charge(&memcg->swap, nr_pages, &counter)) {
memcg_memory_event(memcg, MEMCG_SWAP_MAX);
memcg_memory_event(memcg, MEMCG_SWAP_FAIL);
- mem_cgroup_id_put(memcg);
return -ENOMEM;
}
- /* Get references for the tail pages, too */
- if (nr_pages > 1)
- mem_cgroup_id_get_many(memcg, nr_pages - 1);
mod_memcg_state(memcg, MEMCG_SWAP, nr_pages);
-
- swap_cgroup_record(folio, mem_cgroup_id(memcg), entry);
-
return 0;
}
/**
* __mem_cgroup_uncharge_swap - uncharge swap space
- * @entry: swap entry to uncharge
- * @nr_pages: the amount of swap space to uncharge
- */
-void __mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_pages)
-{
- struct mem_cgroup *memcg;
- unsigned short id;
-
- id = swap_cgroup_clear(entry, nr_pages);
- rcu_read_lock();
- memcg = mem_cgroup_from_id(id);
- if (memcg) {
- if (!mem_cgroup_is_root(memcg)) {
- if (do_memsw_account())
- page_counter_uncharge(&memcg->memsw, nr_pages);
- else
- page_counter_uncharge(&memcg->swap, nr_pages);
- }
- mod_memcg_state(memcg, MEMCG_SWAP, -nr_pages);
- mem_cgroup_id_put_many(memcg, nr_pages);
- }
- rcu_read_unlock();
-}
-
-/**
- * __mem_cgroup_uncharge_swap_by_id - uncharge swap space using memcg id directly
- * @id: mem_cgroup id to uncharge
+ * @memcg: the mem_cgroup to uncharge (may be NULL)
* @nr_pages: the amount of swap space to uncharge
*
- * Same as __mem_cgroup_uncharge_swap() but takes the memcg id directly,
- * skipping the lookup_swap_cgroup_id() call. Use when the caller already
- * knows the memcg id (e.g. from swp_desc->memcgid).
+ * Callers must guarantee @memcg is still alive (swap_cgroup id ref is
+ * normally what pins it).
*/
-void __mem_cgroup_uncharge_swap_by_id(unsigned short id,
- unsigned int nr_pages)
+void __mem_cgroup_uncharge_swap(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
{
- struct mem_cgroup *memcg;
+ if (!memcg)
+ return;
- rcu_read_lock();
- memcg = mem_cgroup_from_id(id);
- if (memcg) {
- if (!mem_cgroup_is_root(memcg)) {
- if (do_memsw_account())
- page_counter_uncharge(&memcg->memsw, nr_pages);
- else
- page_counter_uncharge(&memcg->swap, nr_pages);
- }
- mod_memcg_state(memcg, MEMCG_SWAP, -nr_pages);
- mem_cgroup_id_put_many(memcg, nr_pages);
+ if (!mem_cgroup_is_root(memcg)) {
+ if (do_memsw_account())
+ page_counter_uncharge(&memcg->memsw, nr_pages);
+ else
+ page_counter_uncharge(&memcg->swap, nr_pages);
}
- rcu_read_unlock();
+ mod_memcg_state(memcg, MEMCG_SWAP, -nr_pages);
}
static bool mem_cgroup_may_zswap(struct mem_cgroup *original_memcg);
long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg)
{
- long nr_swap_pages, nr_zswap_pages = 0;
+ long nr_swap_pages;
if (zswap_is_enabled() && (mem_cgroup_disabled() || do_memsw_account() ||
mem_cgroup_may_zswap(memcg))) {
- nr_zswap_pages = PAGE_COUNTER_MAX;
+ /*
+ * No need to check swap cgroup limits, since zswap is not charged
+ * towards swap consumption.
+ */
+ return PAGE_COUNTER_MAX;
}
- nr_swap_pages = max_t(long, nr_zswap_pages, get_nr_swap_pages());
+ nr_swap_pages = get_nr_swap_pages();
if (mem_cgroup_disabled() || do_memsw_account())
return nr_swap_pages;
for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg))
diff --git a/mm/vswap.c b/mm/vswap.c
index 195ba3520e54..3f86bbb3a5ea 100644
--- a/mm/vswap.c
+++ b/mm/vswap.c
@@ -14,6 +14,7 @@
#include <linux/suspend.h>
#include <linux/vmalloc.h>
#include "internal.h"
+#include "memcontrol-v1.h"
#include "swap.h"
#include "swap_table.h"
@@ -297,7 +298,7 @@ static bool cluster_is_alloc_candidate(struct vswap_cluster *cluster)
}
static void __vswap_alloc_from_cluster(struct vswap_cluster *cluster,
- int start, struct folio *folio)
+ int start, struct folio *folio, unsigned short memcgid)
{
int i, nr = 1 << cluster->order;
struct swp_desc *desc;
@@ -307,7 +308,7 @@ static void __vswap_alloc_from_cluster(struct vswap_cluster *cluster,
desc->type = VSWAP_FOLIO;
desc->swap_cache = folio;
#ifdef CONFIG_MEMCG
- desc->memcgid = 0;
+ desc->memcgid = memcgid;
#endif
desc->swap_count = 0;
desc->in_swapcache = true;
@@ -316,7 +317,7 @@ static void __vswap_alloc_from_cluster(struct vswap_cluster *cluster,
}
static unsigned long vswap_alloc_from_cluster(struct vswap_cluster *cluster,
- struct folio *folio)
+ struct folio *folio, unsigned short memcgid)
{
int nr = 1 << cluster->order;
unsigned long i = cluster->id ? 0 : nr;
@@ -335,20 +336,31 @@ static unsigned long vswap_alloc_from_cluster(struct vswap_cluster *cluster,
bitmap_set(cluster->bitmap, i, nr);
refcount_add(nr, &cluster->refcnt);
- __vswap_alloc_from_cluster(cluster, i, folio);
+ __vswap_alloc_from_cluster(cluster, i, folio, memcgid);
return i + (cluster->id << VSWAP_CLUSTER_SHIFT);
}
-/* Allocate a contiguous range of virtual swap slots */
-static swp_entry_t vswap_alloc(struct folio *folio)
+/**
+ * folio_alloc_swap - allocate virtual swap space for a folio.
+ * @folio: the folio.
+ *
+ * Return: 0 on success, -ENOMEM on failure.
+ */
+int folio_alloc_swap(struct folio *folio)
{
struct xa_limit limit = vswap_cluster_map_limit;
struct vswap_cluster *local, *cluster;
+ struct mem_cgroup *memcg;
int order = folio_order(folio), nr = 1 << order;
+ unsigned short memcgid;
bool need_caching = true;
u32 cluster_id;
swp_entry_t entry;
+ VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
+ VM_BUG_ON_FOLIO(!folio_test_uptodate(folio), folio);
+
+ memcgid = mem_cgroup_get_swap_memcgid(folio);
entry.val = 0;
/* first, let's try the locally cached cluster */
@@ -357,7 +369,7 @@ static swp_entry_t vswap_alloc(struct folio *folio)
cluster = this_cpu_read(percpu_vswap_cluster.clusters[order]);
if (cluster) {
spin_lock(&cluster->lock);
- entry.val = vswap_alloc_from_cluster(cluster, folio);
+ entry.val = vswap_alloc_from_cluster(cluster, folio, memcgid);
need_caching = !entry.val;
if (!entry.val || !cluster_is_alloc_candidate(cluster)) {
@@ -384,7 +396,7 @@ static swp_entry_t vswap_alloc(struct folio *folio)
if (!spin_trylock(&cluster->lock))
continue;
- entry.val = vswap_alloc_from_cluster(cluster, folio);
+ entry.val = vswap_alloc_from_cluster(cluster, folio, memcgid);
list_del_init(&cluster->list);
cluster->full = !entry.val || !cluster_is_alloc_candidate(cluster);
need_caching = !cluster->full;
@@ -424,7 +436,7 @@ static swp_entry_t vswap_alloc(struct folio *folio)
if (!cluster_id)
entry.val += nr;
__vswap_alloc_from_cluster(cluster,
- (entry.val & VSWAP_CLUSTER_MASK), folio);
+ (entry.val & VSWAP_CLUSTER_MASK), folio, memcgid);
/* Mark the allocated range in the bitmap */
bitmap_set(cluster->bitmap, (entry.val & VSWAP_CLUSTER_MASK), nr);
need_caching = cluster_is_alloc_candidate(cluster);
@@ -476,10 +488,25 @@ static swp_entry_t vswap_alloc(struct folio *folio)
if (entry.val) {
VM_WARN_ON(entry.val + nr - 1 > MAX_VSWAP);
atomic_add(nr, &vswap_used);
- } else {
- atomic_add(nr, &vswap_alloc_reject);
+
+ folio_ref_add(folio, nr);
+ folio_set_swapcache(folio);
+ folio->swap = entry;
+
+ node_stat_mod_folio(folio, NR_FILE_PAGES, nr);
+ lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr);
+ return 0;
+ }
+
+ atomic_add(nr, &vswap_alloc_reject);
+ if (memcgid) {
+ rcu_read_lock();
+ memcg = mem_cgroup_from_id(memcgid);
+ rcu_read_unlock();
+ if (memcg)
+ mem_cgroup_id_put_many(memcg, nr);
}
- return entry;
+ return -ENOMEM;
}
static void vswap_cluster_free_rcu(struct rcu_head *head)
@@ -570,15 +597,21 @@ static unsigned short swp_desc_memcgid(struct swp_desc *desc);
* but the only vswap function called under ci->lock is vswap_rmap_set(),
* which uses atomic ops and does not take cluster->lock. So there is no
* ABBA deadlock risk.
+ *
+ * 4. Callers must ensure the range belongs to a single memcg, so we can
+ * read the memcg id once from the first descriptor and use it for all
+ * uncharge calls.
*/
static void release_backing(struct vswap_cluster *cluster, swp_entry_t entry,
- int nr)
+ int nr, struct mem_cgroup *memcg)
{
struct swp_desc *desc;
unsigned long flush_nr, phys_swap_start = 0, phys_swap_end = 0;
unsigned int phys_swap_type = 0;
bool need_flushing_phys_swap = false;
swp_slot_t flush_slot;
+ unsigned short batch_memcgid = 0;
+ unsigned int nr_swapfile = 0;
int i;
VM_WARN_ON(!entry.val);
@@ -587,6 +620,9 @@ static void release_backing(struct vswap_cluster *cluster, swp_entry_t entry,
desc = __vswap_iter(cluster, entry.val + i);
VM_WARN_ON(!desc);
+ if (!i && !memcg)
+ batch_memcgid = swp_desc_memcgid(desc);
+
/*
* We batch contiguous physical swap slots for more efficient
* freeing.
@@ -604,6 +640,7 @@ static void release_backing(struct vswap_cluster *cluster, swp_entry_t entry,
if (desc->type == VSWAP_ZSWAP && desc->zswap_entry) {
zswap_entry_free(desc->zswap_entry);
} else if (desc->type == VSWAP_SWAPFILE) {
+ nr_swapfile++;
if (!phys_swap_start) {
/* start a new contiguous range of phys swap */
phys_swap_start = swp_slot_offset(desc->slot);
@@ -629,8 +666,44 @@ static void release_backing(struct vswap_cluster *cluster, swp_entry_t entry,
flush_nr = phys_swap_end - phys_swap_start;
swap_slot_free_nr(flush_slot, flush_nr);
}
+
+ /*
+ * Release the swap-side cgroup accounting for the freed range.
+ *
+ * On cgroup v1 with memsw accounting, memcg1_swapout() carries over a
+ * memsw charge for every swapped-out entry regardless of backing, so
+ * every freed entry needs a matching uncharge here (count = nr).
+ *
+ * On cgroup v2, memcg->swap is only charged via
+ * __mem_cgroup_try_charge_swap() when we allocate a swapfile slot.
+ * Only descs that are currently VSWAP_SWAPFILE need uncharging
+ * (count = nr_swapfile).
+ *
+ * mem_cgroup_uncharge_swap() routes to the right counter and
+ * adjusts the MEMCG_SWAP stat by the same count.
+ *
+ * On v1, desc->memcgid is only non-zero between memcg1_swapout() and
+ * memcg1_swapin(); all non-free release_backing() callers run outside
+ * that window, so the uncharge is a no-op for them. Those callers
+ * pass @memcg == NULL; we resolve it from the recorded id lazily,
+ * only if an uncharge is actually needed.
+ */
+ if (!do_memsw_account() && !nr_swapfile)
+ return;
+
+ if (!memcg && batch_memcgid)
+ memcg = mem_cgroup_from_id(batch_memcgid);
+
+ if (do_memsw_account())
+ mem_cgroup_uncharge_swap(memcg, nr);
+ else
+ mem_cgroup_uncharge_swap(memcg, nr_swapfile);
}
+static void __vswap_swap_cgroup_clear(struct vswap_cluster *cluster,
+ swp_entry_t entry, unsigned int nr_ents,
+ struct mem_cgroup *memcg);
+
/*
* Entered with the cluster locked. The cluster lock is held throughout.
*
@@ -650,64 +723,23 @@ static void release_backing(struct vswap_cluster *cluster, swp_entry_t entry,
static void vswap_free(struct vswap_cluster *cluster, struct swp_desc *desc,
swp_entry_t entry)
{
- unsigned short memcgid = swp_desc_memcgid(desc);
+ unsigned short id = swp_desc_memcgid(desc);
+ struct mem_cgroup *memcg;
- release_backing(cluster, entry, 1);
+ /*
+ * The swap_cgroup id reference taken at swapout time pins this
+ * memcg until swap_cgroup_clear() runs below, so we can resolve
+ * it once here and pass it down. Caller already holds rcu.
+ */
+ memcg = id ? mem_cgroup_from_id(id) : NULL;
- mem_cgroup_uncharge_swap_by_id(memcgid, 1);
+ release_backing(cluster, entry, 1, memcg);
+ __vswap_swap_cgroup_clear(cluster, entry, 1, memcg);
/* erase forward mapping and release the virtual slot for reallocation */
release_vswap_slot(cluster, entry.val);
}
-/**
- * folio_alloc_swap - allocate virtual swap space for a folio.
- * @folio: the folio.
- *
- * Return: 0, if the allocation succeeded, -ENOMEM, if the allocation failed.
- */
-int folio_alloc_swap(struct folio *folio)
-{
- struct vswap_cluster *cluster = NULL;
- int i, nr = folio_nr_pages(folio);
- struct swp_desc *desc;
- swp_entry_t entry;
-
- VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
- VM_BUG_ON_FOLIO(!folio_test_uptodate(folio), folio);
-
- entry = vswap_alloc(folio);
- if (!entry.val)
- return -ENOMEM;
-
- /*
- * XXX: for now, we charge towards the memory cgroup's swap limit on virtual
- * swap slots allocation. This will be changed soon - we will only charge on
- * physical swap slots allocation.
- */
- if (mem_cgroup_try_charge_swap(folio, entry)) {
- rcu_read_lock();
- for (i = 0; i < nr; i++) {
- desc = vswap_iter(&cluster, entry.val + i);
- VM_WARN_ON(!desc);
- vswap_free(cluster, desc, (swp_entry_t){ entry.val + i });
- }
- spin_unlock(&cluster->lock);
- rcu_read_unlock();
- atomic_add(nr, &vswap_alloc_reject);
- entry.val = 0;
- return -ENOMEM;
- }
-
- folio_ref_add(folio, nr);
- folio_set_swapcache(folio);
- folio->swap = entry;
-
- node_stat_mod_folio(folio, NR_FILE_PAGES, nr);
- lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr);
-
- return 0;
-}
/**
* vswap_alloc_swap_slot - allocate physical swap space for a folio that is
@@ -726,19 +758,32 @@ swp_slot_t vswap_alloc_swap_slot(struct folio *folio)
swp_slot_t slot = { .val = 0 };
swp_entry_t entry = folio->swap;
struct swp_desc *desc;
+ struct mem_cgroup *memcg;
+ unsigned short memcgid;
bool fallback = false;
/*
* Check if the first entry already has a physical swap slot. If so,
* the entire range is contiguous (from a previous allocation).
* Keep the cluster pointer around for reuse below.
+ *
+ * Also resolve the swap memcg while we are in the same RCU section:
+ * the folio's memcg almost always matches the recorded swap memcg, so
+ * compare ids first and only fall back to the IDR lookup on mismatch.
+ * The swap memcg is pinned by the swap_cgroup id ref, so we can safely
+ * use it outside RCU after this point.
*/
rcu_read_lock();
desc = vswap_iter(&cluster, entry.val);
VM_WARN_ON(!desc);
if (desc->type == VSWAP_SWAPFILE)
slot = desc->slot;
+ memcgid = swp_desc_memcgid(desc);
spin_unlock(&cluster->lock);
+
+ memcg = folio_memcg(folio);
+ if (!memcg || mem_cgroup_id(memcg) != memcgid)
+ memcg = memcgid ? mem_cgroup_from_id(memcgid) : NULL;
rcu_read_unlock();
if (slot.val)
@@ -750,6 +795,15 @@ swp_slot_t vswap_alloc_swap_slot(struct folio *folio)
if (!slot.val)
return (swp_slot_t){ .val = 0 };
+ if (mem_cgroup_try_charge_swap(memcg, nr)) {
+ /*
+ * We have not updated the backing type of the virtual swap slot.
+ * Simply free up the physical swap slots here!
+ */
+ swap_slot_free_nr(slot, nr);
+ return (swp_slot_t){ .val = 0 };
+ }
+
vswap_rmap_set(__swap_slot_to_cluster(slot), slot, entry.val, nr);
spin_lock(&cluster->lock);
@@ -1181,7 +1235,7 @@ void vswap_store_folio(swp_entry_t entry, struct folio *folio)
rcu_read_lock();
desc = vswap_iter(&cluster, entry.val);
VM_WARN_ON(!desc);
- release_backing(cluster, entry, nr);
+ release_backing(cluster, entry, nr, NULL);
for (i = 0; i < nr; i++) {
desc = __vswap_iter(cluster, entry.val + i);
@@ -1232,7 +1286,7 @@ void vswap_prepare_writeout(swp_entry_t entry, struct folio *folio)
}
/* Release old backing and store the folio, lock already held */
- release_backing(cluster, entry, nr);
+ release_backing(cluster, entry, nr, NULL);
for (i = 0; i < nr; i++) {
desc = __vswap_iter(cluster, entry.val + i);
@@ -1263,7 +1317,7 @@ void swap_zeromap_folio_set(struct folio *folio)
rcu_read_lock();
desc = vswap_iter(&cluster, entry.val);
VM_WARN_ON(!desc);
- release_backing(cluster, entry, nr);
+ release_backing(cluster, entry, nr, NULL);
for (i = 0; i < nr; i++) {
desc = __vswap_iter(cluster, entry.val + i);
@@ -1949,7 +2003,7 @@ void zswap_entry_store(swp_entry_t swpentry, struct zswap_entry *entry)
rcu_read_lock();
desc = vswap_iter(&cluster, swpentry.val);
VM_WARN_ON(!desc);
- release_backing(cluster, swpentry, 1);
+ release_backing(cluster, swpentry, 1, NULL);
desc->zswap_entry = entry;
desc->type = VSWAP_ZSWAP;
spin_unlock(&cluster->lock);
@@ -2023,6 +2077,23 @@ static unsigned short __vswap_cgroup_record(struct vswap_cluster *cluster,
return oldid;
}
+/*
+ * Clear swap cgroup for a range of swap entries.
+ * Entered with the cluster locked. Caller must be under rcu_read_lock().
+ *
+ * @memcg is the mem_cgroup recorded in the descriptors (may be NULL). The
+ * caller is expected to have resolved it once; the swap_cgroup id reference
+ * dropped here.
+ */
+static void __vswap_swap_cgroup_clear(struct vswap_cluster *cluster,
+ swp_entry_t entry, unsigned int nr_ents,
+ struct mem_cgroup *memcg)
+{
+ __vswap_cgroup_record(cluster, entry, 0, nr_ents);
+ if (memcg)
+ mem_cgroup_id_put_many(memcg, nr_ents);
+}
+
static unsigned short vswap_cgroup_record(swp_entry_t entry,
unsigned short memcgid, unsigned int nr_ents)
{
@@ -2040,30 +2111,10 @@ static unsigned short vswap_cgroup_record(swp_entry_t entry,
return oldid;
}
-/**
- * swap_cgroup_record - record mem_cgroup for a set of swap entries.
- * These entries must belong to one single folio, and that folio
- * must be being charged for swap space (swap out), and these
- * entries must not have been charged
- *
- * @folio: the folio that the swap entry belongs to
- * @memcgid: mem_cgroup ID to be recorded
- * @entry: the first swap entry to be recorded
- */
-void swap_cgroup_record(struct folio *folio, unsigned short memcgid,
- swp_entry_t entry)
-{
- unsigned short oldid =
- vswap_cgroup_record(entry, memcgid, folio_nr_pages(folio));
-
- VM_WARN_ON(oldid);
-}
-
/**
* __swap_cgroup_record - record mem_cgroup for a set of swap entries.
*
- * Same as swap_cgroup_record, but assumes the swap cache (vswap cluster)
- * lock is already held.
+ * Assumes the swap cache (vswap cluster) lock is already held.
*
* @folio: the folio that the swap entry belongs to
* @memcgid: mem_cgroup ID to be recorded
@@ -2170,6 +2221,12 @@ static unsigned short swp_desc_memcgid(struct swp_desc *desc)
return desc->memcgid;
}
#else /* !CONFIG_MEMCG */
+static void __vswap_swap_cgroup_clear(struct vswap_cluster *cluster,
+ swp_entry_t entry, unsigned int nr_ents,
+ struct mem_cgroup *memcg)
+{
+}
+
static unsigned short swp_desc_memcgid(struct swp_desc *desc)
{
return 0;
--
2.52.0
^ permalink raw reply related
* [PATCH v6 17/22] swapfile: remove zeromap bitmap
From: Nhat Pham @ 2026-05-05 15:38 UTC (permalink / raw)
To: kasong
Cc: Liam.Howlett, akpm, apopple, axelrasmussen, baohua, baolin.wang,
bhe, byungchul, cgroups, chengming.zhou, chrisl, corbet, david,
dev.jain, gourry, hannes, hughd, jannh, joshua.hahnjy, lance.yang,
lenb, linux-doc, linux-kernel, linux-mm, linux-pm,
lorenzo.stoakes, matthew.brost, mhocko, muchun.song, npache,
nphamcs, pavel, peterx, peterz, pfalcato, rafael, rakie.kim,
roman.gushchin, rppt, ryan.roberts, shakeel.butt, shikemeng,
surenb, tglx, vbabka, weixugc, ying.huang, yosry.ahmed, yuanchu,
zhengqi.arch, ziy, kernel-team, riel, haowenchao22
In-Reply-To: <20260505153854.1612033-1-nphamcs@gmail.com>
Zero swap entries are now treated as a separate, decoupled backend in
the virtual swap layer. The zeromap bitmap of physical swapfile is no
longer used - remove it. This does not have any behavioral change, and
save 1 bit per swap page in terms of memory overhead.
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
include/linux/swap.h | 1 -
mm/swapfile.c | 30 +++++-------------------------
2 files changed, 5 insertions(+), 26 deletions(-)
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 12dd9621b637..4d152fa811f2 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -260,7 +260,6 @@ struct swap_info_struct {
signed char type; /* strange name for an index */
unsigned int max; /* extent of the swap_map */
unsigned char *swap_map; /* vmalloc'ed array of usage counts */
- unsigned long *zeromap; /* kvmalloc'ed bitmap to track zero pages */
struct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */
struct list_head free_clusters; /* free clusters list */
struct list_head full_clusters; /* full clusters list */
diff --git a/mm/swapfile.c b/mm/swapfile.c
index c6a91c657877..ce1254733e96 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -2276,8 +2276,7 @@ static int setup_swap_extents(struct swap_info_struct *sis, sector_t *span)
static void setup_swap_info(struct swap_info_struct *si, int prio,
unsigned char *swap_map,
- struct swap_cluster_info *cluster_info,
- unsigned long *zeromap)
+ struct swap_cluster_info *cluster_info)
{
si->prio = prio;
/*
@@ -2288,7 +2287,6 @@ static void setup_swap_info(struct swap_info_struct *si, int prio,
si->avail_list.prio = -si->prio;
si->swap_map = swap_map;
si->cluster_info = cluster_info;
- si->zeromap = zeromap;
}
static void _enable_swap_info(struct swap_info_struct *si)
@@ -2306,12 +2304,11 @@ static void _enable_swap_info(struct swap_info_struct *si)
static void enable_swap_info(struct swap_info_struct *si, int prio,
unsigned char *swap_map,
- struct swap_cluster_info *cluster_info,
- unsigned long *zeromap)
+ struct swap_cluster_info *cluster_info)
{
spin_lock(&swap_lock);
spin_lock(&si->lock);
- setup_swap_info(si, prio, swap_map, cluster_info, zeromap);
+ setup_swap_info(si, prio, swap_map, cluster_info);
spin_unlock(&si->lock);
spin_unlock(&swap_lock);
/*
@@ -2329,7 +2326,7 @@ static void reinsert_swap_info(struct swap_info_struct *si)
{
spin_lock(&swap_lock);
spin_lock(&si->lock);
- setup_swap_info(si, si->prio, si->swap_map, si->cluster_info, si->zeromap);
+ setup_swap_info(si, si->prio, si->swap_map, si->cluster_info);
_enable_swap_info(si);
spin_unlock(&si->lock);
spin_unlock(&swap_lock);
@@ -2400,7 +2397,6 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
{
struct swap_info_struct *p = NULL;
unsigned char *swap_map;
- unsigned long *zeromap;
struct swap_cluster_info *cluster_info;
struct file *swap_file, *victim;
struct address_space *mapping;
@@ -2495,8 +2491,6 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
p->swap_file = NULL;
swap_map = p->swap_map;
p->swap_map = NULL;
- zeromap = p->zeromap;
- p->zeromap = NULL;
maxpages = p->max;
cluster_info = p->cluster_info;
p->max = 0;
@@ -2508,7 +2502,6 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
kfree(p->global_cluster);
p->global_cluster = NULL;
vfree(swap_map);
- kvfree(zeromap);
free_cluster_info(cluster_info, maxpages);
inode = mapping->host;
@@ -2972,7 +2965,6 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
sector_t span;
unsigned long maxpages;
unsigned char *swap_map = NULL;
- unsigned long *zeromap = NULL;
struct swap_cluster_info *cluster_info = NULL;
struct folio *folio = NULL;
struct inode *inode = NULL;
@@ -3078,17 +3070,6 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
if (error)
goto bad_swap_unlock_inode;
- /*
- * Use kvmalloc_array instead of bitmap_zalloc as the allocation order might
- * be above MAX_PAGE_ORDER incase of a large swap file.
- */
- zeromap = kvmalloc_array(BITS_TO_LONGS(maxpages), sizeof(long),
- GFP_KERNEL | __GFP_ZERO);
- if (!zeromap) {
- error = -ENOMEM;
- goto bad_swap_unlock_inode;
- }
-
if (si->bdev && bdev_stable_writes(si->bdev))
si->flags |= SWP_STABLE_WRITES;
@@ -3155,7 +3136,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
prio = DEF_SWAP_PRIO;
if (swap_flags & SWAP_FLAG_PREFER)
prio = swap_flags & SWAP_FLAG_PRIO_MASK;
- enable_swap_info(si, prio, swap_map, cluster_info, zeromap);
+ enable_swap_info(si, prio, swap_map, cluster_info);
pr_info("Adding %uk swap on %s. Priority:%d extents:%d across:%lluk %s%s%s%s\n",
K(si->pages), name->name, si->prio, nr_extents,
@@ -3183,7 +3164,6 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
si->flags = 0;
spin_unlock(&swap_lock);
vfree(swap_map);
- kvfree(zeromap);
if (cluster_info)
free_cluster_info(cluster_info, maxpages);
if (inced_nr_rotate_swap)
--
2.52.0
^ permalink raw reply related
* [PATCH v6 16/22] swap: do not unnecessarily pin readahead swap entries
From: Nhat Pham @ 2026-05-05 15:38 UTC (permalink / raw)
To: kasong
Cc: Liam.Howlett, akpm, apopple, axelrasmussen, baohua, baolin.wang,
bhe, byungchul, cgroups, chengming.zhou, chrisl, corbet, david,
dev.jain, gourry, hannes, hughd, jannh, joshua.hahnjy, lance.yang,
lenb, linux-doc, linux-kernel, linux-mm, linux-pm,
lorenzo.stoakes, matthew.brost, mhocko, muchun.song, npache,
nphamcs, pavel, peterx, peterz, pfalcato, rafael, rakie.kim,
roman.gushchin, rppt, ryan.roberts, shakeel.butt, shikemeng,
surenb, tglx, vbabka, weixugc, ying.huang, yosry.ahmed, yuanchu,
zhengqi.arch, ziy, kernel-team, riel, haowenchao22
In-Reply-To: <20260505153854.1612033-1-nphamcs@gmail.com>
When we perform swap readahead, the target entry is already pinned by
the caller. No need to pin swap entries in the readahead window that
belongs in the same virtual swap cluster as the target swap entry.
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
mm/swap.h | 1 +
mm/swap_state.c | 22 +++++++++-------------
mm/vswap.c | 10 ++++++++++
3 files changed, 20 insertions(+), 13 deletions(-)
diff --git a/mm/swap.h b/mm/swap.h
index d46b2f243f83..f3f2afc6751b 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -156,6 +156,7 @@ void swap_cache_lock(swp_entry_t entry);
void swap_cache_unlock(swp_entry_t entry);
void vswap_rmap_set(struct swap_cluster_info *ci, swp_slot_t slot,
unsigned long vswap, int nr);
+bool vswap_same_cluster(swp_entry_t entry1, swp_entry_t entry2);
static inline struct address_space *swap_address_space(swp_entry_t entry)
{
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 7fe4d9529e4a..39daea7bed66 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -553,22 +553,18 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask,
pte_unmap(pte);
pte = NULL;
/*
- * Readahead entry may come from a device that we are not
- * holding a reference to, try to grab a reference, or skip.
- *
- * XXX: for now, always try to pin the swap entries in the
- * readahead window to avoid the annoying conversion to physical
- * swap slots. Once we move all swap metadata to virtual swap
- * layer, we can simply compare the clusters of the target
- * swap entry and the current swap entry, and pin the latter
- * swap entry's cluster if it differ from the former's.
+ * The target entry is already pinned - if the readahead entry
+ * belongs to the same cluster, it's already protected.
*/
- swapoff_locked = tryget_swap_entry(entry, &si);
- if (!swapoff_locked)
- continue;
+ if (!vswap_same_cluster(entry, targ_entry)) {
+ swapoff_locked = tryget_swap_entry(entry, &si);
+ if (!swapoff_locked)
+ continue;
+ }
folio = __read_swap_cache_async(entry, gfp_mask, mpol, ilx,
&page_allocated, false);
- put_swap_entry(entry, si);
+ if (swapoff_locked)
+ put_swap_entry(entry, si);
if (!folio)
continue;
if (page_allocated) {
diff --git a/mm/vswap.c b/mm/vswap.c
index 01c336ae252c..195ba3520e54 100644
--- a/mm/vswap.c
+++ b/mm/vswap.c
@@ -1542,6 +1542,16 @@ void put_swap_entry(swp_entry_t entry, struct swap_info_struct *si)
rcu_read_unlock();
}
+/*
+ * Check if two virtual swap entries belong to the same vswap cluster.
+ * Useful for optimizing readahead when entries in the same cluster
+ * share protection from a pinned target entry.
+ */
+bool vswap_same_cluster(swp_entry_t entry1, swp_entry_t entry2)
+{
+ return VSWAP_CLUSTER_IDX(entry1) == VSWAP_CLUSTER_IDX(entry2);
+}
+
static int vswap_cpu_dead(unsigned int cpu)
{
struct vswap_cluster *cluster;
--
2.52.0
^ permalink raw reply related
* [PATCH v6 15/22] zswap: do not start zswap shrinker if there is no physical swap slots
From: Nhat Pham @ 2026-05-05 15:38 UTC (permalink / raw)
To: kasong
Cc: Liam.Howlett, akpm, apopple, axelrasmussen, baohua, baolin.wang,
bhe, byungchul, cgroups, chengming.zhou, chrisl, corbet, david,
dev.jain, gourry, hannes, hughd, jannh, joshua.hahnjy, lance.yang,
lenb, linux-doc, linux-kernel, linux-mm, linux-pm,
lorenzo.stoakes, matthew.brost, mhocko, muchun.song, npache,
nphamcs, pavel, peterx, peterz, pfalcato, rafael, rakie.kim,
roman.gushchin, rppt, ryan.roberts, shakeel.butt, shikemeng,
surenb, tglx, vbabka, weixugc, ying.huang, yosry.ahmed, yuanchu,
zhengqi.arch, ziy, kernel-team, riel, haowenchao22
In-Reply-To: <20260505153854.1612033-1-nphamcs@gmail.com>
When swap is virtualized, we no longer pre-allocate a slot on swapfile
for each zswap entry. Do not start the zswap shrinker if there is no
physical swap slots available.
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
mm/zswap.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/mm/zswap.c b/mm/zswap.c
index e8aa9201ea30..751687b7e2b9 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -1214,6 +1214,14 @@ static unsigned long zswap_shrinker_count(struct shrinker *shrinker,
if (!zswap_shrinker_enabled || !mem_cgroup_zswap_writeback_enabled(memcg))
return 0;
+ /*
+ * When swap is virtualized, we do not have any swap slots on swapfile
+ * preallocated for zswap objects. If there is no slot available, we
+ * cannot writeback and should just bail out here.
+ */
+ if (!get_nr_swap_pages())
+ return 0;
+
/*
* The shrinker resumes swap writeback, which will enter block
* and may enter fs. XXX: Harmonize with vmscan.c __GFP_FS
--
2.52.0
^ permalink raw reply related
* [PATCH v6 14/22] mm: swap: decouple virtual swap slot from backing store
From: Nhat Pham @ 2026-05-05 15:38 UTC (permalink / raw)
To: kasong
Cc: Liam.Howlett, akpm, apopple, axelrasmussen, baohua, baolin.wang,
bhe, byungchul, cgroups, chengming.zhou, chrisl, corbet, david,
dev.jain, gourry, hannes, hughd, jannh, joshua.hahnjy, lance.yang,
lenb, linux-doc, linux-kernel, linux-mm, linux-pm,
lorenzo.stoakes, matthew.brost, mhocko, muchun.song, npache,
nphamcs, pavel, peterx, peterz, pfalcato, rafael, rakie.kim,
roman.gushchin, rppt, ryan.roberts, shakeel.butt, shikemeng,
surenb, tglx, vbabka, weixugc, ying.huang, yosry.ahmed, yuanchu,
zhengqi.arch, ziy, kernel-team, riel, haowenchao22
In-Reply-To: <20260505153854.1612033-1-nphamcs@gmail.com>
This patch presents the first real use case of the new virtual swap
design. It leverages the virtualization of the swap space to decouple a
swap entry and its backing storage. A swap entry can now be backed by
one of the following options:
1. A physical swap slot (i.e on a physical swapfile/swap partition).
2. A "zero swap page", i.e the swapped out page is a zero page.
3. A compressed object in the zswap pool.
4. An in-memory page. This can happen when a page is loaded
(exclusively) from the zswap pool, or if the page is rejected by
zswap and zswap writeback is disabled.
This allows us to use zswap and the zero swap page optimization, without
having to reserved a slot on a swapfile, or a swapfile at all. This
translates to tens to hundreds of GBs of disk saving on hosts and
workloads that have high memory usage, as well as removes this spurious
limit on the usage of these optimizations.
One implication of this change is that we need to be much more careful
with THP swapin and batched swap free operations. The central
requirement is the range of entries we are working with must
have no mixed backing states:
1. For now, zswap-backed entries are not supported for these batched
operations.
2. All the entries must be backed by the same type.
3. If the swap entries in the batch are backed by in-memory folio, it
must be the same folio (i.e they correspond to the subpages of that
folio).
4. If the swap entries in the batch are backed by slots on swapfiles, it
must be the same swapfile, and these physical swap slots must also be
contiguous.
For now, we still charge virtual swap slots towards the memcg's swap
usage. In a following patch, we will change this behavior and only
charge physical (i.e on swapfile) swap slots towards the memcg's swap
usage.
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
include/linux/swap.h | 34 ++-
include/linux/zswap.h | 3 +-
mm/internal.h | 22 +-
mm/memcontrol.c | 65 +++--
mm/memory.c | 84 ++++--
mm/page_io.c | 123 +++-----
mm/shmem.c | 6 +-
mm/swap.h | 35 +--
mm/swap_state.c | 29 +-
mm/swapfile.c | 11 +-
mm/vmscan.c | 19 +-
mm/vswap.c | 646 +++++++++++++++++++++++++++++++++---------
mm/zswap.c | 65 +++--
13 files changed, 787 insertions(+), 355 deletions(-)
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 0ec408a6c6a2..12dd9621b637 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -422,7 +422,14 @@ swp_slot_t swp_entry_to_swp_slot(swp_entry_t entry);
swp_entry_t swp_slot_to_swp_entry(swp_slot_t slot);
bool tryget_swap_entry(swp_entry_t entry, struct swap_info_struct **si);
void put_swap_entry(swp_entry_t entry, struct swap_info_struct *si);
-
+bool vswap_swapfile_backed(swp_entry_t entry, int nr);
+bool vswap_folio_backed(swp_entry_t entry, int nr);
+void vswap_store_folio(swp_entry_t entry, struct folio *folio);
+void vswap_prepare_writeout(swp_entry_t entry, struct folio *folio);
+void swap_zeromap_folio_set(struct folio *folio);
+void vswap_assoc_zswap(swp_entry_t entry, struct zswap_entry *zswap_entry);
+bool vswap_can_swapin_thp(swp_entry_t entry, int nr);
+swp_slot_t vswap_alloc_swap_slot(struct folio *folio);
/* Lifecycle swap API (mm/swapfile.c and mm/vswap.c) */
int folio_alloc_swap(struct folio *folio);
@@ -479,7 +486,15 @@ unsigned int count_swap_pages(int, int);
sector_t swapdev_block(int, pgoff_t);
struct backing_dev_info;
struct swap_info_struct *swap_slot_tryget_swap_info(swp_slot_t slot);
-sector_t swap_folio_sector(struct folio *folio);
+
+static inline struct swap_info_struct *vswap_get_device(swp_entry_t entry)
+{
+ swp_slot_t slot = swp_entry_to_swp_slot(entry);
+
+ return slot.val ? swap_slot_tryget_swap_info(slot) : NULL;
+}
+
+sector_t swap_slot_sector(swp_slot_t slot);
static inline void swap_slot_put_swap_info(struct swap_info_struct *si)
{
@@ -598,6 +613,21 @@ static inline int add_swap_extent(struct swap_info_struct *sis,
return -EINVAL;
}
+static inline bool vswap_swapfile_backed(swp_entry_t entry, int nr)
+{
+ return false;
+}
+
+static inline bool vswap_can_swapin_thp(swp_entry_t entry, int nr)
+{
+ return false;
+}
+
+static inline struct swap_info_struct *vswap_get_device(swp_entry_t entry)
+{
+ return NULL;
+}
+
#endif /* CONFIG_SWAP */
static inline void free_swap_and_cache(swp_entry_t entry)
diff --git a/include/linux/zswap.h b/include/linux/zswap.h
index 07b2936c38f2..f33b4433a5ee 100644
--- a/include/linux/zswap.h
+++ b/include/linux/zswap.h
@@ -33,9 +33,8 @@ void zswap_lruvec_state_init(struct lruvec *lruvec);
void zswap_folio_swapin(struct folio *folio);
bool zswap_is_enabled(void);
bool zswap_never_enabled(void);
-void *zswap_entry_store(swp_entry_t swpentry, struct zswap_entry *entry);
+void zswap_entry_store(swp_entry_t swpentry, struct zswap_entry *entry);
void *zswap_entry_load(swp_entry_t swpentry);
-void *zswap_entry_erase(swp_entry_t swpentry);
bool zswap_empty(swp_entry_t swpentry);
void zswap_entry_free(struct zswap_entry *entry);
diff --git a/mm/internal.h b/mm/internal.h
index 240103c8819b..df67a3c4168a 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -327,20 +327,6 @@ static inline swp_entry_t swap_nth(swp_entry_t entry, long n)
return (swp_entry_t) { entry.val + n };
}
-/* similar to swap_nth, but check the backing physical slots as well. */
-static inline swp_entry_t swap_move(swp_entry_t entry, long delta)
-{
- swp_slot_t slot = swp_entry_to_swp_slot(entry), next_slot;
- swp_entry_t next_entry = swap_nth(entry, delta);
-
- next_slot = swp_entry_to_swp_slot(next_entry);
- if (swp_slot_type(slot) != swp_slot_type(next_slot) ||
- swp_slot_offset(slot) + delta != swp_slot_offset(next_slot))
- next_entry.val = 0;
-
- return next_entry;
-}
-
/**
* pte_move_swp_offset - Move the swap entry offset field of a swap pte
* forward or backward by delta
@@ -354,7 +340,7 @@ static inline swp_entry_t swap_move(swp_entry_t entry, long delta)
static inline pte_t pte_move_swp_offset(pte_t pte, long delta)
{
softleaf_t entry = softleaf_from_pte(pte);
- pte_t new = swp_entry_to_pte(swap_move(entry, delta));
+ pte_t new = swp_entry_to_pte(swap_nth(entry, delta));
if (pte_swp_soft_dirty(pte))
new = pte_swp_mksoft_dirty(new);
@@ -419,6 +405,12 @@ static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte,
nr = ptep - start_ptep;
if (!free_batch)
nr = vswap_cgroup_batch(entry, nr);
+ /*
+ * free_swap_and_cache_nr can handle mixed backends, as long as virtual
+ * swap entries backing these PTEs are contiguous.
+ */
+ if (!free_batch && !vswap_can_swapin_thp(entry, nr))
+ return 1;
return nr;
}
#endif /* CONFIG_MMU */
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index e237cc1e644d..4ff24bd53bd7 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -5276,10 +5276,18 @@ void __mem_cgroup_uncharge_swap_by_id(unsigned short id,
rcu_read_unlock();
}
+static bool mem_cgroup_may_zswap(struct mem_cgroup *original_memcg);
+
long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg)
{
- long nr_swap_pages = get_nr_swap_pages();
+ long nr_swap_pages, nr_zswap_pages = 0;
+
+ if (zswap_is_enabled() && (mem_cgroup_disabled() || do_memsw_account() ||
+ mem_cgroup_may_zswap(memcg))) {
+ nr_zswap_pages = PAGE_COUNTER_MAX;
+ }
+ nr_swap_pages = max_t(long, nr_zswap_pages, get_nr_swap_pages());
if (mem_cgroup_disabled() || do_memsw_account())
return nr_swap_pages;
for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg))
@@ -5448,6 +5456,29 @@ static struct cftype swap_files[] = {
};
#ifdef CONFIG_ZSWAP
+static bool mem_cgroup_may_zswap(struct mem_cgroup *original_memcg)
+{
+ struct mem_cgroup *memcg;
+
+ for (memcg = original_memcg; !mem_cgroup_is_root(memcg);
+ memcg = parent_mem_cgroup(memcg)) {
+ unsigned long max = READ_ONCE(memcg->zswap_max);
+ unsigned long pages;
+
+ if (max == PAGE_COUNTER_MAX)
+ continue;
+ if (max == 0)
+ return false;
+
+ /* Force flush to get accurate stats for charging */
+ __mem_cgroup_flush_stats(memcg, true);
+ pages = memcg_page_state(memcg, MEMCG_ZSWAP_B) / PAGE_SIZE;
+ if (pages >= max)
+ return false;
+ }
+ return true;
+}
+
/**
* obj_cgroup_may_zswap - check if this cgroup can zswap
* @objcg: the object cgroup
@@ -5462,34 +5493,15 @@ static struct cftype swap_files[] = {
*/
bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)
{
- struct mem_cgroup *memcg, *original_memcg;
+ struct mem_cgroup *memcg;
bool ret = true;
if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
return true;
- original_memcg = get_mem_cgroup_from_objcg(objcg);
- for (memcg = original_memcg; !mem_cgroup_is_root(memcg);
- memcg = parent_mem_cgroup(memcg)) {
- unsigned long max = READ_ONCE(memcg->zswap_max);
- unsigned long pages;
-
- if (max == PAGE_COUNTER_MAX)
- continue;
- if (max == 0) {
- ret = false;
- break;
- }
-
- /* Force flush to get accurate stats for charging */
- __mem_cgroup_flush_stats(memcg, true);
- pages = memcg_page_state(memcg, MEMCG_ZSWAP_B) / PAGE_SIZE;
- if (pages < max)
- continue;
- ret = false;
- break;
- }
- mem_cgroup_put(original_memcg);
+ memcg = get_mem_cgroup_from_objcg(objcg);
+ ret = mem_cgroup_may_zswap(memcg);
+ mem_cgroup_put(memcg);
return ret;
}
@@ -5633,6 +5645,11 @@ static struct cftype zswap_files[] = {
},
{ } /* terminate */
};
+#else
+static inline bool mem_cgroup_may_zswap(struct mem_cgroup *original_memcg)
+{
+ return false;
+}
#endif /* CONFIG_ZSWAP */
static int __init mem_cgroup_swap_init(void)
diff --git a/mm/memory.c b/mm/memory.c
index d32d35e25738..5a8d7fe5f1c9 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4362,6 +4362,15 @@ static inline bool should_try_to_free_swap(struct folio *folio,
if (mem_cgroup_swap_full(folio) || (vma->vm_flags & VM_LOCKED) ||
folio_test_mlocked(folio))
return true;
+
+ /*
+ * Non-swapfile backends cannot be re-used for future swapouts anyway.
+ * Try to free swap space unless the folio is backed by contiguous
+ * physical swap slots.
+ */
+ if (!vswap_swapfile_backed(folio->swap, folio_nr_pages(folio)))
+ return true;
+
/*
* If we want to map a page that's in the swapcache writable, we
* have to detect via the refcount if we're really the exclusive
@@ -4623,12 +4632,12 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
struct folio *swapcache, *folio = NULL;
DECLARE_WAITQUEUE(wait, current);
struct page *page;
- struct swap_info_struct *si = NULL;
+ struct swap_info_struct *si = NULL, *stable_si;
rmap_t rmap_flags = RMAP_NONE;
bool need_clear_cache = false;
bool swapoff_locked = false;
bool exclusive = false;
- softleaf_t entry;
+ softleaf_t orig_entry, entry;
pte_t pte;
vm_fault_t ret = 0;
void *shadow = NULL;
@@ -4641,6 +4650,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
goto out;
entry = softleaf_from_pte(vmf->orig_pte);
+ /*
+ * entry might change if we get a large folio - remember the original entry
+ * for unlocking swapoff etc.
+ */
+ orig_entry = entry;
if (unlikely(!softleaf_is_swap(entry))) {
if (softleaf_is_migration(entry)) {
migration_entry_wait(vma->vm_mm, vmf->pmd,
@@ -4705,7 +4719,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
swapcache = folio;
if (!folio) {
- if (data_race(si->flags & SWP_SYNCHRONOUS_IO) &&
+ if (si && data_race(si->flags & SWP_SYNCHRONOUS_IO) &&
__swap_count(entry) == 1) {
/* skip swapcache */
folio = alloc_swap_folio(vmf);
@@ -4736,6 +4750,17 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
}
need_clear_cache = true;
+ /*
+ * Recheck to make sure the entire range is still
+ * THP-swapin-able. Note that before we call
+ * swapcache_prepare(), entries in the range can
+ * still have their backing status changed.
+ */
+ if (!vswap_can_swapin_thp(entry, nr_pages)) {
+ schedule_timeout_uninterruptible(1);
+ goto out_page;
+ }
+
memcg1_swapin(entry, nr_pages);
shadow = swap_cache_get_shadow(entry);
@@ -4916,27 +4941,40 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
* swapcache -> certainly exclusive.
*/
exclusive = true;
- } else if (exclusive && folio_test_writeback(folio) &&
- data_race(si->flags & SWP_STABLE_WRITES)) {
+ } else if (exclusive && folio_test_writeback(folio)) {
/*
- * This is tricky: not all swap backends support
- * concurrent page modifications while under writeback.
- *
- * So if we stumble over such a page in the swapcache
- * we must not set the page exclusive, otherwise we can
- * map it writable without further checks and modify it
- * while still under writeback.
+ * We need to look up the swap device again here, because
+ * the si we got from tryget_swap_entry() might have changed
+ * before we pin the backend.
*
- * For these problematic swap backends, simply drop the
- * exclusive marker: this is perfectly fine as we start
- * writeback only if we fully unmapped the page and
- * there are no unexpected references on the page after
- * unmapping succeeded. After fully unmapped, no
- * further GUP references (FOLL_GET and FOLL_PIN) can
- * appear, so dropping the exclusive marker and mapping
- * it only R/O is fine.
+ * With the folio locked and loaded into the swap cache, we can
+ * now guarantee a stable backing state.
*/
- exclusive = false;
+ stable_si = vswap_get_device(entry);
+ if (stable_si && data_race(stable_si->flags & SWP_STABLE_WRITES)) {
+ /*
+ * This is tricky: not all swap backends support
+ * concurrent page modifications while under writeback.
+ *
+ * So if we stumble over such a page in the swapcache
+ * we must not set the page exclusive, otherwise we can
+ * map it writable without further checks and modify it
+ * while still under writeback.
+ *
+ * For these problematic swap backends, simply drop the
+ * exclusive marker: this is perfectly fine as we start
+ * writeback only if we fully unmapped the page and
+ * there are no unexpected references on the page after
+ * unmapping succeeded. After fully unmapped, no
+ * further GUP references (FOLL_GET and FOLL_PIN) can
+ * appear, so dropping the exclusive marker and mapping
+ * it only R/O is fine.
+ */
+ exclusive = false;
+ }
+
+ if (stable_si)
+ swap_slot_put_swap_info(stable_si);
}
}
@@ -5045,7 +5083,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
wake_up(&swapcache_wq);
}
if (swapoff_locked)
- put_swap_entry(entry, si);
+ put_swap_entry(orig_entry, si);
return ret;
out_nomap:
if (vmf->pte)
@@ -5064,7 +5102,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
wake_up(&swapcache_wq);
}
if (swapoff_locked)
- put_swap_entry(entry, si);
+ put_swap_entry(orig_entry, si);
return ret;
}
diff --git a/mm/page_io.c b/mm/page_io.c
index 5de370557295..8cfe3599f765 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -4,7 +4,7 @@
*
* Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
*
- * Swap reorganised 29.12.95,
+ * Swap reorganised 29.12.95,
* Asynchronous swapping added 30.12.95. Stephen Tweedie
* Removed race in async swapping. 14.4.1996. Bruno Haible
* Add swap of shared pages through the page cache. 20.2.1998. Stephen Tweedie
@@ -201,44 +201,6 @@ static bool is_folio_zero_filled(struct folio *folio)
return true;
}
-static void swap_zeromap_folio_set(struct folio *folio)
-{
- struct obj_cgroup *objcg = get_obj_cgroup_from_folio(folio);
- struct swap_info_struct *sis =
- __swap_slot_to_info(swp_entry_to_swp_slot(folio->swap));
- int nr_pages = folio_nr_pages(folio);
- swp_entry_t entry;
- swp_slot_t slot;
- unsigned int i;
-
- for (i = 0; i < folio_nr_pages(folio); i++) {
- entry = page_swap_entry(folio_page(folio, i));
- slot = swp_entry_to_swp_slot(entry);
- set_bit(swp_slot_offset(slot), sis->zeromap);
- }
-
- count_vm_events(SWPOUT_ZERO, nr_pages);
- if (objcg) {
- count_objcg_events(objcg, SWPOUT_ZERO, nr_pages);
- obj_cgroup_put(objcg);
- }
-}
-
-static void swap_zeromap_folio_clear(struct folio *folio)
-{
- struct swap_info_struct *sis =
- __swap_slot_to_info(swp_entry_to_swp_slot(folio->swap));
- swp_entry_t entry;
- swp_slot_t slot;
- unsigned int i;
-
- for (i = 0; i < folio_nr_pages(folio); i++) {
- entry = page_swap_entry(folio_page(folio, i));
- slot = swp_entry_to_swp_slot(entry);
- clear_bit(swp_slot_offset(slot), sis->zeromap);
- }
-}
-
/*
* We may have stale swap cache pages in memory: notice
* them here and get rid of the unnecessary final write.
@@ -246,6 +208,7 @@ static void swap_zeromap_folio_clear(struct folio *folio)
int swap_writeout(struct folio *folio, struct swap_iocb **swap_plug)
{
int ret = 0;
+ swp_slot_t slot;
if (folio_free_swap(folio))
goto out_unlock;
@@ -260,23 +223,19 @@ int swap_writeout(struct folio *folio, struct swap_iocb **swap_plug)
goto out_unlock;
}
- /*
- * Use a bitmap (zeromap) to avoid doing IO for zero-filled pages.
- * The bits in zeromap are protected by the locked swapcache folio
- * and atomic updates are used to protect against read-modify-write
- * corruption due to other zero swap entries seeing concurrent updates.
- */
if (is_folio_zero_filled(folio)) {
swap_zeromap_folio_set(folio);
goto out_unlock;
}
/*
- * Clear bits this folio occupies in the zeromap to prevent zero data
- * being read in from any previous zero writes that occupied the same
- * swap entries.
+ * Release swap backends to make sure we do not have mixed backends.
+ *
+ * If the folio is already backed by contiguous physical swap slots
+ * (e.g. from a previous swapout attempt when zswap is disabled),
+ * keep that backend to avoid reallocation.
*/
- swap_zeromap_folio_clear(folio);
+ vswap_prepare_writeout(folio->swap, folio);
if (zswap_store(folio)) {
count_mthp_stat(folio_order(folio), MTHP_STAT_ZSWPOUT);
@@ -287,7 +246,15 @@ int swap_writeout(struct folio *folio, struct swap_iocb **swap_plug)
return AOP_WRITEPAGE_ACTIVATE;
}
- __swap_writepage(folio, swap_plug);
+ /* fall back to physical swap device */
+ slot = vswap_alloc_swap_slot(folio);
+
+ if (!slot.val) {
+ folio_mark_dirty(folio);
+ return AOP_WRITEPAGE_ACTIVATE;
+ }
+
+ __swap_writepage(folio, swap_plug, slot);
return 0;
out_unlock:
folio_unlock(folio);
@@ -378,10 +345,10 @@ static void sio_write_complete(struct kiocb *iocb, long ret)
mempool_free(sio, sio_pool);
}
-static void swap_writepage_fs(struct folio *folio, struct swap_iocb **swap_plug)
+static void swap_writepage_fs(struct folio *folio, struct swap_iocb **swap_plug,
+ swp_slot_t slot)
{
struct swap_iocb *sio = swap_plug ? *swap_plug : NULL;
- swp_slot_t slot = swp_entry_to_swp_slot(folio->swap);
struct swap_info_struct *sis = __swap_slot_to_info(slot);
struct file *swap_file = sis->swap_file;
loff_t pos = swap_slot_dev_pos(slot);
@@ -416,13 +383,13 @@ static void swap_writepage_fs(struct folio *folio, struct swap_iocb **swap_plug)
}
static void swap_writepage_bdev_sync(struct folio *folio,
- struct swap_info_struct *sis)
+ struct swap_info_struct *sis, swp_slot_t slot)
{
struct bio_vec bv;
struct bio bio;
bio_init(&bio, sis->bdev, &bv, 1, REQ_OP_WRITE | REQ_SWAP);
- bio.bi_iter.bi_sector = swap_folio_sector(folio);
+ bio.bi_iter.bi_sector = swap_slot_sector(slot);
bio_add_folio_nofail(&bio, folio, folio_size(folio), 0);
bio_associate_blkg_from_page(&bio, folio);
@@ -436,12 +403,12 @@ static void swap_writepage_bdev_sync(struct folio *folio,
}
static void swap_writepage_bdev_async(struct folio *folio,
- struct swap_info_struct *sis)
+ struct swap_info_struct *sis, swp_slot_t slot)
{
struct bio *bio;
bio = bio_alloc(sis->bdev, 1, REQ_OP_WRITE | REQ_SWAP, GFP_NOIO);
- bio->bi_iter.bi_sector = swap_folio_sector(folio);
+ bio->bi_iter.bi_sector = swap_slot_sector(slot);
bio->bi_end_io = end_swap_bio_write;
bio_add_folio_nofail(bio, folio, folio_size(folio), 0);
@@ -452,10 +419,10 @@ static void swap_writepage_bdev_async(struct folio *folio,
submit_bio(bio);
}
-void __swap_writepage(struct folio *folio, struct swap_iocb **swap_plug)
+void __swap_writepage(struct folio *folio, struct swap_iocb **swap_plug,
+ swp_slot_t slot)
{
- struct swap_info_struct *sis =
- __swap_slot_to_info(swp_entry_to_swp_slot(folio->swap));
+ struct swap_info_struct *sis = __swap_slot_to_info(slot);
VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio);
/*
@@ -464,16 +431,16 @@ void __swap_writepage(struct folio *folio, struct swap_iocb **swap_plug)
* is safe.
*/
if (data_race(sis->flags & SWP_FS_OPS))
- swap_writepage_fs(folio, swap_plug);
+ swap_writepage_fs(folio, swap_plug, slot);
/*
* ->flags can be updated non-atomicially (scan_swap_map_slots),
* but that will never affect SWP_SYNCHRONOUS_IO, so the data_race
* is safe.
*/
else if (data_race(sis->flags & SWP_SYNCHRONOUS_IO))
- swap_writepage_bdev_sync(folio, sis);
+ swap_writepage_bdev_sync(folio, sis, slot);
else
- swap_writepage_bdev_async(folio, sis);
+ swap_writepage_bdev_async(folio, sis, slot);
}
void swap_write_unplug(struct swap_iocb *sio)
@@ -544,9 +511,9 @@ static bool swap_read_folio_zeromap(struct folio *folio)
return true;
}
-static void swap_read_folio_fs(struct folio *folio, struct swap_iocb **plug)
+static void swap_read_folio_fs(struct folio *folio, struct swap_iocb **plug,
+ swp_slot_t slot)
{
- swp_slot_t slot = swp_entry_to_swp_slot(folio->swap);
struct swap_info_struct *sis = __swap_slot_to_info(slot);
struct swap_iocb *sio = NULL;
loff_t pos = swap_slot_dev_pos(slot);
@@ -580,13 +547,13 @@ static void swap_read_folio_fs(struct folio *folio, struct swap_iocb **plug)
}
static void swap_read_folio_bdev_sync(struct folio *folio,
- struct swap_info_struct *sis)
+ struct swap_info_struct *sis, swp_slot_t slot)
{
struct bio_vec bv;
struct bio bio;
bio_init(&bio, sis->bdev, &bv, 1, REQ_OP_READ);
- bio.bi_iter.bi_sector = swap_folio_sector(folio);
+ bio.bi_iter.bi_sector = swap_slot_sector(slot);
bio_add_folio_nofail(&bio, folio, folio_size(folio), 0);
/*
* Keep this task valid during swap readpage because the oom killer may
@@ -602,12 +569,12 @@ static void swap_read_folio_bdev_sync(struct folio *folio,
}
static void swap_read_folio_bdev_async(struct folio *folio,
- struct swap_info_struct *sis)
+ struct swap_info_struct *sis, swp_slot_t slot)
{
struct bio *bio;
bio = bio_alloc(sis->bdev, 1, REQ_OP_READ, GFP_KERNEL);
- bio->bi_iter.bi_sector = swap_folio_sector(folio);
+ bio->bi_iter.bi_sector = swap_slot_sector(slot);
bio->bi_end_io = end_swap_bio_read;
bio_add_folio_nofail(bio, folio, folio_size(folio), 0);
count_mthp_stat(folio_order(folio), MTHP_STAT_SWPIN);
@@ -618,14 +585,12 @@ static void swap_read_folio_bdev_async(struct folio *folio,
void swap_read_folio(struct folio *folio, struct swap_iocb **plug)
{
- struct swap_info_struct *sis =
- __swap_slot_to_info(swp_entry_to_swp_slot(folio->swap));
- bool synchronous = sis->flags & SWP_SYNCHRONOUS_IO;
- bool workingset = folio_test_workingset(folio);
+ struct swap_info_struct *sis;
+ bool synchronous, workingset = folio_test_workingset(folio);
unsigned long pflags;
bool in_thrashing;
+ swp_slot_t slot;
- VM_BUG_ON_FOLIO(!folio_test_swapcache(folio) && !synchronous, folio);
VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
VM_BUG_ON_FOLIO(folio_test_uptodate(folio), folio);
@@ -651,12 +616,18 @@ void swap_read_folio(struct folio *folio, struct swap_iocb **plug)
/* We have to read from slower devices. Increase zswap protection. */
zswap_folio_swapin(folio);
+ slot = swp_entry_to_swp_slot(folio->swap);
+
+ sis = __swap_slot_to_info(slot);
+ synchronous = sis->flags & SWP_SYNCHRONOUS_IO;
+ VM_BUG_ON_FOLIO(!folio_test_swapcache(folio) && !synchronous, folio);
+
if (data_race(sis->flags & SWP_FS_OPS)) {
- swap_read_folio_fs(folio, plug);
+ swap_read_folio_fs(folio, plug, slot);
} else if (synchronous) {
- swap_read_folio_bdev_sync(folio, sis);
+ swap_read_folio_bdev_sync(folio, sis, slot);
} else {
- swap_read_folio_bdev_async(folio, sis);
+ swap_read_folio_bdev_async(folio, sis, slot);
}
finish:
diff --git a/mm/shmem.c b/mm/shmem.c
index 780571c830e5..3a346cca114a 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -1459,7 +1459,7 @@ static unsigned int shmem_find_swap_entries(struct address_space *mapping,
* swapin error entries can be found in the mapping. But they're
* deliberately ignored here as we've done everything we can do.
*/
- if (swp_slot_type(slot) != type)
+ if (!slot.val || swp_slot_type(slot) != type)
continue;
indices[folio_batch_count(fbatch)] = xas.xa_index;
@@ -1604,7 +1604,7 @@ int shmem_writeout(struct folio *folio, struct swap_iocb **plug,
if ((info->flags & SHMEM_F_LOCKED) || sbinfo->noswap)
goto redirty;
- if (!total_swap_pages)
+ if (!zswap_is_enabled() && !total_swap_pages)
goto redirty;
/*
@@ -2341,7 +2341,7 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index,
/* Look it up and read it in.. */
folio = swap_cache_get_folio(swap);
if (!folio) {
- if (data_race(si->flags & SWP_SYNCHRONOUS_IO)) {
+ if (si && data_race(si->flags & SWP_SYNCHRONOUS_IO)) {
/* Direct swapin skipping swap cache & readahead */
folio = shmem_swap_alloc_folio(inode, vma, index,
index_entry, order, gfp);
diff --git a/mm/swap.h b/mm/swap.h
index 4b9005224518..d46b2f243f83 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -143,7 +143,8 @@ static inline void swap_read_unplug(struct swap_iocb *plug)
}
void swap_write_unplug(struct swap_iocb *sio);
int swap_writeout(struct folio *folio, struct swap_iocb **swap_plug);
-void __swap_writepage(struct folio *folio, struct swap_iocb **swap_plug);
+void __swap_writepage(struct folio *folio, struct swap_iocb **swap_plug,
+ swp_slot_t slot);
/* linux/mm/swap_state.c */
extern struct address_space swap_space __read_mostly;
@@ -253,35 +254,15 @@ static inline unsigned int folio_swap_flags(struct folio *folio)
{
swp_slot_t swp_slot = swp_entry_to_swp_slot(folio->swap);
+ /* The folio might not be backed by any physical swap slots
+ * (for e.g zswap-backed only).
+ */
+ if (!swp_slot.val)
+ return 0;
return __swap_slot_to_info(swp_slot)->flags;
}
-/*
- * Return the count of contiguous swap entries that share the same
- * zeromap status as the starting entry. If is_zeromap is not NULL,
- * it will return the zeromap status of the starting entry.
- */
-static inline int swap_zeromap_batch(swp_entry_t entry, int max_nr,
- bool *is_zeromap)
-{
- swp_slot_t slot = swp_entry_to_swp_slot(entry);
- struct swap_info_struct *sis = __swap_slot_to_info(slot);
- unsigned long start = swp_slot_offset(slot);
- unsigned long end = start + max_nr;
- bool first_bit;
-
- first_bit = test_bit(start, sis->zeromap);
- if (is_zeromap)
- *is_zeromap = first_bit;
-
- if (max_nr <= 1)
- return max_nr;
- if (first_bit)
- return find_next_zero_bit(sis->zeromap, end, start) - start;
- else
- return find_next_bit(sis->zeromap, end, start) - start;
-}
-
+int swap_zeromap_batch(swp_entry_t entry, int max_nr, bool *is_zeromap);
int non_swapcache_batch(swp_entry_t entry, int max_nr);
#else /* CONFIG_SWAP */
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 235f165c7b30..7fe4d9529e4a 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -179,6 +179,10 @@ struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,
struct folio *result = NULL;
void *shadow = NULL;
+ /* we might get an unused entry from cluster readahead - just skip */
+ if (!entry.val)
+ return NULL;
+
*new_page_allocated = false;
for (;;) {
int err;
@@ -213,8 +217,20 @@ struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,
* Swap entry may have been freed since our caller observed it.
*/
err = swapcache_prepare(entry, 1);
- if (!err)
+ if (!err) {
+ /* This might be invoked by swap_cluster_readahead(), which can
+ * race with shmem_swapin_folio(). The latter might have already
+ * called swap_cache_del_folio(), allowing swapcache_prepare()
+ * to succeed here. This can lead to reading bogus data to populate
+ * the page. To prevent this, skip folio-backed virtual swap slots,
+ * and let caller retry if necessary.
+ */
+ if (vswap_folio_backed(entry, 1)) {
+ swapcache_clear(entry, 1);
+ goto put_and_return;
+ }
break;
+ }
else if (err != -EEXIST)
goto put_and_return;
@@ -391,11 +407,18 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask,
unsigned long offset = slot_offset;
unsigned long start_offset, end_offset;
unsigned long mask;
- struct swap_info_struct *si = __swap_slot_to_info(slot);
+ struct swap_info_struct *si = swap_slot_tryget_swap_info(slot);
struct blk_plug plug;
struct swap_iocb *splug = NULL;
bool page_allocated;
+ /*
+ * The swap entry might not be backed by any physical swap slot. In that
+ * case, just skip readahead and bring in the target entry.
+ */
+ if (!si)
+ goto skip;
+
mask = swapin_nr_pages(offset) - 1;
if (!mask)
goto skip;
@@ -429,6 +452,8 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask,
swap_read_unplug(splug);
lru_add_drain(); /* Push any new pages onto the LRU now */
skip:
+ if (si)
+ swap_slot_put_swap_info(si);
/* The page was likely read above, so no need for plugging here */
folio = __read_swap_cache_async(entry, gfp_mask, mpol, ilx,
&page_allocated, false);
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 96151504a6e1..c6a91c657877 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -338,9 +338,8 @@ offset_to_swap_extent(struct swap_info_struct *sis, unsigned long offset)
BUG();
}
-sector_t swap_folio_sector(struct folio *folio)
+sector_t swap_slot_sector(swp_slot_t slot)
{
- swp_slot_t slot = swp_entry_to_swp_slot(folio->swap);
struct swap_info_struct *sis = __swap_slot_to_info(slot);
struct swap_extent *se;
sector_t sector;
@@ -1218,14 +1217,6 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
{
unsigned long end = offset + nr_entries - 1;
void (*swap_slot_free_notify)(struct block_device *, unsigned long);
- unsigned int i;
-
- /*
- * Use atomic clear_bit operations only on zeromap instead of non-atomic
- * bitmap_clear to prevent adjacent bits corruption due to simultaneous writes.
- */
- for (i = 0; i < nr_entries; i++)
- clear_bit(offset + i, si->zeromap);
if (si->flags & SWP_BLKDEV)
swap_slot_free_notify =
diff --git a/mm/vmscan.c b/mm/vmscan.c
index bd8900d7bb28..0ddf3f646645 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -365,10 +365,11 @@ static inline bool can_reclaim_anon_pages(struct mem_cgroup *memcg,
{
if (memcg == NULL) {
/*
- * For non-memcg reclaim, is there
- * space in any swap device?
+ * For non-memcg reclaim:
+ *
+ * Check if zswap is enabled or if there is space in any swap device?
*/
- if (get_nr_swap_pages() > 0)
+ if (zswap_is_enabled() || get_nr_swap_pages() > 0)
return true;
} else {
/* Is the memcg below its swap limit? */
@@ -2637,12 +2638,12 @@ static void get_scan_count(struct lruvec *lruvec, struct scan_control *sc,
static bool can_age_anon_pages(struct lruvec *lruvec,
struct scan_control *sc)
{
- /* Aging the anon LRU is valuable if swap is present: */
- if (total_swap_pages > 0)
- return true;
-
- /* Also valuable if anon pages can be demoted: */
- return can_demote(lruvec_pgdat(lruvec)->node_id, sc,
+ /*
+ * Aging the anon LRU is valuable if zswap or physical swap is available or
+ * anon pages can be demoted
+ */
+ return zswap_is_enabled() || total_swap_pages > 0 ||
+ can_demote(lruvec_pgdat(lruvec)->node_id, sc,
lruvec_memcg(lruvec));
}
diff --git a/mm/vswap.c b/mm/vswap.c
index 71148323499b..01c336ae252c 100644
--- a/mm/vswap.c
+++ b/mm/vswap.c
@@ -13,6 +13,7 @@
#include <linux/zswap.h>
#include <linux/suspend.h>
#include <linux/vmalloc.h>
+#include "internal.h"
#include "swap.h"
#include "swap_table.h"
@@ -56,22 +57,48 @@
*
* Note that we do not have a reference count field per se - it is derived from
* the swap_count and the in_swapcache fields.
+ *
+ * III. Backing State
+ *
+ * Each virtual swap slot can be backed by:
+ *
+ * 1. A slot on a physical swap device (i.e a swapfile or a swap partition).
+ * 2. A swapped out zero-filled page.
+ * 3. A compressed object in zswap.
+ * 4. An in-memory folio, that is not backed by neither a physical swap device
+ * nor zswap (i.e only in swap cache). This is used for pages that are
+ * rejected by zswap, but not (yet) backed by a physical swap device,
+ * (for e.g, due to zswap.writeback = 0), or for pages that were previously
+ * stored in zswap, but has since been loaded back into memory (and has its
+ * zswap copy invalidated).
*/
+/* The backing state options of a virtual swap slot */
+enum swap_type {
+ VSWAP_SWAPFILE,
+ VSWAP_ZERO,
+ VSWAP_ZSWAP,
+ VSWAP_FOLIO
+};
+
/**
* Swap descriptor - metadata of a swapped out page.
*
* @slot: The handle to the physical swap slot backing this page.
* @zswap_entry: The zswap entry associated with this swap slot.
- * @swap_cache: The folio in swap cache.
+ * @swap_cache: The folio in swap cache. If the swap entry backing type is
+ * VSWAP_FOLIO, the backend is also stored here.
* @shadow: The shadow entry.
- * @memcgid: The memcg id of the owning memcg, if any.
* @swap_count: The number of page table entries that refer to the swap entry.
+ * @memcgid: The memcg id of the owning memcg, if any.
* @in_swapcache: Whether the swap entry is (about to be) pinned in swap cache.
+ * @type: The backing store type of the swap entry.
*/
struct swp_desc {
- swp_slot_t slot;
- struct zswap_entry *zswap_entry;
+ union {
+ swp_slot_t slot;
+ struct zswap_entry *zswap_entry;
+ };
union {
struct folio *swap_cache;
void *shadow;
@@ -80,10 +107,10 @@ struct swp_desc {
unsigned int swap_count;
#ifdef CONFIG_MEMCG
- unsigned short memcgid;
+ unsigned short memcgid:16;
#endif
-
- bool in_swapcache;
+ bool in_swapcache:1;
+ enum swap_type type:2;
};
#define VSWAP_CLUSTER_SHIFT HPAGE_PMD_ORDER
@@ -269,28 +296,27 @@ static bool cluster_is_alloc_candidate(struct vswap_cluster *cluster)
return cluster->count + (1 << (cluster->order)) <= VSWAP_CLUSTER_SIZE;
}
-static void __vswap_alloc_from_cluster(struct vswap_cluster *cluster, int start,
- struct folio *folio)
+static void __vswap_alloc_from_cluster(struct vswap_cluster *cluster,
+ int start, struct folio *folio)
{
int i, nr = 1 << cluster->order;
struct swp_desc *desc;
for (i = 0; i < nr; i++) {
desc = &cluster->descriptors[start + i];
- desc->slot.val = 0;
- desc->zswap_entry = NULL;
+ desc->type = VSWAP_FOLIO;
+ desc->swap_cache = folio;
#ifdef CONFIG_MEMCG
desc->memcgid = 0;
#endif
desc->swap_count = 0;
desc->in_swapcache = true;
- desc->swap_cache = folio;
}
cluster->count += nr;
}
static unsigned long vswap_alloc_from_cluster(struct vswap_cluster *cluster,
- struct folio *folio)
+ struct folio *folio)
{
int nr = 1 << cluster->order;
unsigned long i = cluster->id ? 0 : nr;
@@ -505,24 +531,106 @@ static inline void release_vswap_slot(struct vswap_cluster *cluster,
/*
* Update the physical-to-virtual swap slot mapping.
- * Caller must ensure the physical swap slot's cluster is locked.
+ *
+ * Uses atomic_long_set which is inherently atomic, so no lock is required
+ * for the rmap update itself. Callers hold the physical swap cluster lock
+ * only if they need it for other operations (e.g. swap_slots_free).
*/
void vswap_rmap_set(struct swap_cluster_info *ci, swp_slot_t slot,
unsigned long vswap, int nr)
{
- atomic_long_t *table;
- unsigned long slot_offset = swp_slot_offset(slot);
- unsigned int ci_off = slot_offset % SWAPFILE_CLUSTER;
+ unsigned int ci_off = swp_cluster_offset(slot);
int i;
- table = rcu_dereference_protected(ci->table, lockdep_is_held(&ci->lock));
- VM_WARN_ON(!table);
for (i = 0; i < nr; i++)
__swap_table_set(ci, ci_off + i, vswap ? vswap + i : 0);
}
static unsigned short swp_desc_memcgid(struct swp_desc *desc);
+/*
+ * release_backing - release the backend storage for a given range of virtual
+ * swap slots.
+ *
+ * Entered with the cluster locked. The cluster lock is held throughout.
+ *
+ * This is safe, because:
+ *
+ * 1. Callers ensure no concurrent modification of the swap entry's internal
+ * state can occur. This is guaranteed by one of the following:
+ * - For vswap_free() callers: the swap entry's refcnt (swap count and
+ * swapcache pin) is down to 0.
+ * - For vswap_store_folio(), swap_zeromap_folio_set(), and zswap_entry_store()
+ * callers: the folio is locked and in the swap cache.
+ *
+ * 2. The swap entry still holds a refcnt to the cluster, keeping the cluster
+ * itself valid.
+ *
+ * 3. swap_slot_free_nr() takes the physical swap cluster lock (ci->lock),
+ * but the only vswap function called under ci->lock is vswap_rmap_set(),
+ * which uses atomic ops and does not take cluster->lock. So there is no
+ * ABBA deadlock risk.
+ */
+static void release_backing(struct vswap_cluster *cluster, swp_entry_t entry,
+ int nr)
+{
+ struct swp_desc *desc;
+ unsigned long flush_nr, phys_swap_start = 0, phys_swap_end = 0;
+ unsigned int phys_swap_type = 0;
+ bool need_flushing_phys_swap = false;
+ swp_slot_t flush_slot;
+ int i;
+
+ VM_WARN_ON(!entry.val);
+
+ for (i = 0; i < nr; i++) {
+ desc = __vswap_iter(cluster, entry.val + i);
+ VM_WARN_ON(!desc);
+
+ /*
+ * We batch contiguous physical swap slots for more efficient
+ * freeing.
+ */
+ if (phys_swap_start != phys_swap_end &&
+ (desc->type != VSWAP_SWAPFILE ||
+ swp_slot_type(desc->slot) != phys_swap_type ||
+ swp_slot_offset(desc->slot) != phys_swap_end)) {
+ need_flushing_phys_swap = true;
+ flush_slot = swp_slot(phys_swap_type, phys_swap_start);
+ flush_nr = phys_swap_end - phys_swap_start;
+ phys_swap_start = phys_swap_end = 0;
+ }
+
+ if (desc->type == VSWAP_ZSWAP && desc->zswap_entry) {
+ zswap_entry_free(desc->zswap_entry);
+ } else if (desc->type == VSWAP_SWAPFILE) {
+ if (!phys_swap_start) {
+ /* start a new contiguous range of phys swap */
+ phys_swap_start = swp_slot_offset(desc->slot);
+ phys_swap_end = phys_swap_start + 1;
+ phys_swap_type = swp_slot_type(desc->slot);
+ } else {
+ /* extend the current contiguous range of phys swap */
+ phys_swap_end++;
+ }
+ }
+
+ desc->slot.val = 0;
+
+ if (need_flushing_phys_swap) {
+ swap_slot_free_nr(flush_slot, flush_nr);
+ need_flushing_phys_swap = false;
+ }
+ }
+
+ /* Flush any remaining physical swap range */
+ if (phys_swap_start) {
+ flush_slot = swp_slot(phys_swap_type, phys_swap_start);
+ flush_nr = phys_swap_end - phys_swap_start;
+ swap_slot_free_nr(flush_slot, flush_nr);
+ }
+}
+
/*
* Entered with the cluster locked. The cluster lock is held throughout.
*
@@ -543,29 +651,17 @@ static void vswap_free(struct vswap_cluster *cluster, struct swp_desc *desc,
swp_entry_t entry)
{
unsigned short memcgid = swp_desc_memcgid(desc);
- struct zswap_entry *zswap_entry;
- swp_slot_t slot;
- slot = desc->slot;
- desc->slot.val = 0;
-
- zswap_entry = desc->zswap_entry;
- if (zswap_entry) {
- desc->zswap_entry = NULL;
- zswap_entry_free(zswap_entry);
- }
+ release_backing(cluster, entry, 1);
mem_cgroup_uncharge_swap_by_id(memcgid, 1);
- if (slot.val)
- swap_slot_free_nr(slot, 1);
-
/* erase forward mapping and release the virtual slot for reallocation */
release_vswap_slot(cluster, entry.val);
}
/**
- * folio_alloc_swap - allocate swap space for a folio.
+ * folio_alloc_swap - allocate virtual swap space for a folio.
* @folio: the folio.
*
* Return: 0, if the allocation succeeded, -ENOMEM, if the allocation failed.
@@ -573,12 +669,9 @@ static void vswap_free(struct vswap_cluster *cluster, struct swp_desc *desc,
int folio_alloc_swap(struct folio *folio)
{
struct vswap_cluster *cluster = NULL;
- struct swap_info_struct *si;
- struct swap_cluster_info *ci;
- int i, ret, nr = folio_nr_pages(folio), order = folio_order(folio);
+ int i, nr = folio_nr_pages(folio);
struct swp_desc *desc;
swp_entry_t entry;
- swp_slot_t slot = { 0 };
VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
VM_BUG_ON_FOLIO(!folio_test_uptodate(folio), folio);
@@ -588,53 +681,21 @@ int folio_alloc_swap(struct folio *folio)
return -ENOMEM;
/*
- * XXX: for now, we always allocate a physical swap slot for each virtual
- * swap slot, and their lifetime are coupled. This will change once we
- * decouple virtual swap slots from their backing states, and only allocate
- * physical swap slots for them on demand (i.e on zswap writeback, or
- * fallback from zswap store failure).
+ * XXX: for now, we charge towards the memory cgroup's swap limit on virtual
+ * swap slots allocation. This will be changed soon - we will only charge on
+ * physical swap slots allocation.
*/
- ret = swap_slot_alloc(&slot, order);
- if (ret || !slot.val) {
- /* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. */
- mem_cgroup_try_charge_swap(folio, (swp_entry_t){0});
-
+ if (mem_cgroup_try_charge_swap(folio, entry)) {
+ rcu_read_lock();
for (i = 0; i < nr; i++) {
desc = vswap_iter(&cluster, entry.val + i);
VM_WARN_ON(!desc);
vswap_free(cluster, desc, (swp_entry_t){ entry.val + i });
}
spin_unlock(&cluster->lock);
-
- return ret ? ret : -ENOMEM;
- }
-
- /* establish the virtual <-> physical swap slots linkages. */
- si = __swap_slot_to_info(slot);
- ci = swap_cluster_lock(si, swp_slot_offset(slot));
- vswap_rmap_set(ci, slot, entry.val, nr);
- swap_cluster_unlock(ci);
-
- rcu_read_lock();
- for (i = 0; i < nr; i++) {
- desc = vswap_iter(&cluster, entry.val + i);
- VM_WARN_ON(!desc);
-
- desc->slot.val = slot.val + i;
- }
- if (cluster)
- spin_unlock(&cluster->lock);
- rcu_read_unlock();
-
- /*
- * XXX: for now, we charge towards the memory cgroup's swap limit on virtual
- * swap slots allocation. This is acceptable because as noted above, each
- * virtual swap slot corresponds to a physical swap slot. Once we have
- * decoupled virtual and physical swap slots, we will only charge when we
- * actually allocate a physical swap slot.
- */
- if (mem_cgroup_try_charge_swap(folio, entry)) {
- put_swap_folio(folio, entry);
+ rcu_read_unlock();
+ atomic_add(nr, &vswap_alloc_reject);
+ entry.val = 0;
return -ENOMEM;
}
@@ -648,12 +709,84 @@ int folio_alloc_swap(struct folio *folio)
return 0;
}
+/**
+ * vswap_alloc_swap_slot - allocate physical swap space for a folio that is
+ * already associated with virtual swap slots.
+ * @folio: folio we want to allocate physical swap space for.
+ *
+ * Note that this does NOT release existing swap backends of the folio.
+ * Callers need to handle this themselves.
+ *
+ * Return: the physical swap slot allocated, or zero swap slot on failure.
+ */
+swp_slot_t vswap_alloc_swap_slot(struct folio *folio)
+{
+ int i, nr = folio_nr_pages(folio);
+ struct vswap_cluster *cluster = NULL;
+ swp_slot_t slot = { .val = 0 };
+ swp_entry_t entry = folio->swap;
+ struct swp_desc *desc;
+ bool fallback = false;
+
+ /*
+ * Check if the first entry already has a physical swap slot. If so,
+ * the entire range is contiguous (from a previous allocation).
+ * Keep the cluster pointer around for reuse below.
+ */
+ rcu_read_lock();
+ desc = vswap_iter(&cluster, entry.val);
+ VM_WARN_ON(!desc);
+ if (desc->type == VSWAP_SWAPFILE)
+ slot = desc->slot;
+ spin_unlock(&cluster->lock);
+ rcu_read_unlock();
+
+ if (slot.val)
+ return slot;
+
+ if (swap_slot_alloc(&slot, folio_order(folio)))
+ return (swp_slot_t){ .val = 0 };
+
+ if (!slot.val)
+ return (swp_slot_t){ .val = 0 };
+
+ vswap_rmap_set(__swap_slot_to_cluster(slot), slot, entry.val, nr);
+
+ spin_lock(&cluster->lock);
+ for (i = 0; i < nr; i++) {
+ desc = __vswap_iter(cluster, entry.val + i);
+ VM_WARN_ON(!desc);
+
+ if (desc->type == VSWAP_FOLIO) {
+ /* case 1: fallback from zswap store failure */
+ fallback = true;
+ VM_WARN_ON(folio != desc->swap_cache);
+ } else {
+ /*
+ * Case 2: zswap writeback.
+ *
+ * No need to free zswap entry here - it will be freed
+ * once zswap writeback succeeds.
+ */
+ VM_WARN_ON(desc->type != VSWAP_ZSWAP);
+ VM_WARN_ON(fallback);
+ }
+ desc->type = VSWAP_SWAPFILE;
+ desc->slot.val = slot.val + i;
+ }
+ spin_unlock(&cluster->lock);
+
+ return slot;
+}
+
/**
* swp_entry_to_swp_slot - look up the physical swap slot corresponding to a
* virtual swap slot.
* @entry: the virtual swap slot.
*
- * Return: the physical swap slot corresponding to the virtual swap slot.
+ * Return: the physical swap slot corresponding to the virtual swap slot, if
+ * exists, or the zero physical swap slot if the virtual swap slot is not
+ * backed by any physical slot on a swapfile.
*/
swp_slot_t swp_entry_to_swp_slot(swp_entry_t entry)
{
@@ -672,7 +805,10 @@ swp_slot_t swp_entry_to_swp_slot(swp_entry_t entry)
return (swp_slot_t){0};
}
- slot = desc->slot;
+ if (desc->type != VSWAP_SWAPFILE)
+ slot.val = 0;
+ else
+ slot = desc->slot;
spin_unlock(&cluster->lock);
rcu_read_unlock();
@@ -731,7 +867,7 @@ static bool vswap_free_nr_any_cache_only(swp_entry_t entry, int nr)
desc->shadow = NULL;
vswap_free(cluster, desc, entry);
} else if (!desc->swap_count && desc->in_swapcache &&
- desc->slot.val) {
+ desc->type == VSWAP_SWAPFILE) {
/*
* swap_count just dropped to 0, but still in swap
* cache. If backed by a physical swap slot, mark it
@@ -769,7 +905,7 @@ static int swap_duplicate_nr(swp_entry_t entry, int nr)
desc = vswap_iter(&cluster, entry.val + i);
if (!desc || (!desc->swap_count && !desc->in_swapcache))
goto done;
- if (!desc->swap_count && desc->slot.val)
+ if (!desc->swap_count && desc->type == VSWAP_SWAPFILE)
swap_rmap_clear_cache_only(desc->slot);
desc->swap_count++;
}
@@ -795,7 +931,6 @@ int swap_duplicate(swp_entry_t entry)
return swap_duplicate_nr(entry, 1);
}
-
/**
* folio_free_swap() - Free the swap space used for this folio.
* @folio: The folio to remove.
@@ -1025,6 +1160,274 @@ int non_swapcache_batch(swp_entry_t entry, int max_nr)
static struct folio *vswap_get_swap_cache(swp_entry_t entry, bool swap_cache_only);
+static int vswap_check_backing(swp_entry_t entry, enum swap_type *type, int nr,
+ struct vswap_cluster **clusterp);
+
+/**
+ * vswap_store_folio - set a folio as the backing of a range of virtual swap
+ * slots.
+ * @entry: the first virtual swap slot in the range.
+ * @folio: the folio.
+ */
+void vswap_store_folio(swp_entry_t entry, struct folio *folio)
+{
+ struct vswap_cluster *cluster = NULL;
+ int i, nr = folio_nr_pages(folio);
+ struct swp_desc *desc;
+
+ VM_BUG_ON(!folio_test_locked(folio));
+ VM_BUG_ON(folio->swap.val != entry.val);
+
+ rcu_read_lock();
+ desc = vswap_iter(&cluster, entry.val);
+ VM_WARN_ON(!desc);
+ release_backing(cluster, entry, nr);
+
+ for (i = 0; i < nr; i++) {
+ desc = __vswap_iter(cluster, entry.val + i);
+ VM_WARN_ON(!desc);
+ desc->type = VSWAP_FOLIO;
+ desc->swap_cache = folio;
+ }
+ spin_unlock(&cluster->lock);
+ rcu_read_unlock();
+}
+
+/**
+ * vswap_prepare_writeout - prepare a folio for writeout by releasing
+ * non-swapfile backends in a single lock context.
+ * @entry: the first virtual swap slot in the range.
+ * @folio: the folio being written out.
+ *
+ * If the folio is already backed by contiguous physical swap slots,
+ * this is a no-op (the existing backing is preserved). Otherwise,
+ * release the current backing and store the folio.
+ *
+ * Combines the vswap_swapfile_backed() check and vswap_store_folio()
+ * into a single cluster lock acquisition.
+ */
+void vswap_prepare_writeout(swp_entry_t entry, struct folio *folio)
+{
+ struct vswap_cluster *cluster = NULL;
+ int i, nr = folio_nr_pages(folio);
+ enum swap_type type;
+ struct swp_desc *desc;
+
+ VM_BUG_ON(!folio_test_locked(folio));
+ VM_BUG_ON(folio->swap.val != entry.val);
+
+ /* Check backing with lock held on return */
+ if (vswap_check_backing(entry, &type, nr, &cluster) == nr &&
+ (type == VSWAP_SWAPFILE || type == VSWAP_FOLIO)) {
+ /* Already contiguous swapfile-backed or folio-backed, keep it */
+ if (cluster)
+ spin_unlock(&cluster->lock);
+ rcu_read_unlock();
+ return;
+ }
+
+ if (!cluster) {
+ rcu_read_unlock();
+ return;
+ }
+
+ /* Release old backing and store the folio, lock already held */
+ release_backing(cluster, entry, nr);
+
+ for (i = 0; i < nr; i++) {
+ desc = __vswap_iter(cluster, entry.val + i);
+ VM_WARN_ON(!desc);
+ desc->type = VSWAP_FOLIO;
+ desc->swap_cache = folio;
+ }
+ spin_unlock(&cluster->lock);
+ rcu_read_unlock();
+}
+
+/**
+ * swap_zeromap_folio_set - mark a range of virtual swap slots corresponding to
+ * a folio as zero-filled.
+ * @folio: the folio
+ */
+void swap_zeromap_folio_set(struct folio *folio)
+{
+ struct obj_cgroup *objcg = get_obj_cgroup_from_folio(folio);
+ struct vswap_cluster *cluster = NULL;
+ swp_entry_t entry = folio->swap;
+ int i, nr = folio_nr_pages(folio);
+ struct swp_desc *desc;
+
+ VM_BUG_ON(!folio_test_locked(folio));
+ VM_BUG_ON(!entry.val);
+
+ rcu_read_lock();
+ desc = vswap_iter(&cluster, entry.val);
+ VM_WARN_ON(!desc);
+ release_backing(cluster, entry, nr);
+
+ for (i = 0; i < nr; i++) {
+ desc = __vswap_iter(cluster, entry.val + i);
+ VM_WARN_ON(!desc);
+ desc->type = VSWAP_ZERO;
+ }
+ spin_unlock(&cluster->lock);
+ rcu_read_unlock();
+
+ count_vm_events(SWPOUT_ZERO, nr);
+ if (objcg) {
+ count_objcg_events(objcg, SWPOUT_ZERO, nr);
+ obj_cgroup_put(objcg);
+ }
+}
+
+/*
+ * Iterate through the entire range of virtual swap slots, returning the
+ * longest contiguous range of slots starting from the first slot that satisfies:
+ *
+ * 1. If the first slot is zero-mapped, the entire range should be
+ * zero-mapped.
+ * 2. If the first slot is backed by a swapfile, the entire range should
+ * be backed by a range of contiguous swap slots on the same swapfile.
+ * 3. If the first slot is zswap-backed, the entire range should be
+ * zswap-backed.
+ * 4. If the first slot is backed by a folio, the entire range should
+ * be backed by the same folio.
+ *
+ * Note that this check is racy unless we can ensure that the entire range
+ * has their backing state stable - for instance, if the caller was the one
+ * who set the swap cache pin.
+ */
+static int vswap_check_backing(swp_entry_t entry, enum swap_type *type, int nr,
+ struct vswap_cluster **clusterp)
+{
+ unsigned int swapfile_type;
+ struct vswap_cluster *cluster = NULL;
+ enum swap_type first_type;
+ struct swp_desc *desc;
+ pgoff_t first_offset;
+ struct folio *folio;
+ int i = 0;
+
+ if (!entry.val)
+ return 0;
+
+ rcu_read_lock();
+ for (i = 0; i < nr; i++) {
+ desc = vswap_iter(&cluster, entry.val + i);
+ if (!desc)
+ goto done;
+
+ if (!i) {
+ first_type = desc->type;
+ if (first_type == VSWAP_SWAPFILE) {
+ swapfile_type = swp_slot_type(desc->slot);
+ first_offset = swp_slot_offset(desc->slot);
+ } else if (first_type == VSWAP_FOLIO) {
+ folio = desc->swap_cache;
+ }
+ } else if (desc->type != first_type) {
+ goto done;
+ } else if (first_type == VSWAP_SWAPFILE &&
+ (swp_slot_type(desc->slot) != swapfile_type ||
+ swp_slot_offset(desc->slot) != first_offset + i)) {
+ goto done;
+ } else if (first_type == VSWAP_FOLIO && desc->swap_cache != folio) {
+ goto done;
+ }
+ }
+done:
+ if (clusterp) {
+ /* Caller takes ownership of the lock and rcu_read_lock */
+ *clusterp = cluster;
+ } else {
+ if (cluster)
+ spin_unlock(&cluster->lock);
+ rcu_read_unlock();
+ }
+ if (type)
+ *type = first_type;
+ return i;
+}
+
+/**
+ * vswap_swapfile_backed - check if the virtual swap slots are backed by physical
+ * swap slots.
+ * @entry: the first entry in the range.
+ * @nr: the number of entries in the range.
+ */
+bool vswap_swapfile_backed(swp_entry_t entry, int nr)
+{
+ enum swap_type type;
+
+ return vswap_check_backing(entry, &type, nr, NULL) == nr
+ && type == VSWAP_SWAPFILE;
+}
+
+/**
+ * vswap_folio_backed - check if the virtual swap slots are backed by in-memory
+ * pages.
+ * @entry: the first virtual swap slot in the range.
+ * @nr: the number of slots in the range.
+ */
+bool vswap_folio_backed(swp_entry_t entry, int nr)
+{
+ enum swap_type type;
+
+ return vswap_check_backing(entry, &type, nr, NULL) == nr && type == VSWAP_FOLIO;
+}
+
+/**
+ * vswap_can_swapin_thp - check if the swap entries can be swapped in as a THP.
+ * @entry: the first virtual swap slot in the range.
+ * @nr: the number of slots in the range.
+ *
+ * For now, we can only swap in a THP if the entire range is zero-filled, or if
+ * the entire range is backed by a contiguous range of physical swap slots on a
+ * swapfile.
+ */
+bool vswap_can_swapin_thp(swp_entry_t entry, int nr)
+{
+ enum swap_type type;
+
+ return vswap_check_backing(entry, &type, nr, NULL) == nr &&
+ (type == VSWAP_ZERO || type == VSWAP_SWAPFILE);
+}
+
+/*
+ * Return the count of contiguous swap entries that share the same
+ * VSWAP_ZERO status as the starting entry. If is_zeromap is not NULL,
+ * it will return the VSWAP_ZERO status of the starting entry.
+ */
+int swap_zeromap_batch(swp_entry_t entry, int max_nr, bool *is_zeromap)
+{
+ struct vswap_cluster *cluster = NULL;
+ struct swp_desc *desc;
+ int i = 0;
+ bool is_zero = false;
+
+ VM_WARN_ON(!entry.val);
+
+ rcu_read_lock();
+ for (i = 0; i < max_nr; i++) {
+ desc = vswap_iter(&cluster, entry.val + i);
+ if (!desc)
+ goto done;
+
+ if (!i)
+ is_zero = (desc->type == VSWAP_ZERO);
+ else if ((desc->type == VSWAP_ZERO) != is_zero)
+ goto done;
+ }
+done:
+ if (cluster)
+ spin_unlock(&cluster->lock);
+ rcu_read_unlock();
+ if (i && is_zeromap)
+ *is_zeromap = is_zero;
+
+ return i;
+}
+
/**
* free_swap_and_cache_nr() - Release a swap count on range of swap entries and
* reclaim their cache if no more references remain.
@@ -1088,11 +1491,6 @@ bool tryget_swap_entry(swp_entry_t entry, struct swap_info_struct **si)
struct vswap_cluster *cluster;
swp_slot_t slot;
- slot = swp_entry_to_swp_slot(entry);
- *si = swap_slot_tryget_swap_info(slot);
- if (!*si)
- return false;
-
/*
* Ensure the cluster and its associated data structures (swap cache etc.)
* remain valid.
@@ -1101,11 +1499,30 @@ bool tryget_swap_entry(swp_entry_t entry, struct swap_info_struct **si)
cluster = xa_load(&vswap_cluster_map, VSWAP_CLUSTER_IDX(entry));
if (!cluster || !refcount_inc_not_zero(&cluster->refcnt)) {
rcu_read_unlock();
- swap_slot_put_swap_info(*si);
*si = NULL;
return false;
}
rcu_read_unlock();
+
+ slot = swp_entry_to_swp_slot(entry);
+ /*
+ * Note that this function does not provide any guarantee that the virtual
+ * swap slot's backing state will be stable. This has several implications:
+ *
+ * 1. We have to obtain a reference to the swap device itself, because we
+ * need swap device's metadata in certain scenarios, for example when we
+ * need to inspect the swap device flag in do_swap_page().
+ *
+ * 2. The swap device we are looking up here might be outdated by the time we
+ * return to the caller. It is perfectly OK, if the swap_info_struct is only
+ * used in a best-effort manner (i.e optimization). If we need the precise
+ * backing state, we need to re-check after the entry is pinned in swapcache.
+ */
+ if (slot.val)
+ *si = swap_slot_tryget_swap_info(slot);
+ else
+ *si = NULL;
+
return true;
}
@@ -1354,7 +1771,7 @@ void swap_cache_add_folio(struct folio *folio, swp_entry_t entry, void **shadowp
old = desc->shadow;
/* Warn if slot is already occupied by a folio */
- VM_WARN_ON_FOLIO(old && !xa_is_value(old), folio);
+ VM_WARN_ON_FOLIO(old && !xa_is_value(old) && old != folio, folio);
/* Save shadow if found and not yet saved */
if (shadowp && xa_is_value(old) && !*shadowp)
@@ -1512,29 +1929,21 @@ void __swap_cache_replace_folio(struct folio *old, struct folio *new)
* @entry: the zswap entry to store
*
* Stores a zswap entry in the swap descriptor for the given swap entry.
- * The cluster is locked during the store operation.
- *
- * Return: the old zswap entry if one existed, NULL otherwise
+ * Releases the old backend if one existed.
*/
-void *zswap_entry_store(swp_entry_t swpentry, struct zswap_entry *entry)
+void zswap_entry_store(swp_entry_t swpentry, struct zswap_entry *entry)
{
struct vswap_cluster *cluster = NULL;
struct swp_desc *desc;
- void *old;
rcu_read_lock();
desc = vswap_iter(&cluster, swpentry.val);
- if (!desc) {
- rcu_read_unlock();
- return NULL;
- }
-
- old = desc->zswap_entry;
+ VM_WARN_ON(!desc);
+ release_backing(cluster, swpentry, 1);
desc->zswap_entry = entry;
+ desc->type = VSWAP_ZSWAP;
spin_unlock(&cluster->lock);
rcu_read_unlock();
-
- return old;
}
/**
@@ -1549,6 +1958,7 @@ void *zswap_entry_load(swp_entry_t swpentry)
{
struct vswap_cluster *cluster = NULL;
struct swp_desc *desc;
+ enum swap_type type;
void *zswap_entry;
rcu_read_lock();
@@ -1558,41 +1968,15 @@ void *zswap_entry_load(swp_entry_t swpentry)
return NULL;
}
+ type = desc->type;
zswap_entry = desc->zswap_entry;
spin_unlock(&cluster->lock);
rcu_read_unlock();
- return zswap_entry;
-}
-
-/**
- * zswap_entry_erase - erase a zswap entry for a swap entry
- * @swpentry: the swap entry
- *
- * Erases the zswap entry from the swap descriptor for the given swap entry.
- * The cluster is locked during the erase operation.
- *
- * Return: the zswap entry that was erased, NULL if none existed
- */
-void *zswap_entry_erase(swp_entry_t swpentry)
-{
- struct vswap_cluster *cluster = NULL;
- struct swp_desc *desc;
- void *old;
-
- rcu_read_lock();
- desc = vswap_iter(&cluster, swpentry.val);
- if (!desc) {
- rcu_read_unlock();
+ if (type != VSWAP_ZSWAP)
return NULL;
- }
- old = desc->zswap_entry;
- desc->zswap_entry = NULL;
- spin_unlock(&cluster->lock);
- rcu_read_unlock();
-
- return old;
+ return zswap_entry;
}
bool zswap_empty(swp_entry_t swpentry)
diff --git a/mm/zswap.c b/mm/zswap.c
index 0590a96062fb..e8aa9201ea30 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -991,8 +991,10 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
{
struct folio *folio;
struct mempolicy *mpol;
- bool folio_was_allocated;
+ bool folio_was_allocated, phys_swap_alloced = false;
struct swap_info_struct *si;
+ struct zswap_entry *new_entry = NULL;
+ swp_slot_t slot;
int ret = 0;
/* try to allocate swap cache folio */
@@ -1027,18 +1029,25 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
* old compressed data. Only when this is successful can the entry
* be dereferenced.
*/
- if (entry != zswap_entry_load(swpentry)) {
+ new_entry = zswap_entry_load(swpentry);
+ if (entry != new_entry) {
ret = -ENOMEM;
goto out;
}
+ slot = vswap_alloc_swap_slot(folio);
+
+ if (!slot.val) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ phys_swap_alloced = true;
+
if (!zswap_decompress(entry, folio)) {
ret = -EIO;
goto out;
}
- zswap_entry_erase(swpentry);
-
count_vm_event(ZSWPWB);
if (entry->objcg)
count_objcg_events(entry->objcg, ZSWPWB, 1);
@@ -1052,10 +1061,12 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
folio_set_reclaim(folio);
/* start writeback */
- __swap_writepage(folio, NULL);
+ __swap_writepage(folio, NULL, slot);
out:
if (ret && ret != -EEXIST) {
+ if (phys_swap_alloced)
+ zswap_entry_store(swpentry, new_entry);
swap_cache_del_folio(folio);
folio_unlock(folio);
}
@@ -1401,7 +1412,7 @@ static bool zswap_store_page(struct page *page,
struct zswap_pool *pool)
{
swp_entry_t page_swpentry = page_swap_entry(page);
- struct zswap_entry *entry, *old;
+ struct zswap_entry *entry;
/* allocate entry */
entry = zswap_entry_cache_alloc(GFP_KERNEL, page_to_nid(page));
@@ -1413,15 +1424,12 @@ static bool zswap_store_page(struct page *page,
if (!zswap_compress(page, entry, pool))
goto compress_failed;
- old = zswap_entry_store(page_swpentry, entry);
-
/*
* We may have had an existing entry that became stale when
* the folio was redirtied and now the new version is being
- * swapped out. Get rid of the old.
+ * swapped out. zswap_entry_store() will get rid of the old.
*/
- if (old)
- zswap_entry_free(old);
+ zswap_entry_store(page_swpentry, entry);
/*
* The entry is successfully compressed and stored in vswap, there is
@@ -1473,13 +1481,14 @@ bool zswap_store(struct folio *folio)
struct mem_cgroup *memcg = NULL;
struct zswap_pool *pool;
bool ret = false;
+ bool partial_store = false;
long index;
VM_WARN_ON_ONCE(!folio_test_locked(folio));
VM_WARN_ON_ONCE(!folio_test_swapcache(folio));
if (!zswap_enabled)
- goto check_old;
+ return false;
objcg = get_obj_cgroup_from_folio(folio);
if (objcg && !obj_cgroup_may_zswap(objcg)) {
@@ -1510,8 +1519,10 @@ bool zswap_store(struct folio *folio)
for (index = 0; index < nr_pages; ++index) {
struct page *page = folio_page(folio, index);
- if (!zswap_store_page(page, objcg, pool))
+ if (!zswap_store_page(page, objcg, pool)) {
+ partial_store = index > 0;
goto put_pool;
+ }
}
if (objcg)
@@ -1527,24 +1538,17 @@ bool zswap_store(struct folio *folio)
obj_cgroup_put(objcg);
if (!ret && zswap_pool_reached_full)
queue_work(shrink_wq, &zswap_shrink_work);
-check_old:
/*
- * If the zswap store fails or zswap is disabled, we must invalidate
- * the possibly stale entries which were previously stored at the
- * offsets corresponding to each page of the folio. Otherwise,
- * writeback could overwrite the new data in the swapfile.
+ * If a partial store happened, some pages have stale VSWAP_ZSWAP
+ * entries that must be invalidated. Otherwise, writeback could
+ * overwrite the new data in the swapfile.
+ *
+ * If we never entered the store loop (zswap disabled, cgroup
+ * limits, no pool, etc.), the backing is untouched — no cleanup
+ * needed.
*/
- if (!ret) {
- unsigned type = swp_type(swp);
- pgoff_t offset = swp_offset(swp);
- struct zswap_entry *entry;
-
- for (index = 0; index < nr_pages; ++index) {
- entry = zswap_entry_erase(swp_entry(type, offset + index));
- if (entry)
- zswap_entry_free(entry);
- }
- }
+ if (partial_store)
+ vswap_prepare_writeout(swp, folio);
return ret;
}
@@ -1619,8 +1623,7 @@ int zswap_load(struct folio *folio)
*/
if (swapcache) {
folio_mark_dirty(folio);
- zswap_entry_erase(swp);
- zswap_entry_free(entry);
+ vswap_store_folio(swp, folio);
}
folio_unlock(folio);
--
2.52.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox