From: nakixii <nakixii@gmail.com>
To: linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org,
Andrew Morton <akpm@linux-foundation.org>,
David Hildenbrand <david@kernel.org>,
Johannes Weiner <hannes@cmpxchg.org>,
Chris Li <chrisl@kernel.org>, Kairui Song <kasong@tencent.com>,
Nhat Pham <nphamcs@gmail.com>, Yosry Ahmed <yosry@kernel.org>,
Minchan Kim <minchan@kernel.org>,
Sergey Senozhatsky <senozhatsky@chromium.org>,
Barry Song <baohua@kernel.org>, Hugh Dickins <hughd@google.com>,
Michal Hocko <mhocko@kernel.org>,
Roman Gushchin <roman.gushchin@linux.dev>,
Shakeel Butt <shakeel.butt@linux.dev>,
Mike Rapoport <rppt@kernel.org>, Peter Xu <peterx@redhat.com>,
Suren Baghdasaryan <surenb@google.com>
Subject: [RFC PATCH 06/16] mm: memcomp: store shmem and tmpfs pages
Date: Mon, 16 Nov 2026 19:01:12 +0800 [thread overview]
Message-ID: <20261116110122.309748-7-nakixii@gmail.com> (raw)
In-Reply-To: <20261116110122.309748-1-nakixii@gmail.com>
Anonymous token PTEs do not cover shmem contents held by a page-cache
mapping. Shmem needs an exceptional entry that survives eviction and can
be replaced atomically when the contents are faulted back.
Store eligible single-page shmem folios in memcomp and place their
tokens in the page-cache xarray. Reserve per-key slot storage with cache
ownership distinct from an anonymous producer's resident-record
ownership. Prepare payload and accounting before activating the slot
references that the published shadow will own.
Keep a lookup reference while allocating, charging and decompressing a
refault folio. Confirm that the expected shadow remains installed before
committing the page-cache replacement. Only the winning replacement
consumes the cache owner; an aborted load retains it. Balance inode
accounting, memory charges and temporary references on every exit.
Publish a zero shadow's owner count only when the expected xarray
replacement succeeds. Keep that cache-owner ledger separate from
anonymous zero PTEs, and do not allocate slot metadata for a zero
shadow.
Retain ordinary shmem swap as a fallback. A successful in-memory store
needs no swap device, and truncate or swapin races can retry without
publishing contents from a stale shadow.
Signed-off-by: nakixii <nakixii@gmail.com>
---
include/linux/memcomp.h | 80 +++++++++++++
mm/Kconfig | 3 +-
mm/memcomp.c | 46 ++++++++
mm/memcomp/api.h | 34 ++++++
mm/memcomp/cache_folio.h | 70 ++++++++++++
mm/memcomp/cache_store.h | 70 ++++++++++++
mm/memcomp/consumers.h | 55 +++++++++
mm/shmem.c | 238 +++++++++++++++++++++++++++++++++++++++
8 files changed, 595 insertions(+), 1 deletion(-)
create mode 100644 mm/memcomp/cache_folio.h
create mode 100644 mm/memcomp/cache_store.h
diff --git a/include/linux/memcomp.h b/include/linux/memcomp.h
index 20b553d12151f7bc0202c25d52e4f6832c6ae0b6..20e81dee9062b97274910a1ed3451dd0089ce235 100644
--- a/include/linux/memcomp.h
+++ b/include/linux/memcomp.h
@@ -103,6 +103,9 @@ void memcomp_entry_resident_put(struct memcomp_ref *ref);
/* Consumes and clears the held guard on every return. */
int memcomp_restore_resident(struct vm_fault *vmf, struct memcomp_ref *ref);
void memcomp_invalidate(unsigned long token);
+void memcomp_invalidate_cache(unsigned long token);
+/* Called only by the winning zero-cache XA replacement while locked. */
+void memcomp_publish_cache_zero(void);
void memcomp_invalidate_nr(unsigned long token, unsigned int nr_pages);
void memcomp_invalidate_folio(struct folio *folio);
bool memcomp_release_folio(struct folio *folio);
@@ -125,6 +128,30 @@ void memcomp_entry_clear_refresh_pending(unsigned long token);
bool memcomp_entry_get(unsigned long token);
bool memcomp_entry_present(unsigned long token);
+/*
+ * Page-cache variants for shmem/tmpfs pages. A zero store result prepares
+ * a value only: its winning XA replacement must publish the cache owner
+ * before unlocking. An aborted zero preparation creates no owner to drop.
+ */
+bool memcomp_store_cache(struct folio *folio, unsigned long *token);
+/*
+ * Pin rejects token0 with an empty guard. Explicit zero load consumes no
+ * owner; the winning shadow replacement drops its cache owner separately.
+ * Pin must be resolved with exactly one abort or commit call. Commit also
+ * drops the shadow owner and is valid only after replacing the expected
+ * shadow in the page cache. Failed pins clear the output; abort and commit
+ * consume and clear the guard. Load keeps the guard on every return.
+ */
+bool memcomp_load_folio_pin(unsigned long token, struct memcomp_ref *ref);
+int memcomp_load_folio(struct memcomp_ref *ref, struct folio *folio);
+int memcomp_load_folio_zero(struct folio *folio);
+void memcomp_load_folio_abort(struct memcomp_ref *ref);
+void memcomp_load_folio_commit(struct memcomp_ref *ref);
+
+/* Check if a page-cache xa_value is a memcomp entry. */
+bool is_memcomp_shadow(void *shadow);
+unsigned long memcomp_shadow_token(void *shadow);
+
#else
static inline bool memcomp_available(void)
@@ -225,6 +252,14 @@ static inline void memcomp_invalidate(unsigned long token)
{
}
+static inline void memcomp_invalidate_cache(unsigned long token)
+{
+}
+
+static inline void memcomp_publish_cache_zero(void)
+{
+}
+
static inline void memcomp_invalidate_nr(unsigned long token,
unsigned int nr_pages)
{
@@ -301,6 +336,51 @@ static inline bool memcomp_entry_present(unsigned long token)
return false;
}
+static inline bool memcomp_store_cache(struct folio *folio,
+ unsigned long *token)
+{
+ return false;
+}
+
+static inline bool memcomp_load_folio_pin(unsigned long token,
+ struct memcomp_ref *ref)
+{
+ *ref = (struct memcomp_ref) { };
+ return false;
+}
+
+static inline int memcomp_load_folio(struct memcomp_ref *ref,
+ struct folio *folio)
+{
+ *ref = (struct memcomp_ref) { };
+ return -ENOENT;
+}
+
+static inline int memcomp_load_folio_zero(struct folio *folio)
+{
+ return -ENOENT;
+}
+
+static inline void memcomp_load_folio_abort(struct memcomp_ref *ref)
+{
+ *ref = (struct memcomp_ref) { };
+}
+
+static inline void memcomp_load_folio_commit(struct memcomp_ref *ref)
+{
+ *ref = (struct memcomp_ref) { };
+}
+
+static inline bool is_memcomp_shadow(void *shadow)
+{
+ return false;
+}
+
+static inline unsigned long memcomp_shadow_token(void *shadow)
+{
+ return 0;
+}
+
#endif
#endif
diff --git a/mm/Kconfig b/mm/Kconfig
index d88c27e0235bf8ac79e05e009b8a4625d9d0c693..61356d7e375b4ce4bcf93b5d37108deb8dd655bd 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -195,7 +195,8 @@ config MEMCOMP
select XXHASH
default n
help
- Enable direct in-memory compression for reclaimed anonymous pages.
+ Enable direct in-memory compression for reclaimed anonymous and
+ shmem/tmpfs pages.
Pages are compressed into RAM and mapped through a special
non-swap PTE entry type. This does not require swap block I/O as
diff --git a/mm/memcomp.c b/mm/memcomp.c
index 8944fa081f3e35bfacf00d3a1c09c8bb32a9db18..07242de736f6771d4a6d74a6182b2cad4c9d150d 100644
--- a/mm/memcomp.c
+++ b/mm/memcomp.c
@@ -2234,6 +2234,35 @@ memcomp_prepare_payload_locked(struct memcomp_reclaim_ctx **ctxp,
return 0;
}
+static int memcomp_prepare_payload(const void *src, unsigned int src_len,
+ unsigned int *length,
+ unsigned long *handle,
+ u8 *pool_idx,
+ enum memcomp_payload_mode mode,
+ bool allow_provisional_raw)
+{
+ struct memcomp_reclaim_ctx *ctx;
+ unsigned long element;
+ int ret;
+
+ /* Same-filled shmem pages do not need a compression workspace. */
+ if (page_same_filled(src, &element)) {
+ *length = 0;
+ *handle = element;
+ return 0;
+ }
+
+ ctx = memcomp_get_comp_ctx(mode, NULL, MEMCOMP_RECLAIM_DIRECT);
+ if (!ctx)
+ return -EBUSY;
+ ret = memcomp_prepare_payload_locked(&ctx, src, src_len, length,
+ handle, pool_idx,
+ allow_provisional_raw, NULL);
+ WARN_ON_ONCE(!ctx);
+ memcomp_put_comp_ctx(ctx);
+ return ret;
+}
+
static __always_inline int
memcomp_arch_prepare_to_swap(struct folio *folio)
{
@@ -2242,6 +2271,9 @@ memcomp_arch_prepare_to_swap(struct folio *folio)
/* Allocate @nr_pages consecutive tokens, skipping fragmented free ranges. */
+#include "memcomp/cache_store.h"
+#include "memcomp/cache_folio.h"
+
/*
* Core store - compress @nr_pages pages and insert them into the global
* entries xarray under consecutive tokens. Returns 0 with *token set
@@ -2455,6 +2487,16 @@ static bool memcomp_map_resident(struct vm_area_struct *vma,
#endif
/* Check if a page-cache xa_value is a memcomp entry. */
+bool is_memcomp_shadow(void *shadow)
+{
+ return xa_is_value(shadow) &&
+ is_memcomp_entry(radix_to_swp_entry(shadow));
+}
+
+unsigned long memcomp_shadow_token(void *shadow)
+{
+ return swp_offset(radix_to_swp_entry(shadow));
+}
/*
* Shmem/tmpfs variant: compress a folio and return the token. This does not
@@ -2468,6 +2510,10 @@ static bool memcomp_map_resident(struct vm_area_struct *vma,
* abort. The caller handles locking, page-cache insertion and rmap.
*/
/* Explicit zero transport: no sentinel guard or owner consumption. */
+int memcomp_load_folio_zero(struct folio *folio)
+{
+ return mc16_cache_load_zero(folio);
+}
/* Fill an existing token span after every mapping has been removed. */
diff --git a/mm/memcomp/api.h b/mm/memcomp/api.h
index 5e49d8484f0a303533f5ae04e0477960775c3512..4cc192fba3af9cd858f5752a8e5e931ef4e3af8f 100644
--- a/mm/memcomp/api.h
+++ b/mm/memcomp/api.h
@@ -29,6 +29,16 @@ void memcomp_invalidate_nr(unsigned long token, unsigned int nr)
mc16_pte_put_nr(token, nr);
}
+void memcomp_publish_cache_zero(void)
+{
+ mc16_zero_cache_get(1);
+}
+
+void memcomp_invalidate_cache(unsigned long token)
+{
+ mc16_cache_drop(token);
+}
+
bool memcomp_entry_present(unsigned long token)
{
return mc16_entry_present(token);
@@ -146,4 +156,28 @@ int memcomp_restore_resident(struct vm_fault *vmf, struct memcomp_ref *ref)
return (ret & VM_FAULT_SIGBUS) ? -EFAULT : 0;
}
+bool memcomp_store_cache(struct folio *folio, unsigned long *token)
+{
+ return mc16_store_cache_folio(folio, token);
+}
+
+bool memcomp_load_folio_pin(unsigned long token, struct memcomp_ref *ref)
+{
+ return mc16_cache_pin(token, ref);
+}
+
+int memcomp_load_folio(struct memcomp_ref *ref, struct folio *folio)
+{
+ return mc16_cache_load(ref, folio);
+}
+
+void memcomp_load_folio_abort(struct memcomp_ref *ref)
+{
+ mc16_cache_abort(ref);
+}
+
+void memcomp_load_folio_commit(struct memcomp_ref *ref)
+{
+ mc16_cache_commit(ref);
+}
#endif /* _MM_MEMCOMP_API_H */
diff --git a/mm/memcomp/cache_folio.h b/mm/memcomp/cache_folio.h
new file mode 100644
index 0000000000000000000000000000000000000000..b0b9f154b42f02f54720f44f5c1e15dd2bf4140a
--- /dev/null
+++ b/mm/memcomp/cache_folio.h
@@ -0,0 +1,70 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef _MM_MEMCOMP_CACHE_FOLIO_H
+#define _MM_MEMCOMP_CACHE_FOLIO_H
+
+#include <linux/highmem.h>
+
+/* Include after cache_store.h and memcomp_arch_prepare_to_swap(). */
+#define mc16_cache_folio_arch_prepare memcomp_arch_prepare_to_swap
+
+/*
+ * Caller holds the order-0 source folio lock throughout preparation and XA
+ * replacement. Output is valid only on success and unchanged on failure.
+ * Positive success transfers the initial cache ref, without another get.
+ * Caller drops it if the expected XA replacement loses.
+ * Zero success is preparation only: the winning XA replacement must publish
+ * one zero cache owner under its XA lock. A losing zero preparation owes no put.
+ */
+static inline bool mc16_store_cache_folio(struct folio *folio,
+ unsigned long *token)
+{
+ struct memcomp_ref initial = { };
+#ifdef CONFIG_MEMCOMP_DEBUG_STATS
+ unsigned int nr_pages = folio ? folio_nr_pages(folio) : 1;
+#endif
+ void *src;
+ int ret;
+
+ if (!token || !folio || !folio_test_locked(folio) ||
+ folio_test_large(folio) || folio_mapped(folio) ||
+ folio_test_mlocked(folio) || folio_test_writeback(folio) ||
+ folio_maybe_dma_pinned(folio))
+ goto fail_eligibility;
+ if (!memcomp_is_enabled())
+ goto fail_payload;
+ /* The arch predicate includes tags, unlike a byte-only zero scan. */
+ if (pages_identical(&folio->page, ZERO_PAGE(0))) {
+ *token = 0;
+ goto success;
+ }
+ src = kmap_local_folio(folio, 0);
+ ret = mc16_cache_store(src, folio, &initial);
+ kunmap_local(src);
+ if (ret)
+ goto fail_payload;
+ /* The arch hook keys metadata off page_swap_entry(). */
+ folio->swap = make_memcomp_entry(initial.token);
+ ret = mc16_cache_folio_arch_prepare(folio);
+ folio->swap.val = 0;
+ if (ret) {
+ mc16_ref_put(&initial);
+ goto fail_arch;
+ }
+ *token = initial.token;
+success:
+ memcomp_debug_add(nr_pages, &memcomp_shmem_store_ok_pages);
+ return true;
+
+fail_eligibility:
+ memcomp_debug_add(nr_pages, &memcomp_shmem_eligibility_fail_pages);
+ goto fail;
+fail_payload:
+ memcomp_debug_add(nr_pages, &memcomp_shmem_payload_fail_pages);
+ goto fail;
+fail_arch:
+ memcomp_debug_add(nr_pages, &memcomp_shmem_arch_fail_pages);
+fail:
+ memcomp_debug_add(nr_pages, &memcomp_shmem_store_fail_pages);
+ return false;
+}
+#endif /* _MM_MEMCOMP_CACHE_FOLIO_H */
diff --git a/mm/memcomp/cache_store.h b/mm/memcomp/cache_store.h
new file mode 100644
index 0000000000000000000000000000000000000000..ca2d6097b491aeaaaf2807e367d37ec3f6b8cc8b
--- /dev/null
+++ b/mm/memcomp/cache_store.h
@@ -0,0 +1,70 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef _MM_MEMCOMP_CACHE_STORE_H
+#define _MM_MEMCOMP_CACHE_STORE_H
+
+/*
+ * Prepare one cache payload, retaining its initial cache reference in out.
+ * The caller keeps the source/account folio alive and transfers that reference
+ * only after arch preparation and the expected page-cache replacement succeed.
+ * This lower-level helper does not choose the separate slotless-zero path.
+ */
+/* Private reservation -> counted initial cache ownership, infallibly. */
+static inline void mc16_cache_publish(struct memcomp_ref *ref, unsigned int bytes)
+{
+ memcomp_stored_bytes_inc();
+ memcomp_compressed_bytes_add(bytes);
+ mc16_account_state(ref, true);
+ mc16_cache_activate(ref);
+}
+
+static inline int mc16_cache_store(const void *src, struct folio *account,
+ struct memcomp_ref *out)
+{
+ struct mc16_initial_payload payload = { };
+ struct obj_cgroup *objcg;
+ unsigned long handle = 0;
+ unsigned int bytes = 0;
+ u8 pool = 0;
+ int ret;
+
+ if (!out)
+ return -EINVAL;
+ *out = (struct memcomp_ref) { };
+ if (!src || !account)
+ return -EINVAL;
+ if (!memcomp_is_enabled())
+ return -EOPNOTSUPP;
+ /* Independent source owner, never a borrowed registry key across X1/X2. */
+ objcg = memcomp_objcg_get(account);
+ ret = memcomp_prepare_payload(src, PAGE_SIZE, &bytes, &handle, &pool,
+ MEMCOMP_PAYLOAD_BEST_EFFORT, false);
+ if (ret)
+ goto out_owner;
+ if (!memcomp_payloads_net_reclaimable(&bytes, 1)) {
+ if (bytes == PAGE_SIZE)
+ memcomp_debug_inc(&memcomp_raw_reject_net);
+ ret = -E2BIG;
+ goto out_payload;
+ }
+ ret = memcomp_objcg_charge(objcg, GFP_NOWAIT | __GFP_NOWARN, bytes);
+ if (ret)
+ goto out_payload;
+ payload = (struct mc16_initial_payload) { .word = handle, .bytes = bytes };
+ ret = mc16_reserve_cache(objcg, folio_nid(account), &payload, 1, out);
+ if (ret) {
+ memcomp_objcg_uncharge(objcg, bytes);
+ goto out_payload;
+ }
+ /* No pending/resident/OWNER transition or record/member RMW occurs. */
+ mc16_cache_publish(out, bytes);
+ ret = 0;
+ goto out_owner;
+out_payload:
+ if (bytes)
+ memcomp_zs_free(pool, handle);
+out_owner:
+ if (objcg)
+ memcomp_objcg_put(objcg);
+ return ret;
+}
+#endif /* _MM_MEMCOMP_CACHE_STORE_H */
diff --git a/mm/memcomp/consumers.h b/mm/memcomp/consumers.h
index 929c046bf98d52c878e727e14d859d81c871b743..03aef9ffecb6df83b6314df43739677120fdd8f8 100644
--- a/mm/memcomp/consumers.h
+++ b/mm/memcomp/consumers.h
@@ -46,19 +46,74 @@ static inline void mc16_pte_put_nr(unsigned long token, unsigned int nr)
}
/* The caller removed one cache owner, independently of any PTE owner. */
+static inline void mc16_cache_drop(unsigned long token)
+{
+ struct memcomp_ref ref;
+
+ if (!token) {
+ mc16_zero_cache_put(1);
+ return;
+ }
+ if (mc16_get(token, &ref))
+ mc16_ref_put_count(&ref, 2); /* lookup + removed cache owner */
+}
/* Empty output on failure, including token0; caller handles zero explicitly. */
+static inline bool mc16_cache_pin(unsigned long token, struct memcomp_ref *ref)
+{
+ return mc16_get(token, ref);
+}
/*
* Keep this same guard through load and the caller's xarray transaction.
* Failure leaves both the guard and cache owner intact for abort/retry.
*/
+static inline int mc16_cache_load(struct memcomp_ref *ref, struct folio *folio)
+{
+ void *dst;
+ int ret;
+
+ if (!ref || !ref->slot)
+ return -ENOENT;
+ if (!folio || folio_test_large(folio) || !folio_test_locked(folio))
+ return -EINVAL;
+ dst = kmap_local_folio(folio, 0);
+ ret = mc16_load_page(ref, dst);
+ kunmap_local(dst);
+ if (!ret) {
+ __folio_mark_uptodate(folio);
+ flush_dcache_folio(folio);
+ memcomp_arch_restore(ref->token, folio);
+ }
+ return ret;
+}
+
+static inline void mc16_cache_abort(struct memcomp_ref *ref)
+{
+ if (ref && ref->slot)
+ mc16_ref_put(ref);
+}
/* Only the winner replacing the expected cache shadow may consume its owner. */
+static inline void mc16_cache_commit(struct memcomp_ref *ref)
+{
+ if (ref && ref->slot)
+ mc16_ref_put_count(ref, 2); /* same guard + cache owner */
+}
/*
* Explicit token0 branch: no guard, lookup or owner consumption. After the
* expected zero shadow is replaced successfully, caller must cache_drop(0).
* An aborted xarray transaction leaves the zero cache owner unchanged.
*/
+static inline int mc16_cache_load_zero(struct folio *folio)
+{
+ if (!folio || folio_test_large(folio) || !folio_test_locked(folio))
+ return -EINVAL;
+ clear_highpage(folio_page(folio, 0));
+ __folio_mark_uptodate(folio);
+ flush_dcache_folio(folio);
+ memcomp_arch_restore(0, folio);
+ return 0;
+}
#endif /* _MM_MEMCOMP_CONSUMERS_H */
diff --git a/mm/shmem.c b/mm/shmem.c
index 5960e5035f983571fb806313855f74c61abf8354..a04e68d445d37384b02b56e791586d09b392fcfa 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -80,6 +80,9 @@ static struct vfsmount *shm_mnt __ro_after_init;
#include <linux/uuid.h>
#include <linux/quotaops.h>
#include <linux/rcupdate_wait.h>
+#ifdef CONFIG_MEMCOMP
+#include <linux/memcomp.h>
+#endif
#include <linux/uaccess.h>
@@ -847,6 +850,15 @@ static void shmem_delete_from_page_cache(struct folio *folio, void *radswap)
xa_lock_irq(&mapping->i_pages);
error = shmem_replace_entry(mapping, folio->index, folio, radswap);
+#ifdef CONFIG_MEMCOMP
+ if (!error) {
+ swp_entry_t entry = radix_to_swp_entry(radswap);
+
+ /* Publish ownership before any reader can remove this shadow. */
+ if (is_memcomp_entry(entry) && !swp_offset(entry))
+ memcomp_publish_cache_zero();
+ }
+#endif
folio->mapping = NULL;
mapping->nrpages -= nr;
shmem_update_stats(folio, -nr);
@@ -864,14 +876,29 @@ static long shmem_free_swap(struct address_space *mapping,
pgoff_t index, void *radswap)
{
int order = xa_get_order(&mapping->i_pages, index);
+#ifdef CONFIG_MEMCOMP
+ swp_entry_t entry = radix_to_swp_entry(radswap);
+ long nr_pages = is_memcomp_entry(entry) ? 1 : 1 << order;
+#endif
void *old;
old = xa_cmpxchg_irq(&mapping->i_pages, index, radswap, NULL, 0);
if (old != radswap)
return 0;
+#ifdef CONFIG_MEMCOMP
+ if (is_memcomp_entry(entry))
+ memcomp_invalidate_cache(swp_offset(entry));
+ else
+ free_swap_and_cache_nr(entry, nr_pages);
+#else
free_swap_and_cache_nr(radix_to_swp_entry(radswap), 1 << order);
+#endif
+#ifdef CONFIG_MEMCOMP
+ return nr_pages;
+#else
return 1 << order;
+#endif
}
/*
@@ -893,8 +920,23 @@ unsigned long shmem_partial_swap_usage(struct address_space *mapping,
xas_for_each(&xas, page, max) {
if (xas_retry(&xas, page))
continue;
+#ifdef CONFIG_MEMCOMP
+ if (xa_is_value(page)) {
+ swp_entry_t entry = radix_to_swp_entry(page);
+
+ if (is_memcomp_entry(entry)) {
+ if (memcomp_entry_stat(swp_offset(entry),
+ NULL, NULL))
+ swapped += PAGE_SIZE;
+ } else {
+ swapped += (1UL << xas_get_order(&xas)) <<
+ PAGE_SHIFT;
+ }
+ }
+#else
if (xa_is_value(page))
swapped += 1 << xas_get_order(&xas);
+#endif
if (xas.xa_index == max)
break;
if (need_resched()) {
@@ -904,7 +946,11 @@ unsigned long shmem_partial_swap_usage(struct address_space *mapping,
}
rcu_read_unlock();
+#ifdef CONFIG_MEMCOMP
+ return swapped;
+#else
return swapped << PAGE_SHIFT;
+#endif
}
/*
@@ -932,6 +978,15 @@ unsigned long shmem_swap_usage(struct vm_area_struct *vma)
if (!swapped)
return 0;
+#ifdef CONFIG_MEMCOMP
+ /*
+ * info->swapped counts a memcomp entry as one page, which is what
+ * shmem_partial_swap_usage() also reports for it, so the whole-object
+ * answer is still exact without the walk. Restoring this matters:
+ * smaps_rollup is polled continuously, and the walk is an RCU pass
+ * over every entry of the mapping with an xa_load per memcomp one.
+ */
+#endif
if (!vma->vm_pgoff && vma->vm_end - vma->vm_start >= inode->i_size)
return swapped << PAGE_SHIFT;
@@ -1321,6 +1376,10 @@ static int shmem_find_swap_entries(struct address_space *mapping,
continue;
entry = radix_to_swp_entry(folio);
+#ifdef CONFIG_MEMCOMP
+ if (is_memcomp_entry(entry))
+ continue;
+#endif
/*
* swapin error entries can be found in the mapping. But they're
* deliberately ignored here as we've done everything we can do.
@@ -1459,7 +1518,13 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc)
struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
swp_entry_t swap;
pgoff_t index;
+#ifdef CONFIG_MEMCOMP
+ unsigned long mc_token;
+#endif
int nr_pages;
+#ifdef CONFIG_MEMCOMP
+ bool memcomp_attempted = false;
+#endif
bool split = false;
/*
@@ -1475,8 +1540,84 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc)
if (WARN_ON_ONCE((info->flags & VM_LOCKED) || sbinfo->noswap))
goto redirty;
+#ifdef CONFIG_MEMCOMP
+ /*
+ * Only an uptodate folio. The !uptodate handling below is what gives
+ * shmem_fallocate() its backpressure - it counts the folio in
+ * shmem_falloc->nr_unswapped and redirties it so the fallocation can
+ * give up with -ENOMEM - and what zeroes a folio whose contents were
+ * never initialised. Jumping straight to the store skipped both, so a
+ * fallocate that should have failed ran on to OOM, and on a kernel
+ * booted with init_on_alloc=0 the uninitialised page became file
+ * contents. A folio that arrives !uptodate takes the swap path, and
+ * the retries below reach the store once it has been made uptodate.
+ */
+ if (memcomp_available() && !folio_test_large(folio) &&
+ folio_test_uptodate(folio)) {
+ memcomp_count_initial_compress();
+ goto try_memcomp;
+ }
+ goto try_swap;
+
+try_memcomp:
+ memcomp_attempted = true;
+ if (!folio_test_large(folio)) {
+ if (memcomp_store_cache(folio, &mc_token)) {
+ swp_entry_t mc_entry = make_memcomp_entry(mc_token);
+
+ mutex_lock(&shmem_swaplist_mutex);
+ if (list_empty(&info->swaplist))
+ list_add(&info->swaplist, &shmem_swaplist);
+ mutex_unlock(&shmem_swaplist_mutex);
+
+ /*
+ * Raise swapped before nrpages is lowered, as the
+ * swap path below does. shmem_recalc_inode()
+ * compensates for exactly that order - see the
+ * comment on its `swapped > 0` clause - so calling it
+ * after shmem_delete_from_page_cache() has already
+ * dropped nrpages makes it count the page as freed
+ * and hand a block back that nothing released.
+ */
+ shmem_recalc_inode(inode, 0, 1);
+ shmem_delete_from_page_cache(folio,
+ swp_to_radix_entry(mc_entry));
+ if (folio_test_dirty(folio))
+ folio_clear_dirty(folio);
+ /*
+ * Leave the token on the folio. The folio now has no
+ * mapping and one reference, and shrink_folio_list()
+ * only credits nr_reclaimed for a folio it frees on
+ * its free_it: path. Its recovery arm for exactly
+ * this shape keys on page_swap_entry(), so without a
+ * token the folio takes keep_locked, is dropped
+ * unaccounted by move_folios_to_lru(), and direct
+ * reclaim reports no progress for work it did - which
+ * __alloc_pages_slowpath() counts toward OOM.
+ *
+ * The entry is owned by the page-cache xarray, not by
+ * the folio: memcomp_do_store() was passed no
+ * owner folio, so it carries MC_SHMEM rather than
+ * MC_OWNER and the memcomp_invalidate_folio() that
+ * arm performs drops nothing. This is the same
+ * handover memcomp_reserve() makes for anonymous
+ * folios.
+ */
+ folio->swap = mc_entry;
+ folio_unlock(folio);
+ return 0;
+ }
+ }
+
+try_swap:
+ /* This is the regular shmem fallback, not memcomp writeback. */
+#endif
if (!total_swap_pages)
goto redirty;
+#ifdef CONFIG_MEMCOMP
+ if (memcomp_attempted)
+ memcomp_count_normal_swap_fallback();
+#endif
/*
* If CONFIG_THP_SWAP is not enabled, the large folio should be
@@ -1541,6 +1682,13 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc)
swap = folio_alloc_swap(folio);
if (!swap.val) {
+#ifdef CONFIG_MEMCOMP
+ if (!memcomp_attempted && memcomp_available() &&
+ !folio_test_large(folio)) {
+ memcomp_count_initial_compress();
+ goto try_memcomp;
+ }
+#endif
if (nr_pages > 1)
goto try_split;
@@ -1573,6 +1721,13 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc)
mutex_unlock(&shmem_swaplist_mutex);
put_swap_folio(folio, swap);
+#ifdef CONFIG_MEMCOMP
+ if (!memcomp_attempted && memcomp_available() &&
+ !folio_test_large(folio)) {
+ memcomp_count_initial_compress();
+ goto try_memcomp;
+ }
+#endif
redirty:
folio_mark_dirty(folio);
if (wbc->for_reclaim)
@@ -2110,6 +2265,89 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index,
if (is_poisoned_swp_entry(swap))
return -EIO;
+#ifdef CONFIG_MEMCOMP
+
+ if (is_memcomp_entry(swap)) {
+ struct memcomp_ref mc_ref = { };
+ unsigned long token = swp_offset(swap);
+
+ if (!shmem_confirm_swap(mapping, index, swap))
+ return -EEXIST;
+
+ if (shmem_inode_acct_blocks(inode, 1))
+ return -ENOSPC;
+
+ folio = shmem_alloc_folio(gfp, 0, info, index);
+ if (!folio) {
+ shmem_inode_unacct_blocks(inode, 1);
+ return -ENOMEM;
+ }
+ __folio_set_locked(folio);
+ __folio_set_swapbacked(folio);
+ nr_pages = folio_nr_pages(folio);
+
+ error = mem_cgroup_charge(folio, fault_mm, gfp & GFP_RECLAIM_MASK);
+ if (error) {
+ shmem_inode_unacct_blocks(inode, nr_pages);
+ folio_unlock(folio);
+ folio_put(folio);
+ return error;
+ }
+
+ /*
+ * Allocation and memcg charge may reclaim or wait for OOM.
+ * Pin only after both, then recheck the page-cache entry.
+ */
+ if (token)
+ memcomp_load_folio_pin(token, &mc_ref);
+ if ((token && !mc_ref.slot) ||
+ !shmem_confirm_swap(mapping, index, swap)) {
+ shmem_inode_unacct_blocks(inode, nr_pages);
+ folio_unlock(folio);
+ folio_put(folio);
+ if (mc_ref.slot)
+ memcomp_load_folio_abort(&mc_ref);
+ return -EEXIST;
+ }
+
+ error = token ? memcomp_load_folio(&mc_ref, folio) :
+ memcomp_load_folio_zero(folio);
+ if (error) {
+ shmem_inode_unacct_blocks(inode, nr_pages);
+ folio_unlock(folio);
+ folio_put(folio);
+ if (!shmem_confirm_swap(mapping, index, swap))
+ error = -EEXIST;
+ memcomp_load_folio_abort(&mc_ref);
+ return error;
+ }
+
+ error = shmem_add_to_page_cache(folio, mapping, index,
+ swp_to_radix_entry(swap), gfp);
+ if (error) {
+ shmem_inode_unacct_blocks(inode, nr_pages);
+ folio_unlock(folio);
+ folio_put(folio);
+ memcomp_load_folio_abort(&mc_ref);
+ return error == -EEXIST ? -EEXIST : error;
+ }
+
+ shmem_recalc_inode(inode, 1, -1);
+
+ if (sgp == SGP_WRITE)
+ folio_mark_accessed(folio);
+
+ folio_mark_dirty(folio);
+ if (token)
+ memcomp_load_folio_commit(&mc_ref);
+ else
+ memcomp_invalidate_cache(0);
+ folio_add_lru(folio);
+
+ *foliop = folio;
+ return 0;
+ }
+#endif
si = get_swap_device(swap);
if (!si) {
next prev parent reply other threads:[~2026-11-16 11:04 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 11:01 [RFC PATCH 00/16] mm: add in-memory compression of reclaimed pages nakixii
2026-11-16 11:01 ` [RFC PATCH 01/16] mm: userfaultfd: check the exclusivity of the page being moved nakixii
2026-11-16 11:01 ` [RFC PATCH 02/16] mm: add a swap entry type for in-memory compressed pages nakixii
2026-11-16 11:01 ` [RFC PATCH 03/16] mm: memcomp: compress reclaimed anonymous pages in memory nakixii
2026-11-16 11:01 ` [RFC PATCH 04/16] mm: userfaultfd: let UFFDIO_MOVE move memcomp entries nakixii
2026-11-16 11:01 ` [RFC PATCH 05/16] mm: memcomp: charge compressed pages to the source memcg nakixii
2026-11-16 11:01 ` nakixii [this message]
2026-11-16 11:01 ` [RFC PATCH 07/16] mm: memcomp: write cold pages back to swap nakixii
2026-11-16 11:01 ` [RFC PATCH 08/16] mm: memcomp: compress foreground batches asynchronously nakixii
2026-11-16 11:01 ` [RFC PATCH 09/16] mm: memcomp: batch unmap reserved large folios nakixii
2026-11-16 11:01 ` [RFC PATCH 10/16] mm: memcomp: queue restored large folios for splitting nakixii
2026-11-16 11:01 ` [RFC PATCH 11/16] mm: memcomp: account user eviction and refault nakixii
2026-11-16 11:01 ` [RFC PATCH 12/16] mm: memcomp: report precise token residency nakixii
2026-11-16 11:01 ` [RFC PATCH 13/16] mm: memcomp: add KUnit tests nakixii
2026-11-16 11:01 ` [RFC PATCH 14/16] selftests/mm: add a memcomp pageout and refault test nakixii
2026-11-16 11:01 ` [RFC PATCH 15/16] mm: memcomp: add the design doc nakixii
2026-11-16 11:01 ` [RFC PATCH 16/16] mm: memcomp: add the admin guide nakixii
2026-11-16 11:14 ` [RFC PATCH 00/16] mm: add in-memory compression of reclaimed pages Mike Rapoport
2026-11-17 16:07 ` nakixii
2026-11-16 14:13 ` Nhat Pham
2026-11-17 16:10 ` nakixii
2026-11-17 22:57 ` Nhat Pham
2026-11-17 16:27 ` Kairui Song
2026-11-17 19:10 ` nakixii
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261116110122.309748-7-nakixii@gmail.com \
--to=nakixii@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=baohua@kernel.org \
--cc=cgroups@vger.kernel.org \
--cc=chrisl@kernel.org \
--cc=david@kernel.org \
--cc=hannes@cmpxchg.org \
--cc=hughd@google.com \
--cc=kasong@tencent.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@kernel.org \
--cc=minchan@kernel.org \
--cc=nphamcs@gmail.com \
--cc=peterx@redhat.com \
--cc=roman.gushchin@linux.dev \
--cc=rppt@kernel.org \
--cc=senozhatsky@chromium.org \
--cc=shakeel.butt@linux.dev \
--cc=surenb@google.com \
--cc=yosry@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.