[PATCH] ksm: add merge_only_zero_pages to merge only empty anonymous pages

[PATCH] ksm: add merge_only_zero_pages to merge only empty anonymous pages

[xu.xin]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=y, Size: 8462 bytes --]

From: "Xu Xin (vivo)" <xu.xin@linux.dev>

Add a new sysfs attribute, /sys/kernel/mm/ksm/merge_only_zero_pages,
which, when enabled, makes ksmd merge only empty (zeroed) anonymous
pages with the kernel zero page and skip the stable and unstable trees
entirely for pages that carry data.

The attribute requires use_zero_pages to be 1, and the two are kept
consistent by symmetric constraints: it cannot be enabled while
use_zero_pages is 0 (-EINVAL), and use_zero_pages cannot be turned
off while it is active (-EBUSY). The new attribute can be changed at
any time and is only effective for pages merged after the change.

Benefits in three aspects
=========================
- Memory saving:
  On memory-constrained systems such as Android and embedded devices,
  zero-filled anonymous pages are the largest source of duplicate
  anonymous memory. They are produced by userspace allocators: the
  zeroing of freshly allocated Java objects and buffers in the ART heap
  (new byte[]/int[], Bitmap buffers), calloc/memset in native code, and
  preallocated media buffers. Merging them with the kernel zero page
  frees a significant amount of physical memory, which matters as DRAM
  costs keep rising. With this enabled, we measured 100MB–300MB saving
  tested with several popular applications on aarch64 (Android).

- CPU overhead reduction:
  Merging only zero pages also reduces the overhead of ksmd. A page that
  is not merged with the zero page carries data and can never be merged
  with the zero page, so there is no point in searching the stable tree
  or inserting it into the unstable tree for it. Skipping those tree
  operations for data-bearing pages reduces both the CPU time and the
  rhashtable memory footprint of ksmd, which makes zero page merging
  practical on low-resource devices where full KSM scanning would be too
  expensive.

- Security:
  Finally, this shrinks the KSM side-channel attack surface: only zero
  pages are ever merged, so the sharing status of data-bearing pages is
  no longer observable through KSM timing side channels.

Note
=====
- Suitable for Android, embedded and low-memory devices, where zero-page
  deduplication is wanted but full KSM scanning is too costly, and reduced
  KSM side-channel exposure is desired. Many handset vendors may prefer
  this knob, which can be used as a default fallback policy.

- Not suitable for server and virtualization workloads containing many
  non-zero duplicate pages, where vanilla KSM provides better memory
  savings.

Signed-off-by: Xu Xin (vivo) <xu.xin@linux.dev>
#Reviewed-by: Huan Yang <link@vivo.com>
#Cc: Wang Xiao <xiao.wang@vivo.com>
---
 Documentation/admin-guide/mm/ksm.rst | 40 ++++++++++++++++++++-
 mm/ksm.c                             | 53 ++++++++++++++++++++++++++++
 2 files changed, 92 insertions(+), 1 deletion(-)

diff --git a/Documentation/admin-guide/mm/ksm.rst b/Documentation/admin-guide/mm/ksm.rst
index c329ca747b8c..dd8ef2e4fdc2 100644
--- a/Documentation/admin-guide/mm/ksm.rst
+++ b/Documentation/admin-guide/mm/ksm.rst
@@ -128,10 +128,48 @@ use_zero_pages
         KSM for some workloads, for example if the checksums of pages
         candidate for merging match the checksum of an empty
         page. This setting can be changed at any time, it is only
-        effective for pages merged after the change.
+        effective for pages merged after the change. Turning this off
+        while ``merge_only_zero_pages`` is enabled fails with EBUSY;
+        clear ``merge_only_zero_pages`` first.
 
         Default: 0 (normal KSM behaviour as in earlier releases)
 
+merge_only_zero_pages
+        specifies whether ksmd should only merge empty (zeroed)
+        anonymous pages, with the kernel zero page, and skip the stable
+        and unstable trees entirely for pages that carry data. This
+        requires ``use_zero_pages`` to be 1; it cannot be enabled while
+        ``use_zero_pages`` is 0, and ``use_zero_pages`` cannot be turned
+        off while it is enabled (the write fails with EBUSY).
+
+        This is useful on memory-constrained systems (e.g. Android and
+        embedded devices) where the main source of duplicate anonymous
+        memory is zero-filled pages produced by userspace allocators
+        (for example the zeroing of freshly allocated Java objects and
+        buffers in the ART heap, ``calloc``/``memset`` in native code,
+        and pre-allocated media buffers). Merging only those pages
+        avoids the per-page cost of searching and inserting into the
+        stable and unstable trees for data-bearing pages, which reduces
+        the CPU and memory overhead of ksmd. It also shrinks the KSM
+        side-channel attack surface, since only zero pages are ever
+        merged and the sharing state of data-bearing pages is no longer
+        observable through KSM.
+
+        - Suitable for: Android, embedded and low-memory devices, where
+          zero-page deduplication is wanted but full KSM scanning is
+          too costly, and reduced KSM side-channel exposure is desired.
+          Many handset vendors may prefer this knob, which can be used
+          as a default fallback policy.
+
+        - Not suitable for: Server and virtualization workloads containing
+          many non-zero duplicate pages, where vanilla KSM provides better
+          memory savings.
+
+        This setting can be changed at any time, it is only effective
+        for pages merged after the change.
+
+        Default: 0 (normal KSM behaviour)
+
 max_page_sharing
         Maximum sharing allowed for each KSM page. This enforces a
         deduplication limit to avoid high latency for virtual memory
diff --git a/mm/ksm.c b/mm/ksm.c
index 64f9e803b99e..a89b0f1c1065 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -297,6 +297,13 @@ static unsigned int zero_checksum __read_mostly;
 /* Whether to merge empty (zeroed) pages with actual zero pages */
 static bool ksm_use_zero_pages __read_mostly;
 
+/*
+ * When set, ksmd only merges empty (zeroed) anonymous pages with the kernel
+ * zero page and skips the stable/unstable tree entirely for pages that carry
+ * data. This requires ksm_use_zero_pages to be enabled.
+ */
+static bool ksm_merge_only_zero_pages __read_mostly;
+
 /* Skip pages that couldn't be de-duplicated previously */
 /* Default to true at least temporarily, for testing */
 static bool ksm_smart_scan = true;
@@ -2369,6 +2376,15 @@ static void cmp_and_merge_page(struct page *page, struct ksm_rmap_item *rmap_ite
 
 		if (!try_to_merge_with_zero_page(rmap_item, page))
 			return;
+
+		/*
+		 * merge_only_zero_pages: only empty (zeroed) pages are merged,
+		 * with the kernel zero page. A page that reached here was not
+		 * merged with the zero page, so it carries data; skip the
+		 * stable and unstable trees for it.
+		 */
+		if (ksm_merge_only_zero_pages)
+			return;
 	}
 
 	/* Start by searching for the folio in the stable tree */
@@ -3730,12 +3746,48 @@ static ssize_t use_zero_pages_store(struct kobject *kobj,
 	if (err)
 		return -EINVAL;
 
+	/*
+	 * merge_only_zero_pages depends on use_zero_pages: refuse to turn
+	 * the latter off while the former is active, so the two cannot
+	 * silently diverge.
+	 */
+	if (!value && ksm_merge_only_zero_pages)
+		return -EBUSY;
+
 	ksm_use_zero_pages = value;
 
 	return count;
 }
 KSM_ATTR(use_zero_pages);
 
+static ssize_t merge_only_zero_pages_show(struct kobject *kobj,
+					  struct kobj_attribute *attr, char *buf)
+{
+	return sysfs_emit(buf, "%u\n", ksm_merge_only_zero_pages);
+}
+static ssize_t merge_only_zero_pages_store(struct kobject *kobj,
+					   struct kobj_attribute *attr,
+					   const char *buf, size_t count)
+{
+	int err;
+	bool value;
+
+	err = kstrtobool(buf, &value);
+	if (err)
+		return -EINVAL;
+
+	if (value && !ksm_use_zero_pages) {
+		pr_warn_once("ksm: merge_only_zero_pages can only be enabled "
+			     "when use_zero_pages is 1\n");
+		return -EINVAL;
+	}
+
+	ksm_merge_only_zero_pages = value;
+
+	return count;
+}
+KSM_ATTR(merge_only_zero_pages);
+
 static ssize_t max_page_sharing_show(struct kobject *kobj,
 				     struct kobj_attribute *attr, char *buf)
 {
@@ -4061,6 +4113,7 @@ static struct attribute *ksm_attrs[] = {
 	&stable_node_dups_attr.attr,
 	&stable_node_chains_prune_millisecs_attr.attr,
 	&use_zero_pages_attr.attr,
+	&merge_only_zero_pages_attr.attr,
 	&general_profit_attr.attr,
 	&smart_scan_attr.attr,
 	&advisor_mode_attr.attr,
-- 
2.34.1

Re: [PATCH] ksm: add merge_only_zero_pages to merge only empty anonymous pages

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] mm: ksm: race between use_zero_pages_store() and
  merge_only_zero_pages_store()
--

commit e3394c6484d9d77c2cd89b85bc6fb89ad39b6904
Author: Xu Xin (vivo) <xu.xin@linux.dev>

ksm: add merge_only_zero_pages to merge only empty anonymous pages

This commit adds a new sysfs attribute, /sys/kernel/mm/ksm/merge_only_zero_pages,
which causes ksmd to merge only zeroed anonymous pages with the kernel zero
page and skip the stable and unstable trees for non-zero pages.

> diff --git a/mm/ksm.c b/mm/ksm.c
> index 340c65aa14e58..35ed40747e30e 100644
> --- a/mm/ksm.c
> +++ b/mm/ksm.c

[ ... ]

> @@ -3679,12 +3695,48 @@ static ssize_t use_zero_pages_store(struct kobject *kobj,
>  	if (err)
>  		return -EINVAL;
>  
> +	/*
> +	 * merge_only_zero_pages depends on use_zero_pages: refuse to turn
> +	 * the latter off while the former is active, so the two cannot
> +	 * silently diverge.
> +	 */
> +	if (!value && ksm_merge_only_zero_pages)
> +		return -EBUSY;
> +
>  	ksm_use_zero_pages = value;
>  
>  	return count;
>  }
>  KSM_ATTR(use_zero_pages);
>  
> +static ssize_t merge_only_zero_pages_show(struct kobject *kobj,
> +					  struct kobj_attribute *attr, char *buf)
> +{
> +	return sysfs_emit(buf, "%u\n", ksm_merge_only_zero_pages);
> +}
> +static ssize_t merge_only_zero_pages_store(struct kobject *kobj,
> +					   struct kobj_attribute *attr,
> +					   const char *buf, size_t count)
> +{
> +	int err;
> +	bool value;
> +
> +	err = kstrtobool(buf, &value);
> +	if (err)
> +		return -EINVAL;
> +
> +	if (value && !ksm_use_zero_pages) {
> +		pr_warn_once("ksm: merge_only_zero_pages can only be enabled "
> +			     "when use_zero_pages is 1\n");
> +		return -EINVAL;
> +	}
> +
> +	ksm_merge_only_zero_pages = value;
> +
> +	return count;
> +}

[Severity: Medium]
Can concurrent writes to use_zero_pages and merge_only_zero_pages race with
each other and leave ksm_use_zero_pages set to false while
ksm_merge_only_zero_pages is set to true?

Neither use_zero_pages_store() nor merge_only_zero_pages_store() acquires
ksm_thread_mutex or another shared lock, and kernfs_fop_write_iter() only
serializes writes per open file description. Starting with
ksm_use_zero_pages == true and ksm_merge_only_zero_pages == false, two
concurrent sysfs writes can interleave:

CPU0                                  CPU1
use_zero_pages_store()                merge_only_zero_pages_store()
  if (!value &&
      ksm_merge_only_zero_pages)
    // sees false, passes check
                                        if (value && !ksm_use_zero_pages)
                                          // sees true, passes check
  ksm_use_zero_pages = false;
                                        ksm_merge_only_zero_pages = true;

If that state is reached, cmp_and_merge_page() will see
try_to_merge_with_zero_page() fail with -EFAULT because ksm_use_zero_pages is
false, and then immediately return early at:

	if (!try_to_merge_with_zero_page(rmap_item, page))
		return;

	if (ksm_merge_only_zero_pages)
		return;

causing ksmd to scan and checksum pages without ever merging either zero
pages or non-zero pages.

Should use_zero_pages_store() and merge_only_zero_pages_store() serialize
these checks and updates under ksm_thread_mutex?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106034304.231809-1-xu.xin@linux.dev?part=1