All of lore.kernel.org
 help / color / mirror / Atom feed
From: Sergey Senozhatsky <senozhatsky@chromium.org>
To: Hongfeng Chen <chenhongfeng0212@gmail.com>,
	 Nhat Pham <nphamcs@gmail.com>
Cc: Hongfeng Chen <chenhongfeng0212@163.com>,
	 Andrew Morton <akpm@linux-foundation.org>,
	Barry Song <baohua@kernel.org>,
	 Chengming Zhou <chengming.zhou@linux.dev>,
	Jens Axboe <axboe@kernel.dk>,
	 Johannes Weiner <hannes@cmpxchg.org>,
	Minchan Kim <minchan@kernel.org>,
	 Shakeel Butt <shakeel.butt@linux.dev>,
	Yosry Ahmed <yosry@kernel.org>,
	 Wenchao Hao <haowenchao@xiaomi.com>,
	Wenchao Hao <haowenchao22@gmail.com>,
	linux-mm@kvack.org,  linux-kernel@vger.kernel.org,
	linux-block@vger.kernel.org,
	 Hongfeng Chen <hongfengchen@xiaomi.com>,
	Sergey Senozhatsky <senozhatsky@chromium.org>
Subject: Re: [RFC PATCH v4 0/2] mm/zsmalloc: defer object free to per-cpu batches
Date: Fri, 20 Nov 2026 16:48:30 +0900	[thread overview]
Message-ID: <av_4KLI35RzCQ7zC@google.com> (raw)
In-Reply-To: <av-ltJooUJ0lqvqa@google.com>

On (26/11/20 10:44), Sergey Senozhatsky wrote:
> On (26/11/19 14:35), Sergey Senozhatsky wrote:
> > On (26/11/09 18:59), Hongfeng Chen wrote:
> > > On Fri, Oct 23, 2026 at 11:19 AM Nhat Pham <nphamcs@gmail.com> wrote:
> > > > Is there any more juice we can squeeze from zsmalloc in that case?
> > > >
> > > > It would be nice if you can perf zsmalloc freeing. And yeah, maybe we
> > > > can't optimize any further, but let's eliminate that first before we
> > > > rely on offloading.
> > > 
> > > Here is the perf breakdown of the free path:
> > > 
> > > zram:
> > > 
> > >   __swap_cluster_free_entries        90.28%
> > >     zram_slot_free_notify            86.01%
> > >       slot_free                      74.15%
> > >         zs_free                      56.01%
> > >           __free_frozen_pages        19.85%
> > >           kmem_cache_free            10.15%
> > >           _raw_spin_lock              5.53%
> > >           fix_fullness_group          3.31%
> > >           dec_zone_page_state         2.43%
> > >           obj_free                    2.17%
> > >           trylock_zspage              2.12%
> > 
> > We removed one of kmem_cache_free() from zs_free() [1].
> > 
> > [1] https://lore.kernel.org/linux-kernel/20261110054836.1515030-1-senozhatsky@chromium.org
> 
> I suppose we can batch some operations in zs_free(), something like
> this:

A slightly reworked patch below - handle incompressible and pow-of-two
zspages separately (w/o batching), because those are 1-page zspages,
nothing to batch.

---
 mm/zsmalloc.c | 52 +++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 46 insertions(+), 6 deletions(-)

diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index f571b3c9e9ca..9dcd1549c2fb 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -37,6 +37,7 @@
 #include <linux/debugfs.h>
 #include <linux/zsmalloc.h>
 #include <linux/fs.h>
+#include <linux/folio_batch.h>
 #include <linux/workqueue.h>
 #include "zpdesc.h"
 
@@ -866,6 +867,44 @@ static int trylock_zspage(struct zspage *zspage)
 	return 0;
 }
 
+/*
+ * Free the pages of a multi-page zspage in one go, see folios_put().
+ * Not inlined, so that the folio_batch is only set up (and, with
+ * CONFIG_INIT_STACK_ALL_ZERO, zero-filled) when there is something
+ * to batch.
+ */
+static noinline void free_page_chain(struct zpdesc *zpdesc)
+{
+	struct zone *zone = zpdesc_zone(zpdesc);
+	struct folio_batch fbatch;
+	int nr = 0;
+
+	static_assert(ZS_MAX_PAGES_PER_ZSPAGE <= FOLIO_BATCH_SIZE);
+
+	folio_batch_init(&fbatch);
+	do {
+		struct zpdesc *next = get_next_zpdesc(zpdesc);
+
+		VM_BUG_ON_PAGE(!zpdesc_is_locked(zpdesc), zpdesc_page(zpdesc));
+		reset_zpdesc(zpdesc);
+		zpdesc_unlock(zpdesc);
+
+		/* NR_ZSPAGES is per zone: one update per same-zone run */
+		if (unlikely(zpdesc_zone(zpdesc) != zone)) {
+			mod_zone_page_state(zone, NR_ZSPAGES, -nr);
+			zone = zpdesc_zone(zpdesc);
+			nr = 0;
+		}
+		nr++;
+
+		folio_batch_add(&fbatch, zpdesc_folio(zpdesc));
+		zpdesc = next;
+	} while (zpdesc);
+
+	mod_zone_page_state(zone, NR_ZSPAGES, -nr);
+	folios_put(&fbatch);
+}
+
 /*
  * Three free helpers, kept apart here:
  *
@@ -884,21 +923,22 @@ static int trylock_zspage(struct zspage *zspage)
  */
 static inline void __free_zspage_lockless(struct zspage *zspage)
 {
-	struct zpdesc *zpdesc, *next;
+	struct zpdesc *zpdesc;
 
 	VM_BUG_ON(get_zspage_inuse(zspage));
 	VM_BUG_ON(zspage->fullness != ZS_INUSE_RATIO_0);
 
-	next = zpdesc = get_first_zpdesc(zspage);
-	do {
+	zpdesc = get_first_zpdesc(zspage);
+	if (get_next_zpdesc(zpdesc)) {
+		free_page_chain(zpdesc);
+	} else {
+		/* Single-page zspage, e.g. a huge object: nothing to batch */
 		VM_BUG_ON_PAGE(!zpdesc_is_locked(zpdesc), zpdesc_page(zpdesc));
-		next = get_next_zpdesc(zpdesc);
 		reset_zpdesc(zpdesc);
 		zpdesc_unlock(zpdesc);
 		zpdesc_dec_zone_page_state(zpdesc);
 		zpdesc_put(zpdesc);
-		zpdesc = next;
-	} while (zpdesc != NULL);
+	}
 
 	cache_free_zspage(zspage);
 }
-- 
2.56.1.750.gf70dc682cf-goog



  reply	other threads:[~2026-11-20  7:48 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-10-16 12:09 [RFC PATCH v4 0/2] mm/zsmalloc: defer object free to per-cpu batches Hongfeng Chen
2026-10-16 12:09 ` [RFC PATCH v4 1/2] mm/zsmalloc: defer object free to per-cpu batches and a drain worker Hongfeng Chen
2026-10-16 16:09 ` [RFC PATCH v4 0/2] mm/zsmalloc: defer object free to per-cpu batches Nhat Pham
2026-10-22  2:55   ` Sergey Senozhatsky
2026-10-23 12:37   ` Hongfeng Chen
2026-10-23 18:19     ` Nhat Pham
2026-11-09 10:59       ` Hongfeng Chen
2026-11-19  5:35         ` Sergey Senozhatsky
2026-11-20  1:44           ` Sergey Senozhatsky
2026-11-20  7:48             ` Sergey Senozhatsky [this message]
2026-10-17  3:15 ` [RFC PATCH v4 2/2] zram: keep the hot statistics per cpu Hongfeng Chen

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=av_4KLI35RzCQ7zC@google.com \
    --to=senozhatsky@chromium.org \
    --cc=akpm@linux-foundation.org \
    --cc=axboe@kernel.dk \
    --cc=baohua@kernel.org \
    --cc=chengming.zhou@linux.dev \
    --cc=chenhongfeng0212@163.com \
    --cc=chenhongfeng0212@gmail.com \
    --cc=hannes@cmpxchg.org \
    --cc=haowenchao22@gmail.com \
    --cc=haowenchao@xiaomi.com \
    --cc=hongfengchen@xiaomi.com \
    --cc=linux-block@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=minchan@kernel.org \
    --cc=nphamcs@gmail.com \
    --cc=shakeel.butt@linux.dev \
    --cc=yosry@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.