All of lore.kernel.org
 help / color / mirror / Atom feed
From: Nithurshen <nithurshen.dev@gmail.com>
To: nithurshen.dev@gmail.com
Cc: hsiangkao@linux.alibaba.com, linux-erofs@lists.ozlabs.org,
	xiang@kernel.org
Subject: [PATCH v2 2/2] fsck.erofs: implement algorithm-aware pcluster batching
Date: Mon,  8 Jun 2026 10:37:11 +0530	[thread overview]
Message-ID: <20260608050711.30648-3-nithurshen.dev@gmail.com> (raw)
In-Reply-To: <20260608050711.30648-1-nithurshen.dev@gmail.com>

While static batching successfully overlaps I/O and compute, different
compression algorithms exhibit vastly different scheduling thresholds.
Extremely fast algorithms like LZ4 require large batches (e.g., 32
pclusters) to effectively hide the synchronization overhead of the
thread pool.

Conversely, applying this large batch size to compute-heavy algorithms
like LZMA or ZSTD causes memory bloat and thread starvation, as the
main thread spends too much time reading and accumulating memory before
waking up the background workers.

This patch modifies the workqueue submission logic in
z_erofs_read_one_data to scale the batch size based on the algorithm
format. LZ4 is permitted to utilize the Z_EROFS_PCLUSTER_MAX_BATCH_SIZE,
while other heavier algorithms trigger workqueue submission at a much
lower threshold (8 pclusters) to ensure a steady pipeline of work and a
bounded memory footprint.

Signed-off-by: Nithurshen <nithurshen.dev@gmail.com>
---
 include/erofs/internal.h |  2 +-
 lib/data.c               | 15 +++++++++------
 2 files changed, 10 insertions(+), 7 deletions(-)

diff --git a/include/erofs/internal.h b/include/erofs/internal.h
index 63fd3bf..ffb0adb 100644
--- a/include/erofs/internal.h
+++ b/include/erofs/internal.h
@@ -63,7 +63,7 @@ struct erofs_buf {
 #define erofs_pos(sbi, nr)      ((erofs_off_t)(nr) << (sbi)->blkszbits)
 #define BLK_ROUND_UP(sbi, addr)	\
 	(roundup(addr, erofs_blksiz(sbi)) >> (sbi)->blkszbits)
-#define Z_EROFS_PCLUSTER_BATCH_SIZE 32
+#define Z_EROFS_PCLUSTER_MAX_BATCH_SIZE 32
 
 struct erofs_buffer_head;
 struct erofs_bufmgr;
diff --git a/lib/data.c b/lib/data.c
index 26fdb43..cb882c5 100644
--- a/lib/data.c
+++ b/lib/data.c
@@ -17,11 +17,11 @@ struct erofs_workqueue erofs_wq;
 struct z_erofs_decompress_task {
 	struct erofs_work work;
 	struct z_erofs_read_ctx *ctx;
-	struct z_erofs_decompress_req reqs[Z_EROFS_PCLUSTER_BATCH_SIZE];
-	char *raw_bufs[Z_EROFS_PCLUSTER_BATCH_SIZE];
-	char *out_bufs[Z_EROFS_PCLUSTER_BATCH_SIZE];
-	erofs_off_t out_offsets[Z_EROFS_PCLUSTER_BATCH_SIZE];
-	unsigned int out_lengths[Z_EROFS_PCLUSTER_BATCH_SIZE];
+	struct z_erofs_decompress_req reqs[Z_EROFS_PCLUSTER_MAX_BATCH_SIZE];
+	char *raw_bufs[Z_EROFS_PCLUSTER_MAX_BATCH_SIZE];
+	char *out_bufs[Z_EROFS_PCLUSTER_MAX_BATCH_SIZE];
+	erofs_off_t out_offsets[Z_EROFS_PCLUSTER_MAX_BATCH_SIZE];
+	unsigned int out_lengths[Z_EROFS_PCLUSTER_MAX_BATCH_SIZE];
 	unsigned int nr_reqs;
 };
 
@@ -401,7 +401,10 @@ int z_erofs_read_one_data(struct erofs_inode *inode,
 	task->out_offsets[idx] = out_offset;
 	task->out_lengths[idx] = length;
 
-	if (task->nr_reqs == Z_EROFS_PCLUSTER_BATCH_SIZE) {
+	int batch_limit = (map->m_algorithmformat == Z_EROFS_COMPRESSION_LZ4) ? 
+						Z_EROFS_PCLUSTER_MAX_BATCH_SIZE : 8;
+
+	if (task->nr_reqs >= batch_limit) {
 		z_erofs_read_ctx_enqueue(ctx);
 	}
 	return 0;
-- 
2.52.0



      parent reply	other threads:[~2026-06-08  5:07 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-05-23  0:37 [PATCH 0/2] fsck.erofs: introduce multi-threaded decompression Nithurshen
2026-05-23  0:37 ` [PATCH 1/2] fsck.erofs: introduce multi-threaded decompression with static batching Nithurshen
2026-06-07  1:50   ` Gao Xiang
2026-05-23  0:37 ` [PATCH 2/2] fsck.erofs: implement dynamic pcluster batching based on algorithm complexity Nithurshen
2026-06-07  1:52   ` Gao Xiang
2026-06-08  5:07 ` [PATCH v2 0/2] fsck.erofs: add multi-threaded decompression Nithurshen
2026-06-08  5:07   ` [PATCH v2 1/2] fsck.erofs: introduce multi-threaded decompression with static batching Nithurshen
2026-06-08  6:25     ` Gao Xiang
2026-06-08  5:07   ` Nithurshen [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20260608050711.30648-3-nithurshen.dev@gmail.com \
    --to=nithurshen.dev@gmail.com \
    --cc=hsiangkao@linux.alibaba.com \
    --cc=linux-erofs@lists.ozlabs.org \
    --cc=xiang@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.