All of lore.kernel.org
 help / color / mirror / Atom feed
From: Gao Xiang <xiang@kernel.org>
To: Nithurshen <nithurshen.dev@gmail.com>
Cc: linux-erofs@lists.ozlabs.org, hsiangkao@linux.alibaba.com,
	xiang@kernel.org
Subject: Re: [PATCH 2/2] fsck.erofs: implement concurrent directory traversal
Date: Sun, 5 Jul 2026 22:05:16 +0800	[thread overview]
Message-ID: <akpknJC1zDTfE1I6@debian> (raw)
In-Reply-To: <20260621120121.73114-3-nithurshen.dev@gmail.com>

On Sun, Jun 21, 2026 at 05:31:21PM +0530, Nithurshen wrote:
> Currently, fsck.erofs traverses the filesystem tree and verifies
> inodes synchronously on the main thread. While decompression
> compute is offloaded, the main thread remains a bottleneck
> during the I/O-heavy directory walk.
> 
> This patch parallelizes the directory traversal and inode
> extraction processes. To achieve this safely, the globally shared
> fsckcfg.extract_path and fsckcfg.dirstack states are decoupled
> and localized into individual struct erofsfsck_inode_task
> payloads, which are deep-copied and handed off to the worker
> pool. Global statistics and hardlink tables are secured using
> native erofs_mutex_t primitives.
> 
> To prevent thread pool exhaustion deadlocks—where workers
> processing a deep directory tree occupy all available execution
> slots and block on erofs_cond_wait, starving their own spawned
> decompression tasks—this patch introduces a dedicated
> erofs_traverse_wq. By isolating the producers (traversal and
> verification) from the consumers (pcluster decompression), the
> pipeline avoids gridlock.
> 
> Signed-off-by: Nithurshen <nithurshen.dev@gmail.com>

I really hope you could parallelize the inodes in a single directory
first, and get a minimal commit and show the improvement.

And try to improve parallelization between directories.

The priciple is to keep each patch small so that it's more reviewable
and has low risky.

Also it seems that you could make mutex protection in seperate commits
too so it's also easier to review.

Thanks,
Gao Xiang 


  reply	other threads:[~2026-07-05 14:05 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-06-21 12:01 [PATCH 0/2] fsck.erofs: implement multi-threaded extraction Nithurshen
2026-06-21 12:01 ` [PATCH 1/2] fsck.erofs: add multi-threaded decompression Nithurshen
2026-06-22  2:08   ` Gao Xiang
2026-06-29  9:55   ` [PATCH 1/2 v2] " Nithurshen
2026-07-05 14:00     ` Gao Xiang
2026-07-06  6:05   ` [PATCH 1/2 v3] " Nithurshen
2026-07-06  6:10     ` Nithurshen
2026-07-08  2:31       ` Gao Xiang
2026-07-13  7:02     ` [PATCH 1/2 v4] " Nithurshen
2026-06-21 12:01 ` [PATCH 2/2] fsck.erofs: implement concurrent directory traversal Nithurshen
2026-07-05 14:05   ` Gao Xiang [this message]
2026-07-14  1:10   ` [PATCH v5] fsck.erofs: add multi-threaded decompression Nithurshen
2026-07-14  1:18     ` Nithurshen Karthikeyan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=akpknJC1zDTfE1I6@debian \
    --to=xiang@kernel.org \
    --cc=hsiangkao@linux.alibaba.com \
    --cc=linux-erofs@lists.ozlabs.org \
    --cc=nithurshen.dev@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.