Re: [PATCH 1/2 v3] fsck.erofs: add multi-threaded decompression

[Gao Xiang <hsiangkao@linux.alibaba.com>]

On 2026/7/6 14:10, Nithurshen wrote:
> Hi Xiang,
> 
> Please find the updated multi-threaded decompression implementation for
> fsck.erofs. This version introduces an algorithm-aware asynchronous
> worker pool, dynamically sized based on system CPUs, to significantly
> accelerate the extraction of computationally expensive images.
> 
> Benchmarks were performed on an ARM64 environment, extracting the 8.2 GB
> Linux repository downloaded from Github.
> 
> Extraction Time (Seconds):
> 
> | Algorithm | 4k | 8k | 16k | 32k | 64k |
> | --- | --- | --- | --- | --- | --- |
> | lz4hc(MT) | 9.36 | 8.31 | 8.27 | 8.54 | 6.94 |
> | lz4hc(ST) | 4.40 | 5.77 | 3.65 | 5.45 | 3.36 |
> | zstd(MT) | 9.26 | 8.68 | 8.89 | 8.11 | 7.94 |
> | zstd(ST) | 5.23 | 4.52 | 4.62 | 4.11 | 4.02 |
> | lzma(MT) | 23.72 | 24.79 | 25.90 | 26.92 | 27.77 |
> | lzma(ST) | 56.37 | 65.06 | 71.07 | 74.69 | 81.63 |
> 
> Performance Analysis:
> The implementation provides a significant speedup for LZMA (up to 2.9x)
> as the heavy decompression workload effectively amortizes the thread
> synchronization overhead.
> 
> However, for fast algorithms like LZ4 and ZSTD, the current MT overhead
> (futex contention and scheduling) leads to slower performance compared
> to the synchronous baseline.
> 
> I have tried various batch sizes for fast algorithms, but the time did
> not improve. (I tried from 32 to 256 batch sizes)
> 
> Can we fall back to synchronous extraction here?

Sorry for delay.

We cannot, that is why we need to find a proper way to batch
the pclusters and reschedule.

And that is why I asked you to benchmark each commit.

Thanks,
Gao Xiang

> 
> Verification:
> 
> * Deadlocks/Race Conditions: Verified via GDB backtrace analysis and
> stress testing.
> * Memory Leaks: Verified via rigorous buffer ownership tracking and
> ensuring all task resources are cleaned up upon worker completion.
> * Integrity: All configurations passed bit-for-bit integrity checks
> between the extracted and original directory for all algorithms and
> chunk sizes.
> 
> All concurrency primitives and memory paths have been verified to the
> best of my knowledge.
> 
> Thanks,
> Nithurshen