public inbox for linux-btrfs@vger.kernel.org
 help / color / mirror / Atom feed
* Sharing my faster compsize implementation
@ 2025-04-24 12:45 Sun YangKai
  0 siblings, 0 replies; only message in thread
From: Sun YangKai @ 2025-04-24 12:45 UTC (permalink / raw)
  To: linux-btrfs; +Cc: sunk67188

I love compsize a lot. However, It is a little slow for me. So I'm trying to 
write a faster compsize implementation, just for my use case, which gets a 
great speedup by both reducing syscalls and multi-threading.

I'm sharing this in case it might be of interest to others in the community. 
All feedback and questions are welcome. The project is on
https://github.com/SaltyKitkat/xsz

And followings are the performance comparison on my machine.

On a SATA SSD device, with some snapshots for backing up, and some large media 
files.
Cache is cleared before each run.
```
$ sudo time ./xsz -j1 /mnt/1
Processed 5663444 files, 1097233 regular extents (3859780 refs), 3459956 
inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL       95%      827G         865G         1.3T
none       100%      811G         811G         1.1T
zstd        29%       15G          54G         192G
4.27user 37.46system 1:14.13elapsed 56%CPU (0avgtext+0avgdata 
31420maxresident)k
5060832inputs+0outputs (0major+46434minor)pagefaults 0swaps

$ sudo time ./xsz -j4 /mnt/1
Processed 5663444 files, 1097233 regular extents (3859780 refs), 3459956 
inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL       95%      827G         865G         1.3T
none       100%      811G         811G         1.1T
zstd        29%       15G          54G         192G
4.89user 37.88system 0:20.42elapsed 209%CPU (0avgtext+0avgdata 
35820maxresident)k
5060832inputs+0outputs (0major+79430minor)pagefaults 0swaps

$ sudo time compsize /mnt/1
Processed 5663444 files, 1097233 regular extents (3859780 refs), 3459956 
inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL       95%      827G         865G         1.3T
none       100%      811G         811G         1.1T
zstd        29%       15G          54G         192G
3.72user 72.98system 1:50.07elapsed 69%CPU (0avgtext+0avgdata 
80132maxresident)k
5254008inputs+0outputs (1major+24967minor)pagefaults 0swaps
```

On a HDD device, with a lot of small files, and some program files.

Cache is cleared before each run.
```
$ sudo time ./xsz -j1 /mnt/guest
Processed 393486 files, 215207 regular extents (215207 refs), 178308 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL      100%       29G          29G          29G
none       100%       29G          29G          29G
0.37user 3.47system 1:07.54elapsed 5%CPU (0avgtext+0avgdata 5436maxresident)k
830912inputs+0outputs (0major+7825minor)pagefaults 0swaps

$ sudo time ./xsz -j4 /mnt/guest
Processed 393486 files, 215207 regular extents (215207 refs), 178308 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL      100%       29G          29G          29G
none       100%       29G          29G          29G
0.40user 3.97system 0:57.72elapsed 7%CPU (0avgtext+0avgdata 8064maxresident)k
830912inputs+0outputs (0major+5093minor)pagefaults 0swaps

$ sudo time compsize /mnt/guest
Processed 393486 files, 215207 regular extents (215207 refs), 178308 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL      100%       29G          29G          29G
none       100%       29G          29G          29G
0.42user 9.45system 2:49.78elapsed 5%CPU (0avgtext+0avgdata 12648maxresident)k
954080inputs+0outputs (0major+2990minor)pagefaults 0swaps
```



^ permalink raw reply	[flat|nested] only message in thread

only message in thread, other threads:[~2025-04-24 12:45 UTC | newest]

Thread overview: (only message) (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-04-24 12:45 Sharing my faster compsize implementation Sun YangKai

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox