* compsize reports that filesystem uses zlib compression, while I set zstd compression everywhere @ 2022-10-26 10:21 Nemcev Aleksey 2022-10-26 19:07 ` Chris Murphy 0 siblings, 1 reply; 6+ messages in thread From: Nemcev Aleksey @ 2022-10-26 10:21 UTC (permalink / raw) To: linux-btrfs Hello. Recently I decided to compress the existing Btrfs volume with zstd. To do this, I set property `compression=zstd` for all subvolumes to compress new files with the default zstd:3 compression level. Then I run `btrfs filesystem defragment -c zstd:12 -v -r` on all subvolumes to compress existing files using zstd:12 compression level (to spend more time now and save more space later, but don't want to keep slow zstd:12 level all the time). After defragment, I run `btrfs filesystem balance` on all subvolumes to make Btrfs happy. And after all, I run `compsize` on the root subvolume to check the compression ratio, and compsize shows me the following: Processed 1100578 files, 1393995 regular extents (1649430 refs), 666312 inline. Type Perc Disk Usage Uncompressed Referenced TOTAL 77% 169G 217G 226G none 100% 99G 99G 100G zlib 52% 54G 102G 109G zstd 19% 12M 65M 66M prealloc 100% 15G 15G 16G I'm very confused why almost all compressed data is compressed with zlib while I haven't used zlib at any step. Why do compsize reports this? Should I worry about this? It seems zstd offers a much faster decompression speed than zlib. Thank you. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: compsize reports that filesystem uses zlib compression, while I set zstd compression everywhere 2022-10-26 10:21 compsize reports that filesystem uses zlib compression, while I set zstd compression everywhere Nemcev Aleksey @ 2022-10-26 19:07 ` Chris Murphy 2022-10-26 22:29 ` Nemcev Aleksey 0 siblings, 1 reply; 6+ messages in thread From: Chris Murphy @ 2022-10-26 19:07 UTC (permalink / raw) To: Nemcev Aleksey, Btrfs BTRFS On Wed, Oct 26, 2022, at 6:21 AM, Nemcev Aleksey wrote: > Hello. > > Recently I decided to compress the existing Btrfs volume with zstd. > > To do this, I set property `compression=zstd` for all subvolumes to > compress new files with the default zstd:3 compression level. > > Then I run `btrfs filesystem defragment -c zstd:12 -v -r` on all > subvolumes to compress existing files using zstd:12 compression level > (to spend more time now and save more space later, but don't want to > keep slow zstd:12 level all the time). > > After defragment, I run `btrfs filesystem balance` on all subvolumes to > make Btrfs happy. > > And after all, I run `compsize` on the root subvolume to check the > compression ratio, and compsize shows me the following: > Processed 1100578 files, 1393995 regular extents (1649430 refs), 666312 > inline. > Type Perc Disk Usage Uncompressed Referenced > TOTAL 77% 169G 217G 226G > none 100% 99G 99G 100G > zlib 52% 54G 102G 109G > zstd 19% 12M 65M 66M > prealloc 100% 15G 15G 16G > > I'm very confused why almost all compressed data is compressed with zlib > while I haven't used zlib at any step. > Why do compsize reports this? > Should I worry about this? It seems zstd offers a much faster > decompression speed than zlib. > Thank you. If you use chattr +c anywhere, it uses the btrfs default compression algo which is zlib. There is a way to set a compression algo property, but I'm uncertain if the kernel honors it. So you'll want to recursively remove the compression file attribute. I'm not sure it's worth recompressing but you can use defrag -z for that. -- Chris Murphy ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: compsize reports that filesystem uses zlib compression, while I set zstd compression everywhere 2022-10-26 19:07 ` Chris Murphy @ 2022-10-26 22:29 ` Nemcev Aleksey 2022-10-27 1:03 ` Zygo Blaxell 0 siblings, 1 reply; 6+ messages in thread From: Nemcev Aleksey @ 2022-10-26 22:29 UTC (permalink / raw) To: Chris Murphy, Btrfs BTRFS 26.10.2022 22:07, Chris Murphy пишет: > > On Wed, Oct 26, 2022, at 6:21 AM, Nemcev Aleksey wrote: >> Hello. >> >> Recently I decided to compress the existing Btrfs volume with zstd. >> >> To do this, I set property `compression=zstd` for all subvolumes to >> compress new files with the default zstd:3 compression level. >> >> Then I run `btrfs filesystem defragment -c zstd:12 -v -r` on all >> subvolumes to compress existing files using zstd:12 compression level >> (to spend more time now and save more space later, but don't want to >> keep slow zstd:12 level all the time). >> >> After defragment, I run `btrfs filesystem balance` on all subvolumes to >> make Btrfs happy. >> >> And after all, I run `compsize` on the root subvolume to check the >> compression ratio, and compsize shows me the following: >> Processed 1100578 files, 1393995 regular extents (1649430 refs), 666312 >> inline. >> Type Perc Disk Usage Uncompressed Referenced >> TOTAL 77% 169G 217G 226G >> none 100% 99G 99G 100G >> zlib 52% 54G 102G 109G >> zstd 19% 12M 65M 66M >> prealloc 100% 15G 15G 16G >> >> I'm very confused why almost all compressed data is compressed with zlib >> while I haven't used zlib at any step. >> Why do compsize reports this? >> Should I worry about this? It seems zstd offers a much faster >> decompression speed than zlib. >> Thank you. > > If you use chattr +c anywhere, it uses the btrfs default compression algo which is zlib. There is a way to set a compression algo property, but I'm uncertain if the kernel honors it. > > So you'll want to recursively remove the compression file attribute. I'm not sure it's worth recompressing but you can use defrag -z for that. > Thank you for the idea, I understood where I failed. `btrfs filesystem defragment -c zstd -r /` is not correct command, but `btrfs filesystem defragment -czstd -r /` is correct one (note the space). And the first command will try to defragment the file named zstd using default compression (zlib), not the compression from the `compression` subvolume property. And also it's not possible to specify compression level in defragment - so, I'll change the level in subvolume properties, defragment with -czstd, and change the level back. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: compsize reports that filesystem uses zlib compression, while I set zstd compression everywhere 2022-10-26 22:29 ` Nemcev Aleksey @ 2022-10-27 1:03 ` Zygo Blaxell 2022-10-27 13:21 ` Chris Murphy 0 siblings, 1 reply; 6+ messages in thread From: Zygo Blaxell @ 2022-10-27 1:03 UTC (permalink / raw) To: Nemcev Aleksey; +Cc: Chris Murphy, Btrfs BTRFS On Thu, Oct 27, 2022 at 01:29:35AM +0300, Nemcev Aleksey wrote: > 26.10.2022 22:07, Chris Murphy пишет: > > > > On Wed, Oct 26, 2022, at 6:21 AM, Nemcev Aleksey wrote: > > > Hello. > > > > > > Recently I decided to compress the existing Btrfs volume with zstd. > > > > > > To do this, I set property `compression=zstd` for all subvolumes to > > > compress new files with the default zstd:3 compression level. > > > > > > Then I run `btrfs filesystem defragment -c zstd:12 -v -r` on all > > > subvolumes to compress existing files using zstd:12 compression level > > > (to spend more time now and save more space later, but don't want to > > > keep slow zstd:12 level all the time). > > > > > > After defragment, I run `btrfs filesystem balance` on all subvolumes to > > > make Btrfs happy. > > > > > > And after all, I run `compsize` on the root subvolume to check the > > > compression ratio, and compsize shows me the following: > > > Processed 1100578 files, 1393995 regular extents (1649430 refs), 666312 > > > inline. > > > Type Perc Disk Usage Uncompressed Referenced > > > TOTAL 77% 169G 217G 226G > > > none 100% 99G 99G 100G > > > zlib 52% 54G 102G 109G > > > zstd 19% 12M 65M 66M > > > prealloc 100% 15G 15G 16G > > > > > > I'm very confused why almost all compressed data is compressed with zlib > > > while I haven't used zlib at any step. > > > Why do compsize reports this? > > > Should I worry about this? It seems zstd offers a much faster > > > decompression speed than zlib. > > > Thank you. > > > > If you use chattr +c anywhere, it uses the btrfs default compression algo which is zlib. There is a way to set a compression algo property, but I'm uncertain if the kernel honors it. > > > > So you'll want to recursively remove the compression file attribute. I'm not sure it's worth recompressing but you can use defrag -z for that. > > > Thank you for the idea, I understood where I failed. > > `btrfs filesystem defragment -c zstd -r /` is not correct command, but > `btrfs filesystem defragment -czstd -r /` is correct one (note the space). > > And the first command will try to defragment the file named zstd using > default compression (zlib), not the compression from the `compression` > subvolume property. > > And also it's not possible to specify compression level in defragment - so, > I'll change the level in subvolume properties, defragment with -czstd, and > change the level back. It's not possible to set the level in subvol properties either. Only the compress= mount option can set the level. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: compsize reports that filesystem uses zlib compression, while I set zstd compression everywhere 2022-10-27 1:03 ` Zygo Blaxell @ 2022-10-27 13:21 ` Chris Murphy 2022-10-27 14:32 ` Zygo Blaxell 0 siblings, 1 reply; 6+ messages in thread From: Chris Murphy @ 2022-10-27 13:21 UTC (permalink / raw) To: Zygo Blaxell, Nemcev Aleksey; +Cc: Btrfs BTRFS On Wed, Oct 26, 2022, at 9:03 PM, Zygo Blaxell wrote: >> And also it's not possible to specify compression level in defragment - so, >> I'll change the level in subvolume properties, defragment with -czstd, and >> change the level back. > > It's not possible to set the level in subvol properties either. Only the > compress= mount option can set the level. $ sudo btrfs property set test compression zstd:1 $ sudo btrfs property get test compression=zstd:1 $ sudo btrfs property set test compression zstd:f $ sudo btrfs property get test compression=zstd:f It's allowing the arbitrary setting of compression values. This is with btrfs-progs 6.0. Progs is being too permissive, it shouldn't silently set invalid values. But then even with valid values, I'm pretty sure right now the kernel ignores the compression properties. -- Chris Murphy ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: compsize reports that filesystem uses zlib compression, while I set zstd compression everywhere 2022-10-27 13:21 ` Chris Murphy @ 2022-10-27 14:32 ` Zygo Blaxell 0 siblings, 0 replies; 6+ messages in thread From: Zygo Blaxell @ 2022-10-27 14:32 UTC (permalink / raw) To: Chris Murphy; +Cc: Nemcev Aleksey, Btrfs BTRFS On Thu, Oct 27, 2022 at 09:21:53AM -0400, Chris Murphy wrote: > > > On Wed, Oct 26, 2022, at 9:03 PM, Zygo Blaxell wrote: > > >> And also it's not possible to specify compression level in defragment - so, > >> I'll change the level in subvolume properties, defragment with -czstd, and > >> change the level back. > > > > It's not possible to set the level in subvol properties either. Only the > > compress= mount option can set the level. > > $ sudo btrfs property set test compression zstd:1 > $ sudo btrfs property get test > compression=zstd:1 > $ sudo btrfs property set test compression zstd:f > $ sudo btrfs property get test > compression=zstd:f > > It's allowing the arbitrary setting of compression values. This is with > btrfs-progs 6.0. Progs is being too permissive, it shouldn't silently > set invalid values. But then even with valid values, I'm pretty sure > right now the kernel ignores the compression properties. Correct. You can put whatever you want in the string, but the kernel only ever looks at fs_info for compress_level, which is set from the mount options. e.g. if you mount with 'compress=zlib:5' and you have a file with btrfs.compression property with 'zstd:7', what you get is zstd compression with level 5. > -- > Chris Murphy > ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2022-10-27 14:32 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2022-10-26 10:21 compsize reports that filesystem uses zlib compression, while I set zstd compression everywhere Nemcev Aleksey 2022-10-26 19:07 ` Chris Murphy 2022-10-26 22:29 ` Nemcev Aleksey 2022-10-27 1:03 ` Zygo Blaxell 2022-10-27 13:21 ` Chris Murphy 2022-10-27 14:32 ` Zygo Blaxell
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox