Re: zstd compression

["Austin S. Hemmelgarn" <ahferroin7@gmail.com>]

On 2017-11-16 08:43, Duncan wrote:
> Austin S. Hemmelgarn posted on Thu, 16 Nov 2017 07:30:47 -0500 as
> excerpted:
> 
>> On 2017-11-15 16:31, Duncan wrote:
>>> Austin S. Hemmelgarn posted on Wed, 15 Nov 2017 07:57:06 -0500 as
>>> excerpted:
>>>
>>>> The 'compress' and 'compress-force' mount options only impact newly
>>>> written data.  The compression used is stored with the metadata for
>>>> the extents themselves, so any existing data on the volume will be
>>>> read just fine with whatever compression method it was written with,
>>>> while new data will be written with the specified compression method.
>>>>
>>>> If you want to convert existing files, you can use the '-c' option to
>>>> the defrag command to do so.
>>>
>>> ... Being aware of course that using defrag to recompress files like
>>> that will break 100% of the existing reflinks, effectively (near)
>>> doubling data usage if the files are snapshotted, since the snapshot
>>> will now share 0% of its extents with the newly compressed files.
>> Good point, I forgot to mention that.
>>>
>>> (The actual effect shouldn't be quite that bad, as some files are
>>> likely to be uncompressed due to not compressing well, and I'm not sure
>>> if defrag -c rewrites them or not.  Further, if there's multiple
>>> snapshots data usage should only double with respect to the latest one,
>>> the data delta between it and previous snapshots won't be doubled as
>>> well.)
>> I'm pretty sure defrag is equivalent to 'compress-force', not
>> 'compress', but I may be wrong.
> 
> But... compress-force doesn't actually force compression _all_ the time.
> Rather, it forces btrfs to continue checking whether compression is worth
> it for each "block"[1] of the file, instead of giving up if the first
> quick try at the beginning says that block won't compress.
> 
> So what I'm saying is that if the snapshotted data is already compressed,
> think (pre-)compressed tarballs or image files such as jpeg that are
> unlikely to /easily/ compress further and might well actually be _bigger_
> once the compression algorithm is run over them, defrag -c will likely
> fail to compress them further even if it's the equivalent of compress-
> force, and thus /should/ leave them as-is, not breaking the reflinks of
> the snapshots and thus not doubling the data usage for that file, or more
> exactly, that extent of that file.
> 
> Tho come to think of it, is defrag -c that smart, to actually leave the
> data as-is if it doesn't compress further, or does it still rewrite it
> even if it doesn't compress, thus breaking the reflink and doubling the
> usage regardless?
I'm not certain how compression factors in, but if you aren't 
compressing the file, it will only get rewritten if it's fragmented 
(which is shy defragmenting the system root directory is usually 
insanely fast on most systems, stuff there is almost never fragmented).
> 
> ---
> [1] Block:  I'm not positive it's the usual 4K block in this case.  I
> think I read that it's 16K, but I might be confused on that.  But
> regardless the size, the point is, with compress-force btrfs won't give
> up like simple compress will if the first "block" doesn't compress, it'll
> keep trying.
> 
> Of course the new compression heuristic changes this a bit too, but the
> same general idea holds, compress-force continues to try for the entire
> file, compress will give up much faster.
I'm not actually sure, I would think it checks 128k blocks of data (the 
effective block size for compression), but if it doesn't it should be 
checking at the filesystem block size (which means 16k on most recently 
created filesystems).