Btrfs compression ratio > 34:1 possible?

Btrfs compression ratio > 34:1 possible?

[Ole Tange]

I am trying to use Btrfs' compression on some highly compressible data.

I try to make zstd give better performance than 34:1.

This:

   $ truncate -s 1T /tmp/btrfs
   $ mkfs.btrfs /tmp/btrfs
   $ mount -o compress=zstd:9  /tmp/btrfs /mnt/btrfs
   $ head -c 10G /dev/zero > /mnt/btrfs/zero
   $ du /tmp/btrfs
   313672k

shows a compression ratio of 10737418240/312724/1024 = 33.5:1

But if I run:

   $ head -c 10G /dev/zero | zstd | wc -c
   336655

I get a compression ratio of 10737418240/336655 = 31894:1

That is around 1000 times better.

I understand there is some overhead in Btrfs, so it is expected that
it is not possible to reach the full ratio. But it seems there is
little to no difference in using 'compress=zstd:9' and
'compress=zstd:3' on highly compressible data.

My guess is, that data is chopped up in small blocks (1k?) that are
each compressed. If so: Is it possible to make these blocks bigger? I
think that would make sense in general when using higher compression
values.


/Ole

Re: Btrfs compression ratio > 34:1 possible?

[Qu Wenruo]

On 2021/1/12 上午7:32, Ole Tange wrote:
> I am trying to use Btrfs' compression on some highly compressible data.
>
> I try to make zstd give better performance than 34:1.
>
> This:
>
>     $ truncate -s 1T /tmp/btrfs
>     $ mkfs.btrfs /tmp/btrfs
>     $ mount -o compress=zstd:9  /tmp/btrfs /mnt/btrfs
>     $ head -c 10G /dev/zero > /mnt/btrfs/zero
>     $ du /tmp/btrfs
>     313672k
>
> shows a compression ratio of 10737418240/312724/1024 = 33.5:1
>
> But if I run:
>
>     $ head -c 10G /dev/zero | zstd | wc -c
>     336655
>
> I get a compression ratio of 10737418240/336655 = 31894:1
>
> That is around 1000 times better.
>
> I understand there is some overhead in Btrfs, so it is expected that
> it is not possible to reach the full ratio. But it seems there is
> little to no difference in using 'compress=zstd:9' and
> 'compress=zstd:3' on highly compressible data.
>
> My guess is, that data is chopped up in small blocks (1k?) that are
> each compressed. If so: Is it possible to make these blocks bigger? I
> think that would make sense in general when using higher compression
> values.

For compressed data, btrfs has a size limit for data extent, which is
128K. The number is to balance between compression ratio and extra
decompression for CoWed extents.

On the other hand, btrfs (any fs) has its minimal block size, and it's
4K for x86_64.

So the upper limit you can get is 128K / 4K = 32.

Thanks,
Qu

>
>
> /Ole
>