mount compress=zstd leaves files uncompressed, that used to compress well with before

mount compress=zstd leaves files uncompressed, that used to compress well with before

[Dimitrios Apostolou]

Hello list,

has something changed lately in how btrfs discovers if a file is
compressible?

I am moving a database (pgdump|pgrestore) from and to a compressed
(compress=zstd:3) filesystem.  And while on the old host (kernel 5.15) all
the database files were highly compressed with almost 8:1 ratio, while
doing pg_restore on the new host (kernel 6.11 with newly created btrfs
filesystem) I notice (using compsize) that most files are being written
uncompressed.

I have to issue a defragment -czstd command to fix the situation, and I'm
contemplating whether I should change the mount option to compress-force.

Thanks in advance,
Dimitris

Re: mount compress=zstd leaves files uncompressed, that used to compress well with before

[Dimitrios Apostolou]

I keep seeing the same odd behaviour. The database files are not being
compressed despite having compress=zstd:3 in the mount options. When I
issue a defragment -czstd command, everything gets compacted very well.
And then as files are being modified by the database, uncompressed extents
start appearing again.

This did not happen before. So has something changed between kernels 5.15
and 6.11 regarding how btrfs detects if a file is compressible?

How can I debug this further?

Thanks,
Dimitris




On Wed, 19 Mar 2025, Dimitrios Apostolou wrote:

> Hello list,
>
> has something changed lately in how btrfs discovers if a file is
> compressible?
>
> I am moving a database (pgdump|pgrestore) from and to a compressed
> (compress=zstd:3) filesystem.  And while on the old host (kernel 5.15) all
> the database files were highly compressed with almost 8:1 ratio, while doing
> pg_restore on the new host (kernel 6.11 with newly created btrfs filesystem)
> I notice (using compsize) that most files are being written uncompressed.
>
> I have to issue a defragment -czstd command to fix the situation, and I'm
> contemplating whether I should change the mount option to compress-force.
>
> Thanks in advance,
> Dimitris
>

Re: mount compress=zstd leaves files uncompressed, that used to compress well with before

[Gerhard Wiesinger]

Hello Dimitris,

It's a known bug I also ran into, see the disucssion here:
https://lore.kernel.org/all/b7995589-35a4-4595-baea-1dcdf1011d68@wiesinger.com/T/
(It can't be easily fixed)

There is also another issue with BTRFS:
https://lore.kernel.org/linux-bcachefs/kgdutihyy6durmrtqi5dfk7lhl2duzm4wnf6mlyneiuphf3cck@fxulfyg2ugjf/T/

I guess you always had the issue but you didn't notice it.

Therefore I moved to ZFS as a BTRFS replacement. PostgreSQL works also 
fine there.

Ciao,
Gerhard

On 24.03.2025 15:14, Dimitrios Apostolou wrote:
> I keep seeing the same odd behaviour. The database files are not being
> compressed despite having compress=zstd:3 in the mount options. When I
> issue a defragment -czstd command, everything gets compacted very well.
> And then as files are being modified by the database, uncompressed 
> extents
> start appearing again.
>
> This did not happen before. So has something changed between kernels 5.15
> and 6.11 regarding how btrfs detects if a file is compressible?
>
> How can I debug this further?
>
> Thanks,
> Dimitris
>
>
>
>
> On Wed, 19 Mar 2025, Dimitrios Apostolou wrote:
>
>> Hello list,
>>
>> has something changed lately in how btrfs discovers if a file is
>> compressible?
>>
>> I am moving a database (pgdump|pgrestore) from and to a compressed
>> (compress=zstd:3) filesystem.  And while on the old host (kernel 
>> 5.15) all
>> the database files were highly compressed with almost 8:1 ratio, 
>> while doing
>> pg_restore on the new host (kernel 6.11 with newly created btrfs 
>> filesystem)
>> I notice (using compsize) that most files are being written 
>> uncompressed.
>>
>> I have to issue a defragment -czstd command to fix the situation, and 
>> I'm
>> contemplating whether I should change the mount option to 
>> compress-force.
>>
>> Thanks in advance,
>> Dimitris
>>
>
>

Re: mount compress=zstd leaves files uncompressed, that used to compress well with before

[Dimitrios Apostolou]

On Mon, 24 Mar 2025, Gerhard Wiesinger wrote:

> Hello Dimitris,
>
> It's a known bug I also ran into, see the disucssion here:
> https://lore.kernel.org/all/b7995589-35a4-4595-baea-1dcdf1011d68@wiesinger.com/T/
> (It can't be easily fixed)

Thank you Gerhard, it seems we are facing the same issue.
It's a pity I read that force-compress does not fix the issue.

This "preallocation" mentioned in the thread, how is it achieved in the
application level? Is it with posix_fallocate()? If so, I definitely see
it happening in PostgreSQL:

https://github.com/postgres/postgres/blob/0e3e0ec06b995f6809f315752cbf5ff67902e095/src/backend/storage/smgr/md.c#L575

Relevant commit:

https://github.com/postgres/postgres/commit/4d330a61bb1969df31f2cebfe1ba9d1d004346d8

I can see this code was introduced in PostgreSQL 16.

Maybe a workaround is to recompile with -UHAVE_POSIX_FALLOCATE.

> I guess you always had the issue but you didn't notice it.

I definitely didn't have it with PostgreSQL 15 on Linux 5.15 (Ubuntu
22.04), I don't know for sure if I see it after upgraded to PostgreSQL 16,
but I see the issue with PostgreSQL 17 on Linux 6.11 (HWE kernel in Ubuntu
24.04).

I see in your message that you are on Linux 6.5, but what is your
PostgreSQL version?

> There is also another issue with BTRFS:
> https://lore.kernel.org/linux-bcachefs/kgdutihyy6durmrtqi5dfk7lhl2duzm4wnf6mlyneiuphf3cck@fxulfyg2ugjf/T/

And then there is the issue of abysmal performance for buffered read(8KB)
from compressed files:

https://www.spinics.net/lists/linux-btrfs/msg137200.html


Thanks,
Dimitris

Re: mount compress=zstd leaves files uncompressed, that used to compress well with before

[Gerhard Wiesinger]

On 24.03.2025 20:11, Dimitrios Apostolou wrote:
> On Mon, 24 Mar 2025, Gerhard Wiesinger wrote:
>
>> Hello Dimitris,
>>
>> It's a known bug I also ran into, see the disucssion here:
>> https://lore.kernel.org/all/b7995589-35a4-4595-baea-1dcdf1011d68@wiesinger.com/T/ 
>>
>> (It can't be easily fixed)
>
> Thank you Gerhard, it seems we are facing the same issue.
> It's a pity I read that force-compress does not fix the issue.
>
> This "preallocation" mentioned in the thread, how is it achieved in the
> application level? Is it with posix_fallocate()? If so, I definitely see
> it happening in PostgreSQL:
>
> https://github.com/postgres/postgres/blob/0e3e0ec06b995f6809f315752cbf5ff67902e095/src/backend/storage/smgr/md.c#L575 
>
>
> Relevant commit:
>
> https://github.com/postgres/postgres/commit/4d330a61bb1969df31f2cebfe1ba9d1d004346d8 
>
>
> I can see this code was introduced in PostgreSQL 16.
>
> Maybe a workaround is to recompile with -UHAVE_POSIX_FALLOCATE.
>
>> I guess you always had the issue but you didn't notice it.
>
> I definitely didn't have it with PostgreSQL 15 on Linux 5.15 (Ubuntu
> 22.04), I don't know for sure if I see it after upgraded to PostgreSQL 
> 16,
> but I see the issue with PostgreSQL 17 on Linux 6.11 (HWE kernel in 
> Ubuntu
> 24.04).

Yes, it occours when FALLOCATE (in any form) is used. So it looks like 
it was then introduced with PostgreSQL >= 16 and is NOT kernel dependend.

Are there compile options to disable usage of FALLOCATE at all?

>
> I see in your message that you are on Linux 6.5, but what is your
> PostgreSQL version?

Didn't document exactly used version but I was using version PostgreSQL 
16 (see directory /var/lib/pgsql/16 on the mailing list posting).

I was using the PGDG rpms from postgresql.org to get latest version. For 
support reasons with less upgrades I can recommend them anyway because 
distribution versions are often (far) behind.

>
>> There is also another issue with BTRFS:
>> https://lore.kernel.org/linux-bcachefs/kgdutihyy6durmrtqi5dfk7lhl2duzm4wnf6mlyneiuphf3cck@fxulfyg2ugjf/T/ 
>>
>
> And then there is the issue of abysmal performance for buffered read(8KB)
> from compressed files:
>
> https://www.spinics.net/lists/linux-btrfs/msg137200.html
>
>
Ah, didn't know that. Did you also try ZFS and test also regarding 
performance issues?

Ciao,

Gerhard

Re: mount compress=zstd leaves files uncompressed, that used to compress well with before

[Dimitrios Apostolou]

On Mon, 24 Mar 2025, Gerhard Wiesinger wrote:
> On 24.03.2025 20:11, Dimitrios Apostolou wrote:
>>  On Mon, 24 Mar 2025, Gerhard Wiesinger wrote:
>>
>>>  I guess you always had the issue but you didn't notice it.
>>
>>  I definitely didn't have it with PostgreSQL 15 on Linux 5.15 (Ubuntu
>>  22.04), I don't know for sure if I see it after upgraded to PostgreSQL 16,
>>  but I see the issue with PostgreSQL 17 on Linux 6.11 (HWE kernel in Ubuntu
>>  24.04).
>
> Yes, it occours when FALLOCATE (in any form) is used. So it looks like it was
> then introduced with PostgreSQL >= 16 and is NOT kernel dependend.
>
> Are there compile options to disable usage of FALLOCATE at all?
>

Probably -UHAVE_POSIX_FALLOCATE in CPPFLAGS. Haven't tried yet, so take it
with a grain of salt. :-)

I tested fallocate via its command line tool, and I see the issue in all
possible combinations:


Linux 6.11 - ext4

$ strace fallocate -v -l 1g blah-1g    # returned immediately

blah-1g: 1 GiB (1073741824 bytes) allocated.

fallocate(3, 0, 0, 1073741824)          = 0


Linux 6.11 - btrfs(compress-force=zstd:3)

# strace fallocate -v -l 1g blah-1g    # took ~10s

openat(AT_FDCWD, "blah-1g", O_RDWR|O_CREAT, 0666) = 3
fallocate(3, 0, 0, 1073741824)          = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0x9), ...}) = 0
write(1, "blah-1g: 1 GiB (1073741824 byte"..., 46blah-1g: 1 GiB
(1073741824 bytes) allocated.
) = 46

# compsize blah-1g
Processed 1 file, 4 regular extents (4 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL      100%      1.0G         1.0G         1.0G
prealloc   100%      1.0G         1.0G         1.0G


Linux 5.15 - btrfs(compress=zstd:3)

# strace fallocate -v -l 1g blah-1g    # returned immediately

openat(AT_FDCWD, "blah-1g", O_RDWR|O_CREAT, 0666) = 3
fallocate(3, 0, 0, 1073741824)          = 0
fsync(3)                                = 0
close(3)                                = 0

# compsize blah-1g
Processed 1 file, 4 regular extents (4 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL      100%      1.0G         1.0G         1.0G
prealloc   100%      1.0G         1.0G         1.0G


Both kernels (and libc, as I also tried fallocate --posix) seem to support
fallocate() on Btrfs, and btrfs always fails to compress.

Weirdly enough, I don't think I see the issue with PostgreSQL 16 on this
last case (Linux 5.15). The space growth of the database is looking like
it's well compressed. Can't tell for sure though, the compsize command I
issued more than a couple hours ago hasn't returned yet...

>>
>>  I see in your message that you are on Linux 6.5, but what is your
>>  PostgreSQL version?
>
> Didn't document exactly used version but I was using version PostgreSQL 16
> (see directory /var/lib/pgsql/16 on the mailing list posting).
>
> I was using the PGDG rpms from postgresql.org to get latest version. For
> support reasons with less upgrades I can recommend them anyway because
> distribution versions are often (far) behind.
>
>>
>>>  There is also another issue with BTRFS:
>>>  https://lore.kernel.org/linux-bcachefs/kgdutihyy6durmrtqi5dfk7lhl2duzm4wnf6mlyneiuphf3cck@fxulfyg2ugjf/T/
>>>
>>
>>  And then there is the issue of abysmal performance for buffered read(8KB)
>>  from compressed files:
>>
>>  https://www.spinics.net/lists/linux-btrfs/msg137200.html
>>
>>
> Ah, didn't know that. Did you also try ZFS and test also regarding
> performance issues?

No, haven't tested yet. I'm quite desperate though to find an
OK-performing compressed filesystem for postgreSQL, so I might look that
way.


Regards,
Dimitris

Re: mount compress=zstd leaves files uncompressed, that used to compress well with before

[Dimitrios Apostolou]

On Mon, 24 Mar 2025, Gerhard Wiesinger wrote:
>
> Are there compile options to disable usage of FALLOCATE at all?
>

I just tested, and the only thing that worked for compiling postgresql
without using fallocate, was this autoconf "hack":

   ./configure ac_cv_func_posix_fallocate=no


It would be cool if you re-tested your scenario, and verified that
postgresql files are being compressed by btrfs after it. Note that the
files would need to be re-created, so most likely you would need to create
a new database.


Regards,
Dimitris

Re: mount compress=zstd leaves files uncompressed, that used to compress well with before

[Dimitrios Apostolou]

CC'ing Qu Wenruo from this thread

On Mon, 24 Mar 2025, Gerhard Wiesinger wrote:
>
> It's a known bug I also ran into, see the disucssion here:
> https://lore.kernel.org/all/b7995589-35a4-4595-baea-1dcdf1011d68@wiesinger.com/T/
> (It can't be easily fixed)

Hi Qu, reading this thread I understand that posix_fallocate() ends up
leaving files uncompressed forever in btrfs, regardless of mount options.
I see this in PostgreSQL that recently started using posix_fallocate().

You gave some information on why the solution is hard:
> There is a long existing problem with compression with preallocation.
>
> One easy example is, if we go compression for the preallocated range,
> what we do with the gap (compressed size is always smaller than the real
> size).
>
> If we leave the gap, then the read performance can be even worse, as now
> we have to read several small extents with gaps between them, vs a large
> contig read.

Can't the solution/workaround be way more simple, or stupid even?

* Either have fallocate(2) return EOPNOTSUPP on a force-compress
   filesystem, and leave the work-around to userspace,

* or fill up the holes with compressed zeros, basically implementing the
   work-around in kernelspace. I suspect this would be very cheap in a
   deduplicating filesystem like btrfs, since all the zero-filled
   compressed extents are essentially identical.


In any case, I think this should be documented in the btrfs documentation
about compression, [1] it would have saved me time. I can try to submit a
patch if you point me to the docs repo. Any other known cases where
compression is unexpectedly skipped?

[1] https://btrfs.readthedocs.io/en/latest/Compression.html


Thank you in advance,
Dimitris

Re: mount compress=zstd leaves files uncompressed, that used to compress well with before

[Qu Wenruo]

在 2025/3/26 22:45, Dimitrios Apostolou 写道:
> CC'ing Qu Wenruo from this thread
>
> On Mon, 24 Mar 2025, Gerhard Wiesinger wrote:
>>
>> It's a known bug I also ran into, see the disucssion here:
>> https://lore.kernel.org/all/b7995589-35a4-4595-
>> baea-1dcdf1011d68@wiesinger.com/T/
>> (It can't be easily fixed)
>
> Hi Qu, reading this thread I understand that posix_fallocate() ends up
> leaving files uncompressed forever in btrfs, regardless of mount options.
> I see this in PostgreSQL that recently started using posix_fallocate().
>
> You gave some information on why the solution is hard:
>> There is a long existing problem with compression with preallocation.
>>
>> One easy example is, if we go compression for the preallocated range,
>> what we do with the gap (compressed size is always smaller than the real
>> size).
>>
>> If we leave the gap, then the read performance can be even worse, as now
>> we have to read several small extents with gaps between them, vs a large
>> contig read.
>
> Can't the solution/workaround be way more simple, or stupid even?
>
> * Either have fallocate(2) return EOPNOTSUPP on a force-compress
>    filesystem, and leave the work-around to userspace,

Unfortunately fallocate has higher priority, not vise-verse.

In most cases, compression is a good to have feature, but even with
force-compression, we can still have cases that won't be compressed.

On the other hand, all major upstream fses have support for fallocate,
and although I understand preallocation is no longer as simple as
non-COW filesystems, not supporting it would still be a big surprise to
a lot of user space tools.

Although emotionally I agree with you. Fallocation on btrfs is just
looking for extra problems, and if I have the final call, I will be more
than happier to nuke fallocation support.

>
> * or fill up the holes with compressed zeros, basically implementing the
>    work-around in kernelspace. I suspect this would be very cheap in a
>    deduplicating filesystem like btrfs, since all the zero-filled
>    compressed extents are essentially identical.

Not that easily either. Fallocate itself should mean the next write into
the fallocated range will not fail with ENOSPC.

Although that assumption itself is no longer correct on btrfs, (e.g.
fallocate, then snapshot).

But doing compressed zeros means we got nothing from the old
preallocation behavior, and still waste space on holes.

>
>
> In any case, I think this should be documented in the btrfs documentation
> about compression, [1] it would have saved me time. I can try to submit a
> patch if you point me to the docs repo. Any other known cases where
> compression is unexpectedly skipped?

The document repo is btrfs-progs:

https://github.com/kdave/btrfs-progs

You can either submit regular email patches to this ml, or submit a
github PR.

Thanks,
Qu

>
> [1] https://btrfs.readthedocs.io/en/latest/Compression.html
>
>
> Thank you in advance,
> Dimitris
>

Re: mount compress=zstd leaves files uncompressed, that used to compress well with before

[Dimitrios Apostolou]

[-- Attachment #1: Type: text/plain, Size: 3533 bytes --]

On Thu, 27 Mar 2025, Qu Wenruo wrote:
>
> 在 2025/3/26 22:45, Dimitrios Apostolou 写道:
>>
>>  Can't the solution/workaround be way more simple, or stupid even?
>>
>>  * Either have fallocate(2) return EOPNOTSUPP on a force-compress
>>     filesystem, and leave the work-around to userspace,
>
> Unfortunately fallocate has higher priority, not vise-verse.
>
> In most cases, compression is a good to have feature, but even with
> force-compression, we can still have cases that won't be compressed.

Do you know of other cases besides fallocate?

>
> On the other hand, all major upstream fses have support for fallocate,
> and although I understand preallocation is no longer as simple as
> non-COW filesystems, not supporting it would still be a big surprise to
> a lot of user space tools.

I checked what openzfs does, and here is an excerpt from the commit
message that added support for fallocate:

   Since ZFS does COW and snapshows, preallocating blocks for a file
   cannot guarantee that writes to the file will not run out of space.
   Instead, make a best-effort attempt to check that at least enough
   space is currently available in the pool (12% margin), then create
   a sparse file of the requested size and continue on with life.

The whole commit with some discussion is at [1], while a long issue
discussing alternative is at [2].

[1]  https://github.com/openzfs/zfs/pull/10408
[2]  https://github.com/openzfs/zfs/issues/326

It could be the solution for btrfs too, to just check if such space plus a
margin is available and return a sparse file. We lie to userspace about
guaranteeing that write() can't fail, but as you mentioned, we are already
lying:

> Not that easily either. Fallocate itself should mean the next write into
> the fallocated range will not fail with ENOSPC.
>
> Although that assumption itself is no longer correct on btrfs, (e.g.
> fallocate, then snapshot).

Anyway,

>
> Although emotionally I agree with you. Fallocation on btrfs is just
> looking for extra problems, and if I have the final call, I will be more
> than happier to nuke fallocation support.

From a purist's perspective I also find EOPNOTSUPP as the best solution.

* Better for the kernel: no complicated workarounds, no lies to userspace

* Better for the application: it gets to know that there are no guarantees
   on space allocation

* Better for the admin: the files get compressed as the mount options
   mandate

The only disadvantage I see is breaking the applications that don't
implement fallback code to {posix_,}fallocate() returning
EOPNOTSUPP/EINVAL.
I have to ask here, is posix_fallocate() mandated by some standard?
If not, it's an application bug.

Maybe the best tradeoff is to add a mount option fallocate=off.


>
>>
>>  * or fill up the holes with compressed zeros, basically implementing the
>>     work-around in kernelspace. I suspect this would be very cheap in a
>>     deduplicating filesystem like btrfs, since all the zero-filled
>>     compressed extents are essentially identical.
>
>
> But doing compressed zeros means we got nothing from the old
> preallocation behavior, and still waste space on holes.

I might be misunderstanding the terminology. I thought a "hole" is one
block or extent of zeros. If that's one block referenced (deduplicated)
multiple times, then there is no space wasted, right? It's just a lie:
btrfs allocated no space for the hole.


Thank you,
Dimitris

Re: mount compress=zstd leaves files uncompressed, that used to compress well with before

[Qu Wenruo]

在 2025/3/28 00:10, Dimitrios Apostolou 写道:
> On Thu, 27 Mar 2025, Qu Wenruo wrote:
>>
>> 在 2025/3/26 22:45, Dimitrios Apostolou 写道:
>>>
>>>  Can't the solution/workaround be way more simple, or stupid even?
>>>
>>>  * Either have fallocate(2) return EOPNOTSUPP on a force-compress
>>>     filesystem, and leave the work-around to userspace,
>>
>> Unfortunately fallocate has higher priority, not vise-verse.
>>
>> In most cases, compression is a good to have feature, but even with
>> force-compression, we can still have cases that won't be compressed.
>
> Do you know of other cases besides fallocate?

/dev/urandom or something similar, those kind of data will result the
compressed data to be larger than the original, and btrfs will abort
compression no matter the mount option.

>
>>
>> On the other hand, all major upstream fses have support for fallocate,
>> and although I understand preallocation is no longer as simple as
>> non-COW filesystems, not supporting it would still be a big surprise to
>> a lot of user space tools.
>
> I checked what openzfs does, and here is an excerpt from the commit
> message that added support for fallocate:
>
>    Since ZFS does COW and snapshows, preallocating blocks for a file
>    cannot guarantee that writes to the file will not run out of space.
>    Instead, make a best-effort attempt to check that at least enough
>    space is currently available in the pool (12% margin), then create
>    a sparse file of the requested size and continue on with life.
>
> The whole commit with some discussion is at [1], while a long issue
> discussing alternative is at [2].
>
> [1]  https://github.com/openzfs/zfs/pull/10408
> [2]  https://github.com/openzfs/zfs/issues/326
>
> It could be the solution for btrfs too, to just check if such space plus a
> margin is available and return a sparse file. We lie to userspace about
> guaranteeing that write() can't fail, but as you mentioned, we are already
> lying:

In that case, I'd prefer to return EOPNOTSUPP for fallocate, not even
try to emulate the behavior like ZFS.

At least we have one more fs showing how bad fallocate is on a COW fs.

>
>> Not that easily either. Fallocate itself should mean the next write into
>> the fallocated range will not fail with ENOSPC.
>>
>> Although that assumption itself is no longer correct on btrfs, (e.g.
>> fallocate, then snapshot).
>
> Anyway,
>
>>
>> Although emotionally I agree with you. Fallocation on btrfs is just
>> looking for extra problems, and if I have the final call, I will be more
>> than happier to nuke fallocation support.
>
>  From a purist's perspective I also find EOPNOTSUPP as the best solution.
>
> * Better for the kernel: no complicated workarounds, no lies to userspace
>
> * Better for the application: it gets to know that there are no guarantees
>    on space allocation
>
> * Better for the admin: the files get compressed as the mount options
>    mandate
>
> The only disadvantage I see is breaking the applications that don't
> implement fallback code to {posix_,}fallocate() returning
> EOPNOTSUPP/EINVAL.
> I have to ask here, is posix_fallocate() mandated by some standard?
> If not, it's an application bug.

Nope, it's not a hard requirement, in fact some older fses (still
supported upstream) are not supporting fallocate at all.

E.g. Ext2 doesn't support fallocate.

But suddenly dropping one feature which we originally support, is a
little concerning.

>
> Maybe the best tradeoff is to add a mount option fallocate=off.

That will be feasible.

I can try push that direction after you have updated the docs.


>
>>
>>>
>>>  * or fill up the holes with compressed zeros, basically implementing
>>> the
>>>     work-around in kernelspace. I suspect this would be very cheap in a
>>>     deduplicating filesystem like btrfs, since all the zero-filled
>>>     compressed extents are essentially identical.
>>
>>
>> But doing compressed zeros means we got nothing from the old
>> preallocation behavior, and still waste space on holes.
>
> I might be misunderstanding the terminology. I thought a "hole" is one
> block or extent of zeros. If that's one block referenced (deduplicated)
> multiple times, then there is no space wasted, right? It's just a lie:
> btrfs allocated no space for the hole.

Oh, in that case, a hole really means a hole, there is no space taken on
disk, and all the zero are just filled at read time.

Thus there is no compressed or non-compressed hole, it's really a hole,
void.

And in that case I guess you mean making fallocate fallback to hole
punching (for unallocated range).

Which is still not as good as EOPNOTSUPP IHMO.

Thanks,
Qu

>
>
> Thank you,
> Dimitris

Re: mount compress=zstd leaves files uncompressed, that used to compress well with before

[Dimitrios Apostolou]

[-- Attachment #1: Type: text/plain, Size: 3073 bytes --]

On Fri, 28 Mar 2025, Qu Wenruo wrote:
>
> 在 2025/3/28 00:10, Dimitrios Apostolou 写道:
>>
>>  Do you know of other cases besides fallocate?
>
> /dev/urandom or something similar, those kind of data will result the
> compressed data to be larger than the original, and btrfs will abort
> compression no matter the mount option.

The docs imply that the compression remains enabled for incompressible
data with compress-force:

   If the first portion of data being compressed is not smaller than the
   original, the compression of the file is disabled -- unless the
   filesystem is mounted with compress-force.

>
> E.g. Ext2 doesn't support fallocate.
>
> But suddenly dropping one feature which we originally support, is a
> little concerning.
>
>>
>>  Maybe the best tradeoff is to add a mount option fallocate=off.
>
> That will be feasible.
>
> I can try push that direction after you have updated the docs.


How about this patch:


diff --git a/Documentation/ch-compression.rst
b/Documentation/ch-compression.rst
index 30b8849c..2553a60c 100644
--- a/Documentation/ch-compression.rst
+++ b/Documentation/ch-compression.rst
@@ -92,18 +92,34 @@ The ZSTD support includes levels -15..15, a subset of full range of what ZSTD
  provides. Levels -15..-1 are real-time with worse compression ratio, levels
  1..3 are near real-time with good compression, 4..8 are slower with improved
  compression and 9..15 try even harder though the resulting size may not be
  significantly improved. Higher levels also require more memory and as they need
  more CPU the system performance is affected.

  Level 0 always maps to the default. The compression level does not affect
  compatibility.

+Exceptions
+----------
+
+Any file that has been extended with the *fallocate* system call (which is
+invoked by *posix_fallocate*) will always be excepted from compression, even
+if future file growth is without *fallocate*, even if *force-compress* mount
+option is used.
+
+The reason for this is that a successful *fallocate* call must guarantee that
+writing to the allocated range wil not fail because of lack of space. To
+achieve this, btrfs disables COW (thus compression too) for the file.
+
+As a workaround, one can trigger a compressed rewrite for such a file using
+the *btrfs defrag* command.
+
+
  Incompressible data
  -------------------

  Files with already compressed data or with data that won't compress well with
  the CPU and memory constraints of the kernel implementations are using a simple
  decision logic. If the first portion of data being compressed is not smaller
  than the original, the compression of the file is disabled -- unless the
  filesystem is mounted with *compress-force*. In that case compression will
  always be attempted on the file only to be later discarded. This is not optimal




Thank you for your detailed responses!
Dimitris

PS. I would like to add a command that detects if a file is marked as
     fallocate'd. Is there such a thing?

Re: mount compress=zstd leaves files uncompressed, that used to compress well with before

[Dimitrios Apostolou]

Hi Qu, do you find my wording technically accurate? I'm lacking insight
into Btrfs internals, so I'd appreciate your feedback.

Also I can resend the patch as an attachment generated from git-patch if
you prefer, just let me know if needed.

Thanks, Dimitris


P.S. As I mentioned in the previous mail, I would like to add to the docs
      a userspace command that detects if a file is marked as fallocate'd,
      i.e. never to-be-compressed. Is there such a thing?



On Fri, 28 Mar 2025, Dimitrios Apostolou wrote:
>
> How about this patch:
>
>
> diff --git a/Documentation/ch-compression.rst
> b/Documentation/ch-compression.rst
> index 30b8849c..2553a60c 100644
> --- a/Documentation/ch-compression.rst
> +++ b/Documentation/ch-compression.rst
> @@ -92,18 +92,34 @@ The ZSTD support includes levels -15..15, a subset of
> full range of what ZSTD
>  provides. Levels -15..-1 are real-time with worse compression ratio, levels
>  1..3 are near real-time with good compression, 4..8 are slower with
>  improved
>  compression and 9..15 try even harder though the resulting size may not be
>  significantly improved. Higher levels also require more memory and as they
>  need
>  more CPU the system performance is affected.
>
>  Level 0 always maps to the default. The compression level does not affect
>  compatibility.
>
> +Exceptions
> +----------
> +
> +Any file that has been extended with the *fallocate* system call (which is
> +invoked by *posix_fallocate*) will always be excepted from compression, even
> +if future file growth is without *fallocate*, even if *force-compress* mount
> +option is used.
> +
> +The reason for this is that a successful *fallocate* call must guarantee
> that
> +writing to the allocated range wil not fail because of lack of space. To
> +achieve this, btrfs disables COW (thus compression too) for the file.
> +
> +As a workaround, one can trigger a compressed rewrite for such a file using
> +the *btrfs defrag* command.
> +
> +
>  Incompressible data
>  -------------------
>
>  Files with already compressed data or with data that won't compress well
>  with
>  the CPU and memory constraints of the kernel implementations are using a
>  simple
>  decision logic. If the first portion of data being compressed is not
>  smaller
>  than the original, the compression of the file is disabled -- unless the
>  filesystem is mounted with *compress-force*. In that case compression will
>  always be attempted on the file only to be later discarded. This is not
>  optimal
>
>

Re: mount compress=zstd leaves files uncompressed, that used to compress well with before

[Qu Wenruo]

在 2025/3/29 00:11, Dimitrios Apostolou 写道:
> On Fri, 28 Mar 2025, Qu Wenruo wrote:
>>
>> 在 2025/3/28 00:10, Dimitrios Apostolou 写道:
>>>
>>>  Do you know of other cases besides fallocate?
>>
>> /dev/urandom or something similar, those kind of data will result the
>> compressed data to be larger than the original, and btrfs will abort
>> compression no matter the mount option.
>
> The docs imply that the compression remains enabled for incompressible
> data with compress-force:
>
>    If the first portion of data being compressed is not smaller than the
>    original, the compression of the file is disabled -- unless the
>    filesystem is mounted with compress-force.

But the code shows otherwise, from fs/btrfs/zlib.c:

                 /* we're making it bigger, give up */
                 if (workspace->strm.total_in > 8192 &&
                     workspace->strm.total_in <
                     workspace->strm.total_out) {
                         ret = -E2BIG;
                         goto out;
                 }

The same thing is done for zstd too.

Nor during my local tests:

$ mkfs.btrfs  -f test.img
$ sudo mount test.img -o compress-force=zlib /mnt/btrfs/
$ sudo dd if=/dev/urandom bs=1M count=1 of=/mnt/btrfs/file
$ sudo sync
$ btrfs ins dump-tree -t 5 test.img
	item 5 key (257 INODE_REF 256) itemoff 15869 itemsize 14
		index 2 namelen 4 name: file
	item 6 key (257 EXTENT_DATA 0) itemoff 15816 itemsize 53
		generation 9 type 1 (regular)
		extent data disk byte 13631488 nr 524288
		extent data offset 0 nr 524288 ram 524288
		extent compression 0 (none)
	item 7 key (257 EXTENT_DATA 524288) itemoff 15763 itemsize 53
		generation 9 type 1 (regular)
		extent data disk byte 14155776 nr 524288
		extent data offset 0 nr 524288 ram 524288
		extent compression 0 (none)

Showing the resulted file extents are still uncompressed.

>
>>
>> E.g. Ext2 doesn't support fallocate.
>>
>> But suddenly dropping one feature which we originally support, is a
>> little concerning.
>>
>>>
>>>  Maybe the best tradeoff is to add a mount option fallocate=off.
>>
>> That will be feasible.
>>
>> I can try push that direction after you have updated the docs.
>
>
> How about this patch:
>
>
> diff --git a/Documentation/ch-compression.rst
> b/Documentation/ch-compression.rst
> index 30b8849c..2553a60c 100644
> --- a/Documentation/ch-compression.rst
> +++ b/Documentation/ch-compression.rst
> @@ -92,18 +92,34 @@ The ZSTD support includes levels -15..15, a subset
> of full range of what ZSTD
>   provides. Levels -15..-1 are real-time with worse compression ratio,
> levels
>   1..3 are near real-time with good compression, 4..8 are slower with
> improved
>   compression and 9..15 try even harder though the resulting size may
> not be
>   significantly improved. Higher levels also require more memory and as
> they need
>   more CPU the system performance is affected.
>
>   Level 0 always maps to the default. The compression level does not affect
>   compatibility.
>
> +Exceptions
> +----------
> +
> +Any file that has been extended with the *fallocate* system call (which is

Not a native speaker, and in fact I'm pretty sure I'm the biggest causer
of typos and grammar errors inside btrfs code.

But to me, the word "extended" may be a little too specific.
E.g. fallocate can be called inside a hole, not extending the file size.

Thus I'd recommend to use "touched" instead.

> +invoked by *posix_fallocate*)

And I won't mention the function name, as there are a lot of different
ways to trigger fallocate, and the posix one may not be the most common one.

IMHO fallocate() is more common than the posix wrapper.


> will always be excepted from compression,
> even
> +if future file growth is without *fallocate*, even if *force-compress*
> mount

And since we mentioned that any file touched by fallocate will not be
compressed, we can skip the "future file growth" part.

> +option is used.
> +
> +The reason for this is that a successful *fallocate* call must
> guarantee that
> +writing to the allocated range wil not fail because of lack of space. To
> +achieve this, btrfs disables COW (thus compression too) for the file.

Technically not correct. Btrfs doesn't disable COW for fallocated files.

The fallocated inode has a flag called PREALLOC:

	item 10 key (258 INODE_ITEM 0) itemoff 15531 itemsize 160
		generation 10 transid 10 size 1048576 nbytes 1048576
		block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
		sequence 1 flags 0x10(PREALLOC)

That flag makes all btrfs write paths to try NOCOW first, and if NOCOW
can not be done, it falls back to regular COW.
And such fallback will not go through compression path.

I do not have a short and easy way to explain this to end users.
May be you can extract this part into a better expression?

Thanks,
Qu

> +
> +As a workaround, one can trigger a compressed rewrite for such a file
> using
> +the *btrfs defrag* command.
> +
> +
>   Incompressible data
>   -------------------
>
>   Files with already compressed data or with data that won't compress
> well with
>   the CPU and memory constraints of the kernel implementations are using
> a simple
>   decision logic. If the first portion of data being compressed is not
> smaller
>   than the original, the compression of the file is disabled -- unless the
>   filesystem is mounted with *compress-force*. In that case compression
> will
>   always be attempted on the file only to be later discarded. This is
> not optimal
>
>
>
>
> Thank you for your detailed responses!
> Dimitris
>
> PS. I would like to add a command that detects if a file is marked as
>      fallocate'd. Is there such a thing?
>

Re: mount compress=zstd leaves files uncompressed, that used to compress well with before

[Dimitrios Apostolou]

[-- Attachment #1: Type: text/plain, Size: 223 bytes --]

Thanks for the feedback, please see the attached patches. I decided to
omit some details. I also include another simple patch clarifying that
compress-force might end up writing uncompressed blocks.

Thanks,
Dimitris

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch; name=v1-0001-btrfs-compress-doc-clarify-compress-force-mount-o.patch, Size: 2080 bytes --]

From cad7d9803f5f19a05e8248202ce6e44e8ba5f0bc Mon Sep 17 00:00:00 2001
From: Dimitrios Apostolou <jimis@qt.io>
Date: Fri, 4 Apr 2025 03:02:34 +0200
Subject: [PATCH v1 1/2] btrfs compress doc: clarify compress-force mount
 option

---
 Documentation/ch-compression.rst | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/Documentation/ch-compression.rst b/Documentation/ch-compression.rst
index 30b8849c..bc8fef8b 100644
--- a/Documentation/ch-compression.rst
+++ b/Documentation/ch-compression.rst
@@ -98,22 +98,23 @@ more CPU the system performance is affected.
 Level 0 always maps to the default. The compression level does not affect
 compatibility.
 
 Incompressible data
 -------------------
 
 Files with already compressed data or with data that won't compress well with
 the CPU and memory constraints of the kernel implementations are using a simple
 decision logic. If the first portion of data being compressed is not smaller
-than the original, the compression of the file is disabled -- unless the
-filesystem is mounted with *compress-force*. In that case compression will
-always be attempted on the file only to be later discarded. This is not optimal
-and subject to optimizations and further development.
+than the original, the compression of the whole file is disabled. Unless the
+filesystem is mounted with *compress-force* in which case btrfs will try
+compressing every block, falling back to storing the uncompressed version for
+each block that ends up larger after compression. This is not optimal and
+subject to optimizations and further development.
 
 If a file is identified as incompressible, a flag is set (*NOCOMPRESS*) and it's
 sticky. On that file compression won't be performed unless forced. The flag
 can be also set by **chattr +m** (since e2fsprogs 1.46.2) or by properties with
 value *no* or *none*. Empty value will reset it to the default that's currently
 applicable on the mounted filesystem.
 
 There are two ways to detect incompressible data:
 
-- 
2.49.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: Type: text/x-patch; name=v1-0002-btrfs-compress-doc-mention-that-fallocate-disable.patch, Size: 2416 bytes --]

From cb6d5659eb8afbde68a53570e6b22d931b6affed Mon Sep 17 00:00:00 2001
From: Dimitrios Apostolou <jimis@qt.io>
Date: Fri, 4 Apr 2025 03:06:40 +0200
Subject: [PATCH v1 2/2] btrfs compress doc: mention that fallocate disables
 compression

---
 Documentation/ch-compression.rst | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/Documentation/ch-compression.rst b/Documentation/ch-compression.rst
index bc8fef8b..9174f97c 100644
--- a/Documentation/ch-compression.rst
+++ b/Documentation/ch-compression.rst
@@ -92,18 +92,35 @@ The ZSTD support includes levels -15..15, a subset of full range of what ZSTD
 provides. Levels -15..-1 are real-time with worse compression ratio, levels
 1..3 are near real-time with good compression, 4..8 are slower with improved
 compression and 9..15 try even harder though the resulting size may not be
 significantly improved. Higher levels also require more memory and as they need
 more CPU the system performance is affected.
 
 Level 0 always maps to the default. The compression level does not affect
 compatibility.
 
+Exceptions
+----------
+
+Any file that has been touched by the *fallocate* system call will always be
+excepted from compression even if *force-compress* mount option is used.
+
+The reason for this is that a successful *fallocate* call must guarantee that
+future writes to the allocated range will not fail because of lack of space.
+This is difficult to guarantee in a COW filesystem. To reduce the chances of
+it happening, btrfs preallocates space and disables compression for the file.
+
+As a workaround, one can trigger a compressed rewrite for such a file using the
+*btrfs defrag* command. Be aware that if the file is touched again by the
+*fallocate* system call, it will be excepted again from compression for all the
+new data written to it.
+
+
 Incompressible data
 -------------------
 
 Files with already compressed data or with data that won't compress well with
 the CPU and memory constraints of the kernel implementations are using a simple
 decision logic. If the first portion of data being compressed is not smaller
 than the original, the compression of the whole file is disabled. Unless the
 filesystem is mounted with *compress-force* in which case btrfs will try
 compressing every block, falling back to storing the uncompressed version for
-- 
2.49.0

Re: mount compress=zstd leaves files uncompressed, that used to compress well with before

[Qu Wenruo]

在 2025/4/4 11:53, Dimitrios Apostolou 写道:
> Thanks for the feedback, please see the attached patches. I decided to
> omit some details. I also include another simple patch clarifying that
> compress-force might end up writing uncompressed blocks.

Looks good to me.

Feel free to submit a github PR or both patches to the mailing list for
more review.

Thanks,
Qu

>
> Thanks,
> Dimitris

Re: mount compress=zstd leaves files uncompressed, that used to compress well with before

[Dimitrios Apostolou]

On Fri, 4 Apr 2025, Qu Wenruo wrote:
>
> Looks good to me.
>
> Feel free to submit a github PR or both patches to the mailing list for
> more review.

Forgive my ignorance but I thought I did what was needed. I notice now
that maybe the subject is wrong. Do I just send an email with the same
attachments and [PATCH] in the subject?

Thanks,
Dimitris

Re: mount compress=zstd leaves files uncompressed, that used to compress well with before

[Qu Wenruo]

在 2025/4/5 03:47, Dimitrios Apostolou 写道:
> On Fri, 4 Apr 2025, Qu Wenruo wrote:
>>
>> Looks good to me.
>>
>> Feel free to submit a github PR or both patches to the mailing list for
>> more review.
>
> Forgive my ignorance but I thought I did what was needed. I notice now
> that maybe the subject is wrong. Do I just send an email with the same
> attachments and [PATCH] in the subject?

Oh I mean using git-send-email to send the patches to the mailing list,
that's the common way we do the review in the mailing list.

Some examples:

https://lore.kernel.org/linux-btrfs/cover.1740542229.git.wqu@suse.com/

Although it means you have to setup your mail box to allow SMTP access.

Thus the other solution is to send a PR through github.

Thanks,
Qu

>
> Thanks,
> Dimitris
>

Re: mount compress=zstd leaves files uncompressed, that used to compress well with before

[Dimitrios Apostolou]

[-- Attachment #1: Type: text/plain, Size: 1014 bytes --]

On Sat, 5 Apr 2025, Qu Wenruo wrote:
>
>
> 在 2025/4/5 03:47, Dimitrios Apostolou 写道:
>>  On Fri, 4 Apr 2025, Qu Wenruo wrote:
>>>
>>>  Looks good to me.
>>>
>>>  Feel free to submit a github PR or both patches to the mailing list for
>>>  more review.
>>
>>  Forgive my ignorance but I thought I did what was needed. I notice now
>>  that maybe the subject is wrong. Do I just send an email with the same
>>  attachments and [PATCH] in the subject?
>
> Oh I mean using git-send-email to send the patches to the mailing list,
> that's the common way we do the review in the mailing list.
>
> Some examples:
>
> https://lore.kernel.org/linux-btrfs/cover.1740542229.git.wqu@suse.com/
>
> Although it means you have to setup your mail box to allow SMTP access.
>
> Thus the other solution is to send a PR through github.

It got a bit overwhelming to send by email, so I ended up with submitting
a pull request:

https://github.com/kdave/btrfs-progs/pull/976


Thanks,
Dimitris