Re: Btrfs + compression = slow performance and high cpu usage

[pg@btrfs.list.sabi.co.UK (Peter Grandi)]

> Peter, I don't think the filefrag is showing the correct
> fragmentation status of the file when the compression is used.

As reported on a previous message the output of 'filefrag -v'
which can be used to see what is going on:

>>>> filefrag /mnt/sde3/testfile 
>>>>   /mnt/sde3/testfile: 49287 extents found

>>>> Most the latter extents are mercifully rather contiguous, their
>>>> size is just limited by the compression code, here is an extract
>>>> from 'filefrag -v' from around the middle:

>>>>   24757:  1321888.. 1321919:   11339579..  11339610:     32:   11339594:
>>>>   24758:  1321920.. 1321951:   11339597..  11339628:     32:   11339611:
>>>>   24759:  1321952.. 1321983:   11339615..  11339646:     32:   11339629:
>>>>   24760:  1321984.. 1322015:   11339632..  11339663:     32:   11339647:
>>>>   24761:  1322016.. 1322047:   11339649..  11339680:     32:   11339664:
>>>>   24762:  1322048.. 1322079:   11339667..  11339698:     32:   11339681:
>>>>   24763:  1322080.. 1322111:   11339686..  11339717:     32:   11339699:
>>>>   24764:  1322112.. 1322143:   11339703..  11339734:     32:   11339718:
>>>>   24765:  1322144.. 1322175:   11339720..  11339751:     32:   11339735:
>>>>   24766:  1322176.. 1322207:   11339737..  11339768:     32:   11339752:
>>>>   24767:  1322208.. 1322239:   11339754..  11339785:     32:   11339769:
>>>>   24768:  1322240.. 1322271:   11339771..  11339802:     32:   11339786:
>>>>   24769:  1322272.. 1322303:   11339789..  11339820:     32:   11339803:

>>>> But again this is on a fresh empty Btrfs volume.

As I wrote, "their size is just limited by the compression code"
which results in "128KiB writes". On a "fresh empty Btrfs volume"
the compressed extents limited to 128KiB also happen to be pretty
physically contiguous, but on a more fragmented free space list
they can be more scattered.

As I already wrote the main issue here seems to be that we are
talking about a "RAID5 with 128KiB writes and a 768KiB stripe
size". On MD RAID5 the slowdown because of RMW seems only to be
around 30-40%, but it looks like that several back-to-back 128KiB
writes get merged by the Linux IO subsystem (not sure whether
that's thoroughly legal), and perhaps they get merged by the 3ware
firmware only if it has a persistent cache, and maybe your 3ware
does not have one, but you have kept your counsel as to that.

My impression is that you read the Btrfs documentation and my
replies with a lot less attention than I write them. Some of the
things you have done and said make me think that you did not read
https://btrfs.wiki.kernel.org/index.php/Compression and 'man 5
btrfs', for example:

   "How does compression interact with direct IO or COW?

     Compression does not work with DIO, does work with COW and
     does not work for NOCOW files. If a file is opened in DIO
     mode, it will fall back to buffered IO.

   Are there speed penalties when doing random access to a
   compressed file?

     Yes. The compression processes ranges of a file of maximum
     size 128 KiB and compresses each 4 KiB (or page-sized) block
     separately."

> I am currently defragmenting that mountpoint, ensuring that
> everrything is compressed with zlib.

Defragmenting the used space might help find more contiguous
allocations.

> p.s. any other suggestion that might help with the fragmentation
> and data allocation. Should I try and rebalance the data on the
> drive?

Yes, regularly, as that defragments the unused space.