very poor performance / a lot of writes to disk with space_cache (but not with space_cache=v2)

[Tomasz Chmielewski <tch@virtall.com>]

I have a mysql slave which writes to a RAID-1 btrfs filesystem (with 
4.17.14 kernel) on 3 x ~1.9 TB SSD disks; filesystem is around 40% full.

The slave receives around 0.5-1 MB/s of data from the master over the 
network, which is then saved to MySQL's relay log and executed. In ideal 
conditions (i.e. no filesystem overhead) we should expect some 1-3 MB/s 
of data written to disk.

MySQL directory and files in it are chattr +C (since the directory was 
created, so all files are really +C); there are no snapshots.


Now, an interesting thing.

When the filesystem is mounted with these options in fstab:

defaults,noatime,discard


We can see a *constant* write of 25-100 MB/s to each disk. The system is 
generally unresponsive and it sometimes takes long seconds for a simple 
command executed in bash to return.


However, as soon as we remount the filesystem with space_cache=v2 - 
writes drop to just around 3-10 MB/s to each disk. If we remount to 
space_cache - lots of writes, system unresponsive. Again remount to 
space_cache=v2 - low writes, system responsive.


That's a huuge, 10x overhead! Is it expected? Especially that 
space_cache=v1 is still the default mount option?


Tomasz Chmielewski
https://lxadm.com