CEPH to BTRFS over NFS results in no compression

["Konstantin V. Gavrilenko" <k.gavrilenko@arhont.com>]

Hi list, 

I have noticed an interesting and surprising behaviour of my BTRFS with regards to compression of the files and NFS. 

I have BTRFS RAID10 with 8 disks , that is mounted with the " nofail,noatime,space_cache=v2,compress-force=zstd:9,subvol=@cloudstack-secondary" flags and is exported via NFS. 

When I create a snapshot of a disk in Cloudstack from CEPH and save it to a secondary storage to this BTRFS RAID10 over NFS, the file does not compress, despite the compress-force mount option being set on FS 
So in the below example, the file eeceaf0e-9780-408b-a748-1495d517a9b6 was copied over NFS and is not compressed. When I copy the same file directly on a host, it does get compressed pretty well, as per example below. 


root@backup1 : /mnt/backup1/cloudstack-secondary/snapshots/49/5355: compsize eeceaf0e-9780-408b-a748-1495d517a9b6 
Type Perc Disk Usage Uncompressed Referenced 
TOTAL 99% 9.8G 9.8G 9.8G 
none 100% 9.8G 9.8G 9.8G 
zstd 3% 4.0K 128K 124K 

root@backup1 : /mnt/backup1/cloudstack-secondary/snapshots/49/5355: cp eeceaf0e-9780-408b-a748-1495d517a9b6 testfile 
'eeceaf0e-9780-408b-a748-1495d517a9b6' -> 'testfile' 


root@backup1 : /mnt/backup1/cloudstack-secondary/snapshots/49/5355: compsize testfile 
Type Perc Disk Usage Uncompressed Referenced 
TOTAL 38% 3.7G 9.8G 9.8G 
none 100% 1.1G 1.1G 1.1G 
zstd 30% 2.6G 8.6G 8.6G 

root@backup1: /mnt/backup1/cloudstack-secondary/snapshots/49/5355  btrfs filesystem defrag -v -f -czstd ./eeceaf0e-9780-408b-a748-1495d517a9b6 
./eeceaf0e-9780-408b-a748-1495d517a9b6 

root@backup1: /mnt/backup1/cloudstack-secondary/snapshots/49/5355  compsize eeceaf0e-9780-408b-a748-1495d517a9b6 
Type Perc Disk Usage Uncompressed Referenced 
TOTAL 38% 3.7G 9.8G 9.8G 
none 100% 1.1G 1.1G 1.1G 
zstd 30% 2.6G 8.6G 8.6G 



So what I have checked so far what works 
- after the original files is copied over NFS, the copy of the file using #cp gets compressed. 
- after the original files is copied over NFS, the original file can be compressed using #btrfs defrag -czstd option 
- If I copy the original file to some other host, and copy it back via NFS using cp, it does get compressed. 

So the problem seems to appear only when the file is exported from Ceph and copied to NFS. 

Any hints what could be causing such a behaviour? 



Yours sincerely, 
Kos