btrfs filesystem failing with 'No space left on device' after 4 hours

[Michael Firth <MFirth@nevion.com>]

Hi,

I have a BTRFS filesystem that seems to have become very ill. After 4 hours of being mounted, it will fail with every write attempt saying "No space left on device".

Unmounting and remounting the filesystem clears the issue for another 4 hours

From every check I have done, no messages are logged at the point of the failure to "dmesg" or any system log.

I'm over 99% sure there is not a space issue on the filesystem - it has over 100GB free, and I've run a full "balance" which has not changed the behaviour. A "scrub" on the filesystem hasn't reported any issues.

The output of the three (why on earth are there three?) disk space commands on the filesystem are:

--------------------------------------------------------------------------------------------------------------------------------------
$ sudo btrfs filesystem usage /home
Overall:
    Device size:                                    450.00GiB
    Device allocated:                         319.06GiB
    Device unallocated:                    130.94GiB
    Device missing:                                  0.00B
    Used:                                                305.95GiB
    Free (estimated):                        131.77GiB           (min: 66.30GiB)
    Data ratio:                                            1.00
    Metadata ratio:                                  2.00
    Global reserve:                             512.00MiB          (used: 0.00B)

Data,single: Size:299.00GiB, Used:298.16GiB
   /dev/mapper/VG-HomeVol              299.00GiB

Metadata,DUP: Size:10.00GiB, Used:3.89GiB
   /dev/mapper/VG-HomeVol                20.00GiB

System,DUP: Size:32.00MiB, Used:80.00KiB
   /dev/mapper/VG-HomeVol                64.00MiB

Unallocated:
   /dev/mapper/VG-HomeVol              130.94GiB

$ sudo btrfs filesystem df /home
Data, single: total=299.00GiB, used=298.16GiB
System, DUP: total=32.00MiB, used=80.00KiB
Metadata, DUP: total=10.00GiB, used=3.89GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

$ sudo btrfs filesystem show /home
Label: none  uuid: 550e6e7c-d669-4128-9b0d-b61ef4f3f1c1
                Total devices 1 FS bytes used 302.07GiB
                devid    1 size 450.00GiB used 319.06GiB path /dev/mapper/VG-HomeVol
--------------------------------------------------------------------------------------------------------------------------------------

From my understanding of the output in this, there don't seem to be any areas that are even close to full. And if it was a genuine full condition, even due to running out of metadata or something, then I wouldn't expect unmounting and remounting to clear the issue.

Is there any known issue that may cause this behaviour?

Is there any way to get more debugging from what is going on?

My initial thought was that it might be related to snapshots, as I was generating regular snapshots (for a 'previous versions' feature), and many of the failures were just after a snapshot was created. However, I have now disabled the snapshot creation and I am still seeing regular failures.

The system is running stock Debian 9 (Stretch). It was running their latest 4.9 kernel (Rev 4.9.144-3.1) when the problem first occurred. After two instances of the problem, I rolled back to their previous kernel (Rev 4.9.130-2), which the system had been running error free for several months, but the failures have continued.

I'm happy to get any other information that would be needed to debug this, if someone can point me to how to do it.

Currently my faith in BTRFS is approaching zero (it was knocked after a data loss in October, but had grown again). It has a lot of nice features, but (despite comments on the Wiki) really does not seem stable, at least not in Debian.

Thanks

Michael