Re: yet another "out of space" on a filesystem with >100 GB free space, and strange files which exist but don't exist

["Austin S. Hemmelgarn" <ahferroin7@gmail.com>]

On 2017-10-04 07:13, Tomasz Chmielewski wrote:
> Kernel: 4.13.4, btrfs RAID-1.
> 
> Disk usage more or less like below (yes, I know about btrfs fi df / show 
> / usage):
> 
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/sda3       424G  262G  161G  62% /var/lib/lxd
> 
> 
> Balance would exit immediately with "out of space", but continues to run 
> after I've removed a few gigabytes from the filesystem.
> 
> 
> Now, I'm seeing some files which exist, but don't. Strange, I know.
> 
> 
> root@lxd02 /var/lib/lxd/containers/mongo-repl04b/rootfs/var/lib/mongodb 
> # ls *set
> ls: cannot access 'WiredTiger.turtle.set': No such file or directory
> 
> root@lxd02 /var/lib/lxd/containers/mongo-repl04b/rootfs/var/lib/mongodb 
> # ls -l|grep set
> ls: cannot access 'WiredTiger.turtle.set': No such file or directory
> -????????? ? ?      ?                ?            ? WiredTiger.turtle.set
> 
> root@lxd02 /var/lib/lxd/containers/mongo-repl04b/rootfs/var/lib/mongodb 
> # mv WiredTiger.turtle.set WiredTiger.turtle.set.Ghost.File
> mv: cannot stat 'WiredTiger.turtle.set': No such file or directory
> 
> root@lxd02 /var/lib/lxd/containers/mongo-repl04b/rootfs/var/lib/mongodb 
> # rm -v WiredTiger.turtle.set
> rm: cannot remove 'WiredTiger.turtle.set': No such file or directory
> 
> 
> 
> What is this file, and why does it exist if it doesn't? How do I remove it?
It's got corrupted metadata, probably the inode itself (IIRC, the dentry 
in BTRFS just matches the inode to the file name, and all the other data 
reported by ls -l is stored in the inode).  If you're running with a 
replicated metadata profile (dup, raid1, or raid10), run a scrub, and it 
may fix things.  If not, you will likely have to run a check in repair 
mode (though I would suggest waiting to hear from one of the developers 
before doing so).  Alternatively, if that's in a subvolume, and you can 
afford to just nuke the subvolume and recreate it, deleting the 
subvolume should get rid of it (though you should still run a check).

Either way, this is likely related to the balance issues you're seeing.