Ability to free space on a full btrfs filesystem

Ability to free space on a full btrfs filesystem

[Matthieu Dalstein]

Hello,

I'm currently experiencing some space issue on btrfs with linux kernel 3.10.
My btrfs partition is full and I fail to remove any data to give some 
free space:

athena:~# rm -f /mnt/storage/current/backup/slash/var/log/syslog
rm: cannot remove `/mnt/storage/current/backup/slash/var/log/syslog': No 
space left on device

athena:~# true > /mnt/storage/current/backup/slash/var/log/syslog
bash: /mnt/storage/current/backup/slash/var/log/syslog: No space left on 
device

I tried to follow the wiki page 
https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#I_get_.22No_space_left_on_device.22_errors.2C_but_df_says_I.27ve_got_lots_of_space
but the balance operation fails due to lack of space:
athena:~# btrfs fi balance start -dusage=5 /mnt/storage/
ERROR: error during balancing '/mnt/storage/' - No space left on device

[ Kernel logs ] btrfs: 138 enospc errors during balance

I guess this scenario is not expected, or I am probably missing something.
Is there any way I can free some space and get back a writable filesystem ?

Some more info:
Mount options used for the recovery (from 
https://btrfs.wiki.kernel.org/index.php/Ubuntu_support#ENOSPC.2C_no_space_left_on_device)
/dev/sdb1 on /mnt/storage type btrfs 
(rw,relatime,nodatasum,nodatacow,nospace_cache,clear_cache)

athena:~# btrfs fi show
Label: none  uuid: 8fa5ecc2-fff6-4ad1-a6bb-e384ea2086cb
         Total devices 1 FS bytes used 475.19GB
         devid    1 size 619.33GB used 619.33GB path /dev/sdb1

athena:~# btrfs fi df /mnt/storage/
Data: total=613.30GB, used=472.69GB
System, DUP: total=8.00MB, used=72.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=3.00GB, used=2.50GB
Metadata: total=8.00MB, used=0.00

Thanks in advance
-- 
Matthieu

Re: Ability to free space on a full btrfs filesystem

[Marc MERLIN]

On Sun, Aug 25, 2013 at 06:23:20PM +0200, Matthieu Dalstein wrote:
> Hello,
> 
> I'm currently experiencing some space issue on btrfs with linux kernel 3.10.
> My btrfs partition is full and I fail to remove any data to give some 
> free space:

Did you snapshot your filesystem? If so, delete some/all snapshots first.
Then, note that brtfs frees space after you deleted snapshots, it can take
one or several minutes before the space ends up being garbage collected.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: Ability to free space on a full btrfs filesystem

[Chris Murphy]

On Aug 25, 2013, at 10:23 AM, Matthieu Dalstein <matthieu.dalstein@dalmat.net> wrote:

> athena:~# btrfs fi df /mnt/storage/
> Data: total=613.30GB, used=472.69GB

So it seems the issue is that data chunks once allocated aren't deallocated, and 

> Metadata, DUP: total=3.00GB, used=2.50GB

the file system is thus unable to allocate another metadata chunk. So it's stuck. I don't know if removing a snapshot will necessarily help but it may be worth a try and see if these numbers change enough to allow file deletion. 

Another way out of this that's worked for some people in the past is to btrfs device add another disk to the volume. Maybe just 1GB of USB stick space is enough so that it can allocate another metadata chunk. But I'd give it as much of the fastest storage you have: can be any disk partition, or LV. But I've also done it with a USB stick partition.

After it's added, see if you can do a balance, which should free most of those 141GB of data chunks. And then you can btrfs device delete the recently added device, which will move whatever data/metadata chunks are on it, back to the primary device for the volume.


Chris Murphy

Re: Ability to free space on a full btrfs filesystem

[Matthieu Dalstein]

> I don't know if removing a snapshot will necessarily help but it may be worth a try and see if these numbers change enough to allow file deletion.

Actually the snapshots could not be removed due to the unsufficient 
remaining space.


> Another way out of this that's worked for some people in the past is to btrfs device add another disk to the volume. Maybe just 1GB of USB stick space is enough so that it can allocate another metadata chunk. But I'd give it as much of the fastest storage you have: can be any disk partition, or LV. But I've also done it with a USB stick partition.
>
> After it's added, see if you can do a balance, which should free most of those 141GB of data chunks. And then you can btrfs device delete the recently added device, which will move whatever data/metadata chunks are on it, back to the primary device for the volume.

Thanks for the tip. I would have expected another less intrusive 
recovery but this one worked well. I did not fully rebalance the fs 
(could have lasted days!) but with the d/m usage balance parameters I 
managed to get back to a good state.

I guess this means I should find ways to avoid full fs unless there is a 
plan to recover gracefully from such situations.

Thanks
-- 
Matthieu

Re: Ability to free space on a full btrfs filesystem

[Chris Murphy]

On Aug 26, 2013, at 3:43 PM, Matthieu Dalstein <matthieu.dalstein@dalmat.net> wrote:

> Thanks for the tip. I would have expected another less intrusive recovery but this one worked well. I did not fully rebalance the fs (could have lasted days!) but with the d/m usage balance parameters I managed to get back to a good state.
> 
> I guess this means I should find ways to avoid full fs unless there is a plan to recover gracefully from such situations.

The FAQ describes doing a partial balance to fix this problem as a first step. I think it's a good first step for this problem, rather than full balance, or adding a device. Next step would be add a device and see if a partial balance works, then delete that device and carry on (with normal usage or resuming a full balance).

In my few cases I didn't try partial balances, rather full ones and they almost immediately failed which is how I decided to add a USB flash stick to the volume.


Chris Murphy