Re: BTRFS constantly reports "No space left on device" even with a huge unallocated space

[Jeff Mahoney <jeffm@suse.com>]

[-- Attachment #1.1: Type: text/plain, Size: 4191 bytes --]

On 9/8/16 2:24 PM, Ronan Arraes Jardim Chagas wrote:
> Hi all!
> 
> Em Seg, 2016-09-05 às 16:49 +0800, Qu Wenruo escreveu:
>> Just like what Wang has mentioned, would you please paste all the
>> output 
>> of the contents of /sys/fs/btrfs/<your fs uuid>/allocation?
>>
>> It's recommended to use "grep . -IR <path>" to get all the data as
>> it 
>> will show the file name.
> 
> So, one more time, I see the problem. This time I was just using
> Firefox and I cannot recover using `btrfs balance`. I think that, one
> more time, I will need to reboot this machine. This problem is really
> causing me a lot of troubles :(

I have a hunch the list is about to be flooded with similar reports if
we don't find this one before 4.8.

commit d555b6c380c644af63dbdaa7cc14bba041a4e4dd
Author: Josef Bacik <jbacik@fb.com>
Date:   Fri Mar 25 13:25:51 2016 -0400

    Btrfs: warn_on for unaccounted spaces

This commit isn't the source of the bug, but it's making it a lot more
noisy.  I spent a few hours last night trying to track down why xfstests
was throwing these warnings and I was able to reproduce them at least as
far back as 4.4-vanilla with -oenospc_debug enabled.

Speaking of which, can you turn on mounting with -oenospc_debug if you
haven't already?

In my case, space_info->bytes_may_use was getting accounted incorrectly.

I am able to reproduce that even with the following commit:
commit 18513091af9483ba84328d42092bd4d42a3c958f
Author: Wang Xiaoguang <wangxg.fnst@cn.fujitsu.com>
Date:   Mon Jul 25 15:51:40 2016 +0800

    btrfs: update btrfs_space_info's bytes_may_use timely


> grep . -IR /sys/fs/btrfs/e9efaa0c-d477-4249-830f-
> ee5956768b29/allocation
> allocation/data/flags:1
> allocation/data/bytes_pinned:0
> allocation/data/bytes_may_use:0
> allocation/data/total_bytes_pinned:202973265920

That adds up to ~ 189 GB.  total_bytes is only about 42 GB.

> allocation/data/bytes_reserved:0
> allocation/data/bytes_used:45623730176
> allocation/data/single/used_bytes:45623730176
> allocation/data/single/total_bytes:46179287040
> allocation/data/total_bytes:46179287040
> allocation/data/disk_total:46179287040
> allocation/data/disk_used:45623730176
> allocation/metadata/dup/used_bytes:1120698368
> allocation/metadata/dup/total_bytes:6979321856
> allocation/metadata/flags:4
> allocation/metadata/bytes_pinned:0
> allocation/metadata/bytes_may_use:88521768960
> allocation/metadata/total_bytes_pinned:-44285952

... well that's certainly interesting.  It looks like we'll need to see
how that happened.  It seems like we've messed up at least that portion
of accounting.

-Jeff

> allocation/metadata/bytes_reserved:0
> allocation/metadata/bytes_used:1120698368
> allocation/metadata/total_bytes:6979321856
> allocation/metadata/disk_total:13958643712
> allocation/metadata/disk_used:2241396736
> allocation/global_rsv_size:385875968
> allocation/global_rsv_reserved:385875968
> allocation/system/dup/used_bytes:16384
> allocation/system/dup/total_bytes:33554432
> allocation/system/flags:2
> allocation/system/bytes_pinned:0
> allocation/system/bytes_may_use:0
> allocation/system/total_bytes_pinned:0
> allocation/system/bytes_reserved:0
> allocation/system/bytes_used:16384
> allocation/system/total_bytes:33554432
> allocation/system/disk_total:67108864
> allocation/system/disk_used:32768
> 
> Additional information:
> 
> btrfs fi usage /
> Overall:
>     Device size:		   1.26TiB
>     Device allocated:		  56.07GiB
>     Device unallocated:		   1.20TiB
>     Device missing:		     0.00B
>     Used:			  44.58GiB
>     Free (estimated):		   1.20TiB	(min: 616.41GiB)
>     Data ratio:			      1.00
>     Metadata ratio:		      2.00
>     Global reserve:		 368.00MiB	(used: 0.00B)
> 
> Data,single: Size:43.01GiB, Used:42.49GiB
>    /dev/sda6	  43.01GiB
> 
> Metadata,DUP: Size:6.50GiB, Used:1.04GiB
>    /dev/sda6	  13.00GiB
> 
> System,DUP: Size:32.00MiB, Used:16.00KiB
>    /dev/sda6	  64.00MiB
> 
> Unallocated:
>    /dev/sda6	   1.20TiB
> 
> Can anyone help me?
> 
> Best regards,
> Ronan Arraes
> 


-- 
Jeff Mahoney
SUSE Labs


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 881 bytes --]