Re: BTRFS constantly reports "No space left on device" even with a huge unallocated space

[Wang Xiaoguang <wangxg.fnst@cn.fujitsu.com>]

hello,

On 08/29/2016 11:52 PM, Ronan Arraes Jardim Chagas wrote:
> Hi guys,
>
> I just have the problem again. Now, it happens during the lunch time
> when the machine was idle. Only the system processes were running. It
> was not the first time that I saw this problem just after lunch when
> the machine stayed idle for a long period (+- 1h).
>
> Here is the information requested:
>
> /sys/fs/btrfs/$UUID/allocation/data
>
> ./bytes_may_use
> 0
> ./bytes_pinned
> 0
> ./bytes_reserved
> 0
> ./bytes_used
> 36128374784
> ./disk_total
> 37589352448
> ./disk_used
> 36128374784
> ./flags
> 1
> ./total_bytes
> 37589352448
> ./total_bytes_pinned
> 20339560448
> ./single/total_bytes
> 37589352448
> ./single/used_bytes
> 36128374784
>
> /sys/fs/btrfs/$UUID/allocation/metadata
>
> ./bytes_may_use
> 84974452736
> ./bytes_pinned
> 0
> ./bytes_reserved
> 0
> ./bytes_used
> 977354752
> ./disk_total
> 4294967296
> ./disk_used
> 1954709504
> ./flags
> 4
> ./total_bytes
> 2147483648
> ./total_bytes_pinned
> -57851904
> ./dup/total_bytes
> 2147483648
> ./dup/used_bytes
> 977354752

For metadata, "bytes_may_use" is about 80GB, it's very big,
I think this value is very abnormal.

So this explains why you have huge unallocated space, you still
get ENOSPC error. In kernel btrfs, there is a function should_alloc_chunk()
to determine whether to allocate new chunks(new device space)
  num_bytes = total_bytes - bytes_readonly; it's 2147483648
  num_allocated = bytes_used + bytes_reserved; it's 977354752

if num_allocated < num_bytes * 0.8, it will not allocate new device 
space :) even you
have huge unallocated space.

I think the root reason is that bytes_may_use has some computation error and
is not be converted to bytes_used or bytes_reserved.

I just explain why you get ENOSPC error even with huge unallocated space 
from
codes :)

>
> # btrfs fi usage /
>
> btrfs_destroy_inode+0x23f/0x2b0 [btrfs]
> Ago 29 12:06:07 ronanarraes-osd kernel:  [<ffffffff8121f6d1>]
> do_unlinkat+0x131/0x310
> Ago 29 12:06:07 ronanarraes-osd kernel:  [<ffffffff816bb4f6>]
> entry_SYSCALL_64_fastpath+0x1e/0xa8
> Ago 29 12:06:07 ronanarraes-osd kernel: DWARF2 unwinder stuck at
> entry_SYSCALL_64_fastpath+0x1e/0xa8
> Ago 29 12:06:07 ronanarraes-osd kernel:
> Ago 29 12:06:07 ronanarraes-osd kernel: Leftover inexact backtrace:
> Ago 29 12:06:07 ronanarraes-osd kernel: ---[ end trace 5774bd3049f78a63
> ]---
Yes, I know these WARNINGs, but indeed they are already results,
we don't know the procedures which cause these results.

Can you work out a reproducer for this ENOSPC error, then I can
dig into codes to figure out the true reason.

Regards,
Xiaoguang Wang

>
> Best regards,
> Ronan Arraes
>
>