Out of space error even though there's 100 GB unused?

Out of space error even though there's 100 GB unused?

[Stanislaw Kaminski]

Hi,
I am fighting with this since at least Monday - see
https://superuser.com/questions/1096658/btrfs-out-of-space-even-though-there-should-be-10-left

Here's the data:
#   uname -a
Linux archb3 4.6.3-2-ARCH #1 PREEMPT Wed Jun 29 07:15:33 MDT 2016
armv5tel GNU/Linux

#   btrfs --version
btrfs-progs v4.6

#   btrfs fi show
Label: 'home'  uuid: 1c7e35e8-f013-4f65-9d19-eaa168ac088b
        Total devices 1 FS bytes used 1.71TiB
        devid    1 size 1.81TiB used 1.71TiB path /dev/sda4

#   btrfs fi df /home
Data, single: total=1.71TiB, used=1.71TiB
System, DUP: total=32.00MiB, used=224.00KiB
Metadata, DUP: total=4.00GiB, used=2.07GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

# btrfs f usage -T  /home
Overall:
    Device size:                   1.81TiB
    Device allocated:              1.71TiB
    Device unallocated:           97.89GiB
    Device missing:                  0.00B
    Used:                          1.71TiB
    Free (estimated):             98.22GiB      (min: 49.27GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)

             Data    Metadata System
Id Path      single  DUP      DUP       Unallocated
-- --------- ------- -------- --------- -----------
 1 /dev/sda4 1.71TiB  8.00GiB  64.00MiB    97.89GiB
-- --------- ------- -------- --------- -----------
   Total     1.71TiB  4.00GiB  32.00MiB    97.89GiB
   Used      1.71TiB  2.07GiB 224.00KiB

# btrfs fi du -s /home
Total Exclusive Set shared Filename
1.60TiB 1.60TiB 0.00B /home

# btrfs f resize 1:+1G /home/
Resize '/home/' of '1:+1G'
ERROR: unable to resize '/home/': no enough free space

This all is after closely following:
https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_I_ran_out_of_disk_space.21
http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html

So, already did full volume rebalance, defrag, rebooted multiple times
- still, "Error: out of disk space".

To sum up:
- my files sum to 1.6 TiB
- disk usage is shown to be 1.71 TiB
- volume size is 1.81 TiB
- btrfs util shows I have ~98 GiB free space on the volume
- I am getting "out of space" message

Bonus:
- I removed 50 GB of data from the drive and I still get "out of
space" message after writing ~1 GB.

Help would be very appreciated.

Cheers,
Stan

Re: Out of space error even though there's 100 GB unused?

[Stanislaw Kaminski]

Hi Alex,
Thank for having a look.

"You're trying to resize a fs that is probably already fully using the
block device it's on. I don't see anything incorrect happening here,
but I might be missing something."
This was just to show that I can't do this, I know that it is already
utilizing the entire block device.

"The unallocated space will be allocated if you start writing files to it."
That's what I would expect, unfortunately it's kind of hard to write
files to it, as I get "Out of space" error. Tongue-in-cheek, if you
know how to ignore the issue and start writing files, it would solve
my issue.

Bottom line: if the disk is really full, then none of the tools shows
that. If it is not (and I suspect it's not - as I mentioned, I just
removed 50 GB of data from it), then why am I getting "out of space"?

As for block device size:
# fdisk -l
Disk /dev/sda: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 6070B645-D738-4730-BEF7-989210EF1DD7

Device        Start        End    Sectors  Size Type
/dev/sda1      2048     133119     131072   64M Linux filesystem
/dev/sda2    133120    2230271    2097152    1G Linux swap
/dev/sda3   2230272   19007487   16777216    8G Linux filesystem
/dev/sda4  19007488 3907029134 3888021647  1.8T Linux home


Cheers,
Stan

2016-07-06 12:10 GMT+02:00 Alexander Fougner <fougner89@gmail.com>:
>
> Den 6 juli 2016 12:03 em skrev "Stanislaw Kaminski"
> <stasheck.fora@gmail.com>:
>>
>
>> Hi,
>> I am fighting with this since at least Monday - see
>>
>> https://superuser.com/questions/1096658/btrfs-out-of-space-even-though-there-should-be-10-left
>>
>> Here's the data:
>> #   uname -a
>> Linux archb3 4.6.3-2-ARCH #1 PREEMPT Wed Jun 29 07:15:33 MDT 2016
>> armv5tel GNU/Linux
>>
>> #   btrfs --version
>> btrfs-progs v4.6
>>
>> #   btrfs fi show
>> Label: 'home'  uuid: 1c7e35e8-f013-4f65-9d19-eaa168ac088b
>>         Total devices 1 FS bytes used 1.71TiB
>>         devid    1 size 1.81TiB used 1.71TiB path /dev/sda4
>>
>> #   btrfs fi df /home
>> Data, single: total=1.71TiB, used=1.71TiB
>> System, DUP: total=32.00MiB, used=224.00KiB
>> Metadata, DUP: total=4.00GiB, used=2.07GiB
>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>
>> # btrfs f usage -T  /home
>> Overall:
>>     Device size:                   1.81TiB
>>     Device allocated:              1.71TiB
>>     Device unallocated:           97.89GiB
>>     Device missing:                  0.00B
>>     Used:                          1.71TiB
>>     Free (estimated):             98.22GiB      (min: 49.27GiB)
>>     Data ratio:                       1.00
>>     Metadata ratio:                   2.00
>>     Global reserve:              512.00MiB      (used: 0.00B)
>>
>>              Data    Metadata System
>> Id Path      single  DUP      DUP       Unallocated
>> -- --------- ------- -------- --------- -----------
>>  1 /dev/sda4 1.71TiB  8.00GiB  64.00MiB    97.89GiB
>> -- --------- ------- -------- --------- -----------
>>    Total     1.71TiB  4.00GiB  32.00MiB    97.89GiB
>>    Used      1.71TiB  2.07GiB 224.00KiB
>>
>> # btrfs fi du -s /home
>> Total Exclusive Set shared Filename
>> 1.60TiB 1.60TiB 0.00B /home
>>
>> # btrfs f resize 1:+1G /home/
>> Resize '/home/' of '1:+1G'
>> ERROR: unable to resize '/home/': no enough free space
>>
>
> You're trying to resize a fs that is probably already fully using the block
> device it's on. I don't see anything incorrect happening here, but I might
> be missing something.
>
> The used space amounting to 1.6TiB is not as reliable as the btrfs fi df
> tool.
> The unallocated space will be allocated if you start writing files to it.
> What size is the parent block device?
>
>> This all is after closely following:
>>
>> https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_I_ran_out_of_disk_space.21
>>
>> http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html
>>
>> So, already did full volume rebalance, defrag, rebooted multiple times
>> - still, "Error: out of disk space".
>>
>> To sum up:
>> - my files sum to 1.6 TiB
>> - disk usage is shown to be 1.71 TiB
>> - volume size is 1.81 TiB
>> - btrfs util shows I have ~98 GiB free space on the volume
>> - I am getting "out of space" message
>>
>> Bonus:
>> - I removed 50 GB of data from the drive and I still get "out of
>> space" message after writing ~1 GB.
>>
>> Help would be very appreciated.
>>
>> Cheers,
>> Stan
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Out of space error even though there's 100 GB unused?

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 3216 bytes --]

On Wed, Jul 06, 2016 at 11:55:42AM +0200, Stanislaw Kaminski wrote:
> Hi,
> I am fighting with this since at least Monday - see
> https://superuser.com/questions/1096658/btrfs-out-of-space-even-though-there-should-be-10-left
> 
> Here's the data:
> #   uname -a
> Linux archb3 4.6.3-2-ARCH #1 PREEMPT Wed Jun 29 07:15:33 MDT 2016
> armv5tel GNU/Linux
> 
> #   btrfs --version
> btrfs-progs v4.6
> 
> #   btrfs fi show
> Label: 'home'  uuid: 1c7e35e8-f013-4f65-9d19-eaa168ac088b
>         Total devices 1 FS bytes used 1.71TiB
>         devid    1 size 1.81TiB used 1.71TiB path /dev/sda4

   In this state, you should definitely not be seeing out of space
errors. This is, therefore, a bug you're seeing.

   I've not been following things as closely as I'd like of late, but
I think there was a bug recently involving the free space cache. It
might be worth unmounting the FS and mounting again with the
nospace_cache option, just to see if that helps.

   Hugo.

> #   btrfs fi df /home
> Data, single: total=1.71TiB, used=1.71TiB
> System, DUP: total=32.00MiB, used=224.00KiB
> Metadata, DUP: total=4.00GiB, used=2.07GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> # btrfs f usage -T  /home
> Overall:
>     Device size:                   1.81TiB
>     Device allocated:              1.71TiB
>     Device unallocated:           97.89GiB
>     Device missing:                  0.00B
>     Used:                          1.71TiB
>     Free (estimated):             98.22GiB      (min: 49.27GiB)
>     Data ratio:                       1.00
>     Metadata ratio:                   2.00
>     Global reserve:              512.00MiB      (used: 0.00B)
> 
>              Data    Metadata System
> Id Path      single  DUP      DUP       Unallocated
> -- --------- ------- -------- --------- -----------
>  1 /dev/sda4 1.71TiB  8.00GiB  64.00MiB    97.89GiB
> -- --------- ------- -------- --------- -----------
>    Total     1.71TiB  4.00GiB  32.00MiB    97.89GiB
>    Used      1.71TiB  2.07GiB 224.00KiB
> 
> # btrfs fi du -s /home
> Total Exclusive Set shared Filename
> 1.60TiB 1.60TiB 0.00B /home
> 
> # btrfs f resize 1:+1G /home/
> Resize '/home/' of '1:+1G'
> ERROR: unable to resize '/home/': no enough free space
> 
> This all is after closely following:
> https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_I_ran_out_of_disk_space.21
> http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html
> 
> So, already did full volume rebalance, defrag, rebooted multiple times
> - still, "Error: out of disk space".
> 
> To sum up:
> - my files sum to 1.6 TiB
> - disk usage is shown to be 1.71 TiB
> - volume size is 1.81 TiB
> - btrfs util shows I have ~98 GiB free space on the volume
> - I am getting "out of space" message
> 
> Bonus:
> - I removed 50 GB of data from the drive and I still get "out of
> space" message after writing ~1 GB.
> 
> Help would be very appreciated.
> 
> Cheers,
> Stan

-- 
Hugo Mills             | You can play with your friends' privates, but you
hugo@... carfax.org.uk | can't play with your friends' childrens' privates.
http://carfax.org.uk/  |
PGP: E2AB1DE4          |                                       C++ coding rule

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Out of space error even though there's 100 GB unused?

[Stanislaw Kaminski]

Hi Hugo,
I agree that it seems to be a bug, and I'll be glad to help nail that
down - if only because I have no other drive to move the data to :-)

As for your suggestion - no change:
[root@archb3 stan]# mount | grep home
/dev/sda4 on /home type btrfs
(rw,relatime,nospace_cache,clear_cache,subvolid=5,subvol=/)
[root@archb3 stan]# touch test
touch: cannot touch 'test': No space left on device

Cheers,
Stan

2016-07-06 12:34 GMT+02:00 Hugo Mills <hugo@carfax.org.uk>:
> On Wed, Jul 06, 2016 at 11:55:42AM +0200, Stanislaw Kaminski wrote:
>> Hi,
>> I am fighting with this since at least Monday - see
>> https://superuser.com/questions/1096658/btrfs-out-of-space-even-though-there-should-be-10-left
>>
>> Here's the data:
>> #   uname -a
>> Linux archb3 4.6.3-2-ARCH #1 PREEMPT Wed Jun 29 07:15:33 MDT 2016
>> armv5tel GNU/Linux
>>
>> #   btrfs --version
>> btrfs-progs v4.6
>>
>> #   btrfs fi show
>> Label: 'home'  uuid: 1c7e35e8-f013-4f65-9d19-eaa168ac088b
>>         Total devices 1 FS bytes used 1.71TiB
>>         devid    1 size 1.81TiB used 1.71TiB path /dev/sda4
>
>    In this state, you should definitely not be seeing out of space
> errors. This is, therefore, a bug you're seeing.
>
>    I've not been following things as closely as I'd like of late, but
> I think there was a bug recently involving the free space cache. It
> might be worth unmounting the FS and mounting again with the
> nospace_cache option, just to see if that helps.
>
>    Hugo.
>
>> #   btrfs fi df /home
>> Data, single: total=1.71TiB, used=1.71TiB
>> System, DUP: total=32.00MiB, used=224.00KiB
>> Metadata, DUP: total=4.00GiB, used=2.07GiB
>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>
>> # btrfs f usage -T  /home
>> Overall:
>>     Device size:                   1.81TiB
>>     Device allocated:              1.71TiB
>>     Device unallocated:           97.89GiB
>>     Device missing:                  0.00B
>>     Used:                          1.71TiB
>>     Free (estimated):             98.22GiB      (min: 49.27GiB)
>>     Data ratio:                       1.00
>>     Metadata ratio:                   2.00
>>     Global reserve:              512.00MiB      (used: 0.00B)
>>
>>              Data    Metadata System
>> Id Path      single  DUP      DUP       Unallocated
>> -- --------- ------- -------- --------- -----------
>>  1 /dev/sda4 1.71TiB  8.00GiB  64.00MiB    97.89GiB
>> -- --------- ------- -------- --------- -----------
>>    Total     1.71TiB  4.00GiB  32.00MiB    97.89GiB
>>    Used      1.71TiB  2.07GiB 224.00KiB
>>
>> # btrfs fi du -s /home
>> Total Exclusive Set shared Filename
>> 1.60TiB 1.60TiB 0.00B /home
>>
>> # btrfs f resize 1:+1G /home/
>> Resize '/home/' of '1:+1G'
>> ERROR: unable to resize '/home/': no enough free space
>>
>> This all is after closely following:
>> https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_I_ran_out_of_disk_space.21
>> http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html
>>
>> So, already did full volume rebalance, defrag, rebooted multiple times
>> - still, "Error: out of disk space".
>>
>> To sum up:
>> - my files sum to 1.6 TiB
>> - disk usage is shown to be 1.71 TiB
>> - volume size is 1.81 TiB
>> - btrfs util shows I have ~98 GiB free space on the volume
>> - I am getting "out of space" message
>>
>> Bonus:
>> - I removed 50 GB of data from the drive and I still get "out of
>> space" message after writing ~1 GB.
>>
>> Help would be very appreciated.
>>
>> Cheers,
>> Stan
>
> --
> Hugo Mills             | You can play with your friends' privates, but you
> hugo@... carfax.org.uk | can't play with your friends' childrens' privates.
> http://carfax.org.uk/  |
> PGP: E2AB1DE4          |                                       C++ coding rule

Re: Out of space error even though there's 100 GB unused?

[Chris Murphy]

On Wed, Jul 6, 2016 at 3:55 AM, Stanislaw Kaminski
<stasheck.fora@gmail.com> wrote:

>     Device unallocated:           97.89GiB

There should be no problem creating any type of block group from this
much space. It's a bug.

I would try regression testing. Kernel 4.5.7 has some changes that may
or may not relate to this (they should only relate when there is no
unallocated space left) so you could try 4.5.6 and 4.5.7. And also
4.4.14.

But also the kernel messages are important. There is this obscure
enospc with error -28, so either with or without enospc_debug mount
option is useful to try in 4.6.3 (I think it's less useful in older
kernels).

But do try nospace_cache first. If that works, you could then mount
with clear_cache one time and see if that provides an enduring fix. It
can take some time to rebuild the cache after clear_cache is used.



-- 
Chris Murphy

Re: Out of space error even though there's 100 GB unused?

[Stanislaw Kaminski]

Hi all,
I downgraded to 4.4.1-1 - all fine, 4.5.5.-1 - also fine, then got
back to 4.6.3-2 - and it's still fine. Apparently running under
different kernel somehow fixed the glitch (as far as I can test...).

That leaves me with the other question: before issues, I 1.6 TiB was
used, now all the tools report 1.7 TiB issued (except for btrfs fs du
/home, this reports 1.6 TiB). How is that possible?

Cheers,
Stan

2016-07-06 19:42 GMT+02:00 Chris Murphy <lists@colorremedies.com>:
> On Wed, Jul 6, 2016 at 3:55 AM, Stanislaw Kaminski
> <stasheck.fora@gmail.com> wrote:
>
>>     Device unallocated:           97.89GiB
>
> There should be no problem creating any type of block group from this
> much space. It's a bug.
>
> I would try regression testing. Kernel 4.5.7 has some changes that may
> or may not relate to this (they should only relate when there is no
> unallocated space left) so you could try 4.5.6 and 4.5.7. And also
> 4.4.14.
>
> But also the kernel messages are important. There is this obscure
> enospc with error -28, so either with or without enospc_debug mount
> option is useful to try in 4.6.3 (I think it's less useful in older
> kernels).
>
> But do try nospace_cache first. If that works, you could then mount
> with clear_cache one time and see if that provides an enduring fix. It
> can take some time to rebuild the cache after clear_cache is used.
>
>
>
> --
> Chris Murphy

Re: Out of space error even though there's 100 GB unused?

[Stanislaw Kaminski]

Too early report, the issue is back. Back to testing....

2016-07-07 12:18 GMT+02:00 Stanislaw Kaminski <stasheck.fora@gmail.com>:
> Hi all,
> I downgraded to 4.4.1-1 - all fine, 4.5.5.-1 - also fine, then got
> back to 4.6.3-2 - and it's still fine. Apparently running under
> different kernel somehow fixed the glitch (as far as I can test...).
>
> That leaves me with the other question: before issues, I 1.6 TiB was
> used, now all the tools report 1.7 TiB issued (except for btrfs fs du
> /home, this reports 1.6 TiB). How is that possible?
>
> Cheers,
> Stan
>
> 2016-07-06 19:42 GMT+02:00 Chris Murphy <lists@colorremedies.com>:
>> On Wed, Jul 6, 2016 at 3:55 AM, Stanislaw Kaminski
>> <stasheck.fora@gmail.com> wrote:
>>
>>>     Device unallocated:           97.89GiB
>>
>> There should be no problem creating any type of block group from this
>> much space. It's a bug.
>>
>> I would try regression testing. Kernel 4.5.7 has some changes that may
>> or may not relate to this (they should only relate when there is no
>> unallocated space left) so you could try 4.5.6 and 4.5.7. And also
>> 4.4.14.
>>
>> But also the kernel messages are important. There is this obscure
>> enospc with error -28, so either with or without enospc_debug mount
>> option is useful to try in 4.6.3 (I think it's less useful in older
>> kernels).
>>
>> But do try nospace_cache first. If that works, you could then mount
>> with clear_cache one time and see if that provides an enduring fix. It
>> can take some time to rebuild the cache after clear_cache is used.
>>
>>
>>
>> --
>> Chris Murphy

Re: Out of space error even though there's 100 GB unused?

[Stanislaw Kaminski]

Hi Chris, Alex, Hugo,

Running now: Linux archb3 4.6.2-1-ARCH #1 PREEMPT Mon Jun 13 02:11:34
MDT 2016 armv5tel GNU/Linux

Seems to be working fine. I started a defrag, and it seems I'm getting
my space back:
$ sudo btrfs fi usage /home
Overall:
    Device size:                   1.81TiB
    Device allocated:              1.73TiB
    Device unallocated:           80.89GiB
    Device missing:                  0.00B
    Used:                          1.65TiB
    Free (estimated):            159.63GiB      (min: 119.19GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 240.00KiB)

Data,single: Size:1.72TiB, Used:1.65TiB
   /dev/sda4       1.72TiB

Metadata,DUP: Size:3.50GiB, Used:2.16GiB
   /dev/sda4       7.00GiB

System,DUP: Size:32.00MiB, Used:224.00KiB
   /dev/sda4      64.00MiB

Unallocated:
   /dev/sda4      80.89GiB

I deleted some unfinished torrent, ~10 GB in size, but as you can see,
"Free space" has grown by 60 GB (re-checked now and it's 1 GB more now
- so definitely caused by defrag).

What has changed between 4.6.2 and 4.6.3?

Cheers,
Stan

2016-07-07 12:28 GMT+02:00 Stanislaw Kaminski <stasheck.fora@gmail.com>:
> Too early report, the issue is back. Back to testing....
>
> 2016-07-07 12:18 GMT+02:00 Stanislaw Kaminski <stasheck.fora@gmail.com>:
>> Hi all,
>> I downgraded to 4.4.1-1 - all fine, 4.5.5.-1 - also fine, then got
>> back to 4.6.3-2 - and it's still fine. Apparently running under
>> different kernel somehow fixed the glitch (as far as I can test...).
>>
>> That leaves me with the other question: before issues, I 1.6 TiB was
>> used, now all the tools report 1.7 TiB issued (except for btrfs fs du
>> /home, this reports 1.6 TiB). How is that possible?
>>
>> Cheers,
>> Stan
>>
>> 2016-07-06 19:42 GMT+02:00 Chris Murphy <lists@colorremedies.com>:
>>> On Wed, Jul 6, 2016 at 3:55 AM, Stanislaw Kaminski
>>> <stasheck.fora@gmail.com> wrote:
>>>
>>>>     Device unallocated:           97.89GiB
>>>
>>> There should be no problem creating any type of block group from this
>>> much space. It's a bug.
>>>
>>> I would try regression testing. Kernel 4.5.7 has some changes that may
>>> or may not relate to this (they should only relate when there is no
>>> unallocated space left) so you could try 4.5.6 and 4.5.7. And also
>>> 4.4.14.
>>>
>>> But also the kernel messages are important. There is this obscure
>>> enospc with error -28, so either with or without enospc_debug mount
>>> option is useful to try in 4.6.3 (I think it's less useful in older
>>> kernels).
>>>
>>> But do try nospace_cache first. If that works, you could then mount
>>> with clear_cache one time and see if that provides an enduring fix. It
>>> can take some time to rebuild the cache after clear_cache is used.
>>>
>>>
>>>
>>> --
>>> Chris Murphy

Re: Out of space error even though there's 100 GB unused?

[Henk Slager]

On Thu, Jul 7, 2016 at 5:17 PM, Stanislaw Kaminski
<stasheck.fora@gmail.com> wrote:
> Hi Chris, Alex, Hugo,
>
> Running now: Linux archb3 4.6.2-1-ARCH #1 PREEMPT Mon Jun 13 02:11:34
> MDT 2016 armv5tel GNU/Linux
>
> Seems to be working fine. I started a defrag, and it seems I'm getting
> my space back:
> $ sudo btrfs fi usage /home
> Overall:
>     Device size:                   1.81TiB
>     Device allocated:              1.73TiB
>     Device unallocated:           80.89GiB
>     Device missing:                  0.00B
>     Used:                          1.65TiB
>     Free (estimated):            159.63GiB      (min: 119.19GiB)
>     Data ratio:                       1.00
>     Metadata ratio:                   2.00
>     Global reserve:              512.00MiB      (used: 240.00KiB)
>
> Data,single: Size:1.72TiB, Used:1.65TiB
>    /dev/sda4       1.72TiB
>
> Metadata,DUP: Size:3.50GiB, Used:2.16GiB
>    /dev/sda4       7.00GiB
>
> System,DUP: Size:32.00MiB, Used:224.00KiB
>    /dev/sda4      64.00MiB
>
> Unallocated:
>    /dev/sda4      80.89GiB
>
> I deleted some unfinished torrent, ~10 GB in size, but as you can see,
> "Free space" has grown by 60 GB (re-checked now and it's 1 GB more now
> - so definitely caused by defrag).
>
> What has changed between 4.6.2 and 4.6.3?

https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/diff/?id=v4.6.3&id2=v4.6.2&dt=2

I see no change for Btrfs

> Cheers,
> Stan
>
> 2016-07-07 12:28 GMT+02:00 Stanislaw Kaminski <stasheck.fora@gmail.com>:
>> Too early report, the issue is back. Back to testing....
>>
>> 2016-07-07 12:18 GMT+02:00 Stanislaw Kaminski <stasheck.fora@gmail.com>:
>>> Hi all,
>>> I downgraded to 4.4.1-1 - all fine, 4.5.5.-1 - also fine, then got
>>> back to 4.6.3-2 - and it's still fine. Apparently running under
>>> different kernel somehow fixed the glitch (as far as I can test...).
>>>
>>> That leaves me with the other question: before issues, I 1.6 TiB was
>>> used, now all the tools report 1.7 TiB issued (except for btrfs fs du
>>> /home, this reports 1.6 TiB). How is that possible?
>>>
>>> Cheers,
>>> Stan
>>>
>>> 2016-07-06 19:42 GMT+02:00 Chris Murphy <lists@colorremedies.com>:
>>>> On Wed, Jul 6, 2016 at 3:55 AM, Stanislaw Kaminski
>>>> <stasheck.fora@gmail.com> wrote:
>>>>
>>>>>     Device unallocated:           97.89GiB
>>>>
>>>> There should be no problem creating any type of block group from this
>>>> much space. It's a bug.
>>>>
>>>> I would try regression testing. Kernel 4.5.7 has some changes that may
>>>> or may not relate to this (they should only relate when there is no
>>>> unallocated space left) so you could try 4.5.6 and 4.5.7. And also
>>>> 4.4.14.
>>>>
>>>> But also the kernel messages are important. There is this obscure
>>>> enospc with error -28, so either with or without enospc_debug mount
>>>> option is useful to try in 4.6.3 (I think it's less useful in older
>>>> kernels).
>>>>
>>>> But do try nospace_cache first. If that works, you could then mount
>>>> with clear_cache one time and see if that provides an enduring fix. It
>>>> can take some time to rebuild the cache after clear_cache is used.
>>>>
>>>>
>>>>
>>>> --
>>>> Chris Murphy
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Out of space error even though there's 100 GB unused?

[Henk Slager]

On Fri, Jul 8, 2016 at 9:22 AM, Stanislaw Kaminski
<stasheck.fora@gmail.com> wrote:
> Huh.
>
> I left defrag running overnight, and now I'm back to my >200 GiB free
> space. Also, I got no "out of space" messages in Transmission, and it
> successfully downloaded few GBs.
>
> But in dmesg I have 209 traces, see attached.
>
> Does that say anything to you?

I get some vague clue about the problem, but no-one seems to know
exactly the root cause(s).
The 4.6.2 code where the warning comes from is this:
...
/*
 * Called if we need to clear a data reservation for this inode
 * Normally in a error case.
 *
 * This one will *NOT* use accurate qgroup reserved space API, just for case
 * which we can't sleep and is sure it won't affect qgroup reserved space.
 * Like clear_bit_hook().
 */
void btrfs_free_reserved_data_space_noquota(struct inode *inode, u64 start,
                                            u64 len)
{
   struct btrfs_root *root = BTRFS_I(inode)->root;
   struct btrfs_space_info *data_sinfo;

   /* Make sure the range is aligned to sectorsize */
   len = round_up(start + len, root->sectorsize) -
       round_down(start, root->sectorsize);
   start = round_down(start, root->sectorsize);

   data_sinfo = root->fs_info->data_sinfo;
   spin_lock(&data_sinfo->lock);
   if (WARN_ON(data_sinfo->bytes_may_use < len))
           data_sinfo->bytes_may_use = 0;
   else
           data_sinfo->bytes_may_use -= len;
   trace_btrfs_space_reservation(root->fs_info, "space_info",
                               data_sinfo->flags, len, 0);
   spin_unlock(&data_sinfo->lock);
}
...

I think the system is still 'recovering' from getting stuck earlier.
What exactly that is, I don't know. You would probably need to enable
more debugging facilities in order to figure out from which file or
inode the problem comes from. I don't know if you can compile a 4.7
kernel for this Kirkwood SoC, but that would be one way forward. (BTW,
is it a 88F6281 or  a 88F6282 ?)

As far as I remember, Josef Basik has posted some patches that could
benefit this case, I am not sure if they made it in 4.7, but that is
what I think I would try.

Otherwise, it are workarounds:
- you could have a look at the cpuload during defrag and normal
operation and see how it relates to the rate of issuing this warning
- add mount option noatime
- as it looks like this this fs is (also) a torrent-client target, you
can put the torrents in a directoru ot subvol with noCoW flag set or
mount the whole fs with nodatacow
- again clean cache and then mount with space_cache=v2. Then new
mounts will use v2 then automatically. Only thing I can say it that
helped me getting out of kernel crash situation with 4.6.2. 4.7.0-rc5
did not crash on the same fs, so I got it working again (
de-allocations and cleanups, the fs is almost exclusively a
btrfs-receive target)
- connect the fs to a multi-core x86_64 system running same kernel
version for some time and see if you can reproduce the same type of
WARN_ONs