Again, no space left on device while rebalancing and recipe doesnt work

Again, no space left on device while rebalancing and recipe doesnt work

[Marc Haber]

Hi,

I have again the issue of no space left on device while rebalancing
(with btrfs-tools 4.4.1 on kernel 4.4.2 on Debian unstable):

mh@fan:~$ sudo btrfs balance start /mnt/fanbtr
ERROR: error during balancing '/mnt/fanbtr': No space left on device
mh@fan:~$ sudo btrfs fi show /mnt/fanbtr
mh@fan:~$ sudo btrfs fi show -m
Label: 'fanbtr'  uuid: 4198d1bc-e3ce-40df-a7ee-44a2d120bff3
        Total devices 1 FS bytes used 116.49GiB
        devid    1 size 417.19GiB used 177.06GiB path /dev/mapper/fanbtr
mh@fan:~$ sudo btrfs fi df /mnt/fanbtr
Data, single: total=113.00GiB, used=112.77GiB
System, DUP: total=32.00MiB, used=48.00KiB
Metadata, DUP: total=32.00GiB, used=3.72GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
mh@fan:~$

The filesystem was recently resized from 300 GB to 420 GB.

Why does btrfs fi show /mnt/fanbtr not give any output? Wy does btrfs
fi df /mnt/fanbtr say that my data space is only 113 GiB large?

btrfs balance start -dusage=5 works up to -dusage=100:

mh@fan:~$ sudo btrfs balance start -dusage=100 /mnt/fanbtr
Done, had to relocate 111 out of 179 chunks
mh@fan:~$ sudo btrfs balance start -dusage=100 /mnt/fanbtr
Done, had to relocate 111 out of 179 chunks
mh@fan:~$ sudo btrfs balance start -dusage=100 /mnt/fanbtr
Done, had to relocate 110 out of 179 chunks
mh@fan:~$ sudo btrfs balance start -dusage=100 /mnt/fanbtr
Done, had to relocate 109 out of 179 chunks
mh@fan:~$ sudo btrfs balance start /mnt/fanbtr
ERROR: error during balancing '/mnt/fanbtr': No space left on device
mh@fan:~$

What is going on here? How do I get away from here?

Greetings
Marc


-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Martin Steigerwald]

On Samstag, 27. Februar 2016 22:14:50 CET Marc Haber wrote:
> Hi,

Hi Marc.

> I have again the issue of no space left on device while rebalancing
> (with btrfs-tools 4.4.1 on kernel 4.4.2 on Debian unstable):
> 
> mh@fan:~$ sudo btrfs balance start /mnt/fanbtr
> ERROR: error during balancing '/mnt/fanbtr': No space left on device
> mh@fan:~$ sudo btrfs fi show /mnt/fanbtr
> mh@fan:~$ sudo btrfs fi show -m
> Label: 'fanbtr'  uuid: 4198d1bc-e3ce-40df-a7ee-44a2d120bff3
>         Total devices 1 FS bytes used 116.49GiB
>         devid    1 size 417.19GiB used 177.06GiB path /dev/mapper/fanbtr

Hmmm, thats still a ton of space to allocate chunks from.

> mh@fan:~$ sudo btrfs fi df /mnt/fanbtr
> Data, single: total=113.00GiB, used=112.77GiB
> System, DUP: total=32.00MiB, used=48.00KiB
> Metadata, DUP: total=32.00GiB, used=3.72GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> mh@fan:~$
> 
> The filesystem was recently resized from 300 GB to 420 GB.
> 
> Why does btrfs fi show /mnt/fanbtr not give any output? Wy does btrfs
> fi df /mnt/fanbtr say that my data space is only 113 GiB large?

Cause it is.

The "used" in "devid 1" line is btrfs fi sh is "data + 2x system + 2x metadata 
= 113 GiB + 2 * 32 GiB + 2 * 32 MiB, i.e. what amount of the size of the 
device is allocated for chunks.

The value one line above is what is allocated inside the chunks.

I.e. the line in "devid 1" is "total" of btrfs fi df summed up, and the line 
above is "used" in btrfs fi df summed up. And… with more devices you have more 
fun.

I suggest:

merkaba:~> btrfs fi usage -T /daten
Overall:
    Device size:                 235.00GiB
    Device allocated:            227.04GiB
    Device unallocated:            7.96GiB
    Device missing:                  0.00B
    Used:                        225.84GiB
    Free (estimated):              8.48GiB      (min: 8.48GiB)
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:              128.00MiB      (used: 0.00B)

             Data      Metadata  System              
Id Path      single    single    single   Unallocated
-- --------- --------- --------- -------- -----------
 1 /dev/dm-1 226.00GiB   1.01GiB 32.00MiB     7.96GiB
-- --------- --------- --------- -------- -----------
   Total     226.00GiB   1.01GiB 32.00MiB     7.96GiB
   Used      225.48GiB 371.83MiB 48.00KiB 

as that is much clearer to read IMHO.

and

merkaba:~> btrfs device usage /daten   
/dev/dm-1, ID: 1
   Device size:           235.00GiB
   Data,single:           226.00GiB
   Metadata,single:         1.01GiB
   System,single:          32.00MiB
   Unallocated:             7.96GiB

(although thats include in the filesystem usage output)


Or for a BTRFS RAID 1:

merkaba:~> btrfs fi usage -T /home  
Overall:
    Device size:                 340.00GiB
    Device allocated:            340.00GiB
    Device unallocated:            2.00MiB
    Device missing:                  0.00B
    Used:                        306.47GiB
    Free (estimated):             14.58GiB      (min: 14.58GiB)
    Data ratio:                       2.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)

             Data      Metadata System              
Id Path      RAID1     RAID1    RAID1    Unallocated
-- --------- --------- -------- -------- -----------
 1 /dev/dm-0 163.94GiB  6.03GiB 32.00MiB     1.00MiB
 2 /dev/dm-3 163.94GiB  6.03GiB 32.00MiB     1.00MiB
-- --------- --------- -------- -------- -----------
   Total     163.94GiB  6.03GiB 32.00MiB     2.00MiB
   Used      149.36GiB  3.88GiB 48.00KiB            


merkaba:~> btrfs device usage /home
/dev/dm-0, ID: 1
   Device size:           170.00GiB
   Data,RAID1:            163.94GiB
   Metadata,RAID1:          6.03GiB
   System,RAID1:           32.00MiB
   Unallocated:             1.00MiB

/dev/dm-3, ID: 2
   Device size:           170.00GiB
   Data,RAID1:            163.94GiB
   Metadata,RAID1:          6.03GiB
   System,RAID1:           32.00MiB
   Unallocated:             1.00MiB


(this is actually the situation asking for hung task trouble with kworker 
threads seeking for free space inside chunks, as no new chunks can be 
allocated, lets hope kernel 4.4 finally really has fixes for this)

> btrfs balance start -dusage=5 works up to -dusage=100:
> 
> mh@fan:~$ sudo btrfs balance start -dusage=100 /mnt/fanbtr
> Done, had to relocate 111 out of 179 chunks
> mh@fan:~$ sudo btrfs balance start -dusage=100 /mnt/fanbtr
> Done, had to relocate 111 out of 179 chunks
> mh@fan:~$ sudo btrfs balance start -dusage=100 /mnt/fanbtr
> Done, had to relocate 110 out of 179 chunks
> mh@fan:~$ sudo btrfs balance start -dusage=100 /mnt/fanbtr
> Done, had to relocate 109 out of 179 chunks
> mh@fan:~$ sudo btrfs balance start /mnt/fanbtr
> ERROR: error during balancing '/mnt/fanbtr': No space left on device
> mh@fan:~$
> 
> What is going on here? How do I get away from here?

Others may have better tips, but what can always work is:

Adding a new device temporarily, doing the balance and then removing it.

Before that I´d try to balance the metadata chunks, cause 

> Metadata, DUP: total=32.00GiB, used=3.72GiB

32 GiB chunks allocated, only 3,72 GiB used.

Maybe that way you can gain more free space to have a full balance run.

Also note that it is not necessary to do a full balance in case everything 
works okayish.

Thanks,
-- 
Martin

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Marc Haber]

On Sun, Feb 28, 2016 at 12:15:21AM +0100, Martin Steigerwald wrote:
> On Samstag, 27. Februar 2016 22:14:50 CET Marc Haber wrote:
> > I have again the issue of no space left on device while rebalancing
> > (with btrfs-tools 4.4.1 on kernel 4.4.2 on Debian unstable):
> > 
> > mh@fan:~$ sudo btrfs balance start /mnt/fanbtr
> > ERROR: error during balancing '/mnt/fanbtr': No space left on device
> > mh@fan:~$ sudo btrfs fi show /mnt/fanbtr
> > mh@fan:~$ sudo btrfs fi show -m
> > Label: 'fanbtr'  uuid: 4198d1bc-e3ce-40df-a7ee-44a2d120bff3
> >         Total devices 1 FS bytes used 116.49GiB
> >         devid    1 size 417.19GiB used 177.06GiB path /dev/mapper/fanbtr
> 
> Hmmm, thats still a ton of space to allocate chunks from.
> 
> > mh@fan:~$ sudo btrfs fi df /mnt/fanbtr
> > Data, single: total=113.00GiB, used=112.77GiB
> > System, DUP: total=32.00MiB, used=48.00KiB
> > Metadata, DUP: total=32.00GiB, used=3.72GiB
> > GlobalReserve, single: total=512.00MiB, used=0.00B
> > mh@fan:~$
> > 
> > The filesystem was recently resized from 300 GB to 420 GB.
> > 
> > Why does btrfs fi show /mnt/fanbtr not give any output? Wy does btrfs
> > fi df /mnt/fanbtr say that my data space is only 113 GiB large?
> 
> Cause it is.
> 
> The "used" in "devid 1" line is btrfs fi sh is "data + 2x system + 2x metadata 
> = 113 GiB + 2 * 32 GiB + 2 * 32 MiB, i.e. what amount of the size of the 
> device is allocated for chunks.
> 
> The value one line above is what is allocated inside the chunks.
> 
> I.e. the line in "devid 1" is "total" of btrfs fi df summed up, and the line 
> above is "used" in btrfs fi df summed up. And… with more devices you have more 
> fun.

Why wouldn't btrfs allocate more data chunks from the ample free space?

> I suggest:
> 
> merkaba:~> btrfs fi usage -T /daten


[2/498]mh@fan:~$ sudo btrfs fi usage /mnt/fanbtr
Overall:
    Device size:                 417.19GiB
    Device allocated:            177.06GiB
    Device unallocated:          240.12GiB
    Device missing:                  0.00B
    Used:                        120.23GiB
    Free (estimated):            240.33GiB      (min: 120.27GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)

Data,single: Size:113.00GiB, Used:112.79GiB
   /dev/mapper/fanbtr    113.00GiB

Metadata,DUP: Size:32.00GiB, Used:3.72GiB
   /dev/mapper/fanbtr     64.00GiB

System,DUP: Size:32.00MiB, Used:48.00KiB
   /dev/mapper/fanbtr     64.00MiB

[3/498]mh@fan:~$ sudo btrfs fi usage -T /mnt/fanbtr
Overall:
    Device size:                 417.19GiB
    Device allocated:            177.06GiB
    Device unallocated:          240.12GiB
    Device missing:                  0.00B
    Used:                        120.23GiB
    Free (estimated):            240.33GiB      (min: 120.27GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)

                      Data      Metadata System
Id Path               single    DUP      DUP      Unallocated
-- ------------------ --------- -------- -------- -----------
 1 /dev/mapper/fanbtr 113.00GiB 64.00GiB 64.00MiB   240.12GiB
-- ------------------ --------- -------- -------- -----------
   Total              113.00GiB 32.00GiB 32.00MiB   240.12GiB
   Used               112.79GiB  3.72GiB 48.00KiB
[4/499]mh@fan:~$

> (this is actually the situation asking for hung task trouble with kworker 
> threads seeking for free space inside chunks, as no new chunks can be 
> allocated, lets hope kernel 4.4 finally really has fixes for this)

I am running a 4.4.2 kernel on the system in question.

> Adding a new device temporarily, doing the balance and then removing it.

I currently refuse to do this on a 400 GiB device that has more than
half of its capacity free. I do expect a modern filesystem to get out
of that situation without a manual intervention this invasive.

> Before that I´d try to balance the metadata chunks, cause 
> 
> > Metadata, DUP: total=32.00GiB, used=3.72GiB
> 
> 32 GiB chunks allocated, only 3,72 GiB used.

Why would I rebalance metadata if there is less than 20 % used?

[21/504]mh@fan:~$ sudo btrfs balance start -musage=5 /mnt/fanbtr
ERROR: error during balancing '/mnt/fanbtr': No space left on device
There may be more info in syslog - try dmesg | tail
[22/505]mh@fan:~$ sudo btrfs balance start -musage=1 /mnt/fanbtr
Done, had to relocate 56 out of 179 chunks
[23/506]mh@fan:~$ sudo btrfs balance start -musage=1 /mnt/fanbtr
Done, had to relocate 56 out of 179 chunks
[24/506]mh@fan:~$ sudo btrfs balance start -musage=1 /mnt/fanbtr
Done, had to relocate 56 out of 179 chunks
[25/506]mh@fan:~$ sudo btrfs balance start -musage=5 /mnt/fanbtr
Done, had to relocate 56 out of 179 chunks
[26/507]mh@fan:~$ sudo btrfs balance start -musage=50 /mnt/fanbtr
Done, had to relocate 56 out of 179 chunks
[27/508]mh@fan:~$ sudo btrfs balance start -musage=90 /mnt/fanbtr
Done, had to relocate 61 out of 179 chunks
[29/510]mh@fan:~$ sudo btrfs balance start -musage=90 /mnt/fanbtr
ERROR: error during balancing '/mnt/fanbtr': No space left on device
There may be more info in syslog - try dmesg | tail
[30/511]mh@fan:~$ sudo btrfs balance start -musage=90 /mnt/fanbtr
ERROR: error during balancing '/mnt/fanbtr': No space left on device
There may be more info in syslog - try dmesg | tail
[31/511]mh@fan:~$ sudo btrfs fi usage -T /mnt/fanbtr
Overall:
    Device size:                 417.19GiB
    Device allocated:            177.06GiB
    Device unallocated:          240.12GiB
    Device missing:                  0.00B
    Used:                        120.24GiB
    Free (estimated):            240.32GiB      (min: 120.26GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)

                      Data      Metadata System
Id Path               single    DUP      DUP      Unallocated
-- ------------------ --------- -------- -------- -----------
 1 /dev/mapper/fanbtr 113.00GiB 64.00GiB 64.00MiB   240.12GiB
-- ------------------ --------- -------- -------- -----------
   Total              113.00GiB 32.00GiB 32.00MiB   240.12GiB
   Used               112.80GiB  3.72GiB 48.00KiB
[32/512]mh@fan:~$

Why does the number of reallocated chunks not decrease, why does
-musage=90 finish the first time and throw out of space errors on
subsequent invocations, and why doesn't the balance change anything in
btrfs fi usage?

> Maybe that way you can gain more free space to have a full balance run.

Why would i need more free space? there are 240 GiB free!

> Also note that it is not necessary to do a full balance in case everything 
> works okayish.

The filesystem does not look healthy to me.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2199 bytes --]

On Sun, Feb 28, 2016 at 01:08:29AM +0100, Marc Haber wrote:
> On Sun, Feb 28, 2016 at 12:15:21AM +0100, Martin Steigerwald wrote:
> > On Samstag, 27. Februar 2016 22:14:50 CET Marc Haber wrote:
> > > I have again the issue of no space left on device while rebalancing
> > > (with btrfs-tools 4.4.1 on kernel 4.4.2 on Debian unstable):
> > > 
> > > mh@fan:~$ sudo btrfs balance start /mnt/fanbtr
> > > ERROR: error during balancing '/mnt/fanbtr': No space left on device
> > > mh@fan:~$ sudo btrfs fi show /mnt/fanbtr
> > > mh@fan:~$ sudo btrfs fi show -m
> > > Label: 'fanbtr'  uuid: 4198d1bc-e3ce-40df-a7ee-44a2d120bff3
> > >         Total devices 1 FS bytes used 116.49GiB
> > >         devid    1 size 417.19GiB used 177.06GiB path /dev/mapper/fanbtr
> > 
> > Hmmm, thats still a ton of space to allocate chunks from.
> > 
> > > mh@fan:~$ sudo btrfs fi df /mnt/fanbtr
> > > Data, single: total=113.00GiB, used=112.77GiB
> > > System, DUP: total=32.00MiB, used=48.00KiB
> > > Metadata, DUP: total=32.00GiB, used=3.72GiB
> > > GlobalReserve, single: total=512.00MiB, used=0.00B
> > > mh@fan:~$
> > > 
> > > The filesystem was recently resized from 300 GB to 420 GB.
> > > 
> > > Why does btrfs fi show /mnt/fanbtr not give any output? Wy does btrfs
> > > fi df /mnt/fanbtr say that my data space is only 113 GiB large?
> > 
> > Cause it is.
> > 
> > The "used" in "devid 1" line is btrfs fi sh is "data + 2x system + 2x metadata 
> > = 113 GiB + 2 * 32 GiB + 2 * 32 MiB, i.e. what amount of the size of the 
> > device is allocated for chunks.
> > 
> > The value one line above is what is allocated inside the chunks.
> > 
> > I.e. the line in "devid 1" is "total" of btrfs fi df summed up, and the line 
> > above is "used" in btrfs fi df summed up. And… with more devices you have more 
> > fun.
> 
> Why wouldn't btrfs allocate more data chunks from the ample free space?

   It's a bug. It's been around for years (literally), but nobody's
tracked it down and fixed it yet.

   Hugo.

-- 
Hugo Mills             | We believe in free will because we have no choice.
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Marc Haber]

On Sun, Feb 28, 2016 at 12:22:45AM +0000, Hugo Mills wrote:
> On Sun, Feb 28, 2016 at 01:08:29AM +0100, Marc Haber wrote:
> > Why wouldn't btrfs allocate more data chunks from the ample free space?
> 
>    It's a bug. It's been around for years (literally), but nobody's
> tracked it down and fixed it yet.

Is there a fix/workaround?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Qu Wenruo]

Marc Haber wrote on 2016/02/27 22:14 +0100:
> Hi,
>
> I have again the issue of no space left on device while rebalancing
> (with btrfs-tools 4.4.1 on kernel 4.4.2 on Debian unstable):
>
> mh@fan:~$ sudo btrfs balance start /mnt/fanbtr
> ERROR: error during balancing '/mnt/fanbtr': No space left on device

It seems that, only when balancing all chunks, ENOSPC error happens.

And did you run any other heavy IO at background?

BTW, is there any kernel log when the ENOSPC happens?

> mh@fan:~$ sudo btrfs fi show /mnt/fanbtr
> mh@fan:~$ sudo btrfs fi show -m
> Label: 'fanbtr'  uuid: 4198d1bc-e3ce-40df-a7ee-44a2d120bff3
>          Total devices 1 FS bytes used 116.49GiB
>          devid    1 size 417.19GiB used 177.06GiB path /dev/mapper/fanbtr
> mh@fan:~$ sudo btrfs fi df /mnt/fanbtr
> Data, single: total=113.00GiB, used=112.77GiB
> System, DUP: total=32.00MiB, used=48.00KiB
> Metadata, DUP: total=32.00GiB, used=3.72GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> mh@fan:~$
>
> The filesystem was recently resized from 300 GB to 420 GB.
>
> Why does btrfs fi show /mnt/fanbtr not give any output? Wy does btrfs
> fi df /mnt/fanbtr say that my data space is only 113 GiB large?
>
> btrfs balance start -dusage=5 works up to -dusage=100:
>
> mh@fan:~$ sudo btrfs balance start -dusage=100 /mnt/fanbtr
> Done, had to relocate 111 out of 179 chunks
> mh@fan:~$ sudo btrfs balance start -dusage=100 /mnt/fanbtr
> Done, had to relocate 111 out of 179 chunks
> mh@fan:~$ sudo btrfs balance start -dusage=100 /mnt/fanbtr
> Done, had to relocate 110 out of 179 chunks
> mh@fan:~$ sudo btrfs balance start -dusage=100 /mnt/fanbtr
> Done, had to relocate 109 out of 179 chunks
> mh@fan:~$ sudo btrfs balance start /mnt/fanbtr
> ERROR: error during balancing '/mnt/fanbtr': No space left on device
> mh@fan:~$

Would you please try the following commands to see which one caused the 
problem?
And would you please provide the dmesg of them?

# btrfs balance start -dprofiles=single /mnt/fanbtr
# btrfs balance start -mprofile=dup /mnt/fanbtr
# btrfs balance start -sprofile=dup /mnt/fanbtr

The above three commands should be equal to your "btrfs balance start" 
command without any parameter, but do them separately, so it should tell 
you which chunk type caused the ENOSPC error.

 From your above -dusage=100 parameter, I think data should be OK, and 
since the metadata chunk is larger than 1G, and there are a lot of free 
metadata space, I assume it's the single system chunk causing the problem.

And if my assumption is right, you have nothing to worry as data and 
metadata should be OK to allocate new chunks, and 32M system chunks is 
definitely large enough.
But in that case, we should have a clearer clue to trace and fix the bug.

Thanks,
Qu

>
> What is going on here? How do I get away from here?
>
> Greetings
> Marc
>
>

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Marc Haber]

Hi,

On Mon, Feb 29, 2016 at 09:56:58AM +0800, Qu Wenruo wrote:
> Marc Haber wrote on 2016/02/27 22:14 +0100:
> >I have again the issue of no space left on device while rebalancing
> >(with btrfs-tools 4.4.1 on kernel 4.4.2 on Debian unstable):
> >
> >mh@fan:~$ sudo btrfs balance start /mnt/fanbtr
> >ERROR: error during balancing '/mnt/fanbtr': No space left on device
> 
> It seems that, only when balancing all chunks, ENOSPC error happens.
> 
> And did you run any other heavy IO at background?

Not when running those last commands for the mailing list post.

> BTW, is there any kernel log when the ENOSPC happens?

> Would you please try the following commands to see which one caused the
> problem?
> And would you please provide the dmesg of them?
> 
> # btrfs balance start -dprofiles=single /mnt/fanbtr
> # btrfs balance start -mprofile=dup /mnt/fanbtr
> # btrfs balance start -sprofile=dup /mnt/fanbtr

I have attached the logs. I used logger(1) to have in syslog which
command I executed, and I have piped the userspace's output to logger
so that the syslog entries match the userspace output.

-mprofile gave an error message, I therefore tried -mprofiles, and
-sprofiles wanted me to use the --force, so I did that as well.

The three balance commands above all three finshed alright without
running into ENOSPC, while running a plain balance (which is also part
of the log) errors out every time.

And, the -dprofiles=single log caused a number of INFOs regarding
btrfs-cleaner and btrfa-balance processes gotten stuck for more than
120 seconds during the run.

I now have a kworker and a btfs-transact kernel process taking most of
one CPU core each, even after the userspace programs have terminated.
Is there a way to find out what these threads are actually doing?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Qu Wenruo]

Marc Haber wrote on 2016/02/29 16:33 +0100:
> Hi,
>
> On Mon, Feb 29, 2016 at 09:56:58AM +0800, Qu Wenruo wrote:
>> Marc Haber wrote on 2016/02/27 22:14 +0100:
>>> I have again the issue of no space left on device while rebalancing
>>> (with btrfs-tools 4.4.1 on kernel 4.4.2 on Debian unstable):
>>>
>>> mh@fan:~$ sudo btrfs balance start /mnt/fanbtr
>>> ERROR: error during balancing '/mnt/fanbtr': No space left on device
>>
>> It seems that, only when balancing all chunks, ENOSPC error happens.
>>
>> And did you run any other heavy IO at background?
>
> Not when running those last commands for the mailing list post.
>
>> BTW, is there any kernel log when the ENOSPC happens?
>
>> Would you please try the following commands to see which one caused the
>> problem?
>> And would you please provide the dmesg of them?
>>
>> # btrfs balance start -dprofiles=single /mnt/fanbtr
>> # btrfs balance start -mprofile=dup /mnt/fanbtr
>> # btrfs balance start -sprofile=dup /mnt/fanbtr
>
> I have attached the logs. I used logger(1) to have in syslog which
> command I executed, and I have piped the userspace's output to logger
> so that the syslog entries match the userspace output.

Didn't see the attachment though, seems to be filtered by maillist police.

>
> -mprofile gave an error message, I therefore tried -mprofiles, and
> -sprofiles wanted me to use the --force, so I did that as well.
>
> The three balance commands above all three finshed alright without
> running into ENOSPC, while running a plain balance (which is also part
> of the log) errors out every time.

Strange, but at least some small clue to chase.

>
> And, the -dprofiles=single log caused a number of INFOs regarding
> btrfs-cleaner and btrfa-balance processes gotten stuck for more than
> 120 seconds during the run.

That's not normal and it would be a bug.

>
> I now have a kworker and a btfs-transact kernel process taking most of
> one CPU core each, even after the userspace programs have terminated.
> Is there a way to find out what these threads are actually doing?

Did btrfs balance status gives any hint?

Thanks,
Qu

>
> Greetings
> Marc
>

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Qu Wenruo]

[-- Attachment #1: Type: text/plain, Size: 3784 bytes --]



Marc Haber wrote on 2016/03/01 07:54 +0100:
> On Tue, Mar 01, 2016 at 08:45:21AM +0800, Qu Wenruo wrote:
>> Didn't see the attachment though, seems to be filtered by maillist police.
>
> Trying again.

OK, I got the attachment.

And, surprisingly, btrfs balance on data chunk works without problem, 
but it fails on plain btrfs balance command.

>
>>> I now have a kworker and a btfs-transact kernel process taking most of
>>> one CPU core each, even after the userspace programs have terminated.
>>> Is there a way to find out what these threads are actually doing?
>>
>> Did btrfs balance status gives any hint?
>
> It says 'No balance found on /mnt/fanbtr'. I do have a second btrfs on
> the box, which is acting up as well (it has a five digit number of
> snapshots, and deleting a single snapshot takes about five to ten
> minutes. I was planning to write another mailing list article once
> this balance issue is through).

I assume the large number of snapshots is related to the high CPU usage.
As so many snapshots will make btrfs take so much time to calculate its 
backref, and the backtrace seems to prove that.

I'd like to remove unused snapshots and keep the number of them to 4 
digits, as a workaround.

But still not sure if it's related to the ENOSPC problem.

It would provide great help if you can modify your kernel and add the 
following debug: (same as attachment)

------
 From f2cc7af0aea659a522b97d3776b719f14532bce9 Mon Sep 17 00:00:00 2001
From: Qu Wenruo <quwenruo@cn.fujitsu.com>
Date: Tue, 1 Mar 2016 15:21:18 +0800
Subject: [PATCH] btrfs: debug patch

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
  fs/btrfs/extent-tree.c | 15 +++++++++++++--
  1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 083783b..70b284b 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -9393,8 +9393,10 @@ int btrfs_can_relocate(struct btrfs_root *root, 
u64 bytenr)
  	block_group = btrfs_lookup_block_group(root->fs_info, bytenr);

  	/* odd, couldn't find the block group, leave it alone */
-	if (!block_group)
+	if (!block_group) {
+		pr_info("no such chunk: %llu\n", bytenr);
  		return -1;
+	}

  	min_free = btrfs_block_group_used(&block_group->item);

@@ -9419,6 +9421,11 @@ int btrfs_can_relocate(struct btrfs_root *root, 
u64 bytenr)
  	     space_info->bytes_pinned + space_info->bytes_readonly +
  	     min_free < space_info->total_bytes)) {
  		spin_unlock(&space_info->lock);
+		pr_info("no space: total:%llu, bg_len:%llu, used:%llu, reseved:%llu, 
pinned:%llu, ro:%llu, min_free:%llu\n",
+			space_info->total_bytes, block_group->key.offset,
+			space_info->bytes_used, space_info->bytes_reserved,
+			space_info->bytes_pinned, space_info->bytes_readonly,
+			min_free);
  		goto out;
  	}
  	spin_unlock(&space_info->lock);
@@ -9448,8 +9455,10 @@ int btrfs_can_relocate(struct btrfs_root *root, 
u64 bytenr)
  		 * this is just a balance, so if we were marked as full
  		 * we know there is no space for a new chunk
  		 */
-		if (full)
+		if (full) {
+			pr_info("space full\n");
  			goto out;
+		}

  		index = get_block_group_index(block_group);
  	}
@@ -9496,6 +9505,8 @@ int btrfs_can_relocate(struct btrfs_root *root, 
u64 bytenr)
  			ret = -1;
  		}
  	}
+	if (ret == -1)
+		pr_info("no new chunk allocatable\n");
  	mutex_unlock(&root->fs_info->chunk_mutex);
  	btrfs_end_transaction(trans, root);
  out:
-- 
2.7.2

------

Thanks,
Qu

>
> So I do not even know which filesystem is making two processes run in
> circles. I have noticed that the "btrfs-transact" process is still the
> same that started 24 hours ago, while the "kworker/u16:10" process
> occasionally gets replaced by a new one which runs in circles as well.
>
> Greetings
> Marc
>



[-- Attachment #2: 0001-btrfs-debug-patch.patch --]
[-- Type: text/x-patch, Size: 2026 bytes --]

>From f2cc7af0aea659a522b97d3776b719f14532bce9 Mon Sep 17 00:00:00 2001
From: Qu Wenruo <quwenruo@cn.fujitsu.com>
Date: Tue, 1 Mar 2016 15:21:18 +0800
Subject: [PATCH] btrfs: debug patch

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 fs/btrfs/extent-tree.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 083783b..70b284b 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -9393,8 +9393,10 @@ int btrfs_can_relocate(struct btrfs_root *root, u64 bytenr)
 	block_group = btrfs_lookup_block_group(root->fs_info, bytenr);
 
 	/* odd, couldn't find the block group, leave it alone */
-	if (!block_group)
+	if (!block_group) {
+		pr_info("no such chunk: %llu\n", bytenr);
 		return -1;
+	}
 
 	min_free = btrfs_block_group_used(&block_group->item);
 
@@ -9419,6 +9421,11 @@ int btrfs_can_relocate(struct btrfs_root *root, u64 bytenr)
 	     space_info->bytes_pinned + space_info->bytes_readonly +
 	     min_free < space_info->total_bytes)) {
 		spin_unlock(&space_info->lock);
+		pr_info("no space: total:%llu, bg_len:%llu, used:%llu, reseved:%llu, pinned:%llu, ro:%llu, min_free:%llu\n",
+			space_info->total_bytes, block_group->key.offset,
+			space_info->bytes_used, space_info->bytes_reserved,
+			space_info->bytes_pinned, space_info->bytes_readonly,
+			min_free);
 		goto out;
 	}
 	spin_unlock(&space_info->lock);
@@ -9448,8 +9455,10 @@ int btrfs_can_relocate(struct btrfs_root *root, u64 bytenr)
 		 * this is just a balance, so if we were marked as full
 		 * we know there is no space for a new chunk
 		 */
-		if (full)
+		if (full) {
+			pr_info("space full\n");
 			goto out;
+		}
 
 		index = get_block_group_index(block_group);
 	}
@@ -9496,6 +9505,8 @@ int btrfs_can_relocate(struct btrfs_root *root, u64 bytenr)
 			ret = -1;
 		}
 	}
+	if (ret == -1)
+		pr_info("no new chunk allocatable\n");
 	mutex_unlock(&root->fs_info->chunk_mutex);
 	btrfs_end_transaction(trans, root);
 out:
-- 
2.7.2

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Qu Wenruo]

Qu Wenruo wrote on 2016/03/01 15:24 +0800:
>
>
> Marc Haber wrote on 2016/03/01 07:54 +0100:
>> On Tue, Mar 01, 2016 at 08:45:21AM +0800, Qu Wenruo wrote:
>>> Didn't see the attachment though, seems to be filtered by maillist
>>> police.
>>
>> Trying again.
>
> OK, I got the attachment.
>
> And, surprisingly, btrfs balance on data chunk works without problem,
> but it fails on plain btrfs balance command.
>
>>
>>>> I now have a kworker and a btfs-transact kernel process taking most of
>>>> one CPU core each, even after the userspace programs have terminated.
>>>> Is there a way to find out what these threads are actually doing?
>>>
>>> Did btrfs balance status gives any hint?
>>
>> It says 'No balance found on /mnt/fanbtr'. I do have a second btrfs on
>> the box, which is acting up as well (it has a five digit number of
>> snapshots, and deleting a single snapshot takes about five to ten
>> minutes. I was planning to write another mailing list article once
>> this balance issue is through).
>
> I assume the large number of snapshots is related to the high CPU usage.
> As so many snapshots will make btrfs take so much time to calculate its
> backref, and the backtrace seems to prove that.
>
> I'd like to remove unused snapshots and keep the number of them to 4
> digits, as a workaround.
>
> But still not sure if it's related to the ENOSPC problem.
>
> It would provide great help if you can modify your kernel and add the
> following debug: (same as attachment)
>
> ------
>  From f2cc7af0aea659a522b97d3776b719f14532bce9 Mon Sep 17 00:00:00 2001
> From: Qu Wenruo <quwenruo@cn.fujitsu.com>
> Date: Tue, 1 Mar 2016 15:21:18 +0800
> Subject: [PATCH] btrfs: debug patch
>
> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
> ---
>   fs/btrfs/extent-tree.c | 15 +++++++++++++--
>   1 file changed, 13 insertions(+), 2 deletions(-)
>
> diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
> index 083783b..70b284b 100644
> --- a/fs/btrfs/extent-tree.c
> +++ b/fs/btrfs/extent-tree.c
> @@ -9393,8 +9393,10 @@ int btrfs_can_relocate(struct btrfs_root *root,
> u64 bytenr)
>       block_group = btrfs_lookup_block_group(root->fs_info, bytenr);
>
>       /* odd, couldn't find the block group, leave it alone */
> -    if (!block_group)
> +    if (!block_group) {
> +        pr_info("no such chunk: %llu\n", bytenr);
>           return -1;
> +    }
>
>       min_free = btrfs_block_group_used(&block_group->item);
>
> @@ -9419,6 +9421,11 @@ int btrfs_can_relocate(struct btrfs_root *root,
> u64 bytenr)
>            space_info->bytes_pinned + space_info->bytes_readonly +
>            min_free < space_info->total_bytes)) {
>           spin_unlock(&space_info->lock);
> +        pr_info("no space: total:%llu, bg_len:%llu, used:%llu,
> reseved:%llu, pinned:%llu, ro:%llu, min_free:%llu\n",
> +            space_info->total_bytes, block_group->key.offset,
> +            space_info->bytes_used, space_info->bytes_reserved,
> +            space_info->bytes_pinned, space_info->bytes_readonly,
> +            min_free);
Oh, I'm sorry that the output is not necessary, it's better to use the 
newer patch:
https://patchwork.kernel.org/patch/8462881/

With the newer patch, you will need to use enospc_debug mount option to 
get the debug information.

Sorry for the inconvenience.

Thanks,
Qu

>           goto out;
>       }
>       spin_unlock(&space_info->lock);
> @@ -9448,8 +9455,10 @@ int btrfs_can_relocate(struct btrfs_root *root,
> u64 bytenr)
>            * this is just a balance, so if we were marked as full
>            * we know there is no space for a new chunk
>            */
> -        if (full)
> +        if (full) {
> +            pr_info("space full\n");
>               goto out;
> +        }
>
>           index = get_block_group_index(block_group);
>       }
> @@ -9496,6 +9505,8 @@ int btrfs_can_relocate(struct btrfs_root *root,
> u64 bytenr)
>               ret = -1;
>           }
>       }
> +    if (ret == -1)
> +        pr_info("no new chunk allocatable\n");
>       mutex_unlock(&root->fs_info->chunk_mutex);
>       btrfs_end_transaction(trans, root);
>   out:

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Qu Wenruo]

Thanks for the output.

At least for mprofile enospc error, the problem itself is very 
straightforward, just unable to alloc tree block.

I'll check the codes to see if we can improve it, by finding out why we 
can't alloc a new chunk to resolve the problem.

But I'm still a little concerned about the dprofile case, as this time, 
dprofile doesn't trigger a ENOSPC and my debug info is not triggered.

Thanks,
Qu

Marc Haber wrote on 2016/03/01 17:16 +0100:
> Hi,
>
> On Tue, Mar 01, 2016 at 04:13:35PM +0800, Qu Wenruo wrote:
>> Oh, I'm sorry that the output is not necessary, it's better to use the newer
>> patch:
>> https://patchwork.kernel.org/patch/8462881/
>>
>> With the newer patch, you will need to use enospc_debug mount option to get
>> the debug information.
>
> I'll copy the log this time in the body of the message so that it gets
> through to the list. Let me know if you'd prefer an attachment.
>
> This time, I didn't see the busy kernel threads, and I now see ENOSPC
> during the mprofiles part of the manual balance.
>
> Hope this helps.
>
> Greetings
> Marc
>

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Duncan]

Qu Wenruo posted on Tue, 01 Mar 2016 15:24:03 +0800 as excerpted:


> 
> Marc Haber wrote on 2016/03/01 07:54 +0100:
>> On Tue, Mar 01, 2016 at 08:45:21AM +0800, Qu Wenruo wrote:
>>> Didn't see the attachment though, seems to be filtered by maillist
>>> police.
>>
>> Trying again.
> 
> OK, I got the attachment.
> 
> And, surprisingly, btrfs balance on data chunk works without problem,
> but it fails on plain btrfs balance command.

There has been something bothering me about this thread that I wasn't 
quite pinning down, but here it is.

If you look at the btrfs fi df/usage numbers, data chunk total vs. used 
are very close to one another (113 GiB total, 112.77 GiB used, single 
profile, assuming GiB data chunks, that's only a fraction of a single 
data chunk unused), so balance would seem to be getting thru them just 
fine.

But there's a /huge/ spread between total vs. used metadata (32 GiB 
total, under 4 GiB used, clearly _many_ empty or nearly empty chunks), 
implying that has not been successfully balanced in quite some time, if 
ever.  So I'd surmise the problem is in metadata, not in data.

Which would explain why balancing data works fine, but a whole-filesystem 
balance doesn't, because it's getting stuck on the metadata, not the data.

Now the balance metadata filters include system as well, by default, and 
the -mprofiles=dup and -sprofiles=dup balances finished, apparently 
without error, which throws a wrench into my theory.

But while we have the btrfs fi df from before the attempt with the 
profiles filters, we don't have the same output from after.

If btrfs fi df still shows more than a GiB spread between metadata total 
and used /after/ the supposedly successful profiles filter runs, then 
obviously they're not balancing what they should be balancing, a bug 
right there, which an educated guess suggests if it's fixed, the metadata 
and possibly system balances will likely fail, due to whatever problem on 
the filesystem is keeping the full balance from completing as well.

Of course, if the post-filtered-balance btrfs fi df shows a metadata 
spread of under a gig (given 256 MiB metadata chunks, but dup, so 
possibly nearly a half-gig free, and the 512 MiB global reserve counts as 
unused metadata as well, adding another half-gig that's going to be 
reported free but is actually accounted for, yielding a spread of upto a 
gig even after successful balance), then the problem is elsewhere, but 
I'm guessing it's still going to be well over a gig and may still be the 
full 28+ gig spread, 32 gig total, under 4 gig used, indicating the 
metadata filtered balance actually didn't actually work at all.

Meanwhile, the metadata filters also include system, so while it's 
possible to balance system specifically, without (other) metadata, to my 
knowledge it's impossible to balance (other) metadata exclusively, 
without balancing system.

Which, now assuming we still have that huge metadata spread, means if the 
on-filesystem bug is in the system chunks, both system and metadata 
filtered balances *should* fail, while if it's in non-system metadata, a 
system filtered balance *should* succeed, while a metadata filtered 
balance *should* fail.

>>>> I now have a kworker and a btfs-transact kernel process taking most
>>>> of one CPU core each, even after the userspace programs have
>>>> terminated. Is there a way to find out what these threads are
>>>> actually doing?
>>>
>>> Did btrfs balance status gives any hint?
>>
>> It says 'No balance found on /mnt/fanbtr'. I do have a second btrfs on
>> the box, which is acting up as well (it has a five digit number of
>> snapshots, and deleting a single snapshot takes about five to ten
>> minutes. I was planning to write another mailing list article once this
>> balance issue is through).
> 
> I assume the large number of snapshots is related to the high CPU usage.
> As so many snapshots will make btrfs take so much time to calculate its
> backref, and the backtrace seems to prove that.
> 
> I'd like to remove unused snapshots and keep the number of them to 4
> digits, as a workaround.


I'll strongly second that recommendation.  Btrfs is known to have 
snapshot scaling issues at 10K snapshots and above.  My strong 
recommendation is to limit snapshots per filesystem to 3000 or less, with 
a target of 2000 per filesystem or less if possible, and an ideal of 1000 
per filesystem or less if it's practical to keep it to that, which it 
should be with thinning, if you're only snapshotting 1-2 subvolumes, but 
may not be if you're snapshotting more.

You can actually do scheduled snapshotting on a pretty tight schedule, 
say twice or 3X per hour (every 20-30 minutes), provided you have a good 
snapshot thinning program in place as well, thinning to say a snapshot an 
hour after 2-12 hours, every other hour after say 25 hours (giving you a 
bit over a day of at least hourly), every six hours after 8 days (so you 
have over a week of every other hour), twice a day after a couple weeks, 
daily after four weeks, weekly after 90 days, by which time you should 
have an off-system backup available to fall back on as well if you're so 
concerned about snapshots, such that after six months or a year you can 
delete all snapshots and finally free the space taken by the old 
snapshots.

Having posted the same suggestion and done the math multiple times, 
that's 250-500 snapshots per subvolume depending primarily on how fast 
you thin down in the early stages, which means 2-4 subvolumes of 
snapshotting per thousand snapshots total per filesystem, which means 
with a strict enough thinning program, you can snapshot upto 8 subvolumes 
per filesystem and stay under the 2000 total snapshots target.

By 3000 snapshots per filesystem, you'll be beginning to notice slowdowns 
in some btrfs maintenance commands if you're sensitive to it, tho it's 
still at least practical to work with, and by 10K, it's generally 
noticeable by all, at least once they thin down to 2K or so, as it's 
suddenly faster again!  Above 100K, some btrfs maintenance commands slow 
to a crawl and doing that sort of maintenance really becomes impractical 
enough that it's generally easier to backup what you need to and blow 
away the filesystem to start again with a new one, than it is to try to 
recover the existing filesystem to a workable state, given that 
maintenance can at that point take days to weeks.

So 5-digits of snapshots on a filesystem is definitely well outside of 
the recommended range, to the point that in some cases, particularly 
approaching 6-digits of snapshots, it'll be more practical to simply 
ditch the filesystem and start over, than to try to work with it any 
longer.  Just don't do it; setup your thinning schedule so your peak is 
3000 snapshots per filesystem or under, and you won't have that problem 
to worry about. =:^)

Oh, and btrfs quota management exacerbates the scaling issues 
dramatically.  If you're using btrfs quotas, either half the max 
snapshots per filesystem recommendations, or reconsider whether you need 
quota functionality and turn it off, eliminating the existing quota data, 
if you don't really need that functionality. =:^(

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Marc Haber]

Hi,

I have not seen this message coming back to the mailing list. Was it
again too long?

I have pastebinned the log at http://paste.debian.net/412118/

On Tue, Mar 01, 2016 at 08:51:32PM +0000, Duncan wrote:
> There has been something bothering me about this thread that I wasn't 
> quite pinning down, but here it is.
> 
> If you look at the btrfs fi df/usage numbers, data chunk total vs. used 
> are very close to one another (113 GiB total, 112.77 GiB used, single 
> profile, assuming GiB data chunks, that's only a fraction of a single 
> data chunk unused), so balance would seem to be getting thru them just 
> fine.

Where would you see those numbers? I have those, pre-balance:

Mar  2 20:28:01 fan root: Data, single: total=77.00GiB, used=76.35GiB
Mar  2 20:28:01 fan root: System, DUP: total=32.00MiB, used=48.00KiB
Mar  2 20:28:01 fan root: Metadata, DUP: total=86.50GiB, used=2.11GiB
Mar  2 20:28:01 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B

> But there's a /huge/ spread between total vs. used metadata (32 GiB 
> total, under 4 GiB used, clearly _many_ empty or nearly empty chunks), 
> implying that has not been successfully balanced in quite some time, if 
> ever.

This is possible, yes.

>   So I'd surmise the problem is in metadata, not in data.
> 
> Which would explain why balancing data works fine, but a whole-filesystem 
> balance doesn't, because it's getting stuck on the metadata, not the data.
> 
> Now the balance metadata filters include system as well, by default, and 
> the -mprofiles=dup and -sprofiles=dup balances finished, apparently 
> without error, which throws a wrench into my theory.

Also finishes without changing things, post-balance:
Mar  2 21:55:37 fan root: Data, single: total=77.00GiB, used=76.36GiB
Mar  2 21:55:37 fan root: System, DUP: total=32.00MiB, used=80.00KiB
Mar  2 21:55:37 fan root: Metadata, DUP: total=99.00GiB, used=2.11GiB
Mar  2 21:55:37 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B

Wait, Metadata used actually _grew_???

> But while we have the btrfs fi df from before the attempt with the 
> profiles filters, we don't have the same output from after.
s
We now have everything. New log attached.

> > I'd like to remove unused snapshots and keep the number of them to 4
> > digits, as a workaround.
> 
> I'll strongly second that recommendation.  Btrfs is known to have 
> snapshot scaling issues at 10K snapshots and above.  My strong 
> recommendation is to limit snapshots per filesystem to 3000 or less, with 
> a target of 2000 per filesystem or less if possible, and an ideal of 1000 
> per filesystem or less if it's practical to keep it to that, which it 
> should be with thinning, if you're only snapshotting 1-2 subvolumes, but 
> may not be if you're snapshotting more.

I'm snapshotting /home every 10 minutes, the filesystem that I have
been posting logs from has about 400 snapshots, and snapshot cleanup
works fine. The slow snapshot removal is a different filesystem on the
same host which is on a rotating rust HDD, and is much bigger.

> By 3000 snapshots per filesystem, you'll be beginning to notice slowdowns 
> in some btrfs maintenance commands if you're sensitive to it, tho it's 
> still at least practical to work with, and by 10K, it's generally 
> noticeable by all, at least once they thin down to 2K or so, as it's 
> suddenly faster again!  Above 100K, some btrfs maintenance commands slow 
> to a crawl and doing that sort of maintenance really becomes impractical 
> enough that it's generally easier to backup what you need to and blow 
> away the filesystem to start again with a new one, than it is to try to 
> recover the existing filesystem to a workable state, given that 
> maintenance can at that point take days to weeks.

Ouch. This shold not be the case, or btrfs subvolume snapshot should
at least emit a warning. It is not good that it is so easy to get a
filesystem into a state this bad.

> So 5-digits of snapshots on a filesystem is definitely well outside of 
> the recommended range, to the point that in some cases, particularly 
> approaching 6-digits of snapshots, it'll be more practical to simply 
> ditch the filesystem and start over, than to try to work with it any 
> longer.  Just don't do it; setup your thinning schedule so your peak is 
> 3000 snapshots per filesystem or under, and you won't have that problem 
> to worry about. =:^)

That needs to be documented prominently. Ths ZFS fanbois will love that.

> Oh, and btrfs quota management exacerbates the scaling issues 
> dramatically.  If you're using btrfs quotas

Am not, thankfully.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Dāvis Mosāns]

2016-02-27 23:14 GMT+02:00 Marc Haber <mh+linux-btrfs@zugschlus.de>:
> Hi,
>
> I have again the issue of no space left on device while rebalancing
> (with btrfs-tools 4.4.1 on kernel 4.4.2 on Debian unstable):
>

I've same issue, 4.4.3 kernel on Arch Linux

$ sudo btrfs fi show /mnt/fs/
Label: 'fs'  uuid: a3c66d25-2c25-40e5-a827-5f7e5208e235
        Total devices 1 FS bytes used 396.94GiB
        devid    1 size 435.76GiB used 435.76GiB path /dev/sdi2

$ sudo btrfs fi df /mnt/fs/
Data, single: total=416.70GiB, used=390.62GiB
System, DUP: total=32.00MiB, used=96.00KiB
Metadata, DUP: total=9.50GiB, used=6.32GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

$ sudo btrfs fi usage /mnt/fs/
Overall:
    Device size:                 435.76GiB
    Device allocated:            435.76GiB
    Device unallocated:            1.00MiB
    Device missing:                  0.00B
    Used:                        403.26GiB
    Free (estimated):             26.07GiB      (min: 26.07GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)

Data,single: Size:416.70GiB, Used:390.62GiB
   /dev/sdi2     416.70GiB

Metadata,DUP: Size:9.50GiB, Used:6.32GiB
   /dev/sdi2      19.00GiB

System,DUP: Size:32.00MiB, Used:96.00KiB
   /dev/sdi2      64.00MiB

Unallocated:
   /dev/sdi2       1.00MiB


$ sudo btrfs balance start -v /mnt/fs/
Dumping filters: flags 0x7, state 0x0, force is off
  DATA (flags 0x0): balancing
  METADATA (flags 0x0): balancing
  SYSTEM (flags 0x0): balancing
ERROR: error during balancing '/mnt/fs/': No space left on device
There may be more info in syslog - try dmesg | tail

$ sudo btrfs balance start -dusage=34 /mnt/fs/
Done, had to relocate 0 out of 438 chunks
$ sudo btrfs balance start -dusage=35 /mnt/fs/
ERROR: error during balancing '/mnt/fs/': No space left on device


$ sudo btrfs balance start -musage=0 /mnt/fs/
Done, had to relocate 0 out of 438 chunks
$ sudo btrfs balance start -musage=1 /mnt/fs/
ERROR: error during balancing '/mnt/fs/': No space left on device


$ sudo btrfs balance start -dprofiles=single /mnt/fs
ERROR: error during balancing '/mnt/fs': No space left on device

$ sudo btrfs balance start -mprofiles=dup /mnt/fs
Done, had to relocate 20 out of 438 chunks

$ sudo btrfs balance start -sprofiles=dup --force /mnt/fs
Done, had to relocate 1 out of 433 chunks


$ sudo btrfs fi df /mnt/fs/
Data, single: total=416.70GiB, used=390.62GiB
System, DUP: total=32.00MiB, used=96.00KiB
Metadata, DUP: total=7.00GiB, used=6.31GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

$  sudo btrfs fi usage /mnt/fs/
Overall:
    Device size:                 435.76GiB
    Device allocated:            430.76GiB
    Device unallocated:            5.00GiB
    Device missing:                  0.00B
    Used:                        403.25GiB
    Free (estimated):             31.07GiB      (min: 28.57GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)

Data,single: Size:416.70GiB, Used:390.62GiB
   /dev/sdi2     416.70GiB

Metadata,DUP: Size:7.00GiB, Used:6.31GiB
   /dev/sdi2      14.00GiB

System,DUP: Size:32.00MiB, Used:96.00KiB
   /dev/sdi2      64.00MiB

Unallocated:
   /dev/sdi2       5.00GiB

now balance works

$ sudo btrfs balance start -m /mnt/fs/
Done, had to relocate 15 out of 433 chunks

$ sudo btrfs balance start /mnt/fs/

it's still going but most likley it will finish

$ sudo btrfs balance status /mnt/fs/
Balance on '/mnt/fs/' is running
26 out of about 431 chunks balanced (27 considered),  94% left

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Qu Wenruo]

Dāvis Mosāns wrote on 2016/03/03 02:28 +0200:
> 2016-02-27 23:14 GMT+02:00 Marc Haber <mh+linux-btrfs@zugschlus.de>:
>> Hi,
>>
>> I have again the issue of no space left on device while rebalancing
>> (with btrfs-tools 4.4.1 on kernel 4.4.2 on Debian unstable):
>>
>
> I've same issue, 4.4.3 kernel on Arch Linux
>
> $ sudo btrfs fi show /mnt/fs/
> Label: 'fs'  uuid: a3c66d25-2c25-40e5-a827-5f7e5208e235
>          Total devices 1 FS bytes used 396.94GiB
>          devid    1 size 435.76GiB used 435.76GiB path /dev/sdi2
>
> $ sudo btrfs fi df /mnt/fs/
> Data, single: total=416.70GiB, used=390.62GiB
> System, DUP: total=32.00MiB, used=96.00KiB
> Metadata, DUP: total=9.50GiB, used=6.32GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
>
> $ sudo btrfs fi usage /mnt/fs/
> Overall:
>      Device size:                 435.76GiB
>      Device allocated:            435.76GiB
>      Device unallocated:            1.00MiB

Not quite the same with the problem with OP.
In your case, only 1M is not allocated, which is below minimal chunk 
size 16M.

So, you're really out of unallocated space.
>      Device missing:                  0.00B
>      Used:                        403.26GiB
>      Free (estimated):             26.07GiB      (min: 26.07GiB)
>      Data ratio:                       1.00
>      Metadata ratio:                   2.00
>      Global reserve:              512.00MiB      (used: 0.00B)
>
> Data,single: Size:416.70GiB, Used:390.62GiB
>     /dev/sdi2     416.70GiB
>
> Metadata,DUP: Size:9.50GiB, Used:6.32GiB
>     /dev/sdi2      19.00GiB

But considering your space usage, I didn't see any need to balance though.

You have over 3G metadata space, and about 20G data.
No obvious meta/data unbalance.

>
> System,DUP: Size:32.00MiB, Used:96.00KiB
>     /dev/sdi2      64.00MiB
>
> Unallocated:
>     /dev/sdi2       1.00MiB
>
>
> $ sudo btrfs balance start -v /mnt/fs/
> Dumping filters: flags 0x7, state 0x0, force is off
>    DATA (flags 0x0): balancing
>    METADATA (flags 0x0): balancing
>    SYSTEM (flags 0x0): balancing
> ERROR: error during balancing '/mnt/fs/': No space left on device
> There may be more info in syslog - try dmesg | tail
>
> $ sudo btrfs balance start -dusage=34 /mnt/fs/
> Done, had to relocate 0 out of 438 chunks
> $ sudo btrfs balance start -dusage=35 /mnt/fs/
> ERROR: error during balancing '/mnt/fs/': No space left on device
>
>
> $ sudo btrfs balance start -musage=0 /mnt/fs/
> Done, had to relocate 0 out of 438 chunks
> $ sudo btrfs balance start -musage=1 /mnt/fs/
> ERROR: error during balancing '/mnt/fs/': No space left on device
>
>
> $ sudo btrfs balance start -dprofiles=single /mnt/fs
> ERROR: error during balancing '/mnt/fs': No space left on device
>
> $ sudo btrfs balance start -mprofiles=dup /mnt/fs
> Done, had to relocate 20 out of 438 chunks
>
> $ sudo btrfs balance start -sprofiles=dup --force /mnt/fs
> Done, had to relocate 1 out of 433 chunks
>
>
> $ sudo btrfs fi df /mnt/fs/
> Data, single: total=416.70GiB, used=390.62GiB
> System, DUP: total=32.00MiB, used=96.00KiB
> Metadata, DUP: total=7.00GiB, used=6.31GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
>
> $  sudo btrfs fi usage /mnt/fs/
> Overall:
>      Device size:                 435.76GiB
>      Device allocated:            430.76GiB
>      Device unallocated:            5.00GiB

After balancing data and metadata, you have space for at least 5 chunks.
So you should be OK to continue balance.

>      Device missing:                  0.00B
>      Used:                        403.25GiB
>      Free (estimated):             31.07GiB      (min: 28.57GiB)
>      Data ratio:                       1.00
>      Metadata ratio:                   2.00
>      Global reserve:              512.00MiB      (used: 0.00B)
>
> Data,single: Size:416.70GiB, Used:390.62GiB
>     /dev/sdi2     416.70GiB
>
> Metadata,DUP: Size:7.00GiB, Used:6.31GiB
>     /dev/sdi2      14.00GiB
>
> System,DUP: Size:32.00MiB, Used:96.00KiB
>     /dev/sdi2      64.00MiB
>
> Unallocated:
>     /dev/sdi2       5.00GiB
>
> now balance works
>
> $ sudo btrfs balance start -m /mnt/fs/
> Done, had to relocate 15 out of 433 chunks
>
> $ sudo btrfs balance start /mnt/fs/
>
> it's still going but most likley it will finish
>
> $ sudo btrfs balance status /mnt/fs/
> Balance on '/mnt/fs/' is running
> 26 out of about 431 chunks balanced (27 considered),  94% left
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Duncan]

Dāvis Mosāns posted on Thu, 03 Mar 2016 02:28:36 +0200 as excerpted:

> 2016-02-27 23:14 GMT+02:00 Marc Haber <mh+linux-btrfs@zugschlus.de>:
>> Hi,
>>
>> I have again the issue of no space left on device while rebalancing
>> (with btrfs-tools 4.4.1 on kernel 4.4.2 on Debian unstable):
>>
>>
> I've same issue, 4.4.3 kernel on Arch Linux
> 
> $ sudo btrfs fi show /mnt/fs/
> Label: 'fs'  uuid: a3c66d25-2c25-40e5-a827-5f7e5208e235
>         Total devices 1 FS bytes used 396.94GiB
>         devid    1 size 435.76GiB used 435.76GiB path /dev/sdi2
> 

> $ sudo btrfs fi usage /mnt/fs/
> Overall:
>     Device size:                 435.76GiB
>     Device allocated:            435.76GiB
>     Device unallocated:            1.00MiB

[Snipped the longer story, but the summary is that at first a full 
balance would fail, but after jumping thru some balance filtering hoops, 
you cleared enough space to unallocated that you could then do a full 
balance and it was working, tho it was still running.]


You're issue isn't the same, because all your space was allocated, 
leaving only 1 MiB unallocated, which isn't normally enough to allocate a 
new chunk to rewrite the data or metadata from the old chunks into.

That's a known issue, with known workarounds as dealt with in the FAQ. 

Ideally, you catch it while you have at least a gig or two of unallocated 
space (apparently in some instances with huge filesystems, you may need 
up to 10 GiB free, as data chunk allocations can be larger than the 
nominal 1 GiB on really large filesystems, but yours obviously isn't 
/that/ large) thereby giving you enough room to allocate at least one new 
chunk in ordered to rewrite data from old chunks while consolidating.

You only had 1 MiB of unallocated space in that first report, so you're a 
bit lucky that you didn't have to temporarily add a second device of at 
least several gigs in size, at least long enough to rewrite a few chunks 
and clear some additional space.  (FWIW, a few GiB USB thumb drive can be 
used, or even a loopback file on tmpfs, if you have enough memory for it 
and your system and power is stable enough that you're willing to take 
the gamble of not having an unscheduled reboot in the middle and thus 
losing your loopback, before you're able to btrfs device delete it again 
after you've completed at least enough of the balance to have enough room 
to do so.)  That's what some folks end up having to do if all space is 
allocated and they don't even have enough left to allocate even one more 
new chunk in ordered to do the balance and consolidate the partially-free 
chunks, thereby freeing the space they were using.

The issue of this thread is quite different, as they've triggered a bug 
that the devs have been trying to track down for quite awhile, where even 
with /lots/ of unallocated free space, btrfs will due to the bug refuse 
to allocate more, resulting in ENOSPC errors even when there's tens or 
hundreds of GiB of unallocated space, where you had only that 1 MiB.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Dāvis Mosāns]

2016-03-03 6:57 GMT+02:00 Duncan <1i5t5.duncan@cox.net>:
>
> You're issue isn't the same, because all your space was allocated,
> leaving only 1 MiB unallocated, which isn't normally enough to allocate a
> new chunk to rewrite the data or metadata from the old chunks into.
>
> That's a known issue, with known workarounds as dealt with in the FAQ.
>

Ah, thanks, well it was surprising for me that balance failed with out of
space when both data and metadata had not all been used and I thought
it could just use space from those...

especially as from FAQ:
> If there is a lot of allocated but unused data or metadata chunks,
> a balance may reclaim some of that allocated space. This is the
> main reason for running a balance on a single-device filesystem.

so I think regular balance should be smart enough that it could
solve this on own and wouldn't need to specify any options.

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Duncan]

Dāvis Mosāns posted on Thu, 03 Mar 2016 17:39:12 +0200 as excerpted:

> 2016-03-03 6:57 GMT+02:00 Duncan <1i5t5.duncan@cox.net>:
>>
>> You're issue isn't the same, because all your space was allocated,
>> leaving only 1 MiB unallocated, which isn't normally enough to allocate
>> a new chunk to rewrite the data or metadata from the old chunks into.
>>
>> That's a known issue, with known workarounds as dealt with in the FAQ.
>>
>>
> Ah, thanks, well it was surprising for me that balance failed with out
> of space when both data and metadata had not all been used and I thought
> it could just use space from those...
> 
> especially as from FAQ:
>> If there is a lot of allocated but unused data or metadata chunks,
>> a balance may reclaim some of that allocated space. This is the main
>> reason for running a balance on a single-device filesystem.
> 
> so I think regular balance should be smart enough that it could solve
> this on own and wouldn't need to specify any options.

Well it does solve the problem on its own... to the extent that it 
eliminates empty chunks (kernel 3.17+, it didn't before that).  But if 
there's even a single 4 KiB file block used in the (nominal 1 GiB sized 
data) chunk, it's no longer empty and thus not eliminated by the empty 
chunk cleanup routines.

Additionally, balance, which was originally called the restriper, by 
definition, must have enough space to create at least one empty chunk in 
ordered to copy the data from existing chunks into, such that it can 
consolidate that data into fewer new chunks when the old ones were partly 
empty.  If there's not enough unallocated space left to write even one 
chunk, balance can't do its thing, because there's nowhere to create the 
new chunk it needs to be able to copy over the data from the old chunk.

With nominal 1 GiB data chunk size (up to 10 GiB in some instances if the 
filesystem is large enough), that means you need at least 1 GiB of free 
unallocated space in ordered to have room to create that single chunk 
that starts the rewrite process off.  Without it, you're stuck, tho there 
are workarounds like trying to balance the smaller (256 MiB nominal) 
metadata chunks first, hoping that frees the minimum 1 GiB space needed 
for a data chunk, or temporarily adding another device a few GiB in size 
to the filesystem, to give it somewhere to write the new chunk to so it 
can start off the rewrite and shrinking process.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2947 bytes --]

On Fri, Mar 04, 2016 at 12:31:44PM +0000, Duncan wrote:
> Dāvis Mosāns posted on Thu, 03 Mar 2016 17:39:12 +0200 as excerpted:
> 
> > 2016-03-03 6:57 GMT+02:00 Duncan <1i5t5.duncan@cox.net>:
> >>
> >> You're issue isn't the same, because all your space was allocated,
> >> leaving only 1 MiB unallocated, which isn't normally enough to allocate
> >> a new chunk to rewrite the data or metadata from the old chunks into.
> >>
> >> That's a known issue, with known workarounds as dealt with in the FAQ.
> >>
> >>
> > Ah, thanks, well it was surprising for me that balance failed with out
> > of space when both data and metadata had not all been used and I thought
> > it could just use space from those...
> > 
> > especially as from FAQ:
> >> If there is a lot of allocated but unused data or metadata chunks,
> >> a balance may reclaim some of that allocated space. This is the main
> >> reason for running a balance on a single-device filesystem.
> > 
> > so I think regular balance should be smart enough that it could solve
> > this on own and wouldn't need to specify any options.
> 
> Well it does solve the problem on its own... to the extent that it 
> eliminates empty chunks (kernel 3.17+, it didn't before that).  But if 
> there's even a single 4 KiB file block used in the (nominal 1 GiB sized 
> data) chunk, it's no longer empty and thus not eliminated by the empty 
> chunk cleanup routines.
> 
> Additionally, balance, which was originally called the restriper,

   Nope, balance was always called balance. The restriper was the
balance feature that's now called "convert". :)

   Hugo.

 by 
> definition, must have enough space to create at least one empty chunk in 
> ordered to copy the data from existing chunks into, such that it can 
> consolidate that data into fewer new chunks when the old ones were partly 
> empty.  If there's not enough unallocated space left to write even one 
> chunk, balance can't do its thing, because there's nowhere to create the 
> new chunk it needs to be able to copy over the data from the old chunk.
> 
> With nominal 1 GiB data chunk size (up to 10 GiB in some instances if the 
> filesystem is large enough), that means you need at least 1 GiB of free 
> unallocated space in ordered to have room to create that single chunk 
> that starts the rewrite process off.  Without it, you're stuck, tho there 
> are workarounds like trying to balance the smaller (256 MiB nominal) 
> metadata chunks first, hoping that frees the minimum 1 GiB space needed 
> for a data chunk, or temporarily adding another device a few GiB in size 
> to the filesystem, to give it somewhere to write the new chunk to so it 
> can start off the rewrite and shrinking process.
> 

-- 
Hugo Mills             | Darkling's First Law of Filesystems:
hugo@... carfax.org.uk | The user hates their data
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Martin Steigerwald]

On Freitag, 4. März 2016 12:31:44 CEST Duncan wrote:
> Dāvis Mosāns posted on Thu, 03 Mar 2016 17:39:12 +0200 as excerpted:
> > 2016-03-03 6:57 GMT+02:00 Duncan <1i5t5.duncan@cox.net>:
> >> You're issue isn't the same, because all your space was allocated,
> >> leaving only 1 MiB unallocated, which isn't normally enough to allocate
> >> a new chunk to rewrite the data or metadata from the old chunks into.
> >> 
> >> That's a known issue, with known workarounds as dealt with in the FAQ.
> > 
> > Ah, thanks, well it was surprising for me that balance failed with out
> > of space when both data and metadata had not all been used and I thought
> > it could just use space from those...
> > 
> > especially as from FAQ:
> >> If there is a lot of allocated but unused data or metadata chunks,
> >> a balance may reclaim some of that allocated space. This is the main
> >> reason for running a balance on a single-device filesystem.
> > 
> > so I think regular balance should be smart enough that it could solve
> > this on own and wouldn't need to specify any options.
> 
> Well it does solve the problem on its own... to the extent that it 
> eliminates empty chunks (kernel 3.17+, it didn't before that).  But if 
> there's even a single 4 KiB file block used in the (nominal 1 GiB sized 
> data) chunk, it's no longer empty and thus not eliminated by the empty 
> chunk cleanup routines.

It could theoretically copy part of one almost empty chunk into another chunk 
to free it up, couldn´t it? This way it can free some chunks completely and 
then start the regular balance?

In either case, its unintuitive for the user to fail this. The filesystem 
tools should allow a balance in *any* case without needing special treatment 
by the user.

Thanks,
-- 
Martin

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Duncan]

Martin Steigerwald posted on Sun, 27 Mar 2016 14:10:07 +0200 as excerpted:

> On Freitag, 4. März 2016 12:31:44 CEST Duncan wrote:
>> Dāvis Mosāns posted on Thu, 03 Mar 2016 17:39:12 +0200 as excerpted:
>> > 2016-03-03 6:57 GMT+02:00 Duncan <1i5t5.duncan@cox.net>:
>> >> You're issue isn't the same, because all your space was allocated,
>> >> leaving only 1 MiB unallocated, which isn't normally enough to
>> >> allocate a new chunk to rewrite the data or metadata from the old
>> >> chunks into.
>> >> 
>> >> That's a known issue, with known workarounds as dealt with in the
>> >> FAQ.
>> > 
>> > Ah, thanks, well it was surprising for me that balance failed with
>> > out of space when both data and metadata had not all been used and I
>> > thought it could just use space from those...
>> > 
>> > especially as from FAQ:
>> >> If there is a lot of allocated but unused data or metadata chunks,
>> >> a balance may reclaim some of that allocated space. This is the main
>> >> reason for running a balance on a single-device filesystem.
>> > 
>> > so I think regular balance should be smart enough that it could solve
>> > this on own and wouldn't need to specify any options.
>> 
>> Well it does solve the problem on its own... to the extent that it
>> eliminates empty chunks (kernel 3.17+, it didn't before that).  But if
>> there's even a single 4 KiB file block used in the (nominal 1 GiB sized
>> data) chunk, it's no longer empty and thus not eliminated by the empty
>> chunk cleanup routines.
> 
> It could theoretically copy part of one almost empty chunk into another
> chunk to free it up, couldn´t it? This way it can free some chunks
> completely and then start the regular balance?

To be clear here, as unfortunately I wasn't in the previous reply, "it" 
in this case refers to the kernel's general btrfs handling -- IOW, the 
kernel, since 3.17, routinely deletes entirely empty chunks.

(Tho apparently there are cases when it misses some, as we've had a few 
reports lately of a balance with usage=0 cleaning up more than the 
trivial one or two chunks that could arguably have been "in transit" at 
the time the balance was run... but that would be a bug.)

For the kernel to routinely and automatically move content from one 
partially filled chunk to another in ordered to free the one is a *MUCH* 
higher level of complexity and thus a *MUCH* higher chance of serious 
show-stopping bugs; certainly nothing /I/'d wish to touch, were I a btrfs 
dev.  

It should be noted that btrfs is in general a COW (copy-on-write) 
filesystem, so simply moving content from one chunk into another isn't 
the way it works.  At the individual node level if not at the chunk 
level, the COW nature of btrfs means that modification of the existing 
data in both chunks would require copying the node elsewhere in ordered 
to rewrite it to include the new/modified information, and this must be 
handled atomically such that in the event of a crash, either the old 
version or the new version survives, not a mix of half of one and half of 
the other.  While btrfs is already designed from the ground up with that 
in mind, normal file and metadata updates would handle that within single 
chunks, and coordinating that atomicity across chunks really does add in 
geometric proportion to the complexity of the situation.

Which means there's much more wisdom than might be first appreciated in 
having balance simply stick to the chunk level COW that is its designed 
scope, instead of having it try to do cross-chunk node-level COW, which 
is what you're effectively proposing.  (Of course the complexity is in 
fact rather higher than I'm explaining here, but the fact remains, to the 
extent possible, keeping node level atomic operations to the node level, 
and chunk level atomic operations to the chunk level, **GREATLY** 
simplifies things, and deliberately crossing that level barrier where 
it's not absolutely required is an invitation to bugs so complex and 
severe that they could ultimately collapse the entire filesystem!)

> In either case, its unintuitive for the user to fail this. The
> filesystem tools should allow a balance in *any* case without needing
> special treatment by the user.

In fairness, there's a reason btrfs isn't claiming full stability and 
maturity just yet -- it's stabilizing, but exactly this sort of problems 
need to be worked out, before it can really be called fully stable.  
Meanwhile, as the (borrowed from Latin) saying goes "caveat emptor", "let 
the buyer beware."[1]  It remains the user's responsibility to ensure 
that btrfs is an appropriate filesystem for their use-case, and if so and 
once installed, that it remains within healthy operating parameters, 
enough unallocated space is kept available to complete balances, backups 
are kept in case some bug kills the filesystem, etc.

I think what ultimately needs to and probably will happen, is they'll 
create a new kind of global reserve that will come from unallocated space 
(instead of already allocated metadata chunks, which is where the current 
global reserve comes from, providing the same sort of reserve-COW-space 
functionality to more ordinary metadata fuctions), reserving enough of it 
to allocate at least one more full-size data chunk and one more full size 
metadata chunk, with only balance allowed to actually use that new global 
reserve space.  That way, balance will always have enough space to do 
what it needs to do.

Of course, it may well be necessary to let users tweak this reserve 
space, say at mkfs.btrfs time, so users creating for instance smaller 
mixed-data/metadata-chunk mode filesystems (like the 256 MiB /boot I have 
on one device, with a parallel 256 MiB backup /boot on a second device) 
can use all the space if it's more convenient for them to backup and do a 
new mkfs.btrfs than it is to reserve additional otherwise unusable space 
on tiny filesystems for balances they don't intend to do anyway.  
Similarly, users at the TB scale might want to reserve say 100 GiB 
instead of the default 1.5 GiB or so, and people doing large multi-device 
filesystems might want to do say 20 or 50 GiB per device.  Etc.  But the 
default reserve from unallocated would be enough for at least 1 chunk 
each of data and metadata, two chunks for dup mode on a single device, on 
each device.

---
[1] Caveat Emptor:  https://en.wikipedia.org/wiki/Caveat_emptor

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Marc Haber]

On Thu, Mar 03, 2016 at 02:28:36AM +0200, Dāvis Mosāns wrote:
> I've same issue, 4.4.3 kernel on Arch Linux
> 
> $ sudo btrfs fi show /mnt/fs/
> Label: 'fs'  uuid: a3c66d25-2c25-40e5-a827-5f7e5208e235
>         Total devices 1 FS bytes used 396.94GiB
>         devid    1 size 435.76GiB used 435.76GiB path /dev/sdi2
> 
> $ sudo btrfs fi df /mnt/fs/
> Data, single: total=416.70GiB, used=390.62GiB
> System, DUP: total=32.00MiB, used=96.00KiB
> Metadata, DUP: total=9.50GiB, used=6.32GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> $ sudo btrfs fi usage /mnt/fs/
> Overall:
>     Device size:                 435.76GiB
>     Device allocated:            435.76GiB
>     Device unallocated:            1.00MiB
>     Device missing:                  0.00B
>     Used:                        403.26GiB
>     Free (estimated):             26.07GiB      (min: 26.07GiB)
>     Data ratio:                       1.00
>     Metadata ratio:                   2.00
>     Global reserve:              512.00MiB      (used: 0.00B)
> 
> Data,single: Size:416.70GiB, Used:390.62GiB
>    /dev/sdi2     416.70GiB
> 
> Metadata,DUP: Size:9.50GiB, Used:6.32GiB
>    /dev/sdi2      19.00GiB
> 
> System,DUP: Size:32.00MiB, Used:96.00KiB
>    /dev/sdi2      64.00MiB
> 
> Unallocated:
>    /dev/sdi2       1.00MiB

http://paste.ubuntu.com/15292589/ has another log of mine with btrfs
fi usage calls as well, just in case this helps.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Chris Murphy]

I can't tell what this btrfs-balance script is doing because not every
btrfs balance command is in the log. It may be doing something not
advisable or suboptimal or unexpected that along with some other bug
is causing this to happen

Metadata,DUP: Size:107.00GiB, Used:2.11GiB

I'd try to use -musage filter alone, in whatever increments work. So
try 0. Then 5. If 5 fails, try 2. Increment until size is not much
more than 2-3x used.

Something is happening with the usage of this file system that's out
of the ordinary. This is the first time I've seen such a large amount
of unused metadata allocation. And then for it not only fail to
balance, but for the allocation amount to increase is a first. So
understanding the usage is important to figuring out what's happening.
I'd file a bug and include as much information on how the fs got into
this state as possible. And also if possible make a btrfs-image using
the proper flags to blot out the filenames for privacy. And what
btrfs-progs tools were used to create this file system. Etc.

The alternative if this can't be fixed, is to recreate the filesystem
because there's no practical way yet to migrate so many snapshots to a
new file system.


Chris Murphy

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Marc Haber]

On Sat, Mar 05, 2016 at 12:34:09PM -0700, Chris Murphy wrote:
> I can't tell what this btrfs-balance script is doing because not every
> btrfs balance command is in the log.

It is. I wrote it to produce reproducible logs.

[1/499]mh@fan:~$ cat btrfs-balance
#!/bin/bash

FS="/mnt/fanbtr"

showdf() {
logger -- btrfs fi df $FS
btrfs fi df $FS 2>&1 | logger
logger -- btrfs fi show /
btrfs fi show / | logger
logger -- btrfs fi usage /
btrfs fi usage / | logger
}

logger -- BEGIN btrfs-balance script
showdf

btrfs balance start  $FS 2>&1 | logger
showdf

logger -- BEGIN btrfs balance start -dprofiles=single $FS
btrfs balance start -dprofiles=single $FS 2>&1 | logger
showdf

logger -- BEGIN btrfs balance start -mprofiles=dup $FS
btrfs balance start -mprofiles=dup $FS 2>&1 | logger
showdf

logger -- BEGIN btrfs balance start --force -sprofiles=dup $FS
btrfs balance start --force -sprofiles=dup $FS 2>&1 | logger
showdf

logger -- BEGIN btrfs balance start $FS
btrfs balance start  $FS 2>&1 | logger
showdf

logger -- END btrfs-balance script
[2/500]mh@fan:~$ 

I see. The logger -- BEGIN is missing for the very first command. My
bad.

> Something is happening with the usage of this file system that's out
> of the ordinary. This is the first time I've seen such a large amount
> of unused metadata allocation. And then for it not only fail to
> balance, but for the allocation amount to increase is a first.

It is just a root filesystem of a workstation running Debian Linux, in
daily use, with daily snapshots of the system, and
ten-minute-increment snapshots of /home, with no cleanup happening for
a few months.

>  So understanding the usage is important to figuring out what's
>  happening. I'd file a bug and include as much information on how the
>  fs got into this state as possible. And also if possible make a
>  btrfs-image using the proper flags to blot out the filenames for
>  privacy.

That would btrfs-image -s?

> And what btrfs-progs tools were used to create this file system. Etc.

The file system is at least two years old, I do not remember, which
version of btrfs-tools was in Debian unstable back then. Is this
information somewhere in the filesystem label? How do I obtain this one?

> The alternative if this can't be fixed, is to recreate the filesystem
> because there's no practical way yet to migrate so many snapshots to a
> new file system.

I am now back to a mid three-digit number of snapshots.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Duncan]

Marc Haber posted on Sat, 05 Mar 2016 21:09:09 +0100 as excerpted:

> On Sat, Mar 05, 2016 at 12:34:09PM -0700, Chris Murphy wrote:

>> Something is happening with the usage of this file system that's out of
>> the ordinary. This is the first time I've seen such a large amount of
>> unused metadata allocation. And then for it not only fail to balance,
>> but for the allocation amount to increase is a first.
> 
> It is just a root filesystem of a workstation running Debian Linux, in
> daily use, with daily snapshots of the system, and ten-minute-increment
> snapshots of /home, with no cleanup happening for a few months.
> 
>>  So understanding the usage is important to figuring out what's
>>  happening. I'd file a bug and include as much information on how the
>>  fs got into this state as possible. And also if possible make a
>>  btrfs-image using the proper flags to blot out the filenames for
>>  privacy.

Now you're homing in on what I picked up on.  There's something very 
funky about that metadata, 100+ GiB of metadata total, only just over 2 
GiB metadata used, and attempts to balance it don't help with the spread 
between the two at all, only increasing the total metadata, if anything, 
but still seem to complete without error.  There's gotta be some sort of 
bug going on there, and I'd /bet/ it's the same one that's keeping full 
balances from working, as well.


OK, this question's out of left field, but it's the only thing (well, 
/almost/ only, see below) I've seen do anything /remotely/ like that:

Was the filesystem originally created as a convert from ext*, using btrfs-
convert?  If so, was the ext2_saved or whatever subvolume removed, and a 
successful defrag and balance completed at that time?

Because as I said, problems due to faulty conversion from ext* have been 
the one thing repeatedly reported to trigger balance behavior and spreads 
between total and used that balance doesn't fix, like this.


Tho AFAIK there was in addition a very narrow timeframe in which a bug in 
mkfs.btrfs would create invalid btrfs'.  That was with btrfs-progs 4.1.1, 
released in July 2015, with an urgent bugfix release 4.1.2 in the same 
month to fix the problem, so the timeframe was days or weeks.  Btrfs 
created with that buggy mkfs.btrfs were known to have some pretty wild 
behavior as well, with the recommendation being to simply blow them up 
and recreate them with a mkfs.btrfs from a btrfs-progs without the bug, 
as the btrfs created by the bugged version were simply too bugged out to 
reliably fix, and might well appear to work fine for awhile, until BOOM!  
If there's a chance the filesystem in question was created by that bugged 
mkfs.btrfs, don't even try to fix it, just get what you can off it and 
recreate with a mkfs.btrfs without that bug, ASAP.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Chris Murphy]

On Sat, Mar 5, 2016 at 11:43 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> Marc Haber posted on Sat, 05 Mar 2016 21:09:09 +0100 as excerpted:
>
>> On Sat, Mar 05, 2016 at 12:34:09PM -0700, Chris Murphy wrote:
>
>>> Something is happening with the usage of this file system that's out of
>>> the ordinary. This is the first time I've seen such a large amount of
>>> unused metadata allocation. And then for it not only fail to balance,
>>> but for the allocation amount to increase is a first.
>>
>> It is just a root filesystem of a workstation running Debian Linux, in
>> daily use, with daily snapshots of the system, and ten-minute-increment
>> snapshots of /home, with no cleanup happening for a few months.
>>
>>>  So understanding the usage is important to figuring out what's
>>>  happening. I'd file a bug and include as much information on how the
>>>  fs got into this state as possible. And also if possible make a
>>>  btrfs-image using the proper flags to blot out the filenames for
>>>  privacy.
>
> Now you're homing in on what I picked up on.  There's something very
> funky about that metadata, 100+ GiB of metadata total, only just over 2
> GiB metadata used, and attempts to balance it don't help with the spread
> between the two at all, only increasing the total metadata, if anything,
> but still seem to complete without error.  There's gotta be some sort of
> bug going on there, and I'd /bet/ it's the same one that's keeping full
> balances from working, as well.
>
>
> OK, this question's out of left field, but it's the only thing (well,
> /almost/ only, see below) I've seen do anything /remotely/ like that:
>
> Was the filesystem originally created as a convert from ext*, using btrfs-
> convert?  If so, was the ext2_saved or whatever subvolume removed, and a
> successful defrag and balance completed at that time?
>
> Because as I said, problems due to faulty conversion from ext* have been
> the one thing repeatedly reported to trigger balance behavior and spreads
> between total and used that balance doesn't fix, like this.
>
>
> Tho AFAIK there was in addition a very narrow timeframe in which a bug in
> mkfs.btrfs would create invalid btrfs'.  That was with btrfs-progs 4.1.1,
> released in July 2015, with an urgent bugfix release 4.1.2 in the same
> month to fix the problem, so the timeframe was days or weeks.  Btrfs
> created with that buggy mkfs.btrfs were known to have some pretty wild
> behavior as well, with the recommendation being to simply blow them up
> and recreate them with a mkfs.btrfs from a btrfs-progs without the bug,
> as the btrfs created by the bugged version were simply too bugged out to
> reliably fix, and might well appear to work fine for awhile, until BOOM!
> If there's a chance the filesystem in question was created by that bugged
> mkfs.btrfs, don't even try to fix it, just get what you can off it and
> recreate with a mkfs.btrfs without that bug, ASAP.

Marc said it was created maybe 2 years ago and doesn't remember what
version of the tools were used. Between it being two years ago and
also being Debian, for all we know it could've been 0.19. *shrug*

On the one hand, the practical advice is to just blow it away and use
everything current, go back to the same workload including thousands
of snapshots, and see if this balance problem is reproducible. That's
pretty clearly a bug.

On the other hand, we're approaching the state with Btrfs where the
problems we're seeing are at least as much about aging file systems,
because the stability is permitting file systems to get older. As they
get older though, the issues get more non-deterministic. So it's an
interesting bug from that perspective, the current kernel code ought
to be able to contend with this (as in, the user is right to expect
the code to deal with this scenario, and if it doesn't it's a bug; not
that I expect today's code to actually do this).

That's why I'm suggesting btrfs-image, because at some point the aging
question might get a dev's attention. But in the meantime the user
should get back to a reliable state and also find out sooner than
later if this is still a problem with all current tools and code.

So if it were me, I'd gather all possible data, including complete,
not trimmed, logs. And as for the btrfs-image, it could be huge. It
should be only 1GB based on the used metadata but the fact there's 50x
more allocated, I'm not sure how big it'll actually be or if
btrfs-image even has the granularity to capture this. It might not be
a bad idea to capture a complete btrfs-debug-tree also, and compress
that, add as attachment. For the image, stick it up in dropbox or
google drive, and post the URL in the bug and give it a year... I've
had devs come back and fix stuff that old before.

-- 
Chris Murphy

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Chris Murphy]

On Sun, Mar 6, 2016 at 1:27 PM, Chris Murphy <lists@colorremedies.com> wrote:

> So if it were me, I'd gather all possible data, including complete,
> not trimmed, logs.

Also include in the bug, the balance script being used. It might be a
contributing factor.

I wonder if the ENOSPC is happening just prior to the point where
balance would free up the unused portion of allocated metadata chunks
and that's why this just keeps getting worse? The balance function is
COW, so I wonder if there are a bunch of failed chunk migrations that
are just accumulating due to the ENOSPC stopping the balance?


Anyway, after collecting all data and btrfs-image, I would blow away
this fs using current kernel and tools. And then go back to the
original workload. I would not pare down the number or frequency of
snapshots. If anything increase it. The idea is to reproduce the bug.
While ENOSPC is a certain indicator the bug is present, it's not the
only possibility. If the metadata chunk allocation substantially
starts to exceed the used amount by say 4x or more after a full
metadata balance, that suggests a problem even if there is no ENOSPC.
Right now total is more than 50x used.


-- 
Chris Murphy

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Marc Haber]

On Sun, Mar 06, 2016 at 01:37:31PM -0700, Chris Murphy wrote:
> On Sun, Mar 6, 2016 at 1:27 PM, Chris Murphy <lists@colorremedies.com> wrote:
> > So if it were me, I'd gather all possible data, including complete,
> > not trimmed, logs.
> 
> Also include in the bug, the balance script being used. It might be a
> contributing factor.

The balance script was only written after Duncan asked me to do
filtered balances instead of a full balance. The issue showed itself
while the filesystem was still managed using the procedures from "the
book" ;-)

> I wonder if the ENOSPC is happening just prior to the point where
> balance would free up the unused portion of allocated metadata chunks
> and that's why this just keeps getting worse? The balance function is
> COW, so I wonder if there are a bunch of failed chunk migrations that
> are just accumulating due to the ENOSPC stopping the balance?

How do we find out?

> Anyway, after collecting all data and btrfs-image, I would blow away
> this fs using current kernel and tools. And then go back to the
> original workload. I would not pare down the number or frequency of
> snapshots. If anything increase it. The idea is to reproduce the bug.

... losing another pile of snapshots in the process? This is a
productive machine[1].

Greetings
Marc

[1] yes, with off-line backups being made

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Marc Haber]

On Sun, Mar 06, 2016 at 01:27:10PM -0700, Chris Murphy wrote:
> Marc said it was created maybe 2 years ago and doesn't remember what
> version of the tools were used. Between it being two years ago and
> also being Debian, for all we know it could've been 0.19. *shrug*

You are mixing up Debian unstable and Debian stable *snort*. You're
lucky that I'm not on RHEL 6[1].

> On the one hand, the practical advice is to just blow it away and use
> everything current, go back to the same workload including thousands
> of snapshots, and see if this balance problem is reproducible. That's
> pretty clearly a bug.

To have the same thing happen in half a year again? That's not why I
converted to a snapshottable file system.

> On the other hand, we're approaching the state with Btrfs where the
> problems we're seeing are at least as much about aging file systems,
> because the stability is permitting file systems to get older.

And this is really something to be proud of? I mean, this is a file
system that is part of the vanilla linux kernel, not marked as
experimental or something, and you're still concerned about file
systems that were made a year ago? This is a new experience for me.

>  As they get older though, the issues get more non-deterministic. So
>  it's an interesting bug from that perspective, the current kernel
>  code ought to be able to contend with this (as in, the user is right
>  to expect the code to deal with this scenario, and if it doesn't it's
>  a bug; not that I expect today's code to actually do this).

Kernel 4.4.4 as of the day before yesterday, thanks for considering.

> So if it were me, I'd gather all possible data, including complete,
> not trimmed, logs.

So you seriously want all messages like
Mar  7 09:25:23 fan systemd[1]: Started http per-connection Server, forwarding to 3142 ([2a01:238:4071:328d:5054:ff:fea9:6807]:41060).
Mar  7 09:25:23 fan named[3000]: client 2a01:238:4071:328d:5054:ff:fea9:6807#59920 (debian.debian.zugschlus.de): query: debian.debian.zugschlus.de IN AAAA + (fec0:0:0:ffff::1)
Mar  7 09:21:34 fan dhcpd[2468]: DHCPREQUEST for 192.168.182.29 from 54:04:a6:82:21:00 via eth0: unknown lease 192.168.182.29.
Mar  7 09:17:01 fan CRON[19474]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar  7 09:18:06 fan systemd[1]: Started Session c101 of user mh.
Mar  7 08:21:40 fan smartd[1956]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 31 to 30

I _can_ swamp the bug report literally with gigabytes of logs, but is
that really what you want? If it is not, please state what you mean by
"not trimmed" as I only removed those clutter messages from the logs I
sent.

Greetings
Marc

[1] Does RHEL 6 have btrfs in the first place?

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Chris Murphy]

On Mon, Mar 7, 2016 at 1:42 AM, Marc Haber <mh+linux-btrfs@zugschlus.de> wrote:
> On Sun, Mar 06, 2016 at 01:27:10PM -0700, Chris Murphy wrote:


>> On the one hand, the practical advice is to just blow it away and use
>> everything current, go back to the same workload including thousands
>> of snapshots, and see if this balance problem is reproducible. That's
>> pretty clearly a bug.
>
> To have the same thing happen in half a year again? That's not why I
> converted to a snapshottable file system.

If you want to help make Btfs better, yes. If you don't, that's fine,
use ZFS on Linux. Or if the ZFS Linux glue thing bugs you, use
FreeBSD. We're very lucky in free software to have so many
alternatives.


>> On the other hand, we're approaching the state with Btrfs where the
>> problems we're seeing are at least as much about aging file systems,
>> because the stability is permitting file systems to get older.
>
> And this is really something to be proud of? I mean, this is a file
> system that is part of the vanilla linux kernel, not marked as
> experimental or something, and you're still concerned about file
> systems that were made a year ago? This is a new experience for me.

I'm not a developer so you should take my opinions with a grain of
salt, but having come from something near to your perspective, it's
changed a lot by following the progress over ~5-6 years. Filesystems
are non-trivial. There's a great set of videos here
http://open-zfs.org/wiki/Main_Page about how non-trivial this was for
ZFS. It was a year to first kernel mount at which point it seemed like
90% of the work was done but far from it as it really took ~10 years
to get to a point where they were certain it could be deployed in an
enterprise environment.

Btrfs first kernel mount was in what 2008? So we're not yet done with
year 8? Not that they're exactly comparable. But I don't know who was
saying Btrfs would be done and stable with no bugs in 5 years.

The reality is, you've lost no data and you've probably found a bug.
So, write it up and contribute if you want. It's been a new experience
for me also.

>
>> So if it were me, I'd gather all possible data, including complete,
>> not trimmed, logs.
>
> So you seriously want all messages like
> Mar  7 09:25:23 fan systemd[1]: Started http per-connection Server, forwarding to 3142 ([2a01:238:4071:328d:5054:ff:fea9:6807]:41060).
> Mar  7 09:25:23 fan named[3000]: client 2a01:238:4071:328d:5054:ff:fea9:6807#59920 (debian.debian.zugschlus.de): query: debian.debian.zugschlus.de IN AAAA + (fec0:0:0:ffff::1)
> Mar  7 09:21:34 fan dhcpd[2468]: DHCPREQUEST for 192.168.182.29 from 54:04:a6:82:21:00 via eth0: unknown lease 192.168.182.29.
> Mar  7 09:17:01 fan CRON[19474]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
> Mar  7 09:18:06 fan systemd[1]: Started Session c101 of user mh.
> Mar  7 08:21:40 fan smartd[1956]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 31 to 30
>
> I _can_ swamp the bug report literally with gigabytes of logs, but is
> that really what you want?

Since there's no hardware issue suspect, you could filter for just btrfs.

journalctl -o short-iso | grep -i btrfs

When there's hardware stuff suspect it's better to include all the
SCSI and  libata (and USB if it's a USB drive) messages also.

If you have any logs that include the filesystem mounted with
enospc_debug, that might be useful for a developer?


> [1] Does RHEL 6 have btrfs in the first place?

They do, but you need a decoder ring to figure out what's been
backported to have some vague idea of what equivalent kernel.org
kernel it is.

-- 
Chris Murphy

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Austin S. Hemmelgarn]

On 2016-03-07 13:39, Chris Murphy wrote:
> On Mon, Mar 7, 2016 at 1:42 AM, Marc Haber <mh+linux-btrfs@zugschlus.de> wrote:
>> [1] Does RHEL 6 have btrfs in the first place?
>
> They do, but you need a decoder ring to figure out what's been
> backported to have some vague idea of what equivalent kernel.org
> kernel it is.
>
Yeah, in general, if you want to get good upstream support for BTRFS 
(such as from the mailing lists), you still want to steer clear of 
'Enterprise' branded distros (RHEL (and by extension CentOS) is 
particularly bad about kernel versioning, OEL and and SLES are 
marginally better but still not great).  People don't often think about 
it, but given the degree of code and version divergence due to patches, 
RHEL, SLES, and OEL kernels are strictly speaking, forks of Linux (most 
distro kernels are, but usually not to the extreme degree that 
enterprise kernels are (with the exception of some embedded systems, 
which can be even worse)).

In fact, this is part of the reason I switched from using Gentoo's 
official kernel sources (they're nowhere near as bad about kernel 
versioning as enterprise distros, but they do have back-ports and 
numerous local modifications I don't care about) to my own sources which 
(try to) directly track linux-stable (the other reason being that it's 
easier to maintain my own patches via git than trying to handle user 
patching for gentoo-sources).

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Chris Murphy]

On Mon, Mar 7, 2016 at 11:56 AM, Austin S. Hemmelgarn
<ahferroin7@gmail.com> wrote:
>  People don't often think about it, but given the degree of code and
> version divergence due to patches, RHEL, SLES, and OEL kernels are strictly
> speaking, forks of Linux (most distro kernels are, but usually not to the
> extreme degree that enterprise kernels are (with the exception of some
> embedded systems, which can be even worse)).

Yes. The exception is with Fedora kernels which carry very minimal
well documented patches already in or planned to go in the upstream
kernel anyway. bugzilla.kernel.org has a Tree pop-up menu with a
Fedora option in addition to Mainline and some others, but no other
distros are listed.


-- 
Chris Murphy

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Marc Haber]

On Mon, Mar 07, 2016 at 01:56:54PM -0500, Austin S. Hemmelgarn wrote:
> Yeah, in general, if you want to get good upstream support for BTRFS (such
> as from the mailing lists), you still want to steer clear of 'Enterprise'
> branded distros (RHEL (and by extension CentOS) is particularly bad about
> kernel versioning

Just to get back to this thread's subject, I am using Debian unstable,
with a vanilla kernel, 4.4.3 at the beginning of this thread, and
4.4.4 today.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Marc Haber]

On Mon, Mar 07, 2016 at 11:39:11AM -0700, Chris Murphy wrote:
> Since there's no hardware issue suspect, you could filter for just btrfs.
> 
> journalctl -o short-iso | grep -i btrfs

Which is exactly what I did. Why did you suspect that my logs were
"trimmed"? That's what got me kind of furious. I took great care to
not trim relevant information.

> When there's hardware stuff suspect it's better to include all the
> SCSI and  libata (and USB if it's a USB drive) messages also.

None there.

> If you have any logs that include the filesystem mounted with
> enospc_debug, that might be useful for a developer?

The later logs I posted were actually taken with enospc_debug, the
4.4.3 ones even with Duncan's patch. I think I didn't apply it before
building 4.4.4.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Chris Murphy]

On Mon, Mar 7, 2016 at 1:42 AM, Marc Haber <mh+linux-btrfs@zugschlus.de> wrote:
> And this is really something to be proud of? I mean, this is a file
> system that is part of the vanilla linux kernel, not marked as
> experimental or something, and you're still concerned about file
> systems that were made a year ago? This is a new experience for me.

https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/Documentation/filesystems/btrfs.txt?id=refs/tags/v4.4.4

"Btrfs is under heavy development, and is not suitable for
any uses other than benchmarking and review. The Btrfs disk format is
not yet finalized."

I thought the 2nd sentence was removed a long time ago but I'm seeing
it in the current branch and 4.1.y. Is this a bug?


-- 
Chris Murphy

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Duncan]

Chris Murphy posted on Mon, 07 Mar 2016 12:44:20 -0700 as excerpted:

> On Mon, Mar 7, 2016 at 1:42 AM, Marc Haber <mh+linux-btrfs@zugschlus.de>
> wrote:
>> And this is really something to be proud of? I mean, this is a file
>> system that is part of the vanilla linux kernel, not marked as
>> experimental or something, and you're still concerned about file
>> systems that were made a year ago? This is a new experience for me.
> 
> https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/
tree/Documentation/filesystems/btrfs.txt?id=refs/tags/v4.4.4
> 
> "Btrfs is under heavy development, and is not suitable for any uses
> other than benchmarking and review. The Btrfs disk format is not yet
> finalized."
> 
> I thought the 2nd sentence was removed a long time ago but I'm seeing it
> in the current branch and 4.1.y. Is this a bug?

AFAIK, this is the still "semi-scary" wording that was left after the 
_really_ scary "eat your babies" level experimental warning was stripped, 
and it remains more or less literally the case, certainly the last 
sentence, tho I'd quibble that the "not suitable for any uses other than 
benchmarking and review" is a bit conservative, and the "disk format is 
not yet finalized" bit, while literally true, doesn't have the 
implications one might initially think, because there _is_ a commitment 
to support all disk formats available since btrfs was kernel-mainlined, 
so while the disk format can and does still occasionally have small 
changes (think features like skinny-metadata and the metadata changes 
necessary for raid56 support), these are all handled by incompatibility 
flags such that a kernel that doesn't understand them will refuse to 
mount a filesystem with them enabled, while newer kernels will still 
mount filesystems with them not enabled.

And of course rather famously, early in the second kernel series after 
mainlining, some change made it in that apparently caused some btrfs 
Linus was using _not_ to mount, and he rightly blistered the btrfs devs 
for it, because it screwed up his ability to kernel bisect over that 
period with the system he was using btrfs on.  Linus said that's not the 
way things work in mainline and he was right.  It hasn't happened again.  
And that was when btrfs still had the much more severe experimental label 
applied, so particularly now that it's gone, yes, there may be disk 
format changes, but they are covered by incompatibility flags and all 
kernels going forward from mainlining are going to support older formats 
in addition to any newer formats, because that's the way it is.

But I'd probably word the first sentence somewhat differently, saying 
that you should have backups and be prepared to use them if you're using 
btrfs, and that it's not suitable for production systems yet, but 
omitting the only suitable for benchmarking and review wording.

Regardless, I believe we've definitely established that while it's in the 
mainline kernel and is no longer experimental, there's still quite some 
warning there, contrary to Mark Habor's claim otherwise.  And indeed, if 
following that warning literally, review and benchmarking is all he'd be 
doing with it, so beyond that is certainly on the user... or distro 
making claims to the contrary.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Chris Murphy]

On Mon, Mar 7, 2016 at 1:43 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> Chris Murphy posted on Mon, 07 Mar 2016 12:44:20 -0700 as excerpted:
>
>> On Mon, Mar 7, 2016 at 1:42 AM, Marc Haber <mh+linux-btrfs@zugschlus.de>
>> wrote:
>>> And this is really something to be proud of? I mean, this is a file
>>> system that is part of the vanilla linux kernel, not marked as
>>> experimental or something, and you're still concerned about file
>>> systems that were made a year ago? This is a new experience for me.
>>
>> https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/
> tree/Documentation/filesystems/btrfs.txt?id=refs/tags/v4.4.4
>>
>> "Btrfs is under heavy development, and is not suitable for any uses
>> other than benchmarking and review. The Btrfs disk format is not yet
>> finalized."
>>
>> I thought the 2nd sentence was removed a long time ago but I'm seeing it
>> in the current branch and 4.1.y. Is this a bug?
>
> AFAIK, this is the still "semi-scary" wording that was left after the
> _really_ scary "eat your babies" level experimental warning was stripped,
> and it remains more or less literally the case, certainly the last
> sentence,

The Btrfs wiki main page says the format is no longer unstable. So OK,
I guess those are compatible. Not yet finalized does not mean unstable
I guess?


> And that was when btrfs still had the much more severe experimental label
> applied, so particularly now that it's gone,

Right the experimental label is probably what I'm thinking of.
Nevertheless the above does sound a bit dire. I mean, for raid56 it's
totally reasonable, and maybe even any multiple device stuff just
because of the lack of faulty device handling, in particular
notification. But I'm not sure ZFS on Linux (or even FreeBSD) have any
of that either.


-- 
Chris Murphy

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Marc Haber]

On Sun, Mar 06, 2016 at 01:27:10PM -0700, Chris Murphy wrote:
> So if it were me, I'd gather all possible data, including complete,
> not trimmed, logs. And as for the btrfs-image, it could be huge.

[5/504]mh@q:~/.www/public_html/stuff$ unxz --list 20160307-fanbtr-image.xz
Strms  Blocks   Compressed Uncompressed  Ratio  Check   Filename
    1      19    248.0 MiB   2385.2 MiB  0.104  CRC32   20160307-fanbtr-image.xz

>  It might not be
>  a bad idea to capture a complete btrfs-debug-tree also, and compress
>  that, add as attachment.

How do I do that?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Marc Haber]

On Sun, Mar 06, 2016 at 06:43:46AM +0000, Duncan wrote:
> Marc Haber posted on Sat, 05 Mar 2016 21:09:09 +0100 as excerpted:
> > On Sat, Mar 05, 2016 at 12:34:09PM -0700, Chris Murphy wrote:
> >> Something is happening with the usage of this file system that's out of
> >> the ordinary. This is the first time I've seen such a large amount of
> >> unused metadata allocation. And then for it not only fail to balance,
> >> but for the allocation amount to increase is a first.
> > 
> > It is just a root filesystem of a workstation running Debian Linux, in
> > daily use, with daily snapshots of the system, and ten-minute-increment
> > snapshots of /home, with no cleanup happening for a few months.
> > 
> >>  So understanding the usage is important to figuring out what's
> >>  happening. I'd file a bug and include as much information on how the
> >>  fs got into this state as possible. And also if possible make a
> >>  btrfs-image using the proper flags to blot out the filenames for
> >>  privacy.
> 
> Now you're homing in on what I picked up on.  There's something very 
> funky about that metadata, 100+ GiB of metadata total, only just over 2 
> GiB metadata used, and attempts to balance it don't help with the spread 
> between the two at all, only increasing the total metadata, if anything, 
> but still seem to complete without error.  There's gotta be some sort of 
> bug going on there, and I'd /bet/ it's the same one that's keeping full 
> balances from working, as well.

I don't understand a single word of this, but you seem to understand
it. Good.

> 
> OK, this question's out of left field, but it's the only thing (well, 
> /almost/ only, see below) I've seen do anything /remotely/ like that:
> 
> Was the filesystem originally created as a convert from ext*, using btrfs-
> convert?  If so, was the ext2_saved or whatever subvolume removed, and a 
> successful defrag and balance completed at that time?

I have dug aroud in my auth.logs, and thanks to my not working in a
root shell but using sudo for every single command I can say that the
filesystem was created on September 1, 2015, so it is not _this_ old,
and snapshot.debian.net tells me that Debian unstable had btrfs-tools
4.1.2 uploaded on August 31, so i guess that the filesystem was either
created by the 4.0 version we had since May 2015 or by the brand new
4.1.2.

And it was a mkfs.btrfs with no special options. I suspected this
since I would probably not have made an ext4 filesystem of 300 GB in
size. Back in the ext4 days, I usually made /, /usr, /var, /home and
/boot their own filesystems.

> Tho AFAIK there was in addition a very narrow timeframe in which a bug in 
> mkfs.btrfs would create invalid btrfs'.  That was with btrfs-progs 4.1.1, 
> released in July 2015, with an urgent bugfix release 4.1.2 in the same 
> month to fix the problem, so the timeframe was days or weeks.

Debian is chastized for their allegedly quirky release schedules even
in this thread, I usually ignore that, but this time a smile comes to
my face when I say that btrfs-progs 4.1.1 was never packaged in
Debian, hence we're clear of this bug here. We went from 4.0 straight
to 4.1.2.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Duncan]

Marc Haber posted on Mon, 07 Mar 2016 09:30:43 +0100 as excerpted:

> I have dug aroud in my auth.logs, and thanks to my not working in a root
> shell but using sudo for every single command I can say that the
> filesystem was created on September 1, 2015, so it is not _this_ old,
> and snapshot.debian.net tells me that Debian unstable had btrfs-tools
> 4.1.2 uploaded on August 31, so i guess that the filesystem was either
> created by the 4.0 version we had since May 2015 or by the brand new
> 4.1.2.
> 
> And it was a mkfs.btrfs with no special options.

I doubted that either a convert-from-ext* or a mkfs with the bad 4.1.1 
would turn out to be the cause, but those are the only things /I/ am 
aware of that can trigger wild behavior like we're seeing with this 
metadata, here.

So it's certainly a bug, and not accounted for by anything know at least 
to me, which makes it a new and interesting bug.  But not being a dev, 
that's about as far as I can take it.  As CMurphy said, file a bug with 
all the various information, and hope the devs can replicate and trace it 
down.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Marc Haber]

On Sat, Mar 05, 2016 at 09:09:09PM +0100, Marc Haber wrote:
> On Sat, Mar 05, 2016 at 12:34:09PM -0700, Chris Murphy wrote:
> >  So understanding the usage is important to figuring out what's
> >  happening. I'd file a bug and include as much information on how the
> >  fs got into this state as possible. And also if possible make a
> >  btrfs-image using the proper flags to blot out the filenames for
> >  privacy.
> 
> That would btrfs-image -s?

btrfs-image -s -t 8 -s /dev/mapper/fanbtr

complains about a mounted filesystem. Will an image made from the
running system with the filesyste mounted help, or do I need to take
down the machine while the image is being made?

Also, threading does not seem to work, despite the -t 8 CPU usage
never increases 100 % in atop.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Marc Haber]

On Sat, Mar 05, 2016 at 12:34:09PM -0700, Chris Murphy wrote:
> Something is happening with the usage of this file system that's out
> of the ordinary. This is the first time I've seen such a large amount
> of unused metadata allocation. And then for it not only fail to
> balance, but for the allocation amount to increase is a first. So
> understanding the usage is important to figuring out what's happening.
> I'd file a bug and include as much information on how the fs got into
> this state as possible. And also if possible make a btrfs-image using
> the proper flags to blot out the filenames for privacy. And what
> btrfs-progs tools were used to create this file system. Etc.

https://bugzilla.kernel.org/show_bug.cgi?id=114451

Please advise if there is something missing.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Chris Murphy]

On Sat, Mar 12, 2016 at 12:57 PM, Marc Haber
<mh+linux-btrfs@zugschlus.de> wrote:
> On Sat, Mar 05, 2016 at 12:34:09PM -0700, Chris Murphy wrote:
>> Something is happening with the usage of this file system that's out
>> of the ordinary. This is the first time I've seen such a large amount
>> of unused metadata allocation. And then for it not only fail to
>> balance, but for the allocation amount to increase is a first. So
>> understanding the usage is important to figuring out what's happening.
>> I'd file a bug and include as much information on how the fs got into
>> this state as possible. And also if possible make a btrfs-image using
>> the proper flags to blot out the filenames for privacy. And what
>> btrfs-progs tools were used to create this file system. Etc.
>
> https://bugzilla.kernel.org/show_bug.cgi?id=114451
>
> Please advise if there is something missing.

No enospc_debug mount option used for kernel messages. And no
indication you applied Qu's patch mentioned on March 1 to get more
info with enospc_debug mount:

>Oh, I'm sorry that the output is not necessary, it's better to use the newer patch:
>https://patchwork.kernel.org/patch/8462881/
>With the newer patch, you will need to use enospc_debug mount option to get the debug information.



-- 
Chris Murphy

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Marc Haber]

On Sun, Mar 13, 2016 at 01:43:50PM -0600, Chris Murphy wrote:
> On Sat, Mar 12, 2016 at 12:57 PM, Marc Haber
> <mh+linux-btrfs@zugschlus.de> wrote:
> > On Sat, Mar 05, 2016 at 12:34:09PM -0700, Chris Murphy wrote:
> >> Something is happening with the usage of this file system that's out
> >> of the ordinary. This is the first time I've seen such a large amount
> >> of unused metadata allocation. And then for it not only fail to
> >> balance, but for the allocation amount to increase is a first. So
> >> understanding the usage is important to figuring out what's happening.
> >> I'd file a bug and include as much information on how the fs got into
> >> this state as possible. And also if possible make a btrfs-image using
> >> the proper flags to blot out the filenames for privacy. And what
> >> btrfs-progs tools were used to create this file system. Etc.
> >
> > https://bugzilla.kernel.org/show_bug.cgi?id=114451
> >
> > Please advise if there is something missing.
> 
> No enospc_debug mount option used for kernel messages.

I apologize for not having this mentioned, but why do you think that
it wasn't active?

|[28/527]mh@fan:~$ grep enospc /proc/mounts
|/dev/mapper/fanbtr / btrfs rw,noatime,nodiratime,ssd,space_cache,enospc_debug,subvolid=257,subvol=/fan-root 0 0
|/dev/mapper/fanbtr /mnt/snapshots/fanbtr btrfs rw,noatime,nodiratime,ssd,space_cache,enospc_debug,subvolid=266,subvol=/snapshots 0 0
|[29/528]mh@fan:~$

>  And no indication you applied Qu's patch mentioned on March 1 to get
>  more info with enospc_debug mount:
> 
> >Oh, I'm sorry that the output is not necessary, it's better to use the newer patch:
> >https://patchwork.kernel.org/patch/8462881/
> >With the newer patch, you will need to use enospc_debug mount option to get the debug information.

That one didn't make it in 4.4.5 yet?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Chris Murphy]

On Sun, Mar 13, 2016 at 2:50 PM, Marc Haber <mh+linux-btrfs@zugschlus.de> wrote:
> On Sun, Mar 13, 2016 at 01:43:50PM -0600, Chris Murphy wrote:
>> On Sat, Mar 12, 2016 at 12:57 PM, Marc Haber
>> <mh+linux-btrfs@zugschlus.de> wrote:
>> > On Sat, Mar 05, 2016 at 12:34:09PM -0700, Chris Murphy wrote:
>> >> Something is happening with the usage of this file system that's out
>> >> of the ordinary. This is the first time I've seen such a large amount
>> >> of unused metadata allocation. And then for it not only fail to
>> >> balance, but for the allocation amount to increase is a first. So
>> >> understanding the usage is important to figuring out what's happening.
>> >> I'd file a bug and include as much information on how the fs got into
>> >> this state as possible. And also if possible make a btrfs-image using
>> >> the proper flags to blot out the filenames for privacy. And what
>> >> btrfs-progs tools were used to create this file system. Etc.
>> >
>> > https://bugzilla.kernel.org/show_bug.cgi?id=114451
>> >
>> > Please advise if there is something missing.
>>
>> No enospc_debug mount option used for kernel messages.
>
> I apologize for not having this mentioned, but why do you think that
> it wasn't active?

No additional information in the log attached to the bug. So I guess
this particular problem doesn't trigger any enospc debug options and
hence the patch.


>
> |[28/527]mh@fan:~$ grep enospc /proc/mounts
> |/dev/mapper/fanbtr / btrfs rw,noatime,nodiratime,ssd,space_cache,enospc_debug,subvolid=257,subvol=/fan-root 0 0
> |/dev/mapper/fanbtr /mnt/snapshots/fanbtr btrfs rw,noatime,nodiratime,ssd,space_cache,enospc_debug,subvolid=266,subvol=/snapshots 0 0
> |[29/528]mh@fan:~$
>
>>  And no indication you applied Qu's patch mentioned on March 1 to get
>>  more info with enospc_debug mount:
>>
>> >Oh, I'm sorry that the output is not necessary, it's better to use the newer patch:
>> >https://patchwork.kernel.org/patch/8462881/
>> >With the newer patch, you will need to use enospc_debug mount option to get the debug information.
>
> That one didn't make it in 4.4.5 yet?

I got no indication it was even going to make it into mainline
actually, I was thinking it was a one off patch. But if it does, it'll
need to be in 4.5.0 before it'd be backported to 4.4.x. Maybe Qu can
clarify?

-- 
Chris Murphy

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Marc Haber]

On Sat, Mar 05, 2016 at 12:34:09PM -0700, Chris Murphy wrote:
> I'd try to use -musage filter alone, in whatever increments work. So
> try 0. Then 5. If 5 fails, try 2. Increment until size is not much
> more than 2-3x used.

Here we go:

[7/506]mh@fan:~$ sudo btrfs balance -usage=0 /media/tempdisk
Done, had to relocate 32 out of 134 chunks
[8/507]mh@fan:~$ sudo btrfs fi df /media/tempdisk/
Data, single: total=79.00GiB, used=78.33GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=27.00GiB, used=2.30GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
[9/508]mh@fan:~$ sudo btrfs balance start -musage=0 /media/tempdisk
Done, had to relocate 32 out of 134 chunks
[10/509]mh@fan:~$ sudo btrfs fi df /media/tempdisk/
Data, single: total=79.00GiB, used=78.33GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=27.00GiB, used=2.30GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
[11/510]mh@fan:~$ sudo btrfs balance start -musage=0 /media/tempdisk
Done, had to relocate 36 out of 134 chunks
[12/511]mh@fan:~$ sudo btrfs balance fi df /media/tempdisk/
Data, single: total=79.00GiB, used=78.33GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=27.00GiB, used=2.30GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
[13/512]mh@fan:~$ sudo btrfs balance start -musage=10 /media/tempdisk
Done, had to relocate 41 out of 134 chunks
[14/513]mh@fan:~$ sudo btrfs fi df /media/tempdisk/
Data, single: total=79.00GiB, used=78.32GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=27.00GiB, used=2.30GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
[15/514]mh@fan:~$ sudo btrfs balance start -musage=20 /media/tempdisk
Done, had to relocate 43 out of 134 chunks
[16/515]mh@fan:~$ sudo btrfs fi df /media/tempdisk/
Data, single: total=79.00GiB, used=78.32GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=27.00GiB, used=2.30GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
[17/516]mh@fan:~$ sudo btrfs balance start -musage=30 /media/tempdisk
Done, had to relocate 48 out of 134 chunks

real    0m48.604s
user    0m0.008s
sys     0m15.756s
[18/517]mh@fan:~$ sudo btrfs fi df /media/tempdisk/
Data, single: total=79.00GiB, used=78.32GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=27.00GiB, used=2.30GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
[19/518]mh@fan:~$ time sudo btrfs balance start -musage=40 /media/tempdisk
Done, had to relocate 49 out of 134 chunks

real    0m17.076s
user    0m0.000s
sys     0m5.188s
[20/519]mh@fan:~$ sudo btrfs fi df /media/tempdisk/
Data, single: total=79.00GiB, used=78.32GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=27.00GiB, used=2.30GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
[21/520]mh@fan:~$ time sudo btrfs balance start -musage=50 /media/tempdisk
Done, had to relocate 49 out of 134 chunks

real    0m11.536s
user    0m0.004s
sys     0m3.512s
[22/521]mh@fan:~$ sudo btrfs fi df /media/tempdisk/
Data, single: total=79.00GiB, used=78.32GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=27.00GiB, used=2.30GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
[23/522]mh@fan:~$ time sudo btrfs balance start -musage=60 /media/tempdisk
Done, had to relocate 49 out of 134 chunks

real    0m4.485s
user    0m0.008s
sys     0m0.664s
[24/523]mh@fan:~$ sudo btrfs fi df /media/tempdisk/
Data, single: total=79.00GiB, used=78.32GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=27.00GiB, used=2.30GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
[25/524]mh@fan:~$ time sudo btrfs balance start -musage=70 /media/tempdisk
Done, had to relocate 50 out of 134 chunks

real    0m16.581s
user    0m0.012s
sys     0m5.456s
[26/525]mh@fan:~$ sudo btrfs fi df /media/tempdisk/
Data, single: total=79.00GiB, used=78.32GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=26.50GiB, used=2.30GiB
GlobalReserve, single: total=512.00MiB, used=3.33MiB
[27/526]mh@fan:~$ time sudo btrfs balance start -musage=80 /media/tempdisk
ERROR: error during balancing '/media/tempdisk': No space left on device
There may be more info in syslog - try dmesg | tail

real    0m16.775s
user    0m0.004s
sys     0m5.800s
[28/527]mh@fan:~$ sudo btrfs fi df /media/tempdisk/
Data, single: total=79.00GiB, used=78.32GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=26.50GiB, used=2.30GiB
GlobalReserve, single: total=512.00MiB, used=33.69MiB
[29/528]mh@fan:~$ time sudo btrfs balance start -musage=75 /media/tempdisk
ERROR: error during balancing '/media/tempdisk': No space left on device
There may be more info in syslog - try dmesg | tail

real    0m4.363s
user    0m0.004s
sys     0m0.600s
[30/529]mh@fan:~$ sudo btrfs fi df /media/tempdisk/
Data, single: total=79.00GiB, used=78.32GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=27.00GiB, used=2.30GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
[31/530]mh@fan:~$ time sudo btrfs balance start -musage=74 /media/tempdisk
Done, had to relocate 50 out of 134 chunks

real    0m4.546s
user    0m0.000s
sys     0m0.620s
[32/531]mh@fan:~$ sudo btrfs fi df /media/tempdisk/
Data, single: total=79.00GiB, used=78.32GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=27.00GiB, used=2.30GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

So one does not see a decrease in total Metadata size until -musage
has gone up to 70, then it decreases by half a gig. -musage=75 is the
first musage value that leads to the ENOSPC condition, with total
Metadata size going up to 27 GiB again, and -musage=74 being the
biggest musage value that finishs without ENOSPC, but no visible
decrease of total Metadata size.

Greetings
Marc


-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Marc Haber]

Hi,

On Sat, Mar 05, 2016 at 12:34:09PM -0700, Chris Murphy wrote:
> The alternative if this can't be fixed, is to recreate the filesystem
> because there's no practical way yet to migrate so many snapshots to a
> new file system.

I recreated the file system on March 7, with 200 GiB in size, using
btrfs-tools 4.4. The snapshot-taking process has been running since
then, but I also regularly cleaned up. The number of snapshots on the
new filesystem has never exceeded 1000, with the current count being
at 148.

And btrfs balance runs into the same ENOSPC issues as the old one:

[9/508]mh@fan:~$ grep -v 'device dm-15' 20160313-fanbtr-btrfs-syslog
Mar 13 11:05:45 fan mh: BEGIN btrfs-balance script
Mar 13 11:05:45 fan mh: btrfs fi df /
Mar 13 11:05:45 fan root: Data, single: total=80.00GiB, used=77.71GiB
Mar 13 11:05:45 fan root: System, single: total=32.00MiB, used=16.00KiB
Mar 13 11:05:45 fan root: Metadata, single: total=27.00GiB, used=2.38GiB
Mar 13 11:05:45 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B
Mar 13 11:05:45 fan mh: btrfs fi show /
Mar 13 11:05:45 fan root: Label: 'fanbtr'  uuid: 90f8d728-6bae-4fca-8cda-b368ba2c008e
Mar 13 11:05:45 fan root: #011Total devices 1 FS bytes used 80.09GiB
Mar 13 11:05:45 fan root: #011devid    1 size 200.00GiB used 107.03GiB path /dev/mapper/fanbtr
Mar 13 11:05:45 fan root:
Mar 13 11:05:45 fan mh: btrfs fi usage /
Mar 13 11:05:45 fan root: Overall:
Mar 13 11:05:45 fan root:     Device size:#011#011 200.00GiB
Mar 13 11:05:45 fan root:     Device allocated:#011#011 107.03GiB
Mar 13 11:05:45 fan root:     Device unallocated:#011#011  92.97GiB
Mar 13 11:05:45 fan root:     Device missing:#011#011     0.00B
Mar 13 11:05:45 fan root:     Used:#011#011#011  80.09GiB
Mar 13 11:05:45 fan root:     Free (estimated):#011#011  95.26GiB#011(min: 95.26GiB)
Mar 13 11:05:45 fan root:     Data ratio:#011#011#011      1.00
Mar 13 11:05:45 fan root:     Metadata ratio:#011#011      1.00
Mar 13 11:05:45 fan root:     Global reserve:#011#011 512.00MiB#011(used: 0.00B)
Mar 13 11:05:45 fan root:
Mar 13 11:05:45 fan root: Data,single: Size:80.00GiB, Used:77.71GiB
Mar 13 11:05:45 fan root:    /dev/mapper/fanbtr#011  80.00GiB
Mar 13 11:05:45 fan root:
Mar 13 11:05:45 fan root: Metadata,single: Size:27.00GiB, Used:2.38GiB
Mar 13 11:05:45 fan root:    /dev/mapper/fanbtr#011  27.00GiB
Mar 13 11:05:45 fan root:
Mar 13 11:05:45 fan root: System,single: Size:32.00MiB, Used:16.00KiB
Mar 13 11:05:45 fan root:    /dev/mapper/fanbtr#011  32.00MiB
Mar 13 11:05:45 fan root:
Mar 13 11:05:45 fan root: Unallocated:
Mar 13 11:05:45 fan root:    /dev/mapper/fanbtr#011  92.97GiB
Mar 13 11:05:45 fan mh: BEGIN btrfs balance start /
Mar 13 11:20:30 fan root: ERROR: error during balancing '/': No space left on device
Mar 13 11:20:30 fan root: There may be more info in syslog - try dmesg | tail
Mar 13 11:20:30 fan root: btrfs fi df /
Mar 13 11:20:30 fan root: Data, single: total=78.00GiB, used=77.70GiB
Mar 13 11:20:30 fan root: System, single: total=32.00MiB, used=16.00KiB
Mar 13 11:20:30 fan root: Metadata, single: total=15.00GiB, used=2.38GiB
Mar 13 11:20:30 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B
Mar 13 11:20:30 fan root: btrfs fi show /
Mar 13 11:20:30 fan root: Label: 'fanbtr'  uuid: 90f8d728-6bae-4fca-8cda-b368ba2c008e
Mar 13 11:20:30 fan root: #011Total devices 1 FS bytes used 80.08GiB
Mar 13 11:20:30 fan root: #011devid    1 size 200.00GiB used 93.03GiB path /dev/mapper/fanbtr
Mar 13 11:20:30 fan root:
Mar 13 11:20:30 fan root: btrfs fi usage /
Mar 13 11:20:30 fan root: Overall:
Mar 13 11:20:30 fan root:     Device size:#011#011 200.00GiB
Mar 13 11:20:30 fan root:     Device allocated:#011#011  93.03GiB
Mar 13 11:20:30 fan root:     Device unallocated:#011#011 106.97GiB
Mar 13 11:20:30 fan root:     Device missing:#011#011     0.00B
Mar 13 11:20:30 fan root:     Used:#011#011#011  80.08GiB
Mar 13 11:20:30 fan root:     Free (estimated):#011#011 107.27GiB#011(min: 107.27GiB)
Mar 13 11:20:30 fan root:     Data ratio:#011#011#011      1.00
Mar 13 11:20:30 fan root:     Metadata ratio:#011#011      1.00
Mar 13 11:20:30 fan root:     Global reserve:#011#011 512.00MiB#011(used: 0.00B)
Mar 13 11:20:30 fan root:
Mar 13 11:20:30 fan root: Data,single: Size:78.00GiB, Used:77.70GiB
Mar 13 11:20:30 fan root:    /dev/mapper/fanbtr#011  78.00GiB
Mar 13 11:20:30 fan root:
Mar 13 11:20:30 fan root: Metadata,single: Size:15.00GiB, Used:2.38GiB
Mar 13 11:20:30 fan root:    /dev/mapper/fanbtr#011  15.00GiB
Mar 13 11:20:30 fan root:
Mar 13 11:20:30 fan root: System,single: Size:32.00MiB, Used:16.00KiB
Mar 13 11:20:30 fan root:    /dev/mapper/fanbtr#011  32.00MiB
Mar 13 11:20:30 fan root:
Mar 13 11:20:30 fan root: Unallocated:
Mar 13 11:20:30 fan root:    /dev/mapper/fanbtr#011 106.97GiB
Mar 13 11:20:30 fan root: BEGIN btrfs balance start -dprofiles=single /
Mar 13 11:36:17 fan root: Done, had to relocate 78 out of 94 chunks
Mar 13 11:36:17 fan root: btrfs fi df /
Mar 13 11:36:17 fan root: Data, single: total=78.00GiB, used=77.71GiB
Mar 13 11:36:17 fan root: System, single: total=32.00MiB, used=16.00KiB
Mar 13 11:36:17 fan root: Metadata, single: total=15.00GiB, used=2.38GiB
Mar 13 11:36:17 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B
Mar 13 11:36:17 fan root: btrfs fi show /
Mar 13 11:36:17 fan root: Label: 'fanbtr'  uuid: 90f8d728-6bae-4fca-8cda-b368ba2c008e
Mar 13 11:36:17 fan root: #011Total devices 1 FS bytes used 80.09GiB
Mar 13 11:36:17 fan root: #011devid    1 size 200.00GiB used 93.03GiB path /dev/mapper/fanbtr
Mar 13 11:36:17 fan root:
Mar 13 11:36:17 fan root: btrfs fi usage /
Mar 13 11:36:17 fan root: Overall:
Mar 13 11:36:17 fan root:     Device size:#011#011 200.00GiB
Mar 13 11:36:17 fan root:     Device allocated:#011#011  93.03GiB
Mar 13 11:36:17 fan root:     Device unallocated:#011#011 106.97GiB
Mar 13 11:36:17 fan root:     Device missing:#011#011     0.00B
Mar 13 11:36:17 fan root:     Used:#011#011#011  80.09GiB
Mar 13 11:36:17 fan root:     Free (estimated):#011#011 107.26GiB#011(min: 107.26GiB)
Mar 13 11:36:17 fan root:     Data ratio:#011#011#011      1.00
Mar 13 11:36:17 fan root:     Metadata ratio:#011#011      1.00
Mar 13 11:36:17 fan root:     Global reserve:#011#011 512.00MiB#011(used: 0.00B)
Mar 13 11:36:17 fan root:
Mar 13 11:36:17 fan root: Data,single: Size:78.00GiB, Used:77.71GiB
Mar 13 11:36:17 fan root:    /dev/mapper/fanbtr#011  78.00GiB
Mar 13 11:36:17 fan root:
Mar 13 11:36:17 fan root: Metadata,single: Size:15.00GiB, Used:2.38GiB
Mar 13 11:36:17 fan root:    /dev/mapper/fanbtr#011  15.00GiB
Mar 13 11:36:17 fan root:
Mar 13 11:36:17 fan root: System,single: Size:32.00MiB, Used:16.00KiB
Mar 13 11:36:17 fan root:    /dev/mapper/fanbtr#011  32.00MiB
Mar 13 11:36:17 fan root:
Mar 13 11:36:17 fan root: Unallocated:
Mar 13 11:36:17 fan root:    /dev/mapper/fanbtr#011 106.97GiB
Mar 13 11:36:17 fan root: BEGIN btrfs balance start -mprofiles=dup /
Mar 13 11:36:17 fan root: Done, had to relocate 0 out of 94 chunks
Mar 13 11:36:17 fan root: btrfs fi df /
Mar 13 11:36:17 fan root: Data, single: total=78.00GiB, used=77.71GiB
Mar 13 11:36:17 fan root: System, single: total=32.00MiB, used=16.00KiB
Mar 13 11:36:17 fan root: Metadata, single: total=15.00GiB, used=2.38GiB
Mar 13 11:36:17 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B
Mar 13 11:36:17 fan root: btrfs fi show /
Mar 13 11:36:17 fan root: Label: 'fanbtr'  uuid: 90f8d728-6bae-4fca-8cda-b368ba2c008e
Mar 13 11:36:17 fan root: #011Total devices 1 FS bytes used 80.09GiB
Mar 13 11:36:17 fan root: #011devid    1 size 200.00GiB used 93.03GiB path /dev/mapper/fanbtr
Mar 13 11:36:17 fan root:
Mar 13 11:36:17 fan root: btrfs fi usage /
Mar 13 11:36:17 fan root: Overall:
Mar 13 11:36:17 fan root:     Device size:#011#011 200.00GiB
Mar 13 11:36:17 fan root:     Device allocated:#011#011  93.03GiB
Mar 13 11:36:17 fan root:     Device unallocated:#011#011 106.97GiB
Mar 13 11:36:17 fan root:     Device missing:#011#011     0.00B
Mar 13 11:36:17 fan root:     Used:#011#011#011  80.09GiB
Mar 13 11:36:17 fan root:     Free (estimated):#011#011 107.26GiB#011(min: 107.26GiB)
Mar 13 11:36:17 fan root:     Data ratio:#011#011#011      1.00
Mar 13 11:36:17 fan root:     Metadata ratio:#011#011      1.00
Mar 13 11:36:17 fan root:     Global reserve:#011#011 512.00MiB#011(used: 0.00B)
Mar 13 11:36:17 fan root:
Mar 13 11:36:17 fan root: Data,single: Size:78.00GiB, Used:77.71GiB
Mar 13 11:36:17 fan root:    /dev/mapper/fanbtr#011  78.00GiB
Mar 13 11:36:17 fan root:
Mar 13 11:36:17 fan root: Metadata,single: Size:15.00GiB, Used:2.38GiB
Mar 13 11:36:17 fan root:    /dev/mapper/fanbtr#011  15.00GiB
Mar 13 11:36:17 fan root:
Mar 13 11:36:17 fan root: System,single: Size:32.00MiB, Used:16.00KiB
Mar 13 11:36:17 fan root:    /dev/mapper/fanbtr#011  32.00MiB
Mar 13 11:36:17 fan root:
Mar 13 11:36:17 fan root: Unallocated:
Mar 13 11:36:17 fan root:    /dev/mapper/fanbtr#011 106.97GiB
Mar 13 11:36:17 fan root: BEGIN btrfs balance start --force -sprofiles=dup /
Mar 13 11:36:17 fan root: Done, had to relocate 0 out of 94 chunks
Mar 13 11:36:17 fan root: btrfs fi df /
Mar 13 11:36:17 fan root: Data, single: total=78.00GiB, used=77.71GiB
Mar 13 11:36:17 fan root: System, single: total=32.00MiB, used=16.00KiB
Mar 13 11:36:17 fan root: Metadata, single: total=15.00GiB, used=2.38GiB
Mar 13 11:36:17 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B
Mar 13 11:36:17 fan root: btrfs fi show /
Mar 13 11:36:17 fan root: Label: 'fanbtr'  uuid: 90f8d728-6bae-4fca-8cda-b368ba2c008e
Mar 13 11:36:17 fan root: #011Total devices 1 FS bytes used 80.09GiB
Mar 13 11:36:17 fan root: #011devid    1 size 200.00GiB used 93.03GiB path /dev/mapper/fanbtr
Mar 13 11:36:17 fan root:
Mar 13 11:36:17 fan root: btrfs fi usage /
Mar 13 11:36:17 fan root: Overall:
Mar 13 11:36:17 fan root:     Device size:#011#011 200.00GiB
Mar 13 11:36:17 fan root:     Device allocated:#011#011  93.03GiB
Mar 13 11:36:17 fan root:     Device unallocated:#011#011 106.97GiB
Mar 13 11:36:17 fan root:     Device missing:#011#011     0.00B
Mar 13 11:36:17 fan root:     Used:#011#011#011  80.09GiB
Mar 13 11:36:17 fan root:     Free (estimated):#011#011 107.26GiB#011(min: 107.26GiB)
Mar 13 11:36:17 fan root:     Data ratio:#011#011#011      1.00
Mar 13 11:36:17 fan root:     Metadata ratio:#011#011      1.00
Mar 13 11:36:17 fan root:     Global reserve:#011#011 512.00MiB#011(used: 0.00B)
Mar 13 11:36:17 fan root:
Mar 13 11:36:17 fan root: Data,single: Size:78.00GiB, Used:77.71GiB
Mar 13 11:36:17 fan root:    /dev/mapper/fanbtr#011  78.00GiB
Mar 13 11:36:17 fan root:
Mar 13 11:36:17 fan root: Metadata,single: Size:15.00GiB, Used:2.38GiB
Mar 13 11:36:17 fan root:    /dev/mapper/fanbtr#011  15.00GiB
Mar 13 11:36:17 fan root:
Mar 13 11:36:17 fan root: System,single: Size:32.00MiB, Used:16.00KiB
Mar 13 11:36:17 fan root:    /dev/mapper/fanbtr#011  32.00MiB
Mar 13 11:36:17 fan root:
Mar 13 11:36:17 fan root: Unallocated:
Mar 13 11:36:17 fan root:    /dev/mapper/fanbtr#011 106.97GiB
Mar 13 11:36:17 fan root: BEGIN btrfs balance start /
Mar 13 11:51:23 fan root: ERROR: error during balancing '/': No space left on device
Mar 13 11:51:23 fan root: There may be more info in syslog - try dmesg | tail
Mar 13 11:51:23 fan root: btrfs fi df /
Mar 13 11:51:23 fan root: Data, single: total=78.00GiB, used=77.70GiB
Mar 13 11:51:23 fan root: System, single: total=32.00MiB, used=16.00KiB
Mar 13 11:51:23 fan root: Metadata, single: total=23.00GiB, used=2.38GiB
Mar 13 11:51:23 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B
Mar 13 11:51:23 fan root: btrfs fi show /
Mar 13 11:51:23 fan root: Label: 'fanbtr'  uuid: 90f8d728-6bae-4fca-8cda-b368ba2c008e
Mar 13 11:51:23 fan root: #011Total devices 1 FS bytes used 80.08GiB
Mar 13 11:51:23 fan root: #011devid    1 size 200.00GiB used 101.03GiB path /dev/mapper/fanbtr
Mar 13 11:51:23 fan root:
Mar 13 11:51:23 fan root: btrfs fi usage /
Mar 13 11:51:23 fan root: Overall:
Mar 13 11:51:23 fan root:     Device size:#011#011 200.00GiB
Mar 13 11:51:23 fan root:     Device allocated:#011#011 101.03GiB
Mar 13 11:51:23 fan root:     Device unallocated:#011#011  98.97GiB
Mar 13 11:51:23 fan root:     Device missing:#011#011     0.00B
Mar 13 11:51:23 fan root:     Used:#011#011#011  80.08GiB
Mar 13 11:51:23 fan root:     Free (estimated):#011#011  99.26GiB#011(min: 99.26GiB)
Mar 13 11:51:23 fan root:     Data ratio:#011#011#011      1.00
Mar 13 11:51:23 fan root:     Metadata ratio:#011#011      1.00
Mar 13 11:51:23 fan root:     Global reserve:#011#011 512.00MiB#011(used: 0.00B)
Mar 13 11:51:23 fan root:
Mar 13 11:51:23 fan root: Data,single: Size:78.00GiB, Used:77.70GiB
Mar 13 11:51:23 fan root:    /dev/mapper/fanbtr#011  78.00GiB
Mar 13 11:51:23 fan root:
Mar 13 11:51:23 fan root: Metadata,single: Size:23.00GiB, Used:2.38GiB
Mar 13 11:51:23 fan root:    /dev/mapper/fanbtr#011  23.00GiB
Mar 13 11:51:23 fan root:
Mar 13 11:51:23 fan root: System,single: Size:32.00MiB, Used:16.00KiB
Mar 13 11:51:23 fan root:    /dev/mapper/fanbtr#011  32.00MiB
Mar 13 11:51:23 fan root:
Mar 13 11:51:23 fan root: Unallocated:
Mar 13 11:51:23 fan root:    /dev/mapper/fanbtr#011  98.97GiB
Mar 13 11:51:23 fan root: END btrfs-balance script
[10/509]mh@fan:~$

I see the same metadata spread as with the old filesystem in btrfs fi
df, totl at 23 and used at 2.38 GiB. What I find strange is that this
filesystem has Data, System and Metadata in "single" profile, is this
the new default for a 200 GiB file system?

Full log is at http://q.bofh.de/~mh/stuff/20160313-fanbtr-btrfs-syslog

The log was taken with enospc_debug active on the file system and all
file system, block device and storage relevant log lines were left in.

Is there anything missing? Is this the same issue? Would the log help
as addition in https://bugzilla.kernel.org/show_bug.cgi?id=114451?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Andrew Vaughan]

Hi Marc

On 13 March 2016 at 22:58, Marc Haber <mh+linux-btrfs@zugschlus.de> wrote:
> Hi,
>
> On Sat, Mar 05, 2016 at 12:34:09PM -0700, Chris Murphy wrote:
>> The alternative if this can't be fixed, is to recreate the filesystem
>> because there's no practical way yet to migrate so many snapshots to a
>> new file system.
>
> I recreated the file system on March 7, with 200 GiB in size, using
> btrfs-tools 4.4. The snapshot-taking process has been running since
> then, but I also regularly cleaned up. The number of snapshots on the
> new filesystem has never exceeded 1000, with the current count being
> at 148.
>
<snip>


I'm not a dev, so I'll just thouw out a random, and possibly naive idea.

How much i/o load is this filesystem under?
What type of access pattern(s), how frequent and large are the changes?

Are you still making snapshots every 10m?
How often do you delete old snapshots?  Also every 10m, or do you
delete them in batchs every hour or so?

How long does "btrfs subvolume delete -c <one_snapshot>" take?
What does "time btrfs subvolume delete -C <one_snapshot> ; time btrfs
subvolume sync <mount_point>" print ?

The reason for asking is that even on a lightly loaded filesystem I
have seen btrfs subvolume delete take more than 30 seconds.  On a more
heavily load filesystem  I have seen 5+ minutes before btrfs subvolume
delete had finished.

If you have a high enough i/o load, plus large enough changes per
snapshot, it might be possible to get btrfs into a situation were it
never actually finishes cleaning up deleted snapshots.  (I'm also not
sure what happens if you shutdown or unmount whilst btrfs is still
cleaning up, but I expect the devs thought of that).

Andrew

Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Marc Haber]

On Mon, Mar 14, 2016 at 12:17:24AM +1100, Andrew Vaughan wrote:
> On 13 March 2016 at 22:58, Marc Haber <mh+linux-btrfs@zugschlus.de> wrote:
> > Hi,
> >
> > On Sat, Mar 05, 2016 at 12:34:09PM -0700, Chris Murphy wrote:
> >> The alternative if this can't be fixed, is to recreate the filesystem
> >> because there's no practical way yet to migrate so many snapshots to a
> >> new file system.
> >
> > I recreated the file system on March 7, with 200 GiB in size, using
> > btrfs-tools 4.4. The snapshot-taking process has been running since
> > then, but I also regularly cleaned up. The number of snapshots on the
> > new filesystem has never exceeded 1000, with the current count being
> > at 148.
> >
> <snip>
> 
> 
> I'm not a dev, so I'll just thouw out a random, and possibly naive idea.
> 
> How much i/o load is this filesystem under?
> What type of access pattern(s), how frequent and large are the changes?

Nearly none. It's a workstation which I have avoided using in the last
days due to the filesystem trouble and to avoid impact of local work
to the filesystem behavior. I even log out after working on the box
for a few minutes.

There is a Debian apt-cacher running on the box and writing its cache
to this btrfs, but /var is on its own subvolume that is only
snapshotted once a day. I'll move /var/cache to its own subvolume and
set this subvolume on a "no snapshots" schedule.

The box itself is running a couple of KVM VMs, but the virtual disks
of the VMs are on dedicated LVs.

> Are you still making snapshots every 10m?

I am snapshotting the subvolume /home/mh, with the obvious contents,
every ten minutes, yes. Most of the other subvolumes is snapshotted
once daily, with some of them not getting snapshotted at all.

> How often do you delete old snapshots?  Also every 10m, or do you
> delete them in batchs every hour or so?

I delete them in batches about every ohter day.

> How long does "btrfs subvolume delete -c <one_snapshot>" take?
> What does "time btrfs subvolume delete -C <one_snapshot> ;

[4/504]mh@fan:~$ time sudo btrfs subvolume delete -c /mnt/snapshots/fanbtr/user/subdaily/2016/03/13/07/5001/-home-mh
Delete subvolume (commit): '/mnt/snapshots/fanbtr/user/subdaily/2016/03/13/07/5001/-home-mh'

real    0m0.100s
user    0m0.000s
sys     0m0.016s
[5/505]mh@fan:~$ time sudo btrfs subvolume delete -C /mnt/snapshots/fanbtr/user/subdaily/2016/03/13/07/4001/-home-mh
Delete subvolume (commit): '/mnt/snapshots/fanbtr/user/subdaily/2016/03/13/07/4001/-home-mh'

real    0m0.079s
user    0m0.012s
sys     0m0.000s
[6/506]mh@fan:~$

The difference between -c and -C does only show when there is more
than one snapshot to be deleted.

>  time btrfs subvolume sync <mount_point>" print ?

[8/508]mh@fan:~$ time sudo btrfs subvolume sync /

real    0m0.030s
user    0m0.004s
sys     0m0.008s
[9/509]mh@fan:~$

> The reason for asking is that even on a lightly loaded filesystem I
> have seen btrfs subvolume delete take more than 30 seconds.  On a more
> heavily load filesystem  I have seen 5+ minutes before btrfs subvolume
> delete had finished.

In my experience, deleting snapshot in huge batches slows down quite a
bit, but this btrfs does not suffer from this disease.

> If you have a high enough i/o load, plus large enough changes per
> snapshot, it might be possible to get btrfs into a situation were it
> never actually finishes cleaning up deleted snapshots.  (I'm also not
> sure what happens if you shutdown or unmount whilst btrfs is still
> cleaning up, but I expect the devs thought of that).

It is a COW filesystem, I'd expect it to be consistent no matter what.
But that's the theory.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Duncan]

Marc Haber posted on Sun, 13 Mar 2016 12:58:10 +0100 as excerpted:

> I see the same metadata spread as with the old filesystem in btrfs fi
> df,
> totl at 23 and used at 2.38 GiB. What I find strange is that this
> filesystem has Data, System and Metadata in "single" profile, is this
> the new default for a 200 GiB file system?

Single is default for data.  Metadata (and system) will normally default 
to dup on a single device, raid1 on multi-device, EXCEPT on detected 
SSDs, where it defaults to single as well, because the firmware on some 
ssds will dedup it in any case.  If you know your ssd isn't one of the 
deduping ones (as I do, here), you can of course overrule that by 
specifying modes at mkfs.btrfs time.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Marc Haber]

On Sun, Mar 13, 2016 at 05:12:35PM +0000, Duncan wrote:
> Marc Haber posted on Sun, 13 Mar 2016 12:58:10 +0100 as excerpted:
> > I see the same metadata spread as with the old filesystem in btrfs fi
> > df,
> > totl at 23 and used at 2.38 GiB. What I find strange is that this
> > filesystem has Data, System and Metadata in "single" profile, is this
> > the new default for a 200 GiB file system?
> 
> Single is default for data.  Metadata (and system) will normally default 
> to dup on a single device, raid1 on multi-device, EXCEPT on detected 
> SSDs, where it defaults to single as well, because the firmware on some 
> ssds will dedup it in any case.  If you know your ssd isn't one of the 
> deduping ones (as I do, here), you can of course overrule that by 
> specifying modes at mkfs.btrfs time.

It was both times the same Samsung 840 EVO. Has this SSD detection
been added recently, or did older versions of mkfs.btrfs not detect an
SSD through a crypto layer, maybe?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Duncan]

Marc Haber posted on Sun, 13 Mar 2016 22:05:37 +0100 as excerpted:

> On Sun, Mar 13, 2016 at 05:12:35PM +0000, Duncan wrote:
>> Marc Haber posted on Sun, 13 Mar 2016 12:58:10 +0100 as excerpted:
>> > I see the same metadata spread as with the old filesystem in btrfs fi
>> > df,
>> > totl at 23 and used at 2.38 GiB. What I find strange is that this
>> > filesystem has Data, System and Metadata in "single" profile, is this
>> > the new default for a 200 GiB file system?
>> 
>> Single is default for data.  Metadata (and system) will normally
>> default to dup on a single device, raid1 on multi-device, EXCEPT on
>> detected SSDs, where it defaults to single as well, because the
>> firmware on some ssds will dedup it in any case.  If you know your ssd
>> isn't one of the deduping ones (as I do, here), you can of course
>> overrule that by specifying modes at mkfs.btrfs time.
> 
> It was both times the same Samsung 840 EVO. Has this SSD detection been
> added recently, or did older versions of mkfs.btrfs not detect an SSD
> through a crypto layer, maybe?

Btrfs' ssd detection has been there for quite some time now (for 
userspace, since well before the releases synced to kernel version with 
3.12 or so, which effectively makes it ancient history in btrfs terms).

But according to the mkfs.btrfs manpage, the detection is based on 
/sys/block/DEV/queue/rotational (with DEV substituted appropriately), and 
various layers got support for correctly passing that thru at various 
times, some before btrfs, some after.  So that's very likely why btrfs 
didn't detect it originally, if it was on top of crypto and/or some other 
layer that might not have been passing that thru.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Marc Haber]

On Mon, Mar 14, 2016 at 01:05:39AM +0000, Duncan wrote:
> But according to the mkfs.btrfs manpage, the detection is based on 
> /sys/block/DEV/queue/rotational (with DEV substituted appropriately), and 
> various layers got support for correctly passing that thru at various 
> times, some before btrfs, some after.  So that's very likely why btrfs 
> didn't detect it originally, if it was on top of crypto and/or some other 
> layer that might not have been passing that thru.

That explains it, thanks.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Henk Slager]

On Sun, Mar 13, 2016 at 12:58 PM, Marc Haber
<mh+linux-btrfs@zugschlus.de> wrote:
> Hi,
>
> On Sat, Mar 05, 2016 at 12:34:09PM -0700, Chris Murphy wrote:
>> The alternative if this can't be fixed, is to recreate the filesystem
>> because there's no practical way yet to migrate so many snapshots to a
>> new file system.
>
> I recreated the file system on March 7, with 200 GiB in size, using
> btrfs-tools 4.4. The snapshot-taking process has been running since
> then, but I also regularly cleaned up. The number of snapshots on the
> new filesystem has never exceeded 1000, with the current count being
> at 148.

Is the snapshotting still read-write?
You mentioned earlier that you treated the snapshots as read-only, so
maybe create them also as read-only, in an attempt to mitigate the
problem of growing metadata and enospc issues.
Also, If some part of the OS or tools scans through the snapshot dirs
every now and then with atime creation on, metadata grows without a
real need.

Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Henk Slager]

On Sun, Mar 13, 2016 at 8:14 PM, Henk Slager <eye1tm@gmail.com> wrote:
> On Sun, Mar 13, 2016 at 12:58 PM, Marc Haber
> <mh+linux-btrfs@zugschlus.de> wrote:
>> Hi,
>>
>> On Sat, Mar 05, 2016 at 12:34:09PM -0700, Chris Murphy wrote:
>>> The alternative if this can't be fixed, is to recreate the filesystem
>>> because there's no practical way yet to migrate so many snapshots to a
>>> new file system.
>>
>> I recreated the file system on March 7, with 200 GiB in size, using
>> btrfs-tools 4.4. The snapshot-taking process has been running since
>> then, but I also regularly cleaned up. The number of snapshots on the
>> new filesystem has never exceeded 1000, with the current count being
>> at 148.
>
> Is the snapshotting still read-write?
> You mentioned earlier that you treated the snapshots as read-only, so
> maybe create them also as read-only, in an attempt to mitigate the
> problem of growing metadata and enospc issues.

Sorry, I mixed it up with another thread; Forget about the question.

Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Marc Haber]

On Sun, Mar 13, 2016 at 08:14:45PM +0100, Henk Slager wrote:
> On Sun, Mar 13, 2016 at 12:58 PM, Marc Haber
> <mh+linux-btrfs@zugschlus.de> wrote:
> > Hi,
> >
> > On Sat, Mar 05, 2016 at 12:34:09PM -0700, Chris Murphy wrote:
> >> The alternative if this can't be fixed, is to recreate the filesystem
> >> because there's no practical way yet to migrate so many snapshots to a
> >> new file system.
> >
> > I recreated the file system on March 7, with 200 GiB in size, using
> > btrfs-tools 4.4. The snapshot-taking process has been running since
> > then, but I also regularly cleaned up. The number of snapshots on the
> > new filesystem has never exceeded 1000, with the current count being
> > at 148.
> 
> Is the snapshotting still read-write?

Yes, I want to keep the possibility to remove huge files from
snapshots that shouldnt have been on a snapshotted volume in the first
place without having to ditch the entire snapshot.

> Also, If some part of the OS or tools scans through the snapshot dirs
> every now and then with atime creation on, metadata grows without a
> real need.

I mount with noatime and nodiratime anyway, and the directory the
snapshots are mounted to (/mnt/snapshots) are excluded in
updatedb.conf. Any other idea which tool might scan filesystems and
that might not be noticed when it's running about a five digit number
of snapshots?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Henk Slager]

On Sun, Mar 13, 2016 at 9:56 PM, Marc Haber <mh+linux-btrfs@zugschlus.de> wrote:
> On Sun, Mar 13, 2016 at 08:14:45PM +0100, Henk Slager wrote:
>> On Sun, Mar 13, 2016 at 12:58 PM, Marc Haber
>> <mh+linux-btrfs@zugschlus.de> wrote:
>> > Hi,
>> >
>> > On Sat, Mar 05, 2016 at 12:34:09PM -0700, Chris Murphy wrote:
>> >> The alternative if this can't be fixed, is to recreate the filesystem
>> >> because there's no practical way yet to migrate so many snapshots to a
>> >> new file system.
>> >
>> > I recreated the file system on March 7, with 200 GiB in size, using
>> > btrfs-tools 4.4. The snapshot-taking process has been running since
>> > then, but I also regularly cleaned up. The number of snapshots on the
>> > new filesystem has never exceeded 1000, with the current count being
>> > at 148.
>>
>> Is the snapshotting still read-write?
>
> Yes, I want to keep the possibility to remove huge files from
> snapshots that shouldnt have been on a snapshotted volume in the first
> place without having to ditch the entire snapshot.

You could do ro snapshotting and in case you want to modify something
inside a snapshot/subvolume:
# btrfs property set <subvolume> ro false
# rm <subvolume>/<somefile>
# btrfs property set <subvolume> ro true

>> Also, If some part of the OS or tools scans through the snapshot dirs
>> every now and then with atime creation on, metadata grows without a
>> real need.
>
> I mount with noatime and nodiratime anyway, and the directory the
> snapshots are mounted to (/mnt/snapshots) are excluded in
> updatedb.conf. Any other idea which tool might scan filesystems and
> that might not be noticed when it's running about a five digit number
> of snapshots?

Maybe baloo or so if you use KDE. Someone else reporting on this
maillist was searching snapshots dirs for latest before creating new
snapshot. That was more related to fs performance rather then the
metadata issue you experience.

The rw vs. ro snapshotting is the only think I can think of in order
to stop having the issue. I'm using both snapper and my own
scripts+btrfspogs since 2+ years on various small and big filesystems
with 100-1500 snapshots per fs without any metadata issues or so, but
it is all ro snapshots. And actually, if I want some modification
(rarely), I create another rw snapshot of the particular snapshot.

Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Marc Haber]

On Mon, Mar 14, 2016 at 01:00:13AM +0100, Henk Slager wrote:
> On Sun, Mar 13, 2016 at 9:56 PM, Marc Haber <mh+linux-btrfs@zugschlus.de> wrote:
> > Yes, I want to keep the possibility to remove huge files from
> > snapshots that shouldnt have been on a snapshotted volume in the first
> > place without having to ditch the entire snapshot.
> 
> You could do ro snapshotting and in case you want to modify something
> inside a snapshot/subvolume:
> # btrfs property set <subvolume> ro false
> # rm <subvolume>/<somefile>
> # btrfs property set <subvolume> ro true

I was not aware that it is possible to fiddle with the ro property of
an already existing snapshot. I am not yet sure whether I love or hate
this.

> >> Also, If some part of the OS or tools scans through the snapshot dirs
> >> every now and then with atime creation on, metadata grows without a
> >> real need.
> >
> > I mount with noatime and nodiratime anyway, and the directory the
> > snapshots are mounted to (/mnt/snapshots) are excluded in
> > updatedb.conf. Any other idea which tool might scan filesystems and
> > that might not be noticed when it's running about a five digit number
> > of snapshots?
> 
> Maybe baloo or so if you use KDE.

I usually do those tests via ssh without even being logged in to a
local desktop.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Marc Haber]

On Sun, Mar 13, 2016 at 12:58:09PM +0100, Marc Haber wrote:
> On Sat, Mar 05, 2016 at 12:34:09PM -0700, Chris Murphy wrote:
> > The alternative if this can't be fixed, is to recreate the filesystem
> > because there's no practical way yet to migrate so many snapshots to a
> > new file system.
> 
> I recreated the file system on March 7, with 200 GiB in size, using
> btrfs-tools 4.4. The snapshot-taking process has been running since
> then, but I also regularly cleaned up. The number of snapshots on the
> new filesystem has never exceeded 1000, with the current count being
> at 148.
> 
> And btrfs balance runs into the same ENOSPC issues as the old one:

... with Qu's patch, I now get a reproducible kernel trace:

Mar 14 10:23:49 fan mh: BEGIN btrfs-balance script
Mar 14 10:23:49 fan mh: btrfs fi df /
Mar 14 10:23:49 fan root: Data, single: total=79.00GiB, used=78.42GiB
Mar 14 10:23:49 fan root: System, single: total=32.00MiB, used=16.00KiB
Mar 14 10:23:49 fan root: Metadata, single: total=10.00GiB, used=2.46GiB
Mar 14 10:23:49 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B
Mar 14 10:23:49 fan mh: btrfs fi show /
Mar 14 10:23:49 fan root: Label: 'fanbtr'  uuid: 90f8d728-6bae-4fca-8cda-b368ba2c008e
Mar 14 10:23:49 fan root: #011Total devices 1 FS bytes used 80.89GiB
Mar 14 10:23:49 fan root: #011devid    1 size 200.00GiB used 89.03GiB path /dev/mapper/fanbtr
Mar 14 10:23:49 fan root: 
Mar 14 10:23:49 fan mh: btrfs fi usage /
Mar 14 10:23:49 fan root: Overall:
Mar 14 10:23:49 fan root:     Device size:#011#011 200.00GiB
Mar 14 10:23:49 fan root:     Device allocated:#011#011  89.03GiB
Mar 14 10:23:49 fan root:     Device unallocated:#011#011 110.97GiB
Mar 14 10:23:49 fan root:     Device missing:#011#011     0.00B
Mar 14 10:23:49 fan root:     Used:#011#011#011  80.89GiB
Mar 14 10:23:49 fan root:     Free (estimated):#011#011 111.54GiB#011(min: 111.54GiB)
Mar 14 10:23:49 fan root:     Data ratio:#011#011#011      1.00
Mar 14 10:23:49 fan root:     Metadata ratio:#011#011      1.00
Mar 14 10:23:49 fan root:     Global reserve:#011#011 512.00MiB#011(used: 0.00B)
Mar 14 10:23:49 fan root: 
Mar 14 10:23:49 fan root: Data,single: Size:79.00GiB, Used:78.42GiB
Mar 14 10:23:49 fan root:    /dev/mapper/fanbtr#011  79.00GiB
Mar 14 10:23:49 fan root: 
Mar 14 10:23:49 fan root: Metadata,single: Size:10.00GiB, Used:2.46GiB
Mar 14 10:23:49 fan root:    /dev/mapper/fanbtr#011  10.00GiB
Mar 14 10:23:49 fan root: 
Mar 14 10:23:49 fan root: System,single: Size:32.00MiB, Used:16.00KiB
Mar 14 10:23:49 fan root:    /dev/mapper/fanbtr#011  32.00MiB
Mar 14 10:23:49 fan root: 
Mar 14 10:23:49 fan root: Unallocated:
Mar 14 10:23:49 fan root:    /dev/mapper/fanbtr#011 110.97GiB
Mar 14 10:23:49 fan mh: BEGIN btrfs balance start /
Mar 14 10:36:46 fan kernel: [  890.995815] BTRFS info (device dm-15): 6 enospc errors during balance
Mar 14 10:36:46 fan root: ERROR: error during balancing '/': No space left on device
Mar 14 10:36:46 fan root: There may be more info in syslog - try dmesg | tail
Mar 14 10:36:46 fan root: btrfs fi df /
Mar 14 10:36:46 fan root: Data, single: total=79.00GiB, used=78.42GiB
Mar 14 10:36:46 fan root: System, single: total=32.00MiB, used=16.00KiB
Mar 14 10:36:46 fan root: Metadata, single: total=12.00GiB, used=2.46GiB
Mar 14 10:36:46 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B
Mar 14 10:36:46 fan root: btrfs fi show /
Mar 14 10:36:46 fan root: Label: 'fanbtr'  uuid: 90f8d728-6bae-4fca-8cda-b368ba2c008e
Mar 14 10:36:46 fan root: #011Total devices 1 FS bytes used 80.89GiB
Mar 14 10:36:46 fan root: #011devid    1 size 200.00GiB used 91.03GiB path /dev/mapper/fanbtr
Mar 14 10:36:46 fan root: 
Mar 14 10:36:46 fan root: btrfs fi usage /
Mar 14 10:36:46 fan root: Overall:
Mar 14 10:36:46 fan root:     Device size:#011#011 200.00GiB
Mar 14 10:36:46 fan root:     Device allocated:#011#011  91.03GiB
Mar 14 10:36:46 fan root:     Device unallocated:#011#011 108.97GiB
Mar 14 10:36:46 fan root:     Device missing:#011#011     0.00B
Mar 14 10:36:46 fan root:     Used:#011#011#011  80.89GiB
Mar 14 10:36:46 fan root:     Free (estimated):#011#011 109.54GiB#011(min: 109.54GiB)
Mar 14 10:36:46 fan root:     Data ratio:#011#011#011      1.00
Mar 14 10:36:46 fan root:     Metadata ratio:#011#011      1.00
Mar 14 10:36:46 fan root:     Global reserve:#011#011 512.00MiB#011(used: 0.00B)
Mar 14 10:36:46 fan root: 
Mar 14 10:36:46 fan root: Data,single: Size:79.00GiB, Used:78.42GiB
Mar 14 10:36:46 fan root:    /dev/mapper/fanbtr#011  79.00GiB
Mar 14 10:36:46 fan root: 
Mar 14 10:36:46 fan root: Metadata,single: Size:12.00GiB, Used:2.46GiB
Mar 14 10:36:46 fan root:    /dev/mapper/fanbtr#011  12.00GiB
Mar 14 10:36:46 fan root: 
Mar 14 10:36:46 fan root: System,single: Size:32.00MiB, Used:16.00KiB
Mar 14 10:36:46 fan root:    /dev/mapper/fanbtr#011  32.00MiB
Mar 14 10:36:46 fan root: 
Mar 14 10:36:46 fan root: Unallocated:
Mar 14 10:36:46 fan root:    /dev/mapper/fanbtr#011 108.97GiB
Mar 14 10:36:46 fan root: BEGIN btrfs balance start -dprofiles=single /
Mar 14 10:48:16 fan kernel: [ 1581.228727] ------------[ cut here ]------------
Mar 14 10:48:16 fan kernel: [ 1581.228794] WARNING: CPU: 1 PID: 121 at fs/btrfs/extent-tree.c:7897 btrfs_alloc_tree_block+0xeb/0x3d6 [btrfs]()
Mar 14 10:48:16 fan kernel: [ 1581.228800] BTRFS: block rsv returned -28
Mar 14 10:48:16 fan kernel: [ 1581.228804] Modules linked in: iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp dummy ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative bridge stp llc snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi amd64_edac_mod edac_mce_amd kvm_amd input_leds pcspkr edac_core snd_hda_intel kvm snd_cmipci snd_hda_codec snd_mpu401_uart snd_opl3_lib snd_rawmidi snd_hda_core snd_seq_device irqbypass k10temp snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_timer evdev snd asus_atk0110 soundcore acpi_cpufreq i2c_piix4 tpm_tis sg tpm processor shpchp hwmon_vid autofs4 crc32c_generic btrfs xor raid6_pq ext4 crc16 mbcache jbd2 hmac sha256_ssse3 sha256_generic drbg ansi_cprng xts gf128mul algif_skcipher af_alg dm_crypt dm_mod hid_generic usbhid hid usb_storage sr_mod sd_mod cdrom ohci_pci ahci libahci amdkfd r8169 mii radeon xhci_pci i2c_algo_bit xhci_hcd ttm sym53c8xx ehci_pci scsi_transport_spi ohci_hcd drm_kms_helper libata ehci_hcd drm usbcore scsi_mod usb_common i2c_core button
Mar 14 10:48:16 fan kernel: [ 1581.228957] CPU: 1 PID: 121 Comm: kworker/u16:2 Not tainted 4.4.5-zgws1 #2
Mar 14 10:48:16 fan kernel: [ 1581.228963] Hardware name: System manufacturer System Product Name/M5A88-V EVO, BIOS 1603    10/12/2012
Mar 14 10:48:16 fan kernel: [ 1581.229012] Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs]
Mar 14 10:48:16 fan kernel: [ 1581.229018]  0000000000000000 ffffffff811dd418 ffff880613aafb58 0000000000000009
Mar 14 10:48:16 fan kernel: [ 1581.229026]  ffffffff81051e21 ffffffffa047d29f ffff8804350d6800 ffff880613aafbb0
Mar 14 10:48:16 fan kernel: [ 1581.229033]  ffff880610322000 ffff880531466700 ffffffff81051e79 ffffffffa04f065b
Mar 14 10:48:16 fan kernel: [ 1581.229039] Call Trace:
Mar 14 10:48:16 fan kernel: [ 1581.229052]  [<ffffffff811dd418>] ? dump_stack+0x5a/0x6f
Mar 14 10:48:16 fan kernel: [ 1581.229061]  [<ffffffff81051e21>] ? warn_slowpath_common+0x8e/0xa3
Mar 14 10:48:16 fan kernel: [ 1581.229104]  [<ffffffffa047d29f>] ? btrfs_alloc_tree_block+0xeb/0x3d6 [btrfs]
Mar 14 10:48:16 fan kernel: [ 1581.229112]  [<ffffffff81051e79>] ? warn_slowpath_fmt+0x43/0x4b
Mar 14 10:48:16 fan kernel: [ 1581.229155]  [<ffffffffa047d29f>] ? btrfs_alloc_tree_block+0xeb/0x3d6 [btrfs]
Mar 14 10:48:16 fan kernel: [ 1581.229194]  [<ffffffffa046b05b>] ? btrfs_copy_root+0x9e/0x1c1 [btrfs]
Mar 14 10:48:16 fan kernel: [ 1581.229239]  [<ffffffffa04c7844>] ? create_reloc_root+0x6c/0x196 [btrfs]
Mar 14 10:48:16 fan kernel: [ 1581.229285]  [<ffffffffa04cb3cd>] ? btrfs_init_reloc_root+0x71/0x9a [btrfs]
Mar 14 10:48:16 fan kernel: [ 1581.229329]  [<ffffffffa0489d74>] ? record_root_in_trans+0xc8/0xd5 [btrfs]
Mar 14 10:48:16 fan kernel: [ 1581.229374]  [<ffffffffa048b192>] ? btrfs_record_root_in_trans+0x42/0x5a [btrfs]
Mar 14 10:48:16 fan kernel: [ 1581.229418]  [<ffffffffa048c383>] ? start_transaction+0x2de/0x455 [btrfs]
Mar 14 10:48:16 fan kernel: [ 1581.229464]  [<ffffffffa0492c67>] ? btrfs_finish_ordered_io+0x1e2/0x4d7 [btrfs]
Mar 14 10:48:16 fan kernel: [ 1581.229474]  [<ffffffff81078953>] ? pick_next_task_fair+0x1f8/0x357
Mar 14 10:48:16 fan kernel: [ 1581.229482]  [<ffffffff810752af>] ? account_entity_dequeue+0x46/0x67
Mar 14 10:48:16 fan kernel: [ 1581.229529]  [<ffffffffa04b1ec4>] ? btrfs_scrubparity_helper+0xf4/0x233 [btrfs]
Mar 14 10:48:16 fan kernel: [ 1581.229538]  [<ffffffff81063b4f>] ? process_one_work+0x178/0x27b
Mar 14 10:48:16 fan kernel: [ 1581.229545]  [<ffffffff810640d3>] ? worker_thread+0x1da/0x280
Mar 14 10:48:16 fan kernel: [ 1581.229553]  [<ffffffff81063ef9>] ? rescuer_thread+0x284/0x284
Mar 14 10:48:16 fan kernel: [ 1581.229559]  [<ffffffff81067e59>] ? kthread+0x95/0x9d
Mar 14 10:48:16 fan kernel: [ 1581.229566]  [<ffffffff81067dc4>] ? kthread_parkme+0x16/0x16
Mar 14 10:48:16 fan kernel: [ 1581.229574]  [<ffffffff8140dfff>] ? ret_from_fork+0x3f/0x70
Mar 14 10:48:16 fan kernel: [ 1581.229580]  [<ffffffff81067dc4>] ? kthread_parkme+0x16/0x16
Mar 14 10:48:16 fan kernel: [ 1581.229586] ---[ end trace 69a5cc7238dc3665 ]---
Mar 14 10:51:06 fan root: Done, had to relocate 79 out of 92 chunks
Mar 14 10:51:06 fan root: btrfs fi df /
Mar 14 10:51:06 fan root: Data, single: total=79.00GiB, used=78.43GiB
Mar 14 10:51:06 fan root: System, single: total=32.00MiB, used=16.00KiB
Mar 14 10:51:06 fan root: Metadata, single: total=12.00GiB, used=2.46GiB
Mar 14 10:51:06 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B
Mar 14 10:51:06 fan root: btrfs fi show /
Mar 14 10:51:06 fan root: Label: 'fanbtr'  uuid: 90f8d728-6bae-4fca-8cda-b368ba2c008e
Mar 14 10:51:06 fan root: #011Total devices 1 FS bytes used 80.89GiB
Mar 14 10:51:06 fan root: #011devid    1 size 200.00GiB used 91.03GiB path /dev/mapper/fanbtr
Mar 14 10:51:06 fan root: 
Mar 14 10:51:06 fan root: btrfs fi usage /
Mar 14 10:51:06 fan root: Overall:
Mar 14 10:51:06 fan root:     Device size:#011#011 200.00GiB
Mar 14 10:51:06 fan root:     Device allocated:#011#011  91.03GiB
Mar 14 10:51:06 fan root:     Device unallocated:#011#011 108.97GiB
Mar 14 10:51:06 fan root:     Device missing:#011#011     0.00B
Mar 14 10:51:06 fan root:     Used:#011#011#011  80.89GiB
Mar 14 10:51:06 fan root:     Free (estimated):#011#011 109.54GiB#011(min: 109.54GiB)
Mar 14 10:51:06 fan root:     Data ratio:#011#011#011      1.00
Mar 14 10:51:06 fan root:     Metadata ratio:#011#011      1.00
Mar 14 10:51:06 fan root:     Global reserve:#011#011 512.00MiB#011(used: 0.00B)
Mar 14 10:51:06 fan root: 
Mar 14 10:51:06 fan root: Data,single: Size:79.00GiB, Used:78.43GiB
Mar 14 10:51:06 fan root:    /dev/mapper/fanbtr#011  79.00GiB
Mar 14 10:51:06 fan root: 
Mar 14 10:51:06 fan root: Metadata,single: Size:12.00GiB, Used:2.46GiB
Mar 14 10:51:06 fan root:    /dev/mapper/fanbtr#011  12.00GiB
Mar 14 10:51:06 fan root: 
Mar 14 10:51:06 fan root: System,single: Size:32.00MiB, Used:16.00KiB
Mar 14 10:51:06 fan root:    /dev/mapper/fanbtr#011  32.00MiB
Mar 14 10:51:06 fan root: 
Mar 14 10:51:06 fan root: Unallocated:
Mar 14 10:51:06 fan root:    /dev/mapper/fanbtr#011 108.97GiB
Mar 14 10:51:06 fan root: BEGIN btrfs balance start -mprofiles=dup /
Mar 14 10:51:06 fan root: Done, had to relocate 0 out of 92 chunks
Mar 14 10:51:06 fan root: btrfs fi df /
Mar 14 10:51:06 fan root: Data, single: total=79.00GiB, used=78.43GiB
Mar 14 10:51:06 fan root: System, single: total=32.00MiB, used=16.00KiB
Mar 14 10:51:06 fan root: Metadata, single: total=12.00GiB, used=2.46GiB
Mar 14 10:51:06 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B
Mar 14 10:51:06 fan root: btrfs fi show /
Mar 14 10:51:06 fan root: Label: 'fanbtr'  uuid: 90f8d728-6bae-4fca-8cda-b368ba2c008e
Mar 14 10:51:06 fan root: #011Total devices 1 FS bytes used 80.89GiB
Mar 14 10:51:06 fan root: #011devid    1 size 200.00GiB used 91.03GiB path /dev/mapper/fanbtr
Mar 14 10:51:06 fan root: 
Mar 14 10:51:06 fan root: btrfs fi usage /
Mar 14 10:51:06 fan root: Overall:
Mar 14 10:51:06 fan root:     Device size:#011#011 200.00GiB
Mar 14 10:51:06 fan root:     Device allocated:#011#011  91.03GiB
Mar 14 10:51:06 fan root:     Device unallocated:#011#011 108.97GiB
Mar 14 10:51:06 fan root:     Device missing:#011#011     0.00B
Mar 14 10:51:06 fan root:     Used:#011#011#011  80.89GiB
Mar 14 10:51:06 fan root:     Free (estimated):#011#011 109.54GiB#011(min: 109.54GiB)
Mar 14 10:51:06 fan root:     Data ratio:#011#011#011      1.00
Mar 14 10:51:06 fan root:     Metadata ratio:#011#011      1.00
Mar 14 10:51:06 fan root:     Global reserve:#011#011 512.00MiB#011(used: 0.00B)
Mar 14 10:51:06 fan root: 
Mar 14 10:51:06 fan root: Data,single: Size:79.00GiB, Used:78.43GiB
Mar 14 10:51:06 fan root:    /dev/mapper/fanbtr#011  79.00GiB
Mar 14 10:51:06 fan root: 
Mar 14 10:51:06 fan root: Metadata,single: Size:12.00GiB, Used:2.46GiB
Mar 14 10:51:06 fan root:    /dev/mapper/fanbtr#011  12.00GiB
Mar 14 10:51:06 fan root: 
Mar 14 10:51:06 fan root: System,single: Size:32.00MiB, Used:16.00KiB
Mar 14 10:51:06 fan root:    /dev/mapper/fanbtr#011  32.00MiB
Mar 14 10:51:06 fan root: 
Mar 14 10:51:06 fan root: Unallocated:
Mar 14 10:51:06 fan root:    /dev/mapper/fanbtr#011 108.97GiB
Mar 14 10:51:06 fan root: BEGIN btrfs balance start --force -sprofiles=dup /
Mar 14 10:51:06 fan root: Done, had to relocate 0 out of 92 chunks
Mar 14 10:51:06 fan root: btrfs fi df /
Mar 14 10:51:06 fan root: Data, single: total=79.00GiB, used=78.43GiB
Mar 14 10:51:06 fan root: System, single: total=32.00MiB, used=16.00KiB
Mar 14 10:51:06 fan root: Metadata, single: total=12.00GiB, used=2.46GiB
Mar 14 10:51:06 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B
Mar 14 10:51:06 fan root: btrfs fi show /
Mar 14 10:51:06 fan root: Label: 'fanbtr'  uuid: 90f8d728-6bae-4fca-8cda-b368ba2c008e
Mar 14 10:51:06 fan root: #011Total devices 1 FS bytes used 80.89GiB
Mar 14 10:51:06 fan root: #011devid    1 size 200.00GiB used 91.03GiB path /dev/mapper/fanbtr
Mar 14 10:51:06 fan root: 
Mar 14 10:51:06 fan root: btrfs fi usage /
Mar 14 10:51:06 fan root: Overall:
Mar 14 10:51:06 fan root:     Device size:#011#011 200.00GiB
Mar 14 10:51:06 fan root:     Device allocated:#011#011  91.03GiB
Mar 14 10:51:06 fan root:     Device unallocated:#011#011 108.97GiB
Mar 14 10:51:06 fan root:     Device missing:#011#011     0.00B
Mar 14 10:51:06 fan root:     Used:#011#011#011  80.89GiB
Mar 14 10:51:06 fan root:     Free (estimated):#011#011 109.54GiB#011(min: 109.54GiB)
Mar 14 10:51:06 fan root:     Data ratio:#011#011#011      1.00
Mar 14 10:51:06 fan root:     Metadata ratio:#011#011      1.00
Mar 14 10:51:06 fan root:     Global reserve:#011#011 512.00MiB#011(used: 0.00B)
Mar 14 10:51:06 fan root: 
Mar 14 10:51:06 fan root: Data,single: Size:79.00GiB, Used:78.43GiB
Mar 14 10:51:06 fan root:    /dev/mapper/fanbtr#011  79.00GiB
Mar 14 10:51:06 fan root: 
Mar 14 10:51:06 fan root: Metadata,single: Size:12.00GiB, Used:2.46GiB
Mar 14 10:51:06 fan root:    /dev/mapper/fanbtr#011  12.00GiB
Mar 14 10:51:06 fan root: 
Mar 14 10:51:06 fan root: System,single: Size:32.00MiB, Used:16.00KiB
Mar 14 10:51:06 fan root:    /dev/mapper/fanbtr#011  32.00MiB
Mar 14 10:51:06 fan root: 
Mar 14 10:51:06 fan root: Unallocated:
Mar 14 10:51:06 fan root:    /dev/mapper/fanbtr#011 108.97GiB
Mar 14 10:51:06 fan root: BEGIN btrfs balance start /
Mar 14 11:08:36 fan kernel: [ 2800.565044] BTRFS info (device dm-15): 7 enospc errors during balance
Mar 14 11:08:36 fan root: ERROR: error during balancing '/': No space left on device
Mar 14 11:08:36 fan root: There may be more info in syslog - try dmesg | tail
Mar 14 11:08:36 fan mh: btrfs fi df /
Mar 14 11:08:36 fan root: Data, single: total=79.00GiB, used=78.43GiB
Mar 14 11:08:36 fan root: System, single: total=32.00MiB, used=16.00KiB
Mar 14 11:08:36 fan root: Metadata, single: total=19.00GiB, used=2.46GiB
Mar 14 11:08:36 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B
Mar 14 11:08:36 fan mh: btrfs fi show /
Mar 14 11:08:36 fan root: Label: 'fanbtr'  uuid: 90f8d728-6bae-4fca-8cda-b368ba2c008e
Mar 14 11:08:36 fan root: #011Total devices 1 FS bytes used 80.89GiB
Mar 14 11:08:36 fan root: #011devid    1 size 200.00GiB used 98.03GiB path /dev/mapper/fanbtr
Mar 14 11:08:36 fan root: 
Mar 14 11:08:36 fan mh: btrfs fi usage /
Mar 14 11:08:36 fan root: Overall:
Mar 14 11:08:36 fan root:     Device size:#011#011 200.00GiB
Mar 14 11:08:36 fan root:     Device allocated:#011#011  98.03GiB
Mar 14 11:08:36 fan root:     Device unallocated:#011#011 101.97GiB
Mar 14 11:08:36 fan root:     Device missing:#011#011     0.00B
Mar 14 11:08:36 fan root:     Used:#011#011#011  80.89GiB
Mar 14 11:08:36 fan root:     Free (estimated):#011#011 102.54GiB#011(min: 102.54GiB)
Mar 14 11:08:36 fan root:     Data ratio:#011#011#011      1.00
Mar 14 11:08:36 fan root:     Metadata ratio:#011#011      1.00
Mar 14 11:08:36 fan root:     Global reserve:#011#011 512.00MiB#011(used: 0.00B)
Mar 14 11:08:36 fan root: 
Mar 14 11:08:36 fan root: Data,single: Size:79.00GiB, Used:78.43GiB
Mar 14 11:08:36 fan root:    /dev/mapper/fanbtr#011  79.00GiB
Mar 14 11:08:36 fan root: 
Mar 14 11:08:36 fan root: Metadata,single: Size:19.00GiB, Used:2.46GiB
Mar 14 11:08:36 fan root:    /dev/mapper/fanbtr#011  19.00GiB
Mar 14 11:08:36 fan root: 
Mar 14 11:08:36 fan root: System,single: Size:32.00MiB, Used:16.00KiB
Mar 14 11:08:36 fan root:    /dev/mapper/fanbtr#011  32.00MiB
Mar 14 11:08:36 fan root: 
Mar 14 11:08:36 fan root: Unallocated:
Mar 14 11:08:36 fan root:    /dev/mapper/fanbtr#011 101.97GiB
Mar 14 11:08:36 fan mh: END btrfs-balance script

Full log is at http://q.bofh.de/~mh/stuff/20160314-fanbtr-btrfs-syslog

The log was taken with enospc_debug active on the file system and all
file system, block device and storage relevant log lines were left in.

What does the trace tell us here?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: New file system with same issue

[Holger Hoffstätte]

On 03/14/16 13:07, Marc Haber wrote:
>> And btrfs balance runs into the same ENOSPC issues as the old one:
> 
> ... with Qu's patch, I now get a reproducible kernel trace:

<snip>

That is interesting and useful. Sorry if this was asked before, but
did you ever try to clear the free-space cache via -o clear_cache
on mount?

Give it a try, let it run for a while and then try balancing
again. There is definitely something wrong with the chunk
allocation on your system since you still have allegedly
~100G free, so it's not the notorious (and a lot less common these
days) edge case of sparsely populated chunks that would require
a 'compaction'.

IMHO you are also looking at two different, possibly unrelated
problems: failure to allocate more chunks vs. metadata bloat,
so let's not confuse the two.

Uncle Occam's razor also suggests that the involvement of dm
doesn't help. Why not just use the device/partition directly?
_Someone_ is lying to btrfs in terms of device size and/or allocated
chunks, otherwise you wouldn't get the ENOSPC.

-h

Re: New file system with same issue

[Marc Haber]

On Mon, Mar 14, 2016 at 01:48:18PM +0100, Holger Hoffstätte wrote:
> On 03/14/16 13:07, Marc Haber wrote:
> >> And btrfs balance runs into the same ENOSPC issues as the old one:
> > 
> > ... with Qu's patch, I now get a reproducible kernel trace:
> 
> <snip>
> 
> That is interesting and useful. Sorry if this was asked before, but
> did you ever try to clear the free-space cache via -o clear_cache
> on mount?

This was not asked, and I didn't try. Since this is an encrypted root
filesystem, is it a workable way to add clear_cache to /etc/fstab,
rebuild initramfs and reboot? Or do you recommend using a rescue system?

> Give it a try, let it run for a while and then try balancing
> again.

Do I need to wait for clear_cache to finish, like until I see disk
usage dropping?

> Uncle Occam's razor also suggests that the involvement of dm
> doesn't help. Why not just use the device/partition directly?

I need the dm intermediate since I don't want to repartition the
expensive SSD and the entire system is crypted.

> _Someone_ is lying to btrfs in terms of device size and/or allocated
> chunks, otherwise you wouldn't get the ENOSPC.

Which properties does a block device report other than size?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: New file system with same issue

[Holger Hoffstätte]

On 03/14/16 21:13, Marc Haber wrote:
> On Mon, Mar 14, 2016 at 01:48:18PM +0100, Holger Hoffstätte wrote:
>> did you ever try to clear the free-space cache via -o clear_cache
>> on mount?
> 
> This was not asked, and I didn't try. Since this is an encrypted root
> filesystem, is it a workable way to add clear_cache to /etc/fstab,
> rebuild initramfs and reboot? Or do you recommend using a rescue system?

If you can do it via a rescue system that might be easiest, but adding
it to fstab and rebooting once has the same effect. Whatever you know
how to do safely.

>> Give it a try, let it run for a while and then try balancing
>> again.
> 
> Do I need to wait for clear_cache to finish, like until I see disk
> usage dropping?

The cache isn't that big, so you won't see a huge drop. Just use the
disk normally for a few minutes, after some time the cache will be
written out again.

>> _Someone_ is lying to btrfs in terms of device size and/or allocated
>> chunks, otherwise you wouldn't get the ENOSPC.
> 
> Which properties does a block device report other than size?

Well..at least all you can find in /sys/block/sdX/*. However, reading
the other subthread about the mismatching image size I'm now none the
wiser what else to suggest. :/

-h

Re: New file system with same issue

[Marc Haber]

On Tue, Mar 15, 2016 at 11:52:30AM +0100, Holger Hoffstätte wrote:
> On 03/14/16 21:13, Marc Haber wrote:
> > Do I need to wait for clear_cache to finish, like until I see disk
> > usage dropping?
> 
> The cache isn't that big, so you won't see a huge drop. Just use the
> disk normally for a few minutes, after some time the cache will be
> written out again.

Is it necessary to actually cause activity on the file system or is it
ok to just let it sit there for an hour or so?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: New file system with same issue

[Austin S. Hemmelgarn]

On 2016-03-15 09:46, Marc Haber wrote:
> On Tue, Mar 15, 2016 at 11:52:30AM +0100, Holger Hoffstätte wrote:
>> On 03/14/16 21:13, Marc Haber wrote:
>>> Do I need to wait for clear_cache to finish, like until I see disk
>>> usage dropping?
>>
>> The cache isn't that big, so you won't see a huge drop. Just use the
>> disk normally for a few minutes, after some time the cache will be
>> written out again.
>
> Is it necessary to actually cause activity on the file system or is it
> ok to just let it sit there for an hour or so?
It should be OK to just let it sit there for ten or fifteen minutes. I'm 
pretty certain that the free space cache gets rebuilt relatively 
quickly, and I'm almost 100% certain that the old one gets dropped 
within seconds of the FS being mounted with -o clear_cache.  I've 
rebuilt the cache on the 64G root filesystem on my laptop a couple of 
times before, and it consistently appears to take about 2-3 minutes to 
do so at most (based on disk usage from the kernel itself).

Re: New file system with same issue

[Marc Haber]

On Tue, Mar 15, 2016 at 09:54:06AM -0400, Austin S. Hemmelgarn wrote:
> On 2016-03-15 09:46, Marc Haber wrote:
> >On Tue, Mar 15, 2016 at 11:52:30AM +0100, Holger Hoffstätte wrote:
> >>On 03/14/16 21:13, Marc Haber wrote:
> >>>Do I need to wait for clear_cache to finish, like until I see disk
> >>>usage dropping?
> >>
> >>The cache isn't that big, so you won't see a huge drop. Just use the
> >>disk normally for a few minutes, after some time the cache will be
> >>written out again.
> >
> >Is it necessary to actually cause activity on the file system or is it
> >ok to just let it sit there for an hour or so?
> It should be OK to just let it sit there for ten or fifteen minutes. I'm
> pretty certain that the free space cache gets rebuilt relatively quickly,
> and I'm almost 100% certain that the old one gets dropped within seconds of
> the FS being mounted with -o clear_cache.  I've rebuilt the cache on the 64G
> root filesystem on my laptop a couple of times before, and it consistently
> appears to take about 2-3 minutes to do so at most (based on disk usage from
> the kernel itself).

In my case, atop has not seen any notable disk activity after mounting
with -o clerar_cache.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

A good "Boot Maintenance" scheme (WAS: New file system with same issue)

[Robert White]

On 03/14/2016 01:13 PM, Marc Haber wrote:
> This was not asked, and I didn't try. Since this is an encrypted root
> filesystem, is it a workable way to add clear_cache to /etc/fstab,
> rebuild initramfs and reboot? Or do you recommend using a rescue system?


You should be able to boot to single user mode and man-handle the 
filesystem from there.

but if that's a problem...

I have a small (woefully incomplete) pre-alpha project on Sourceforge 
called "underdog".

I currently use it to build the initramfs to boot all my systems.

So even if yo don't use it for production you can use it to make the 
initramfs you'll need to have full BTRFS maintenance before any of your 
filesystems are mounted.

You want this tool if you want to do pre-boot surgery without hassles. 
(or you want a nice way to convert from extN to btrfs and back without 
having a bunch of co-variant CDs.)

There's a README.txt, it's long... But basically:


(1) use git to download project

(2) go to your kernel build directory

(3) run /path/utility/make_initramfs_description.bash > .initramfs

where "path" is the path to the download location

(4) set CONFIG_INITRAMFS_SOURCE=".initramfs" in your kernel config

(5) build new kernel.

This creates a kernel which:

(A) Booted with a normal initramfs will run as if you didn't do the above.

(B1) Booted _without_ another initramfs will _probably_ boot normally as 
long as all your disk related drivers are not kernel modules. (It 
handles most layouts I've tried but some hosts have "the UID problem" 
described below

(B2) Booted _without_ another initramfs but _with_ "bash" on the kernel 
command line results in several opportunities to execute arbitrary 
commands during the boot sequence, and provides the commands you need.

You are after mode B2.

So during a normal boot without another initramfs the system will 
recursively examine all the disk block devices, identifying and 
attaching cryptsetup, mdadm, and lvm2, and btrfs devices as it finds them.

Some of the prompting will get lost in the kernel boot messages because, 
you know, multi-threading and all that.

Prompts:

"pre-loop#"; you've used "bash" option and it's about to start the disk 
search.

A password prompt for a device; it's found a cryptsetup device it wants 
to open. It'll give you three tries before moving on.

"post-loop#"; After the recursive search. _hopefully_ your root device 
is mounted as "/root". The FS is (hopefully) mounted but the system is 
completely independent of the mount.

So either use "pre-loop#" to do all your maintenance by hand, or go 
through to "post-loop#" and unmount /root and then do as you must.

These prompts are from the uber-priviliged PID1 and you can exec into 
other things (like an alternate init) as well as just running programs.

(control-d from "pre-loop#" or "post-loop#" will continue the script.

If you drop out to a bash shell prompt in B1 or B2 at the end then the 
script could not deduce your layout. You can then do fixups and 
whatever, and then mount anything as /root and then do the "exec busybox 
switch_root /root /sbin/init" (or /init or whatever) to continue the 
boot. That emergency bash prompt is still PID #1 and all the initramfs 
conditions for init are still in force.

Anyway, having used "bash" you can use the pre- or post- loop prompts to 
do any maintenance you wish. Up-arrow/history is _not_ available. But 
you have btrfs, btrfsck, fsck, mount, blkid, mdadm, and lvm available to 
you. (you can also issue "busybox --instal" to get all of the commands 
like cp and vi and such).

Other simple emergency commands can be added to the foundational list by 
creating/editing a utility/something.mod file or the script itself.

The magic of the script is that it will scour the system for dependent 
interpreters and libraries for whatever commands you add to the 
initramfs so you don't have to have "staticly linked" or otherwise 
special utilities built.

Notes:

I've been using the kernel-inbuilt initramfs this creates in all my 
development systems for years, it's quite stable... except where it's 
not. 8-)

DO NOT USE "utility/make_initramfs.bash", it's depricated. If you want 
to make a not-inbuilt initramfs image then follow the README.txt 
instructions from item 3b (which is also easier).

Full Disclosure: It's a little wonky about using UIDs to find 
filesystems right now because BASH likes to try to do math near the 
minuses in the UIDs when looking up arrays. /sigh

Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Henk Slager]

On Mon, Mar 14, 2016 at 1:07 PM, Marc Haber <mh+linux-btrfs@zugschlus.de> wrote:
> On Sun, Mar 13, 2016 at 12:58:09PM +0100, Marc Haber wrote:
>> On Sat, Mar 05, 2016 at 12:34:09PM -0700, Chris Murphy wrote:
>> > The alternative if this can't be fixed, is to recreate the filesystem
>> > because there's no practical way yet to migrate so many snapshots to a
>> > new file system.
>>
>> I recreated the file system on March 7, with 200 GiB in size, using
>> btrfs-tools 4.4. The snapshot-taking process has been running since
>> then, but I also regularly cleaned up. The number of snapshots on the
>> new filesystem has never exceeded 1000, with the current count being
>> at 148.
>>
>> And btrfs balance runs into the same ENOSPC issues as the old one:
>
> ... with Qu's patch, I now get a reproducible kernel trace:
>
> Mar 14 10:23:49 fan mh: BEGIN btrfs-balance script
> Mar 14 10:23:49 fan mh: btrfs fi df /
> Mar 14 10:23:49 fan root: Data, single: total=79.00GiB, used=78.42GiB
> Mar 14 10:23:49 fan root: System, single: total=32.00MiB, used=16.00KiB
> Mar 14 10:23:49 fan root: Metadata, single: total=10.00GiB, used=2.46GiB
> Mar 14 10:23:49 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B
> Mar 14 10:23:49 fan mh: btrfs fi show /
> Mar 14 10:23:49 fan root: Label: 'fanbtr'  uuid: 90f8d728-6bae-4fca-8cda-b368ba2c008e
> Mar 14 10:23:49 fan root: #011Total devices 1 FS bytes used 80.89GiB
> Mar 14 10:23:49 fan root: #011devid    1 size 200.00GiB used 89.03GiB path /dev/mapper/fanbtr
> Mar 14 10:23:49 fan root:
> Mar 14 10:23:49 fan mh: btrfs fi usage /
> Mar 14 10:23:49 fan root: Overall:
> Mar 14 10:23:49 fan root:     Device size:#011#011 200.00GiB
> Mar 14 10:23:49 fan root:     Device allocated:#011#011  89.03GiB
> Mar 14 10:23:49 fan root:     Device unallocated:#011#011 110.97GiB
> Mar 14 10:23:49 fan root:     Device missing:#011#011     0.00B
> Mar 14 10:23:49 fan root:     Used:#011#011#011  80.89GiB
> Mar 14 10:23:49 fan root:     Free (estimated):#011#011 111.54GiB#011(min: 111.54GiB)
> Mar 14 10:23:49 fan root:     Data ratio:#011#011#011      1.00
> Mar 14 10:23:49 fan root:     Metadata ratio:#011#011      1.00
> Mar 14 10:23:49 fan root:     Global reserve:#011#011 512.00MiB#011(used: 0.00B)
It it looks a bit strange to me that this is already 512MiB for and fs
of 200GiB. Just after creation (4.4 tools) it should be something like
16MiB. And grows when fs is used, but 512MiB... An fs created with
older tools had 512MiB from start AFAIK

> Mar 14 10:23:49 fan root:
> Mar 14 10:23:49 fan root: Data,single: Size:79.00GiB, Used:78.42GiB
> Mar 14 10:23:49 fan root:    /dev/mapper/fanbtr#011  79.00GiB
> Mar 14 10:23:49 fan root:
> Mar 14 10:23:49 fan root: Metadata,single: Size:10.00GiB, Used:2.46GiB
> Mar 14 10:23:49 fan root:    /dev/mapper/fanbtr#011  10.00GiB
> Mar 14 10:23:49 fan root:
> Mar 14 10:23:49 fan root: System,single: Size:32.00MiB, Used:16.00KiB
> Mar 14 10:23:49 fan root:    /dev/mapper/fanbtr#011  32.00MiB
> Mar 14 10:23:49 fan root:
> Mar 14 10:23:49 fan root: Unallocated:
> Mar 14 10:23:49 fan root:    /dev/mapper/fanbtr#011 110.97GiB
> Mar 14 10:23:49 fan mh: BEGIN btrfs balance start /
> Mar 14 10:36:46 fan kernel: [  890.995815] BTRFS info (device dm-15): 6 enospc errors during balance
> Mar 14 10:36:46 fan root: ERROR: error during balancing '/': No space left on device
> Mar 14 10:36:46 fan root: There may be more info in syslog - try dmesg | tail
> Mar 14 10:36:46 fan root: btrfs fi df /
> Mar 14 10:36:46 fan root: Data, single: total=79.00GiB, used=78.42GiB
> Mar 14 10:36:46 fan root: System, single: total=32.00MiB, used=16.00KiB
> Mar 14 10:36:46 fan root: Metadata, single: total=12.00GiB, used=2.46GiB
> Mar 14 10:36:46 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B
> Mar 14 10:36:46 fan root: btrfs fi show /
> Mar 14 10:36:46 fan root: Label: 'fanbtr'  uuid: 90f8d728-6bae-4fca-8cda-b368ba2c008e
> Mar 14 10:36:46 fan root: #011Total devices 1 FS bytes used 80.89GiB
> Mar 14 10:36:46 fan root: #011devid    1 size 200.00GiB used 91.03GiB path /dev/mapper/fanbtr
> Mar 14 10:36:46 fan root:
> Mar 14 10:36:46 fan root: btrfs fi usage /
> Mar 14 10:36:46 fan root: Overall:
> Mar 14 10:36:46 fan root:     Device size:#011#011 200.00GiB
> Mar 14 10:36:46 fan root:     Device allocated:#011#011  91.03GiB
> Mar 14 10:36:46 fan root:     Device unallocated:#011#011 108.97GiB
> Mar 14 10:36:46 fan root:     Device missing:#011#011     0.00B
> Mar 14 10:36:46 fan root:     Used:#011#011#011  80.89GiB
> Mar 14 10:36:46 fan root:     Free (estimated):#011#011 109.54GiB#011(min: 109.54GiB)
> Mar 14 10:36:46 fan root:     Data ratio:#011#011#011      1.00
> Mar 14 10:36:46 fan root:     Metadata ratio:#011#011      1.00
> Mar 14 10:36:46 fan root:     Global reserve:#011#011 512.00MiB#011(used: 0.00B)
> Mar 14 10:36:46 fan root:
> Mar 14 10:36:46 fan root: Data,single: Size:79.00GiB, Used:78.42GiB
> Mar 14 10:36:46 fan root:    /dev/mapper/fanbtr#011  79.00GiB
> Mar 14 10:36:46 fan root:
> Mar 14 10:36:46 fan root: Metadata,single: Size:12.00GiB, Used:2.46GiB
> Mar 14 10:36:46 fan root:    /dev/mapper/fanbtr#011  12.00GiB
> Mar 14 10:36:46 fan root:
> Mar 14 10:36:46 fan root: System,single: Size:32.00MiB, Used:16.00KiB
> Mar 14 10:36:46 fan root:    /dev/mapper/fanbtr#011  32.00MiB
> Mar 14 10:36:46 fan root:
> Mar 14 10:36:46 fan root: Unallocated:
> Mar 14 10:36:46 fan root:    /dev/mapper/fanbtr#011 108.97GiB
> Mar 14 10:36:46 fan root: BEGIN btrfs balance start -dprofiles=single /
> Mar 14 10:48:16 fan kernel: [ 1581.228727] ------------[ cut here ]------------
> Mar 14 10:48:16 fan kernel: [ 1581.228794] WARNING: CPU: 1 PID: 121 at fs/btrfs/extent-tree.c:7897 btrfs_alloc_tree_block+0xeb/0x3d6 [btrfs]()
> Mar 14 10:48:16 fan kernel: [ 1581.228800] BTRFS: block rsv returned -28
> Mar 14 10:48:16 fan kernel: [ 1581.228804] Modules linked in: iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp dummy ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative bridge stp llc snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi amd64_edac_mod edac_mce_amd kvm_amd input_leds pcspkr edac_core snd_hda_intel kvm snd_cmipci snd_hda_codec snd_mpu401_uart snd_opl3_lib snd_rawmidi snd_hda_core snd_seq_device irqbypass k10temp snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_timer evdev snd asus_atk0110 soundcore acpi_cpufreq i2c_piix4 tpm_tis sg tpm processor shpchp hwmon_vid autofs4 crc32c_generic btrfs xor raid6_pq ext4 crc16 mbcache jbd2 hmac sha256_ssse3 sha256_generic drbg ansi_cprng xts gf128mul algif_skcipher af_alg dm_crypt dm_mod hid_generic usbhid hid usb_storage sr_mod sd_mod cdrom ohci_pci ahci libahci amdkfd r8169 mii radeon xhci_pci i2c_algo_bit xhci_hcd ttm sym53c8xx ehci_pci scsi_transport_spi ohci_hcd drm_kms_helper libata ehci_hcd drm usbcore scsi_mod usb_common i2c_core button
> Mar 14 10:48:16 fan kernel: [ 1581.228957] CPU: 1 PID: 121 Comm: kworker/u16:2 Not tainted 4.4.5-zgws1 #2
> Mar 14 10:48:16 fan kernel: [ 1581.228963] Hardware name: System manufacturer System Product Name/M5A88-V EVO, BIOS 1603    10/12/2012
> Mar 14 10:48:16 fan kernel: [ 1581.229012] Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs]
> Mar 14 10:48:16 fan kernel: [ 1581.229018]  0000000000000000 ffffffff811dd418 ffff880613aafb58 0000000000000009
> Mar 14 10:48:16 fan kernel: [ 1581.229026]  ffffffff81051e21 ffffffffa047d29f ffff8804350d6800 ffff880613aafbb0
> Mar 14 10:48:16 fan kernel: [ 1581.229033]  ffff880610322000 ffff880531466700 ffffffff81051e79 ffffffffa04f065b
> Mar 14 10:48:16 fan kernel: [ 1581.229039] Call Trace:
> Mar 14 10:48:16 fan kernel: [ 1581.229052]  [<ffffffff811dd418>] ? dump_stack+0x5a/0x6f
> Mar 14 10:48:16 fan kernel: [ 1581.229061]  [<ffffffff81051e21>] ? warn_slowpath_common+0x8e/0xa3
> Mar 14 10:48:16 fan kernel: [ 1581.229104]  [<ffffffffa047d29f>] ? btrfs_alloc_tree_block+0xeb/0x3d6 [btrfs]
> Mar 14 10:48:16 fan kernel: [ 1581.229112]  [<ffffffff81051e79>] ? warn_slowpath_fmt+0x43/0x4b
> Mar 14 10:48:16 fan kernel: [ 1581.229155]  [<ffffffffa047d29f>] ? btrfs_alloc_tree_block+0xeb/0x3d6 [btrfs]
> Mar 14 10:48:16 fan kernel: [ 1581.229194]  [<ffffffffa046b05b>] ? btrfs_copy_root+0x9e/0x1c1 [btrfs]
> Mar 14 10:48:16 fan kernel: [ 1581.229239]  [<ffffffffa04c7844>] ? create_reloc_root+0x6c/0x196 [btrfs]
> Mar 14 10:48:16 fan kernel: [ 1581.229285]  [<ffffffffa04cb3cd>] ? btrfs_init_reloc_root+0x71/0x9a [btrfs]
> Mar 14 10:48:16 fan kernel: [ 1581.229329]  [<ffffffffa0489d74>] ? record_root_in_trans+0xc8/0xd5 [btrfs]
> Mar 14 10:48:16 fan kernel: [ 1581.229374]  [<ffffffffa048b192>] ? btrfs_record_root_in_trans+0x42/0x5a [btrfs]
> Mar 14 10:48:16 fan kernel: [ 1581.229418]  [<ffffffffa048c383>] ? start_transaction+0x2de/0x455 [btrfs]
> Mar 14 10:48:16 fan kernel: [ 1581.229464]  [<ffffffffa0492c67>] ? btrfs_finish_ordered_io+0x1e2/0x4d7 [btrfs]
> Mar 14 10:48:16 fan kernel: [ 1581.229474]  [<ffffffff81078953>] ? pick_next_task_fair+0x1f8/0x357
> Mar 14 10:48:16 fan kernel: [ 1581.229482]  [<ffffffff810752af>] ? account_entity_dequeue+0x46/0x67
> Mar 14 10:48:16 fan kernel: [ 1581.229529]  [<ffffffffa04b1ec4>] ? btrfs_scrubparity_helper+0xf4/0x233 [btrfs]
> Mar 14 10:48:16 fan kernel: [ 1581.229538]  [<ffffffff81063b4f>] ? process_one_work+0x178/0x27b
> Mar 14 10:48:16 fan kernel: [ 1581.229545]  [<ffffffff810640d3>] ? worker_thread+0x1da/0x280
> Mar 14 10:48:16 fan kernel: [ 1581.229553]  [<ffffffff81063ef9>] ? rescuer_thread+0x284/0x284
> Mar 14 10:48:16 fan kernel: [ 1581.229559]  [<ffffffff81067e59>] ? kthread+0x95/0x9d
> Mar 14 10:48:16 fan kernel: [ 1581.229566]  [<ffffffff81067dc4>] ? kthread_parkme+0x16/0x16
> Mar 14 10:48:16 fan kernel: [ 1581.229574]  [<ffffffff8140dfff>] ? ret_from_fork+0x3f/0x70
> Mar 14 10:48:16 fan kernel: [ 1581.229580]  [<ffffffff81067dc4>] ? kthread_parkme+0x16/0x16
> Mar 14 10:48:16 fan kernel: [ 1581.229586] ---[ end trace 69a5cc7238dc3665 ]---
> Mar 14 10:51:06 fan root: Done, had to relocate 79 out of 92 chunks
> Mar 14 10:51:06 fan root: btrfs fi df /
> Mar 14 10:51:06 fan root: Data, single: total=79.00GiB, used=78.43GiB
> Mar 14 10:51:06 fan root: System, single: total=32.00MiB, used=16.00KiB
> Mar 14 10:51:06 fan root: Metadata, single: total=12.00GiB, used=2.46GiB
> Mar 14 10:51:06 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B
> Mar 14 10:51:06 fan root: btrfs fi show /
> Mar 14 10:51:06 fan root: Label: 'fanbtr'  uuid: 90f8d728-6bae-4fca-8cda-b368ba2c008e
> Mar 14 10:51:06 fan root: #011Total devices 1 FS bytes used 80.89GiB
> Mar 14 10:51:06 fan root: #011devid    1 size 200.00GiB used 91.03GiB path /dev/mapper/fanbtr
> Mar 14 10:51:06 fan root:
> Mar 14 10:51:06 fan root: btrfs fi usage /
> Mar 14 10:51:06 fan root: Overall:
> Mar 14 10:51:06 fan root:     Device size:#011#011 200.00GiB
> Mar 14 10:51:06 fan root:     Device allocated:#011#011  91.03GiB
> Mar 14 10:51:06 fan root:     Device unallocated:#011#011 108.97GiB
> Mar 14 10:51:06 fan root:     Device missing:#011#011     0.00B
> Mar 14 10:51:06 fan root:     Used:#011#011#011  80.89GiB
> Mar 14 10:51:06 fan root:     Free (estimated):#011#011 109.54GiB#011(min: 109.54GiB)
> Mar 14 10:51:06 fan root:     Data ratio:#011#011#011      1.00
> Mar 14 10:51:06 fan root:     Metadata ratio:#011#011      1.00
> Mar 14 10:51:06 fan root:     Global reserve:#011#011 512.00MiB#011(used: 0.00B)
> Mar 14 10:51:06 fan root:
> Mar 14 10:51:06 fan root: Data,single: Size:79.00GiB, Used:78.43GiB
> Mar 14 10:51:06 fan root:    /dev/mapper/fanbtr#011  79.00GiB
> Mar 14 10:51:06 fan root:
> Mar 14 10:51:06 fan root: Metadata,single: Size:12.00GiB, Used:2.46GiB
> Mar 14 10:51:06 fan root:    /dev/mapper/fanbtr#011  12.00GiB
> Mar 14 10:51:06 fan root:
> Mar 14 10:51:06 fan root: System,single: Size:32.00MiB, Used:16.00KiB
> Mar 14 10:51:06 fan root:    /dev/mapper/fanbtr#011  32.00MiB
> Mar 14 10:51:06 fan root:
> Mar 14 10:51:06 fan root: Unallocated:
> Mar 14 10:51:06 fan root:    /dev/mapper/fanbtr#011 108.97GiB
> Mar 14 10:51:06 fan root: BEGIN btrfs balance start -mprofiles=dup /

This probably should have been  -mprofiles=single
So that its gets more clear where and when the enospc errors occur

> Mar 14 10:51:06 fan root: Done, had to relocate 0 out of 92 chunks
> Mar 14 10:51:06 fan root: btrfs fi df /
> Mar 14 10:51:06 fan root: Data, single: total=79.00GiB, used=78.43GiB
> Mar 14 10:51:06 fan root: System, single: total=32.00MiB, used=16.00KiB
> Mar 14 10:51:06 fan root: Metadata, single: total=12.00GiB, used=2.46GiB
> Mar 14 10:51:06 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B
> Mar 14 10:51:06 fan root: btrfs fi show /
> Mar 14 10:51:06 fan root: Label: 'fanbtr'  uuid: 90f8d728-6bae-4fca-8cda-b368ba2c008e
> Mar 14 10:51:06 fan root: #011Total devices 1 FS bytes used 80.89GiB
> Mar 14 10:51:06 fan root: #011devid    1 size 200.00GiB used 91.03GiB path /dev/mapper/fanbtr
> Mar 14 10:51:06 fan root:
> Mar 14 10:51:06 fan root: btrfs fi usage /
> Mar 14 10:51:06 fan root: Overall:
> Mar 14 10:51:06 fan root:     Device size:#011#011 200.00GiB
> Mar 14 10:51:06 fan root:     Device allocated:#011#011  91.03GiB
> Mar 14 10:51:06 fan root:     Device unallocated:#011#011 108.97GiB
> Mar 14 10:51:06 fan root:     Device missing:#011#011     0.00B
> Mar 14 10:51:06 fan root:     Used:#011#011#011  80.89GiB
> Mar 14 10:51:06 fan root:     Free (estimated):#011#011 109.54GiB#011(min: 109.54GiB)
> Mar 14 10:51:06 fan root:     Data ratio:#011#011#011      1.00
> Mar 14 10:51:06 fan root:     Metadata ratio:#011#011      1.00
> Mar 14 10:51:06 fan root:     Global reserve:#011#011 512.00MiB#011(used: 0.00B)
> Mar 14 10:51:06 fan root:
> Mar 14 10:51:06 fan root: Data,single: Size:79.00GiB, Used:78.43GiB
> Mar 14 10:51:06 fan root:    /dev/mapper/fanbtr#011  79.00GiB
> Mar 14 10:51:06 fan root:
> Mar 14 10:51:06 fan root: Metadata,single: Size:12.00GiB, Used:2.46GiB
> Mar 14 10:51:06 fan root:    /dev/mapper/fanbtr#011  12.00GiB
> Mar 14 10:51:06 fan root:
> Mar 14 10:51:06 fan root: System,single: Size:32.00MiB, Used:16.00KiB
> Mar 14 10:51:06 fan root:    /dev/mapper/fanbtr#011  32.00MiB
> Mar 14 10:51:06 fan root:
> Mar 14 10:51:06 fan root: Unallocated:
> Mar 14 10:51:06 fan root:    /dev/mapper/fanbtr#011 108.97GiB
> Mar 14 10:51:06 fan root: BEGIN btrfs balance start --force -sprofiles=dup /
Same here; you have no dup profiles

> Mar 14 10:51:06 fan root: Done, had to relocate 0 out of 92 chunks
> Mar 14 10:51:06 fan root: btrfs fi df /
> Mar 14 10:51:06 fan root: Data, single: total=79.00GiB, used=78.43GiB
> Mar 14 10:51:06 fan root: System, single: total=32.00MiB, used=16.00KiB
> Mar 14 10:51:06 fan root: Metadata, single: total=12.00GiB, used=2.46GiB
> Mar 14 10:51:06 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B
> Mar 14 10:51:06 fan root: btrfs fi show /
> Mar 14 10:51:06 fan root: Label: 'fanbtr'  uuid: 90f8d728-6bae-4fca-8cda-b368ba2c008e
> Mar 14 10:51:06 fan root: #011Total devices 1 FS bytes used 80.89GiB
> Mar 14 10:51:06 fan root: #011devid    1 size 200.00GiB used 91.03GiB path /dev/mapper/fanbtr
> Mar 14 10:51:06 fan root:
> Mar 14 10:51:06 fan root: btrfs fi usage /
> Mar 14 10:51:06 fan root: Overall:
> Mar 14 10:51:06 fan root:     Device size:#011#011 200.00GiB
> Mar 14 10:51:06 fan root:     Device allocated:#011#011  91.03GiB
> Mar 14 10:51:06 fan root:     Device unallocated:#011#011 108.97GiB
> Mar 14 10:51:06 fan root:     Device missing:#011#011     0.00B
> Mar 14 10:51:06 fan root:     Used:#011#011#011  80.89GiB
> Mar 14 10:51:06 fan root:     Free (estimated):#011#011 109.54GiB#011(min: 109.54GiB)
> Mar 14 10:51:06 fan root:     Data ratio:#011#011#011      1.00
> Mar 14 10:51:06 fan root:     Metadata ratio:#011#011      1.00
> Mar 14 10:51:06 fan root:     Global reserve:#011#011 512.00MiB#011(used: 0.00B)
> Mar 14 10:51:06 fan root:
> Mar 14 10:51:06 fan root: Data,single: Size:79.00GiB, Used:78.43GiB
> Mar 14 10:51:06 fan root:    /dev/mapper/fanbtr#011  79.00GiB
> Mar 14 10:51:06 fan root:
> Mar 14 10:51:06 fan root: Metadata,single: Size:12.00GiB, Used:2.46GiB
> Mar 14 10:51:06 fan root:    /dev/mapper/fanbtr#011  12.00GiB
> Mar 14 10:51:06 fan root:
> Mar 14 10:51:06 fan root: System,single: Size:32.00MiB, Used:16.00KiB
> Mar 14 10:51:06 fan root:    /dev/mapper/fanbtr#011  32.00MiB
> Mar 14 10:51:06 fan root:
> Mar 14 10:51:06 fan root: Unallocated:
> Mar 14 10:51:06 fan root:    /dev/mapper/fanbtr#011 108.97GiB
> Mar 14 10:51:06 fan root: BEGIN btrfs balance start /
> Mar 14 11:08:36 fan kernel: [ 2800.565044] BTRFS info (device dm-15): 7 enospc errors during balance
> Mar 14 11:08:36 fan root: ERROR: error during balancing '/': No space left on device
> Mar 14 11:08:36 fan root: There may be more info in syslog - try dmesg | tail
> Mar 14 11:08:36 fan mh: btrfs fi df /
> Mar 14 11:08:36 fan root: Data, single: total=79.00GiB, used=78.43GiB
> Mar 14 11:08:36 fan root: System, single: total=32.00MiB, used=16.00KiB
> Mar 14 11:08:36 fan root: Metadata, single: total=19.00GiB, used=2.46GiB
> Mar 14 11:08:36 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B
> Mar 14 11:08:36 fan mh: btrfs fi show /
> Mar 14 11:08:36 fan root: Label: 'fanbtr'  uuid: 90f8d728-6bae-4fca-8cda-b368ba2c008e
> Mar 14 11:08:36 fan root: #011Total devices 1 FS bytes used 80.89GiB
> Mar 14 11:08:36 fan root: #011devid    1 size 200.00GiB used 98.03GiB path /dev/mapper/fanbtr
> Mar 14 11:08:36 fan root:
> Mar 14 11:08:36 fan mh: btrfs fi usage /
> Mar 14 11:08:36 fan root: Overall:
> Mar 14 11:08:36 fan root:     Device size:#011#011 200.00GiB
> Mar 14 11:08:36 fan root:     Device allocated:#011#011  98.03GiB
> Mar 14 11:08:36 fan root:     Device unallocated:#011#011 101.97GiB
> Mar 14 11:08:36 fan root:     Device missing:#011#011     0.00B
> Mar 14 11:08:36 fan root:     Used:#011#011#011  80.89GiB
> Mar 14 11:08:36 fan root:     Free (estimated):#011#011 102.54GiB#011(min: 102.54GiB)
> Mar 14 11:08:36 fan root:     Data ratio:#011#011#011      1.00
> Mar 14 11:08:36 fan root:     Metadata ratio:#011#011      1.00
> Mar 14 11:08:36 fan root:     Global reserve:#011#011 512.00MiB#011(used: 0.00B)
> Mar 14 11:08:36 fan root:
> Mar 14 11:08:36 fan root: Data,single: Size:79.00GiB, Used:78.43GiB
> Mar 14 11:08:36 fan root:    /dev/mapper/fanbtr#011  79.00GiB
> Mar 14 11:08:36 fan root:
> Mar 14 11:08:36 fan root: Metadata,single: Size:19.00GiB, Used:2.46GiB
> Mar 14 11:08:36 fan root:    /dev/mapper/fanbtr#011  19.00GiB
> Mar 14 11:08:36 fan root:
> Mar 14 11:08:36 fan root: System,single: Size:32.00MiB, Used:16.00KiB
> Mar 14 11:08:36 fan root:    /dev/mapper/fanbtr#011  32.00MiB
> Mar 14 11:08:36 fan root:
> Mar 14 11:08:36 fan root: Unallocated:
> Mar 14 11:08:36 fan root:    /dev/mapper/fanbtr#011 101.97GiB
> Mar 14 11:08:36 fan mh: END btrfs-balance script
>
> Full log is at http://q.bofh.de/~mh/stuff/20160314-fanbtr-btrfs-syslog
>
> The log was taken with enospc_debug active on the file system and all
> file system, block device and storage relevant log lines were left in.
>
> What does the trace tell us here?

Although a warning, data balance seems to work. The problem still
points to metadata.

BTW, I restored and mounted your 20160307-fanbtr-image:

[266169.207952] BTRFS: device label fanbtr devid 1 transid 22215732 /dev/loop0
[266203.734804] BTRFS info (device loop0): disk space caching is enabled
[266203.734806] BTRFS: has skinny extents
[266204.022175] BTRFS: checking UUID tree
[266239.407249] attempt to access beyond end of device
[266239.407252] loop0: rw=1073, want=715202688, limit=705760000
[266239.407254] BTRFS error (device loop0): bdev /dev/loop0 errs: wr
1, rd 0, flush 0, corrupt 0, gen 0
[266239.407272] attempt to access beyond end of device
.. and 16 more

As a quick fix/workaround, I truncated the image to 1T

After re-loop and mount and while doing a balance of the metadata I got this:
[266667.431704] BTRFS error (device loop0): bad tree block start 0 5827368812544

So something is/was wrong with the fs. Did you do a btrfs check before imaging?

Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Marc Haber]

Hi Henk,

On Mon, Mar 14, 2016 at 02:46:54PM +0100, Henk Slager wrote:
> On Mon, Mar 14, 2016 at 1:07 PM, Marc Haber <mh+linux-btrfs@zugschlus.de> wrote:
> > Mar 14 10:23:49 fan mh: BEGIN btrfs-balance script
> > Mar 14 10:23:49 fan mh: btrfs fi df /
> > Mar 14 10:23:49 fan root: Data, single: total=79.00GiB, used=78.42GiB
> > Mar 14 10:23:49 fan root: System, single: total=32.00MiB, used=16.00KiB
> > Mar 14 10:23:49 fan root: Metadata, single: total=10.00GiB, used=2.46GiB
> > Mar 14 10:23:49 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B
> > Mar 14 10:23:49 fan mh: btrfs fi show /
> > Mar 14 10:23:49 fan root: Label: 'fanbtr'  uuid: 90f8d728-6bae-4fca-8cda-b368ba2c008e
> > Mar 14 10:23:49 fan root: #011Total devices 1 FS bytes used 80.89GiB
> > Mar 14 10:23:49 fan root: #011devid    1 size 200.00GiB used 89.03GiB path /dev/mapper/fanbtr
> > Mar 14 10:23:49 fan root:
> > Mar 14 10:23:49 fan mh: btrfs fi usage /
> > Mar 14 10:23:49 fan root: Overall:
> > Mar 14 10:23:49 fan root:     Device size:#011#011 200.00GiB
> > Mar 14 10:23:49 fan root:     Device allocated:#011#011  89.03GiB
> > Mar 14 10:23:49 fan root:     Device unallocated:#011#011 110.97GiB
> > Mar 14 10:23:49 fan root:     Device missing:#011#011     0.00B
> > Mar 14 10:23:49 fan root:     Used:#011#011#011  80.89GiB
> > Mar 14 10:23:49 fan root:     Free (estimated):#011#011 111.54GiB#011(min: 111.54GiB)
> > Mar 14 10:23:49 fan root:     Data ratio:#011#011#011      1.00
> > Mar 14 10:23:49 fan root:     Metadata ratio:#011#011      1.00
> > Mar 14 10:23:49 fan root:     Global reserve:#011#011 512.00MiB#011(used: 0.00B)
> It it looks a bit strange to me that this is already 512MiB for and fs
> of 200GiB. Just after creation (4.4 tools) it should be something like
> 16MiB. And grows when fs is used, but 512MiB... An fs created with
> older tools had 512MiB from start AFAIK

Confirmed, a new btrfs of 200 GB made on a rotating disk has 16 MiB of
global reserve. Unfortunately, I do not have history about how this
grew over time. The first btrfs fi usage I have on file was about half
a day into this fs' existence on Mar 7, after copying data on to it,
and Global reserve was already at 512 MiB.

> > Mar 14 10:51:06 fan root: BEGIN btrfs balance start -mprofiles=dup /
> 
> This probably should have been  -mprofiles=single
> So that its gets more clear where and when the enospc errors occur

Good catch. So I'd need to parse btrfs fi df's output to call the
right balance option. I blindly copied that over from the script I
wrote for the older btrfs which still has DUP metadata and system.

> BTW, I restored and mounted your 20160307-fanbtr-image:
> 
> [266169.207952] BTRFS: device label fanbtr devid 1 transid 22215732 /dev/loop0
> [266203.734804] BTRFS info (device loop0): disk space caching is enabled
> [266203.734806] BTRFS: has skinny extents
> [266204.022175] BTRFS: checking UUID tree
> [266239.407249] attempt to access beyond end of device
> [266239.407252] loop0: rw=1073, want=715202688, limit=705760000
> [266239.407254] BTRFS error (device loop0): bdev /dev/loop0 errs: wr
> 1, rd 0, flush 0, corrupt 0, gen 0
> [266239.407272] attempt to access beyond end of device
> .. and 16 more
> 
> As a quick fix/workaround, I truncated the image to 1T

The original fs was 417 GiB in size. What size does the image claim?

> After re-loop and mount and while doing a balance of the metadata I got this:
> [266667.431704] BTRFS error (device loop0): bad tree block start 0 5827368812544
>
> So something is/was wrong with the fs. Did you do a btrfs check before imaging?

No, I didn't. And there is indeed something wrong:

[10/509]mh@fan:~$ sudo btrfs check /media/tempdisk/
Superblock bytenr is larger than device size
Couldn't open file system
[11/509]mh@fan:~$

Can this be fixed?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Henk Slager]

>> BTW, I restored and mounted your 20160307-fanbtr-image:
>>
>> [266169.207952] BTRFS: device label fanbtr devid 1 transid 22215732 /dev/loop0
>> [266203.734804] BTRFS info (device loop0): disk space caching is enabled
>> [266203.734806] BTRFS: has skinny extents
>> [266204.022175] BTRFS: checking UUID tree
>> [266239.407249] attempt to access beyond end of device
>> [266239.407252] loop0: rw=1073, want=715202688, limit=705760000
>> [266239.407254] BTRFS error (device loop0): bdev /dev/loop0 errs: wr
>> 1, rd 0, flush 0, corrupt 0, gen 0
>> [266239.407272] attempt to access beyond end of device
>> .. and 16 more
>>
>> As a quick fix/workaround, I truncated the image to 1T
>
> The original fs was 417 GiB in size. What size does the image claim?

ls -alFh  of the restored image showed 337G I remember.
btrfs fi us showed also a number over 400G, I don't have the
files/loopdev anymore.
It could some side effect of btrfs-image, I only have used it for
multi-device, where dev id's are ignore, but total image size did not
lead to problems.

> [10/509]mh@fan:~$ sudo btrfs check /media/tempdisk/
> Superblock bytenr is larger than device size
> Couldn't open file system
> [11/509]mh@fan:~$
>
> Can this be fixed?

What I would do in order to fix it, is resize the fs to let's say
190GiB. That should write correct values to the superblocks I /hope/.
And then resize back to max.
Maybe btrfs check --repair can also fix it, but before doing --repair
or other actions, I would see what else besides btrfs could be wrong,
see also suggestion of Holger. --repair can repair certain things, but
also destroy the fs.

Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Chris Murphy]

I'm a little mystified how btrfs check reports a problem with the
superblock, and yet this filesystem can still be mounted and used? If
it mounts rw then resize is possible but why would it be wrong in the
first place?


Chris Murphy

Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Henk Slager]

On Mon, Mar 14, 2016 at 10:59 PM, Chris Murphy <lists@colorremedies.com> wrote:
> I'm a little mystified how btrfs check reports a problem with the
> superblock, and yet this filesystem can still be mounted and used? If
> it mounts rw then resize is possible but why would it be wrong in the
> first place?

Yes you are right. I just focussed on the 'Superblock bytenr is larger
than device size' part.
btrfs check  will actually output that text on a mounted filesystem anyhow.
The other question is: What is mounted on /media/tempdisk/  ?

At least I think a check of the current 200GiB fs is needed. As it is
a rootfs and encrypted, some work is needed to make that happen.

Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Marc Haber]

On Tue, Mar 15, 2016 at 12:22:00AM +0100, Henk Slager wrote:
> The other question is: What is mounted on /media/tempdisk/  ?

The "old" btrfs filesystem "ofanbtr", formerly 417 GB in size, now
resized to 300 GB. Does it need to be umounted to be checked?

> At least I think a check of the current 200GiB fs is needed. As it is
> a rootfs and encrypted, some work is needed to make that happen.

You suggested a btrfs check after looking at the image of "ofanbtr".
Do you want me to check the new "fanbtr" also?

Too bad that we went back to looking at "ofanbtr" after I changed the
subject to avoid mixing up both instances.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Henk Slager]

On Tue, Mar 15, 2016 at 8:16 AM, Marc Haber <mh+linux-btrfs@zugschlus.de> wrote:
> On Tue, Mar 15, 2016 at 12:22:00AM +0100, Henk Slager wrote:
>> The other question is: What is mounted on /media/tempdisk/  ?
>
> The "old" btrfs filesystem "ofanbtr", formerly 417 GB in size, now
> resized to 300 GB. Does it need to be umounted to be checked?

Yes, that's the whole point

>> At least I think a check of the current 200GiB fs is needed. As it is
>> a rootfs and encrypted, some work is needed to make that happen.
>
> You suggested a btrfs check after looking at the image of "ofanbtr".
> Do you want me to check the new "fanbtr" also?

I was not sure if 'ofanbtr' is an image created by btrfs-image or a
extra dd created image you might have locally. Both 'ofanbtr' and
'fanbtr' have the same balance issue, but 'fanbtr' is created with
newer and known kernel+tools version I assume, so that's why the
suggestion.

Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Marc Haber]

On Tue, Mar 15, 2016 at 01:15:33PM +0100, Henk Slager wrote:
> On Tue, Mar 15, 2016 at 8:16 AM, Marc Haber <mh+linux-btrfs@zugschlus.de> wrote:
> > On Tue, Mar 15, 2016 at 12:22:00AM +0100, Henk Slager wrote:
> >> The other question is: What is mounted on /media/tempdisk/  ?
> >
> > The "old" btrfs filesystem "ofanbtr", formerly 417 GB in size, now
> > resized to 300 GB. Does it need to be umounted to be checked?
> 
> Yes, that's the whole point
> 
> >> At least I think a check of the current 200GiB fs is needed. As it is
> >> a rootfs and encrypted, some work is needed to make that happen.
> >
> > You suggested a btrfs check after looking at the image of "ofanbtr".
> > Do you want me to check the new "fanbtr" also?
> 
> I was not sure if 'ofanbtr' is an image created by btrfs-image or a
> extra dd created image you might have locally. Both 'ofanbtr' and
> 'fanbtr' have the same balance issue, but 'fanbtr' is created with
> newer and known kernel+tools version I assume, so that's why the
> suggestion.

ofanbtr is the old btrfs, on /dev/mapper/ofanbtr:
Label: 'ofanbtr'  uuid: 4198d1bc-e3ce-40df-a7ee-44a2d120bff3
        Total devices 1 FS bytes used 80.63GiB
        devid    1 size 300.00GiB used 122.06GiB path /dev/mapper/ofanbtr
it was created as 'fanbtr' in September, 300 GiB in Size, then - in
February, I think, resized to 417 MiB to make room for more data and
for balancing, used until March 7, and then renamed to ofanbtr with
lvrename and btrfs fi label. It was then imaged, and then resized back
to 300 GiB in the hope that this will fix the size issue.

fanbtr is the new btrfs, on /dev/mapper/fanbtr:
Label: 'fanbtr'  uuid: 90f8d728-6bae-4fca-8cda-b368ba2c008e
        Total devices 1 FS bytes used 82.45GiB
        devid    1 size 200.00GiB used 113.03GiB path /dev/mapper/fanbtr
it was created on march 7, had the data from ofanbtr cp'ed over, and
is being used as the active filesystem since then. It is smaller
because I don't have much more room on the SSD.

Both do have the same balance issue, yes.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Marc Haber]

On Mon, Mar 14, 2016 at 09:39:51PM +0100, Henk Slager wrote:
> >> BTW, I restored and mounted your 20160307-fanbtr-image:
> >>
> >> [266169.207952] BTRFS: device label fanbtr devid 1 transid 22215732 /dev/loop0
> >> [266203.734804] BTRFS info (device loop0): disk space caching is enabled
> >> [266203.734806] BTRFS: has skinny extents
> >> [266204.022175] BTRFS: checking UUID tree
> >> [266239.407249] attempt to access beyond end of device
> >> [266239.407252] loop0: rw=1073, want=715202688, limit=705760000
> >> [266239.407254] BTRFS error (device loop0): bdev /dev/loop0 errs: wr
> >> 1, rd 0, flush 0, corrupt 0, gen 0
> >> [266239.407272] attempt to access beyond end of device
> >> .. and 16 more
> >>
> >> As a quick fix/workaround, I truncated the image to 1T
> >
> > The original fs was 417 GiB in size. What size does the image claim?
> 
> ls -alFh  of the restored image showed 337G I remember.
> btrfs fi us showed also a number over 400G, I don't have the
> files/loopdev anymore.

sounds legit.

> It could some side effect of btrfs-image, I only have used it for
> multi-device, where dev id's are ignore, but total image size did not
> lead to problems.

The original "ofanbtr" seems to have a problem, since btrfs check
/media/tempdisk says:

> > [10/509]mh@fan:~$ sudo btrfs check /media/tempdisk/
> > Superblock bytenr is larger than device size
> > Couldn't open file system
> > [11/509]mh@fan:~$
> >
> > Can this be fixed?
> 
> What I would do in order to fix it, is resize the fs to let's say
> 190GiB. That should write correct values to the superblocks I /hope/.
> And then resize back to max.

It doesn't:
[20/518]mh@fan:~$ sudo btrfs filesystem resize 300G /media/tempdisk/
Resize '/media/tempdisk/' of '300G'
[22/520]mh@fan:~$ sudo btrfs check /media/tempdisk/
Superblock bytenr is larger than device size
Couldn't open file system
[23/521]mh@fan:~$ df -h

> Maybe btrfs check --repair can also fix it, but before doing --repair
> or other actions, I would see what else besides btrfs could be wrong,
> see also suggestion of Holger.

Like putting the filesystem on an unencrypted medium? Sorry, no,
private data, paranoia.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Martin Steigerwald]

On Dienstag, 15. März 2016 08:07:22 CEST Marc Haber wrote:
> On Mon, Mar 14, 2016 at 09:39:51PM +0100, Henk Slager wrote:
> > >> BTW, I restored and mounted your 20160307-fanbtr-image:
> > >> 
> > >> [266169.207952] BTRFS: device label fanbtr devid 1 transid 22215732
> > >> /dev/loop0 [266203.734804] BTRFS info (device loop0): disk space
> > >> caching is enabled [266203.734806] BTRFS: has skinny extents
> > >> [266204.022175] BTRFS: checking UUID tree
> > >> [266239.407249] attempt to access beyond end of device
> > >> [266239.407252] loop0: rw=1073, want=715202688, limit=705760000
> > >> [266239.407254] BTRFS error (device loop0): bdev /dev/loop0 errs: wr
> > >> 1, rd 0, flush 0, corrupt 0, gen 0
> > >> [266239.407272] attempt to access beyond end of device
> > >> .. and 16 more
> > >> 
> > >> As a quick fix/workaround, I truncated the image to 1T
> > > 
> > > The original fs was 417 GiB in size. What size does the image claim?
> > 
> > ls -alFh  of the restored image showed 337G I remember.
> > btrfs fi us showed also a number over 400G, I don't have the
> > files/loopdev anymore.
> 
> sounds legit.
> 
> > It could some side effect of btrfs-image, I only have used it for
> > multi-device, where dev id's are ignore, but total image size did not
> > lead to problems.
> 
> The original "ofanbtr" seems to have a problem, since btrfs check
> 
> /media/tempdisk says:
> > > [10/509]mh@fan:~$ sudo btrfs check /media/tempdisk/
> > > Superblock bytenr is larger than device size
> > > Couldn't open file system
> > > [11/509]mh@fan:~$
> > > 
> > > Can this be fixed?
> > 
> > What I would do in order to fix it, is resize the fs to let's say
> > 190GiB. That should write correct values to the superblocks I /hope/.
> > And then resize back to max.
> 
> It doesn't:
> [20/518]mh@fan:~$ sudo btrfs filesystem resize 300G /media/tempdisk/
> Resize '/media/tempdisk/' of '300G'
> [22/520]mh@fan:~$ sudo btrfs check /media/tempdisk/
> Superblock bytenr is larger than device size
> Couldn't open file system
> [23/521]mh@fan:~$ df -h

Are you trying the check on the *mounted* filesystem? "media/tempdisk" appears 
to be a mount point, not a device file.

Unmount it and use the / one device file of the filesystem to check.

Thanks,
-- 
Martin

Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Marc Haber]

On Mon, Mar 14, 2016 at 09:05:46PM +0100, Marc Haber wrote:
> [10/509]mh@fan:~$ sudo btrfs check /media/tempdisk/
> Superblock bytenr is larger than device size
> Couldn't open file system
> [11/509]mh@fan:~$

After umounting and btrfs check the block device, things seem to be
fine now:

[34/532]mh@fan:~$ sudo btrfs check /dev/mapper/ofanbtr
Checking filesystem on /dev/mapper/ofanbtr
UUID: 4198d1bc-e3ce-40df-a7ee-44a2d120bff3
checking extents
checking free space cache
checking fs roots
checking csums
checking root refs
found 86554574954 bytes used err is 0
total csum bytes: 81815012
total tree bytes: 2476670976
total fs tree bytes: 2246311936
total extent tree bytes: 133201920
btree space waste bytes: 452859567
file data blocks allocated: 292994375680
 referenced 132664688640
[35/533]mh@fan:~$ sudo btrfs check /dev/mapper/ofanbtr
Checking filesystem on /dev/mapper/ofanbtr
UUID: 4198d1bc-e3ce-40df-a7ee-44a2d120bff3
checking extents
checking free space cache
checking fs roots
checking csums
checking root refs
found 86554574954 bytes used err is 0
total csum bytes: 81815012
total tree bytes: 2476670976
total fs tree bytes: 2246311936
total extent tree bytes: 133201920
btree space waste bytes: 452859567
file data blocks allocated: 292994375680
 referenced 132664688640
[36/533]mh@fan:~$

This does not indicate an error, does it?

Greetings
Marc, who would like to the tools a bit more explicit and consistent
in whether they want the fs mounted, umounted, the mountpoint or the
device on their command line

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Marc Haber]

On Tue, Mar 15, 2016 at 02:29:32PM +0100, Marc Haber wrote:
> After umounting and btrfs check the block device, things seem to be
> fine now

But, umounting the btrfs seemed to trigger the following kernel traces:

Mar 15 14:21:30 fan kernel: [92308.377104] ------------[ cut here ]------------
Mar 15 14:21:30 fan kernel: [92308.377135] WARNING: CPU: 5 PID: 28243 at fs/btrfs/extent-tree.c:5380 bt
rfs_free_block_groups+0x1bc/0x36f [btrfs]()
Mar 15 14:21:30 fan kernel: [92308.377137] Modules linked in: vhost_net vhost macvtap macvlan tun iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp dummy ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative bridge stp llc snd_cmipci snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi kvm_amd snd_mpu401_uart snd_opl3_lib snd_rawmidi kvm snd_hda_intel snd_seq_device snd_hda_codec snd_hda_core snd_hwdep amd64_edac_mod snd_pcm_oss edac_mce_amd irqbypass input_leds snd_mixer_oss pcspkr k10temp edac_core snd_pcm snd_timer snd i2c_piix4 asus_atk0110 soundcore acpi_cpufreq tpm_tis tpm sg processor evdev shpchp hwmon_vid autofs4 crc32c_generic btrfs xor raid6_pq ext4 crc16 mbcache jbd2 hmac sha256_ssse3 sha256_generic drbg ansi_cprng xts gf128mul algif_skcipher af_alg dm_crypt dm_mod hid_generic usbhid hid usb_storage sr_mod sd_mod cdrom ohci_pci r8169 mii amdkfd radeon i2c_algo_bit ahci ttm sym53c8xx libahci xhci_pci scsi_transport_spi drm_kms_helper ohci_hcd ehci_pci xhci_hcd libata ehci_hcd drm usbcore scsi_mod usb_common i2c_core button
Mar 15 14:21:30 fan kernel: [92308.377203] CPU: 5 PID: 28243 Comm: umount Not tainted 4.4.5-zgws1 #2
Mar 15 14:21:30 fan kernel: [92308.377205] Hardware name: System manufacturer System Product Name/M5A88-V EVO, BIOS 1603    10/12/2012
Mar 15 14:21:30 fan kernel: [92308.377207]  000000000000005b ffffffff811dd418 0000000000000000 0000000000000009
Mar 15 14:21:30 fan kernel: [92308.377210]  ffffffff81051e21 ffffffffa047a147 ffff880600a28000 0000000000000000
Mar 15 14:21:30 fan kernel: [92308.377212]  ffff880600a28080 ffff8805af7eea00 ffffffffa047a147 ffff880600a28000
Mar 15 14:21:30 fan kernel: [92308.377215] Call Trace:
Mar 15 14:21:30 fan kernel: [92308.377221]  [<ffffffff811dd418>] ? dump_stack+0x5a/0x6f
Mar 15 14:21:30 fan kernel: [92308.377224]  [<ffffffff81051e21>] ? warn_slowpath_common+0x8e/0xa3
Mar 15 14:21:30 fan kernel: [92308.377239]  [<ffffffffa047a147>] ? btrfs_free_block_groups+0x1bc/0x36f[btrfs]
Mar 15 14:21:30 fan kernel: [92308.377252]  [<ffffffffa047a147>] ? btrfs_free_block_groups+0x1bc/0x36f[btrfs]
Mar 15 14:21:30 fan kernel: [92308.377267]  [<ffffffffa0487c72>] ? close_ctree+0x1e6/0x2f2 [btrfs]
Mar 15 14:21:30 fan kernel: [92308.377271]  [<ffffffff8113dfd9>] ? generic_shutdown_super+0x64/0xdf
Mar 15 14:21:30 fan kernel: [92308.377273]  [<ffffffff8113e181>] ? kill_anon_super+0x9/0xe
Mar 15 14:21:30 fan kernel: [92308.377285]  [<ffffffffa04660db>] ? btrfs_kill_super+0xd/0x16 [btrfs]
Mar 15 14:21:30 fan kernel: [92308.377288]  [<ffffffff8113e286>] ? deactivate_locked_super+0x2f/0x56
Mar 15 14:21:30 fan kernel: [92308.377291]  [<ffffffff81152aff>] ? cleanup_mnt+0x4f/0x6b
Mar 15 14:21:30 fan kernel: [92308.377293]  [<ffffffff81066ab7>] ? task_work_run+0x5d/0x71
Mar 15 14:21:30 fan kernel: [92308.377296]  [<ffffffff810036dc>] ? prepare_exit_to_usermode+0x70/0x99
Mar 15 14:21:30 fan kernel: [92308.377300]  [<ffffffff8140de08>] ? int_ret_from_sys_call+0x25/0x8f
Mar 15 14:21:30 fan kernel: [92308.377302] ---[ end trace 18c6bb90b0c6c689 ]---

Mar 15 14:21:30 fan kernel: [92308.377303] ------------[ cut here ]------------
Mar 15 14:21:30 fan kernel: [92308.377318] WARNING: CPU: 5 PID: 28243 at fs/btrfs/extent-tree.c:5381 btrfs_free_block_groups+0x1d7/0x36f [btrfs]()
Mar 15 14:21:30 fan kernel: [92308.377319] Modules linked in: vhost_net vhost macvtap macvlan tun iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp dummy ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative bridge stp llc snd_cmipci snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi kvm_amd snd_mpu401_uart snd_opl3_lib snd_rawmidi kvm snd_hda_intel snd_seq_device snd_hda_codec snd_hda_core snd_hwdep amd64_edac_mod snd_pcm_oss edac_mce_amd irqbypass input_leds snd_mixer_oss pcspkr k10temp edac_core snd_pcm snd_timer snd i2c_piix4 asus_atk0110 soundcore acpi_cpufreq tpm_tis tpm sg processor evdev shpchp hwmon_vid autofs4 crc32c_generic btrfs xor raid6_pq ext4 crc16 mbcache jbd2 hmac sha256_ssse3 sha256_generic drbg ansi_cprng xts gf128mul algif_skcipher af_alg dm_crypt dm_mod hid_generic usbhid hid usb_storage sr_mod sd_mod cdrom ohci_pci r8169 mii amdkfd radeon i2c_algo_bit ahci ttm sym53c8xx libahci xhci_pci scsi_transport_spi drm_kms_helper ohci_hcd ehci_pci xhci_hcd libata ehci_hcd drm usbcore scsi_mod usb_common i2c_core button
Mar 15 14:21:30 fan kernel: [92308.377362] CPU: 5 PID: 28243 Comm: umount Tainted: G        W       4.4.5-zgws1 #2
Mar 15 14:21:30 fan kernel: [92308.377364] Hardware name: System manufacturer System Product Name/M5A88-V EVO, BIOS 1603    10/12/2012
Mar 15 14:21:30 fan kernel: [92308.377365]  000000000000005b ffffffff811dd418 0000000000000000 0000000000000009
Mar 15 14:21:30 fan kernel: [92308.377367]  ffffffff81051e21 ffffffffa047a162 ffff880600a28000 0000000000000000
Mar 15 14:21:30 fan kernel: [92308.377369]  ffff880600a28080 ffff8805af7eea00 ffffffffa047a162 ffff880600a28000
Mar 15 14:21:30 fan kernel: [92308.377372] Call Trace:
Mar 15 14:21:30 fan kernel: [92308.377374]  [<ffffffff811dd418>] ? dump_stack+0x5a/0x6f
Mar 15 14:21:30 fan kernel: [92308.377377]  [<ffffffff81051e21>] ? warn_slowpath_common+0x8e/0xa3
Mar 15 14:21:30 fan kernel: [92308.377390]  [<ffffffffa047a162>] ? btrfs_free_block_groups+0x1d7/0x36f [btrfs]
Mar 15 14:21:30 fan kernel: [92308.377404]  [<ffffffffa047a162>] ? btrfs_free_block_groups+0x1d7/0x36f [btrfs]
Mar 15 14:21:30 fan kernel: [92308.377419]  [<ffffffffa0487c72>] ? close_ctree+0x1e6/0x2f2 [btrfs]
Mar 15 14:21:30 fan kernel: [92308.377421]  [<ffffffff8113dfd9>] ? generic_shutdown_super+0x64/0xdf
Mar 15 14:21:30 fan kernel: [92308.377423]  [<ffffffff8113e181>] ? kill_anon_super+0x9/0xe
Mar 15 14:21:30 fan kernel: [92308.377435]  [<ffffffffa04660db>] ? btrfs_kill_super+0xd/0x16 [btrfs]
Mar 15 14:21:30 fan kernel: [92308.377438]  [<ffffffff8113e286>] ? deactivate_locked_super+0x2f/0x56
Mar 15 14:21:30 fan kernel: [92308.377440]  [<ffffffff81152aff>] ? cleanup_mnt+0x4f/0x6b
Mar 15 14:21:30 fan kernel: [92308.377442]  [<ffffffff81066ab7>] ? task_work_run+0x5d/0x71
Mar 15 14:21:30 fan kernel: [92308.377444]  [<ffffffff810036dc>] ? prepare_exit_to_usermode+0x70/0x99
Mar 15 14:21:30 fan kernel: [92308.377446]  [<ffffffff8140de08>] ? int_ret_from_sys_call+0x25/0x8f
Mar 15 14:21:30 fan kernel: [92308.377448] ---[ end trace 18c6bb90b0c6c68a ]---

Mar 15 14:21:30 fan kernel: [92308.377455] ------------[ cut here ]------------
Mar 15 14:21:30 fan kernel: [92308.377469] WARNING: CPU: 5 PID: 28243 at fs/btrfs/extent-tree.c:9595 btrfs_free_block_groups+0x34f/0x36f [btrfs]()
Mar 15 14:21:30 fan kernel: [92308.377471] Modules linked in: vhost_net vhost macvtap macvlan tun iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp dummy ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative bridge stp llc snd_cmipci snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi kvm_amd snd_mpu401_uart snd_opl3_lib snd_rawmidi kvm snd_hda_intel snd_seq_device snd_hda_codec snd_hda_core snd_hwdep amd64_edac_mod snd_pcm_oss edac_mce_amd irqbypass input_leds snd_mixer_oss pcspkr k10temp edac_core snd_pcm snd_timer snd i2c_piix4 asus_atk0110 soundcore acpi_cpufreq tpm_tis tpm sg processor evdev shpchp hwmon_vid autofs4 crc32c_generic btrfs xor raid6_pq ext4 crc16 mbcache jbd2 hmac sha256_ssse3 sha256_generic drbg ansi_cprng xts gf128mul algif_skcipher af_alg dm_crypt dm_mod hid_generic usbhid hid usb_storage sr_mod sd_mod cdrom ohci_pci r8169 mii amdkfd radeon i2c_algo_bit ahci ttm sym53c8xx libahci xhci_pci scsi_transport_spi drm_kms_helper ohci_hcd ehci_pci xhci_hcd libata ehci_hcd drm usbcore scsi_mod usb_common i2c_core button
Mar 15 14:21:30 fan kernel: [92308.377514] CPU: 5 PID: 28243 Comm: umount Tainted: G        W       4.4.5-zgws1 #2
Mar 15 14:21:30 fan kernel: [92308.377515] Hardware name: System manufacturer System Product Name/M5A88-V EVO, BIOS 1603    10/12/2012
Mar 15 14:21:30 fan kernel: [92308.377517]  000000000000005b ffffffff811dd418 0000000000000000 0000000000000009
Mar 15 14:21:30 fan kernel: [92308.377519]  ffffffff81051e21 ffffffffa047a2da ffff880600a28000 ffff8806173f6e88
Mar 15 14:21:30 fan kernel: [92308.377521]  0000000000000038 0000000000000000 ffffffffa047a2da ffff880600a28000
Mar 15 14:21:30 fan kernel: [92308.377523] Call Trace:
Mar 15 14:21:30 fan kernel: [92308.377525]  [<ffffffff811dd418>] ? dump_stack+0x5a/0x6f
Mar 15 14:21:30 fan kernel: [92308.377528]  [<ffffffff81051e21>] ? warn_slowpath_common+0x8e/0xa3
Mar 15 14:21:30 fan kernel: [92308.377542]  [<ffffffffa047a2da>] ? btrfs_free_block_groups+0x34f/0x36f [btrfs]
Mar 15 14:21:30 fan kernel: [92308.377555]  [<ffffffffa047a2da>] ? btrfs_free_block_groups+0x34f/0x36f [btrfs]
Mar 15 14:21:30 fan kernel: [92308.377570]  [<ffffffffa0487c72>] ? close_ctree+0x1e6/0x2f2 [btrfs]
Mar 15 14:21:30 fan kernel: [92308.377573]  [<ffffffff8113dfd9>] ? generic_shutdown_super+0x64/0xdf
Mar 15 14:21:30 fan kernel: [92308.377574]  [<ffffffff8113e181>] ? kill_anon_super+0x9/0xe
Mar 15 14:21:30 fan kernel: [92308.377587]  [<ffffffffa04660db>] ? btrfs_kill_super+0xd/0x16 [btrfs]
Mar 15 14:21:30 fan kernel: [92308.377589]  [<ffffffff8113e286>] ? deactivate_locked_super+0x2f/0x56
Mar 15 14:21:30 fan kernel: [92308.377591]  [<ffffffff81152aff>] ? cleanup_mnt+0x4f/0x6b
Mar 15 14:21:30 fan kernel: [92308.377593]  [<ffffffff81066ab7>] ? task_work_run+0x5d/0x71
Mar 15 14:21:30 fan kernel: [92308.377596]  [<ffffffff810036dc>] ? prepare_exit_to_usermode+0x70/0x99
Mar 15 14:21:30 fan kernel: [92308.377598]  [<ffffffff8140de08>] ? int_ret_from_sys_call+0x25/0x8f
Mar 15 14:21:30 fan kernel: [92308.377600] ---[ end trace 18c6bb90b0c6c68b ]---
Mar 15 14:21:30 fan kernel: [92308.377601] BTRFS: space_info 4 has 20608794624 free, is not full
Mar 15 14:21:30 fan kernel: [92308.377604] BTRFS: space_info total=23085449216, used=2476654592, pinned=0, reserved=0, may_use=25769803776, readonly=0

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Henk Slager]

On Tue, Mar 15, 2016 at 2:42 PM, Marc Haber <mh+linux-btrfs@zugschlus.de> wrote:
> On Tue, Mar 15, 2016 at 02:29:32PM +0100, Marc Haber wrote:
>> After umounting and btrfs check the block device, things seem to be
>> fine now
>
> But, umounting the btrfs seemed to trigger the following kernel traces:
>
> Mar 15 14:21:30 fan kernel: [92308.377104] ------------[ cut here ]------------
> Mar 15 14:21:30 fan kernel: [92308.377135] WARNING: CPU: 5 PID: 28243 at fs/btrfs/extent-tree.c:5380 bt
> rfs_free_block_groups+0x1bc/0x36f [btrfs]()
> Mar 15 14:21:30 fan kernel: [92308.377137] Modules linked in: vhost_net vhost macvtap macvlan tun iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp dummy ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative bridge stp llc snd_cmipci snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi kvm_amd snd_mpu401_uart snd_opl3_lib snd_rawmidi kvm snd_hda_intel snd_seq_device snd_hda_codec snd_hda_core snd_hwdep amd64_edac_mod snd_pcm_oss edac_mce_amd irqbypass input_leds snd_mixer_oss pcspkr k10temp edac_core snd_pcm snd_timer snd i2c_piix4 asus_atk0110 soundcore acpi_cpufreq tpm_tis tpm sg processor evdev shpchp hwmon_vid autofs4 crc32c_generic btrfs xor raid6_pq ext4 crc16 mbcache jbd2 hmac sha256_ssse3 sha256_generic drbg ansi_cprng xts gf128mul algif_skcipher af_alg dm_crypt dm_mod hid_generic usbhid hid usb_storage sr_mod sd_mod cdrom ohci_pci r8169 mii amdkfd radeon i2c_algo_bit ahci ttm sym53c8xx libahci xhci_pci scsi_transport_spi drm_kms_helper ohci_hcd ehci_pci xhci_hcd libata ehci_hcd drm usbcore scsi_mod usb_common i2c_core button
> Mar 15 14:21:30 fan kernel: [92308.377203] CPU: 5 PID: 28243 Comm: umount Not tainted 4.4.5-zgws1 #2
> Mar 15 14:21:30 fan kernel: [92308.377205] Hardware name: System manufacturer System Product Name/M5A88-V EVO, BIOS 1603    10/12/2012
> Mar 15 14:21:30 fan kernel: [92308.377207]  000000000000005b ffffffff811dd418 0000000000000000 0000000000000009
> Mar 15 14:21:30 fan kernel: [92308.377210]  ffffffff81051e21 ffffffffa047a147 ffff880600a28000 0000000000000000
> Mar 15 14:21:30 fan kernel: [92308.377212]  ffff880600a28080 ffff8805af7eea00 ffffffffa047a147 ffff880600a28000
> Mar 15 14:21:30 fan kernel: [92308.377215] Call Trace:
> Mar 15 14:21:30 fan kernel: [92308.377221]  [<ffffffff811dd418>] ? dump_stack+0x5a/0x6f
> Mar 15 14:21:30 fan kernel: [92308.377224]  [<ffffffff81051e21>] ? warn_slowpath_common+0x8e/0xa3
> Mar 15 14:21:30 fan kernel: [92308.377239]  [<ffffffffa047a147>] ? btrfs_free_block_groups+0x1bc/0x36f[btrfs]
> Mar 15 14:21:30 fan kernel: [92308.377252]  [<ffffffffa047a147>] ? btrfs_free_block_groups+0x1bc/0x36f[btrfs]
> Mar 15 14:21:30 fan kernel: [92308.377267]  [<ffffffffa0487c72>] ? close_ctree+0x1e6/0x2f2 [btrfs]
> Mar 15 14:21:30 fan kernel: [92308.377271]  [<ffffffff8113dfd9>] ? generic_shutdown_super+0x64/0xdf
> Mar 15 14:21:30 fan kernel: [92308.377273]  [<ffffffff8113e181>] ? kill_anon_super+0x9/0xe
> Mar 15 14:21:30 fan kernel: [92308.377285]  [<ffffffffa04660db>] ? btrfs_kill_super+0xd/0x16 [btrfs]
> Mar 15 14:21:30 fan kernel: [92308.377288]  [<ffffffff8113e286>] ? deactivate_locked_super+0x2f/0x56
> Mar 15 14:21:30 fan kernel: [92308.377291]  [<ffffffff81152aff>] ? cleanup_mnt+0x4f/0x6b
> Mar 15 14:21:30 fan kernel: [92308.377293]  [<ffffffff81066ab7>] ? task_work_run+0x5d/0x71
> Mar 15 14:21:30 fan kernel: [92308.377296]  [<ffffffff810036dc>] ? prepare_exit_to_usermode+0x70/0x99
> Mar 15 14:21:30 fan kernel: [92308.377300]  [<ffffffff8140de08>] ? int_ret_from_sys_call+0x25/0x8f
> Mar 15 14:21:30 fan kernel: [92308.377302] ---[ end trace 18c6bb90b0c6c689 ]---
>
> Mar 15 14:21:30 fan kernel: [92308.377303] ------------[ cut here ]------------
> Mar 15 14:21:30 fan kernel: [92308.377318] WARNING: CPU: 5 PID: 28243 at fs/btrfs/extent-tree.c:5381 btrfs_free_block_groups+0x1d7/0x36f [btrfs]()
> Mar 15 14:21:30 fan kernel: [92308.377319] Modules linked in: vhost_net vhost macvtap macvlan tun iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp dummy ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative bridge stp llc snd_cmipci snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi kvm_amd snd_mpu401_uart snd_opl3_lib snd_rawmidi kvm snd_hda_intel snd_seq_device snd_hda_codec snd_hda_core snd_hwdep amd64_edac_mod snd_pcm_oss edac_mce_amd irqbypass input_leds snd_mixer_oss pcspkr k10temp edac_core snd_pcm snd_timer snd i2c_piix4 asus_atk0110 soundcore acpi_cpufreq tpm_tis tpm sg processor evdev shpchp hwmon_vid autofs4 crc32c_generic btrfs xor raid6_pq ext4 crc16 mbcache jbd2 hmac sha256_ssse3 sha256_generic drbg ansi_cprng xts gf128mul algif_skcipher af_alg dm_crypt dm_mod hid_generic usbhid hid usb_storage sr_mod sd_mod cdrom ohci_pci r8169 mii amdkfd radeon i2c_algo_bit ahci ttm sym53c8xx libahci xhci_pci scsi_transport_spi drm_kms_helper ohci_hcd ehci_pci xhci_hcd libata ehci_hcd drm usbcore scsi_mod usb_common i2c_core button
> Mar 15 14:21:30 fan kernel: [92308.377362] CPU: 5 PID: 28243 Comm: umount Tainted: G        W       4.4.5-zgws1 #2
> Mar 15 14:21:30 fan kernel: [92308.377364] Hardware name: System manufacturer System Product Name/M5A88-V EVO, BIOS 1603    10/12/2012
> Mar 15 14:21:30 fan kernel: [92308.377365]  000000000000005b ffffffff811dd418 0000000000000000 0000000000000009
> Mar 15 14:21:30 fan kernel: [92308.377367]  ffffffff81051e21 ffffffffa047a162 ffff880600a28000 0000000000000000
> Mar 15 14:21:30 fan kernel: [92308.377369]  ffff880600a28080 ffff8805af7eea00 ffffffffa047a162 ffff880600a28000
> Mar 15 14:21:30 fan kernel: [92308.377372] Call Trace:
> Mar 15 14:21:30 fan kernel: [92308.377374]  [<ffffffff811dd418>] ? dump_stack+0x5a/0x6f
> Mar 15 14:21:30 fan kernel: [92308.377377]  [<ffffffff81051e21>] ? warn_slowpath_common+0x8e/0xa3
> Mar 15 14:21:30 fan kernel: [92308.377390]  [<ffffffffa047a162>] ? btrfs_free_block_groups+0x1d7/0x36f [btrfs]
> Mar 15 14:21:30 fan kernel: [92308.377404]  [<ffffffffa047a162>] ? btrfs_free_block_groups+0x1d7/0x36f [btrfs]
> Mar 15 14:21:30 fan kernel: [92308.377419]  [<ffffffffa0487c72>] ? close_ctree+0x1e6/0x2f2 [btrfs]
> Mar 15 14:21:30 fan kernel: [92308.377421]  [<ffffffff8113dfd9>] ? generic_shutdown_super+0x64/0xdf
> Mar 15 14:21:30 fan kernel: [92308.377423]  [<ffffffff8113e181>] ? kill_anon_super+0x9/0xe
> Mar 15 14:21:30 fan kernel: [92308.377435]  [<ffffffffa04660db>] ? btrfs_kill_super+0xd/0x16 [btrfs]
> Mar 15 14:21:30 fan kernel: [92308.377438]  [<ffffffff8113e286>] ? deactivate_locked_super+0x2f/0x56
> Mar 15 14:21:30 fan kernel: [92308.377440]  [<ffffffff81152aff>] ? cleanup_mnt+0x4f/0x6b
> Mar 15 14:21:30 fan kernel: [92308.377442]  [<ffffffff81066ab7>] ? task_work_run+0x5d/0x71
> Mar 15 14:21:30 fan kernel: [92308.377444]  [<ffffffff810036dc>] ? prepare_exit_to_usermode+0x70/0x99
> Mar 15 14:21:30 fan kernel: [92308.377446]  [<ffffffff8140de08>] ? int_ret_from_sys_call+0x25/0x8f
> Mar 15 14:21:30 fan kernel: [92308.377448] ---[ end trace 18c6bb90b0c6c68a ]---

OK, this corresponds with the symptoms from the balances.

You hit the first to warnings in here:

static void release_global_block_rsv(struct btrfs_fs_info *fs_info)
{
        block_rsv_release_bytes(fs_info, &fs_info->global_block_rsv, NULL,
                                                (u64)-1);
        WARN_ON(fs_info->delalloc_block_rsv.size > 0);
        WARN_ON(fs_info->delalloc_block_rsv.reserved > 0);
        WARN_ON(fs_info->trans_block_rsv.size > 0);
        WARN_ON(fs_info->trans_block_rsv.reserved > 0);
        WARN_ON(fs_info->chunk_block_rsv.size > 0);
        WARN_ON(fs_info->chunk_block_rsv.reserved > 0);
        WARN_ON(fs_info->delayed_block_rsv.size > 0);
        WARN_ON(fs_info->delayed_block_rsv.reserved > 0);
}

and would need to traceback the code further to figure out why this
function is failing. That is one track.

Another track maybe, assuming something outside btrfs is causing the
trouble, would be to copy the fs device (so on top of dm) with dd to
some large secure local storage (maybe on another PC)  so that you can
mount it there via a loopdev. (Make at least sure a kernel has no
possibility to work on the same UUID ).

Then you can run the balance commands etc in more isolated
environment, at least not affected by possible
dm/root-crypt/bootloader etc issues. Hopefully that gives hints to the
root cause.


> Mar 15 14:21:30 fan kernel: [92308.377455] ------------[ cut here ]------------
> Mar 15 14:21:30 fan kernel: [92308.377469] WARNING: CPU: 5 PID: 28243 at fs/btrfs/extent-tree.c:9595 btrfs_free_block_groups+0x34f/0x36f [btrfs]()
> Mar 15 14:21:30 fan kernel: [92308.377471] Modules linked in: vhost_net vhost macvtap macvlan tun iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp dummy ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative bridge stp llc snd_cmipci snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi kvm_amd snd_mpu401_uart snd_opl3_lib snd_rawmidi kvm snd_hda_intel snd_seq_device snd_hda_codec snd_hda_core snd_hwdep amd64_edac_mod snd_pcm_oss edac_mce_amd irqbypass input_leds snd_mixer_oss pcspkr k10temp edac_core snd_pcm snd_timer snd i2c_piix4 asus_atk0110 soundcore acpi_cpufreq tpm_tis tpm sg processor evdev shpchp hwmon_vid autofs4 crc32c_generic btrfs xor raid6_pq ext4 crc16 mbcache jbd2 hmac sha256_ssse3 sha256_generic drbg ansi_cprng xts gf128mul algif_skcipher af_alg dm_crypt dm_mod hid_generic usbhid hid usb_storage sr_mod sd_mod cdrom ohci_pci r8169 mii amdkfd radeon i2c_algo_bit ahci ttm sym53c8xx libahci xhci_pci scsi_transport_spi drm_kms_helper ohci_hcd ehci_pci xhci_hcd libata ehci_hcd drm usbcore scsi_mod usb_common i2c_core button
> Mar 15 14:21:30 fan kernel: [92308.377514] CPU: 5 PID: 28243 Comm: umount Tainted: G        W       4.4.5-zgws1 #2
> Mar 15 14:21:30 fan kernel: [92308.377515] Hardware name: System manufacturer System Product Name/M5A88-V EVO, BIOS 1603    10/12/2012
> Mar 15 14:21:30 fan kernel: [92308.377517]  000000000000005b ffffffff811dd418 0000000000000000 0000000000000009
> Mar 15 14:21:30 fan kernel: [92308.377519]  ffffffff81051e21 ffffffffa047a2da ffff880600a28000 ffff8806173f6e88
> Mar 15 14:21:30 fan kernel: [92308.377521]  0000000000000038 0000000000000000 ffffffffa047a2da ffff880600a28000
> Mar 15 14:21:30 fan kernel: [92308.377523] Call Trace:
> Mar 15 14:21:30 fan kernel: [92308.377525]  [<ffffffff811dd418>] ? dump_stack+0x5a/0x6f
> Mar 15 14:21:30 fan kernel: [92308.377528]  [<ffffffff81051e21>] ? warn_slowpath_common+0x8e/0xa3
> Mar 15 14:21:30 fan kernel: [92308.377542]  [<ffffffffa047a2da>] ? btrfs_free_block_groups+0x34f/0x36f [btrfs]
> Mar 15 14:21:30 fan kernel: [92308.377555]  [<ffffffffa047a2da>] ? btrfs_free_block_groups+0x34f/0x36f [btrfs]
> Mar 15 14:21:30 fan kernel: [92308.377570]  [<ffffffffa0487c72>] ? close_ctree+0x1e6/0x2f2 [btrfs]
> Mar 15 14:21:30 fan kernel: [92308.377573]  [<ffffffff8113dfd9>] ? generic_shutdown_super+0x64/0xdf
> Mar 15 14:21:30 fan kernel: [92308.377574]  [<ffffffff8113e181>] ? kill_anon_super+0x9/0xe
> Mar 15 14:21:30 fan kernel: [92308.377587]  [<ffffffffa04660db>] ? btrfs_kill_super+0xd/0x16 [btrfs]
> Mar 15 14:21:30 fan kernel: [92308.377589]  [<ffffffff8113e286>] ? deactivate_locked_super+0x2f/0x56
> Mar 15 14:21:30 fan kernel: [92308.377591]  [<ffffffff81152aff>] ? cleanup_mnt+0x4f/0x6b
> Mar 15 14:21:30 fan kernel: [92308.377593]  [<ffffffff81066ab7>] ? task_work_run+0x5d/0x71
> Mar 15 14:21:30 fan kernel: [92308.377596]  [<ffffffff810036dc>] ? prepare_exit_to_usermode+0x70/0x99
> Mar 15 14:21:30 fan kernel: [92308.377598]  [<ffffffff8140de08>] ? int_ret_from_sys_call+0x25/0x8f
> Mar 15 14:21:30 fan kernel: [92308.377600] ---[ end trace 18c6bb90b0c6c68b ]---
> Mar 15 14:21:30 fan kernel: [92308.377601] BTRFS: space_info 4 has 20608794624 free, is not full
> Mar 15 14:21:30 fan kernel: [92308.377604] BTRFS: space_info total=23085449216, used=2476654592, pinned=0, reserved=0, may_use=25769803776, readonly=0
>
> Greetings
> Marc
>
> --
> -----------------------------------------------------------------------------
> Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
> Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
> Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Current state of old filesystem (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Marc Haber]

Hi,

I have worked on my script and have interesting new findings about the
"old" filesystem, the one that used to have the huge spread between
Size and Used for Metadata.

I have uploaded the script to
http://q.bofh.de/~mh/stuff/20160327-btrfs-balance
and a current full syslog to
http://q.bofh.de/~mh/stuff/20160326-linux-btrfs-ofanbtr

Current state is:

(1)
Filesystem has been resized to 300 GB. This has solved the "Superblock
bytenr is larger than device size" message on btrfs check. btrfs check
now runs through fine, output below.

(2)
Umounting the filesystem results in a kernel trace, given below as well.

(3)
Metadata spread is down to 27GiB/2.31 GiB, decreases to 10Gib/2.30GiB
and increases again during the run of my btrfs-balance script:
[7/507]mh@fan:~$ grep "Metadata, DUP" 20160326-linux-btrfs-ofanbtr
Mar 26 09:49:40 fan root: Metadata, DUP: total=27.00GiB, used=2.31GiB
Mar 26 10:26:15 fan root: Metadata, DUP: total=10.50GiB, used=2.30GiB
Mar 26 11:01:52 fan root: Metadata, DUP: total=10.50GiB, used=2.31GiB
Mar 26 11:03:36 fan root: Metadata, DUP: total=14.50GiB, used=2.30GiB
Mar 26 11:03:36 fan root: Metadata, DUP: total=14.50GiB, used=2.30GiB
Mar 26 11:38:40 fan root: Metadata, DUP: total=19.00GiB, used=2.31GiB
[8/508]mh@fan:~$

(4)
The attached log gives the following indications:
Mar 26 09:49:40 fan root: Metadata,DUP: Size:27.00GiB, Used:2.31GiB
Mar 26 09:49:40 fan mh: BEGIN btrfs balance start /mnt/ofanbtr
Mar 26 10:26:15 fan root: Done, had to relocate 134 out of 134 chunks
Mar 26 10:26:15 fan root: Metadata, DUP: total=10.50GiB, used=2.30GiB
Mar 26 10:26:15 fan mh: BEGIN btrfs balance start -dprofiles=single /mnt/ofanbtr
Mar 26 11:01:52 fan root: Done, had to relocate 79 out of 101 chunks
Mar 26 11:01:52 fan root: Metadata, DUP: total=10.50GiB, used=2.31GiB
Mar 26 11:01:52 fan mh: BEGIN btrfs balance start -mprofiles=dup
Mar 26 11:03:36 fan kernel: [ 4693.269593] BTRFS info (device dm-31): 9 enospc errors during balance
Mar 26 11:03:36 fan root: ERROR: error during balancing '/mnt/ofanbtr': No space left on device
Mar 26 11:03:36 fan root: There may be more info in syslog - try dmesg | tail
Mar 26 11:03:36 fan root: Metadata, DUP: total=14.50GiB, used=2.30GiB
Mar 26 11:03:36 fan mh: BEGIN btrfs balance start --force -sprofiles=dup /mnt/ofanbtr
Mar 26 11:03:36 fan root: Done, had to relocate 1 out of 109 chunks
Mar 26 11:03:36 fan root: Metadata, DUP: total=14.50GiB, used=2.30GiB
Mar 26 11:03:36 fan mh: BEGIN btrfs balance start /mnt/ofanbtr
Mar 26 11:38:40 fan kernel: [ 6796.907286] BTRFS info (device dm-31): 9 enospc errors during balance
Mar 26 11:38:40 fan root: ERROR: error during balancing '/mnt/ofanbtr': No space left on device
Mar 26 11:38:40 fan root: There may be more info in syslog - try dmesg | tail
Mar 26 11:38:40 fan root: Metadata, DUP: total=19.00GiB, used=2.31GiB

Note, that the initial btrfs balance start runs without ENOSPC, the
btrfs balance start -mprofiles=dup errors out, and the final btrfs
balance start errors out again. We thus have the case where a script
run has actually worsened the state of the filesystem.

Is this worth adding to the bug report
(https://bugzilla.kernel.org/show_bug.cgi?id=114451)? Should I include
a new btrfs image? What else should I include?

Any ideas?

Greetings
Marc



btrfs check:
Checking filesystem on /dev/mapper/ofanbtr
UUID: 4198d1bc-e3ce-40df-a7ee-44a2d120bff3
checking extents
checking free space cache
checking fs roots
checking csums
checking root refs
found 86554099816 bytes used err is 0
total csum bytes: 81815012
total tree bytes: 2476228608
total fs tree bytes: 2246311936
total extent tree bytes: 133005312
btree space waste bytes: 452682835
file data blocks allocated: 292993851392
 referenced 132664164352

Kernel trace on umount:
Mar 27 10:06:32 fan kernel: [84067.902831] ------------[ cut here ]------------
Mar 27 10:06:32 fan kernel: [84067.902861] WARNING: CPU: 5 PID: 13409 at fs/btrfs/extent-tree.c:5380 btrfs_free_block_groups+0x1bc/0x36f [btrfs]()
Mar 27 10:06:32 fan kernel: [84067.902862] Modules linked in: iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp dummy ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative bridge stp llc kvm_amd kvm irqbypass amd64_edac_mod edac_mce_amd pcspkr edac_core k10temp snd_cmipci snd_mpu401_uart snd_opl3_lib snd_rawmidi snd_hda_codec_realtek snd_seq_device input_leds snd_hda_codec_generic snd_hda_codec_hdmi snd_hda_intel asus_atk0110 snd_hda_codec snd_hda_core snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm sg evdev snd_timer acpi_cpufreq snd tpm_tis tpm soundcore processor shpchp i2c_piix4 hwmon_vid autofs4 crc32c_generic btrfs xor raid6_pq ext4 crc16 mbcache jbd2 hmac sha256_ssse3 sha256_generic drbg ansi_cprng xts gf128mul algif_skcipher af_alg dm_crypt dm_mod hid_generic usbhid hid usb_storage sr_mod sd_mod cdrom ohci_pci amdkfd radeon r8169 i2c_algo_bit mii sym53c8xx ahci ttm xhci_pci scsi_transport_spi libahci ehci_pci ohci_hcd xhci_hcd drm_kms_helper ehci_hcd libata drm usbcore scsi_mod usb_common i2c_core button
Mar 27 10:06:32 fan kernel: [84067.902909] CPU: 5 PID: 13409 Comm: umount Tainted: G        W    L  4.4.5-zgws1 #2
Mar 27 10:06:32 fan kernel: [84067.902911] Hardware name: System manufacturer System Product Name/M5A88-V EVO, BIOS 1603    10/12/2012
Mar 27 10:06:32 fan kernel: [84067.902913]  000000000000005b ffffffff811dd418 0000000000000000 0000000000000009
Mar 27 10:06:32 fan kernel: [84067.902915]  ffffffff81051e21 ffffffffa0481147 ffff8805f5b00000 0000000000000000
Mar 27 10:06:32 fan kernel: [84067.902917]  ffff8805f5b00080 ffff880608b5c800 ffffffffa0481147 ffff8805f5b00000
Mar 27 10:06:32 fan kernel: [84067.902919] Call Trace:
Mar 27 10:06:32 fan kernel: [84067.902923]  [<ffffffff811dd418>] ? dump_stack+0x5a/0x6f
Mar 27 10:06:32 fan kernel: [84067.902926]  [<ffffffff81051e21>] ? warn_slowpath_common+0x8e/0xa3
Mar 27 10:06:32 fan kernel: [84067.902937]  [<ffffffffa0481147>] ? btrfs_free_block_groups+0x1bc/0x36f [btrfs]
Mar 27 10:06:32 fan kernel: [84067.902947]  [<ffffffffa0481147>] ? btrfs_free_block_groups+0x1bc/0x36f [btrfs]
Mar 27 10:06:32 fan kernel: [84067.902959]  [<ffffffffa048ec72>] ? close_ctree+0x1e6/0x2f2 [btrfs]
Mar 27 10:06:32 fan kernel: [84067.902961]  [<ffffffff8113dfd9>] ? generic_shutdown_super+0x64/0xdf
Mar 27 10:06:32 fan kernel: [84067.902963]  [<ffffffff8113e181>] ? kill_anon_super+0x9/0xe
Mar 27 10:06:32 fan kernel: [84067.902972]  [<ffffffffa046d0db>] ? btrfs_kill_super+0xd/0x16 [btrfs]
Mar 27 10:06:32 fan kernel: [84067.902974]  [<ffffffff8113e286>] ? deactivate_locked_super+0x2f/0x56
Mar 27 10:06:32 fan kernel: [84067.902976]  [<ffffffff81152aff>] ? cleanup_mnt+0x4f/0x6b
Mar 27 10:06:32 fan kernel: [84067.902978]  [<ffffffff81066ab7>] ? task_work_run+0x5d/0x71
Mar 27 10:06:32 fan kernel: [84067.902981]  [<ffffffff810036dc>] ? prepare_exit_to_usermode+0x70/0x99
Mar 27 10:06:32 fan kernel: [84067.902983]  [<ffffffff8140de08>] ? int_ret_from_sys_call+0x25/0x8f
Mar 27 10:06:32 fan kernel: [84067.902985] ---[ end trace 095a538bb634d33d ]---
Mar 27 10:06:32 fan kernel: [84067.902986] ------------[ cut here ]------------
Mar 27 10:06:32 fan kernel: [84067.902997] WARNING: CPU: 5 PID: 13409 at fs/btrfs/extent-tree.c:5381 btrfs_free_block_groups+0x1d7/0x36f [btrfs]()
Mar 27 10:06:32 fan kernel: [84067.902998] Modules linked in: iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp dummy ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative bridge stp llc kvm_amd kvm irqbypass amd64_edac_mod edac_mce_amd pcspkr edac_core k10temp snd_cmipci snd_mpu401_uart snd_opl3_lib snd_rawmidi snd_hda_codec_realtek snd_seq_device input_leds snd_hda_codec_generic snd_hda_codec_hdmi snd_hda_intel asus_atk0110 snd_hda_codec snd_hda_core snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm sg evdev snd_timer acpi_cpufreq snd tpm_tis tpm soundcore processor shpchp i2c_piix4 hwmon_vid autofs4 crc32c_generic btrfs xor raid6_pq ext4 crc16 mbcache jbd2 hmac sha256_ssse3 sha256_generic drbg ansi_cprng xts gf128mul algif_skcipher af_alg dm_crypt dm_mod hid_generic usbhid hid usb_storage sr_mod sd_mod cdrom ohci_pci amdkfd radeon r8169 i2c_algo_bit mii sym53c8xx ahci ttm xhci_pci scsi_transport_spi libahci ehci_pci ohci_hcd xhci_hcd drm_kms_helper ehci_hcd libata drm usbcore scsi_mod usb_common i2c_core button
Mar 27 10:06:32 fan kernel: [84067.903030] CPU: 5 PID: 13409 Comm: umount Tainted: G        W    L  4.4.5-zgws1 #2
Mar 27 10:06:32 fan kernel: [84067.903031] Hardware name: System manufacturer System Product Name/M5A88-V EVO, BIOS 1603    10/12/2012
Mar 27 10:06:32 fan kernel: [84067.903032]  000000000000005b ffffffff811dd418 0000000000000000 0000000000000009
Mar 27 10:06:32 fan kernel: [84067.903034]  ffffffff81051e21 ffffffffa0481162 ffff8805f5b00000 0000000000000000
Mar 27 10:06:32 fan kernel: [84067.903035]  ffff8805f5b00080 ffff880608b5c800 ffffffffa0481162 ffff8805f5b00000
Mar 27 10:06:32 fan kernel: [84067.903037] Call Trace:
Mar 27 10:06:32 fan kernel: [84067.903039]  [<ffffffff811dd418>] ? dump_stack+0x5a/0x6f
Mar 27 10:06:32 fan kernel: [84067.903041]  [<ffffffff81051e21>] ? warn_slowpath_common+0x8e/0xa3
Mar 27 10:06:32 fan kernel: [84067.903055]  [<ffffffffa0481162>] ? btrfs_free_block_groups+0x1d7/0x36f [btrfs]
Mar 27 10:06:32 fan kernel: [84067.903064]  [<ffffffffa0481162>] ? btrfs_free_block_groups+0x1d7/0x36f [btrfs]
Mar 27 10:06:32 fan kernel: [84067.903074]  [<ffffffffa048ec72>] ? close_ctree+0x1e6/0x2f2 [btrfs]
Mar 27 10:06:32 fan kernel: [84067.903076]  [<ffffffff8113dfd9>] ? generic_shutdown_super+0x64/0xdf
Mar 27 10:06:32 fan kernel: [84067.903077]  [<ffffffff8113e181>] ? kill_anon_super+0x9/0xe
Mar 27 10:06:32 fan kernel: [84067.903085]  [<ffffffffa046d0db>] ? btrfs_kill_super+0xd/0x16 [btrfs]
Mar 27 10:06:32 fan kernel: [84067.903086]  [<ffffffff8113e286>] ? deactivate_locked_super+0x2f/0x56
Mar 27 10:06:32 fan kernel: [84067.903088]  [<ffffffff81152aff>] ? cleanup_mnt+0x4f/0x6b
Mar 27 10:06:32 fan kernel: [84067.903089]  [<ffffffff81066ab7>] ? task_work_run+0x5d/0x71
Mar 27 10:06:32 fan kernel: [84067.903091]  [<ffffffff810036dc>] ? prepare_exit_to_usermode+0x70/0x99
Mar 27 10:06:32 fan kernel: [84067.903092]  [<ffffffff8140de08>] ? int_ret_from_sys_call+0x25/0x8f
Mar 27 10:06:32 fan kernel: [84067.903093] ---[ end trace 095a538bb634d33e ]---

Full log sans boring BTRFS info lines (full log including them on
http://q.bofh.de/~mh/stuff/20160326-linux-btrfs-ofanbtr):
[15/514]mh@fan:~$ grep -v -E '(relocating block group|found [[:digit:]]+ extents)' 20160326-linux-btrfs-ofanbtr
Mar 26 09:49:26 fan kernel: [  242.951247] BTRFS: device label ofanbtr devid 1 transid 22237711 /dev/dm-31
Mar 26 09:49:28 fan kernel: [  245.232833] BTRFS info (device dm-31): force clearing of disk cache
Mar 26 09:49:28 fan kernel: [  245.232846] BTRFS info (device dm-31): disk space caching is enabled
Mar 26 09:49:28 fan kernel: [  245.232851] BTRFS: has skinny extents
Mar 26 09:49:28 fan kernel: [  245.333392] BTRFS: detected SSD devices, enabling SSD mode
Mar 26 09:49:40 fan mh: BEGIN btrfs-balance script
Mar 26 09:49:40 fan root: Linux fan 4.4.5-zgws1 #2 SMP Sun Mar 13 21:00:02 UTC 2016 x86_64 GNU/Linux
Mar 26 09:49:40 fan root: /dev/mapper/ofanbtr /mnt/ofanbtr btrfs rw,relatime,ssd,space_cache,clear_cache,subvolid=5,subvol=/ 0 0
Mar 26 09:49:40 fan mh: btrfs fi df /mnt/ofanbtr
Mar 26 09:49:40 fan root: Data, single: total=79.00GiB, used=78.33GiB
Mar 26 09:49:40 fan root: System, DUP: total=32.00MiB, used=16.00KiB
Mar 26 09:49:40 fan root: Metadata, DUP: total=27.00GiB, used=2.31GiB
Mar 26 09:49:40 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B
Mar 26 09:49:40 fan mh: btrfs fi show /mnt/ofanbtr
Mar 26 09:49:40 fan root: Label: 'ofanbtr'  uuid: 4198d1bc-e3ce-40df-a7ee-44a2d120bff3
Mar 26 09:49:40 fan root: #011Total devices 1 FS bytes used 80.63GiB
Mar 26 09:49:40 fan root: #011devid    1 size 300.00GiB used 133.06GiB path /dev/mapper/ofanbtr
Mar 26 09:49:40 fan root:
Mar 26 09:49:40 fan mh: btrfs fi usage /mnt/ofanbtr
Mar 26 09:49:40 fan root: Overall:
Mar 26 09:49:40 fan root:     Device size:#011#011 300.00GiB
Mar 26 09:49:40 fan root:     Device allocated:#011#011 133.06GiB
Mar 26 09:49:40 fan root:     Device unallocated:#011#011 166.94GiB
Mar 26 09:49:40 fan root:     Device missing:#011#011     0.00B
Mar 26 09:49:40 fan root:     Used:#011#011#011  82.94GiB
Mar 26 09:49:40 fan root:     Free (estimated):#011#011 167.61GiB#011(min: 84.14GiB)
Mar 26 09:49:40 fan root:     Data ratio:#011#011#011      1.00
Mar 26 09:49:40 fan root:     Metadata ratio:#011#011      2.00
Mar 26 09:49:40 fan root:     Global reserve:#011#011 512.00MiB#011(used: 0.00B)
Mar 26 09:49:40 fan root:
Mar 26 09:49:40 fan root: Data,single: Size:79.00GiB, Used:78.33GiB
Mar 26 09:49:40 fan root:    /dev/mapper/ofanbtr#011  79.00GiB
Mar 26 09:49:40 fan root:
Mar 26 09:49:40 fan root: Metadata,DUP: Size:27.00GiB, Used:2.31GiB
Mar 26 09:49:40 fan root:    /dev/mapper/ofanbtr#011  54.00GiB
Mar 26 09:49:40 fan root:
Mar 26 09:49:40 fan root: System,DUP: Size:32.00MiB, Used:16.00KiB
Mar 26 09:49:40 fan root:    /dev/mapper/ofanbtr#011  64.00MiB
Mar 26 09:49:40 fan root:
Mar 26 09:49:40 fan root: Unallocated:
Mar 26 09:49:40 fan root:    /dev/mapper/ofanbtr#011 166.94GiB
Mar 26 09:49:40 fan mh: BEGIN btrfs balance start /mnt/ofanbtr
Mar 26 10:26:15 fan root: Done, had to relocate 134 out of 134 chunks
Mar 26 10:26:15 fan mh: btrfs fi df /mnt/ofanbtr
Mar 26 10:26:15 fan root: Data, single: total=79.00GiB, used=78.32GiB
Mar 26 10:26:15 fan root: System, DUP: total=32.00MiB, used=16.00KiB
Mar 26 10:26:15 fan root: Metadata, DUP: total=10.50GiB, used=2.30GiB
Mar 26 10:26:15 fan root: GlobalReserve, single: total=512.00MiB, used=2.70MiB
Mar 26 10:26:15 fan mh: btrfs fi show /mnt/ofanbtr
Mar 26 10:26:15 fan root: Label: 'ofanbtr'  uuid: 4198d1bc-e3ce-40df-a7ee-44a2d120bff3
Mar 26 10:26:15 fan root: #011Total devices 1 FS bytes used 80.63GiB
Mar 26 10:26:15 fan root: #011devid    1 size 300.00GiB used 100.06GiB path /dev/mapper/ofanbtr
Mar 26 10:26:15 fan root:
Mar 26 10:26:15 fan mh: btrfs fi usage /mnt/ofanbtr
Mar 26 10:26:15 fan root: Overall:
Mar 26 10:26:15 fan root:     Device size:#011#011 300.00GiB
Mar 26 10:26:15 fan root:     Device allocated:#011#011 100.06GiB
Mar 26 10:26:15 fan root:     Device unallocated:#011#011 199.94GiB
Mar 26 10:26:15 fan root:     Device missing:#011#011     0.00B
Mar 26 10:26:15 fan root:     Used:#011#011#011  82.93GiB
Mar 26 10:26:15 fan root:     Free (estimated):#011#011 200.61GiB#011(min: 100.64GiB)
Mar 26 10:26:15 fan root:     Data ratio:#011#011#011      1.00
Mar 26 10:26:15 fan root:     Metadata ratio:#011#011      2.00
Mar 26 10:26:15 fan root:     Global reserve:#011#011 512.00MiB#011(used: 2.70MiB)
Mar 26 10:26:15 fan root:
Mar 26 10:26:15 fan root: Data,single: Size:79.00GiB, Used:78.32GiB
Mar 26 10:26:15 fan root:    /dev/mapper/ofanbtr#011  79.00GiB
Mar 26 10:26:15 fan root:
Mar 26 10:26:15 fan root: Metadata,DUP: Size:10.50GiB, Used:2.30GiB
Mar 26 10:26:15 fan root:    /dev/mapper/ofanbtr#011  21.00GiB
Mar 26 10:26:15 fan root:
Mar 26 10:26:15 fan root: System,DUP: Size:32.00MiB, Used:16.00KiB
Mar 26 10:26:15 fan root:    /dev/mapper/ofanbtr#011  64.00MiB
Mar 26 10:26:15 fan root:
Mar 26 10:26:15 fan root: Unallocated:
Mar 26 10:26:15 fan root:    /dev/mapper/ofanbtr#011 199.94GiB
Mar 26 10:26:15 fan mh: BEGIN btrfs balance start -dprofiles=single /mnt/ofanbtr
Mar 26 11:01:52 fan root: Done, had to relocate 79 out of 101 chunks
Mar 26 11:01:52 fan mh: btrfs fi df /mnt/ofanbtr
Mar 26 11:01:52 fan root: Data, single: total=79.00GiB, used=78.33GiB
Mar 26 11:01:52 fan root: System, DUP: total=32.00MiB, used=16.00KiB
Mar 26 11:01:52 fan root: Metadata, DUP: total=10.50GiB, used=2.31GiB
Mar 26 11:01:52 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B
Mar 26 11:01:52 fan mh: btrfs fi show /mnt/ofanbtr
Mar 26 11:01:52 fan root: Label: 'ofanbtr'  uuid: 4198d1bc-e3ce-40df-a7ee-44a2d120bff3
Mar 26 11:01:52 fan root: #011Total devices 1 FS bytes used 80.63GiB
Mar 26 11:01:52 fan root: #011devid    1 size 300.00GiB used 100.06GiB path /dev/mapper/ofanbtr
Mar 26 11:01:52 fan root:
Mar 26 11:01:52 fan mh: btrfs fi usage /mnt/ofanbtr
Mar 26 11:01:52 fan root: Overall:
Mar 26 11:01:52 fan root:     Device size:#011#011 300.00GiB
Mar 26 11:01:52 fan root:     Device allocated:#011#011 100.06GiB
Mar 26 11:01:52 fan root:     Device unallocated:#011#011 199.94GiB
Mar 26 11:01:52 fan root:     Device missing:#011#011     0.00B
Mar 26 11:01:52 fan root:     Used:#011#011#011  82.94GiB
Mar 26 11:01:52 fan root:     Free (estimated):#011#011 200.61GiB#011(min: 100.64GiB)
Mar 26 11:01:52 fan root:     Data ratio:#011#011#011      1.00
Mar 26 11:01:52 fan root:     Metadata ratio:#011#011      2.00
Mar 26 11:01:52 fan root:     Global reserve:#011#011 512.00MiB#011(used: 0.00B)
Mar 26 11:01:52 fan root:
Mar 26 11:01:52 fan root: Data,single: Size:79.00GiB, Used:78.33GiB
Mar 26 11:01:52 fan root:    /dev/mapper/ofanbtr#011  79.00GiB
Mar 26 11:01:52 fan root:
Mar 26 11:01:52 fan root: Metadata,DUP: Size:10.50GiB, Used:2.31GiB
Mar 26 11:01:52 fan root:    /dev/mapper/ofanbtr#011  21.00GiB
Mar 26 11:01:52 fan root:
Mar 26 11:01:52 fan root: System,DUP: Size:32.00MiB, Used:16.00KiB
Mar 26 11:01:52 fan root:    /dev/mapper/ofanbtr#011  64.00MiB
Mar 26 11:01:52 fan root:
Mar 26 11:01:52 fan root: Unallocated:
Mar 26 11:01:52 fan root:    /dev/mapper/ofanbtr#011 199.94GiB
Mar 26 11:01:52 fan mh: BEGIN btrfs balance start -mprofiles=dup /mnt/ofanbtr
Mar 26 11:03:36 fan kernel: [ 4693.269593] BTRFS info (device dm-31): 9 enospc errors during balance
Mar 26 11:03:36 fan root: ERROR: error during balancing '/mnt/ofanbtr': No space left on device
Mar 26 11:03:36 fan root: There may be more info in syslog - try dmesg | tail
Mar 26 11:03:36 fan mh: btrfs fi df /mnt/ofanbtr
Mar 26 11:03:36 fan root: Data, single: total=79.00GiB, used=78.32GiB
Mar 26 11:03:36 fan root: System, DUP: total=32.00MiB, used=16.00KiB
Mar 26 11:03:36 fan root: Metadata, DUP: total=14.50GiB, used=2.30GiB
Mar 26 11:03:36 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B
Mar 26 11:03:36 fan mh: btrfs fi show /mnt/ofanbtr
Mar 26 11:03:36 fan root: Label: 'ofanbtr'  uuid: 4198d1bc-e3ce-40df-a7ee-44a2d120bff3
Mar 26 11:03:36 fan root: #011Total devices 1 FS bytes used 80.63GiB
Mar 26 11:03:36 fan root: #011devid    1 size 300.00GiB used 108.06GiB path /dev/mapper/ofanbtr
Mar 26 11:03:36 fan root:
Mar 26 11:03:36 fan mh: btrfs fi usage /mnt/ofanbtr
Mar 26 11:03:36 fan root: Overall:
Mar 26 11:03:36 fan root:     Device size:#011#011 300.00GiB
Mar 26 11:03:36 fan root:     Device allocated:#011#011 108.06GiB
Mar 26 11:03:36 fan root:     Device unallocated:#011#011 191.94GiB
Mar 26 11:03:36 fan root:     Device missing:#011#011     0.00B
Mar 26 11:03:36 fan root:     Used:#011#011#011  82.93GiB
Mar 26 11:03:36 fan root:     Free (estimated):#011#011 192.61GiB#011(min: 96.64GiB)
Mar 26 11:03:36 fan root:     Data ratio:#011#011#011      1.00
Mar 26 11:03:36 fan root:     Metadata ratio:#011#011      2.00
Mar 26 11:03:36 fan root:     Global reserve:#011#011 512.00MiB#011(used: 0.00B)
Mar 26 11:03:36 fan root:
Mar 26 11:03:36 fan root: Data,single: Size:79.00GiB, Used:78.32GiB
Mar 26 11:03:36 fan root:    /dev/mapper/ofanbtr#011  79.00GiB
Mar 26 11:03:36 fan root:
Mar 26 11:03:36 fan root: Metadata,DUP: Size:14.50GiB, Used:2.30GiB
Mar 26 11:03:36 fan root:    /dev/mapper/ofanbtr#011  29.00GiB
Mar 26 11:03:36 fan root:
Mar 26 11:03:36 fan root: System,DUP: Size:32.00MiB, Used:16.00KiB
Mar 26 11:03:36 fan root:    /dev/mapper/ofanbtr#011  64.00MiB
Mar 26 11:03:36 fan root:
Mar 26 11:03:36 fan root: Unallocated:
Mar 26 11:03:36 fan root:    /dev/mapper/ofanbtr#011 191.94GiB
Mar 26 11:03:36 fan mh: BEGIN btrfs balance start --force -sprofiles=dup /mnt/ofanbtr
Mar 26 11:03:36 fan root: Done, had to relocate 1 out of 109 chunks
Mar 26 11:03:36 fan mh: btrfs fi df /mnt/ofanbtr
Mar 26 11:03:36 fan root: Data, single: total=79.00GiB, used=78.32GiB
Mar 26 11:03:36 fan root: System, DUP: total=32.00MiB, used=16.00KiB
Mar 26 11:03:36 fan root: Metadata, DUP: total=14.50GiB, used=2.30GiB
Mar 26 11:03:36 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B
Mar 26 11:03:36 fan mh: btrfs fi show /mnt/ofanbtr
Mar 26 11:03:36 fan root: Label: 'ofanbtr'  uuid: 4198d1bc-e3ce-40df-a7ee-44a2d120bff3
Mar 26 11:03:36 fan root: #011Total devices 1 FS bytes used 80.63GiB
Mar 26 11:03:36 fan root: #011devid    1 size 300.00GiB used 108.06GiB path /dev/mapper/ofanbtr
Mar 26 11:03:36 fan root:
Mar 26 11:03:36 fan mh: btrfs fi usage /mnt/ofanbtr
Mar 26 11:03:36 fan root: Overall:
Mar 26 11:03:36 fan root:     Device size:#011#011 300.00GiB
Mar 26 11:03:36 fan root:     Device allocated:#011#011 108.06GiB
Mar 26 11:03:36 fan root:     Device unallocated:#011#011 191.94GiB
Mar 26 11:03:36 fan root:     Device missing:#011#011     0.00B
Mar 26 11:03:36 fan root:     Used:#011#011#011  82.93GiB
Mar 26 11:03:36 fan root:     Free (estimated):#011#011 192.61GiB#011(min: 96.64GiB)
Mar 26 11:03:36 fan root:     Data ratio:#011#011#011      1.00
Mar 26 11:03:36 fan root:     Metadata ratio:#011#011      2.00
Mar 26 11:03:36 fan root:     Global reserve:#011#011 512.00MiB#011(used: 0.00B)
Mar 26 11:03:36 fan root:
Mar 26 11:03:36 fan root: Data,single: Size:79.00GiB, Used:78.32GiB
Mar 26 11:03:36 fan root:    /dev/mapper/ofanbtr#011  79.00GiB
Mar 26 11:03:36 fan root:
Mar 26 11:03:36 fan root: Metadata,DUP: Size:14.50GiB, Used:2.30GiB
Mar 26 11:03:36 fan root:    /dev/mapper/ofanbtr#011  29.00GiB
Mar 26 11:03:36 fan root:
Mar 26 11:03:36 fan root: System,DUP: Size:32.00MiB, Used:16.00KiB
Mar 26 11:03:36 fan root:    /dev/mapper/ofanbtr#011  64.00MiB
Mar 26 11:03:36 fan root:
Mar 26 11:03:36 fan root: Unallocated:
Mar 26 11:03:36 fan root:    /dev/mapper/ofanbtr#011 191.94GiB
Mar 26 11:03:36 fan mh: BEGIN btrfs balance start /mnt/ofanbtr
Mar 26 11:38:40 fan kernel: [ 6796.907286] BTRFS info (device dm-31): 9 enospc errors during balance
Mar 26 11:38:40 fan root: ERROR: error during balancing '/mnt/ofanbtr': No space left on device
Mar 26 11:38:40 fan root: There may be more info in syslog - try dmesg | tail
Mar 26 11:38:40 fan mh: btrfs fi df /mnt/ofanbtr
Mar 26 11:38:40 fan root: Data, single: total=79.00GiB, used=78.33GiB
Mar 26 11:38:40 fan root: System, DUP: total=32.00MiB, used=16.00KiB
Mar 26 11:38:40 fan root: Metadata, DUP: total=19.00GiB, used=2.31GiB
Mar 26 11:38:40 fan root: GlobalReserve, single: total=512.00MiB, used=0.00B
Mar 26 11:38:40 fan mh: btrfs fi show /mnt/ofanbtr
Mar 26 11:38:40 fan root: Label: 'ofanbtr'  uuid: 4198d1bc-e3ce-40df-a7ee-44a2d120bff3
Mar 26 11:38:40 fan root: #011Total devices 1 FS bytes used 80.63GiB
Mar 26 11:38:40 fan root: #011devid    1 size 300.00GiB used 117.06GiB path /dev/mapper/ofanbtr
Mar 26 11:38:40 fan root:
Mar 26 11:38:40 fan mh: btrfs fi usage /mnt/ofanbtr
Mar 26 11:38:40 fan root: Overall:
Mar 26 11:38:40 fan root:     Device size:#011#011 300.00GiB
Mar 26 11:38:40 fan root:     Device allocated:#011#011 117.06GiB
Mar 26 11:38:40 fan root:     Device unallocated:#011#011 182.94GiB
Mar 26 11:38:40 fan root:     Device missing:#011#011     0.00B
Mar 26 11:38:40 fan root:     Used:#011#011#011  82.94GiB
Mar 26 11:38:40 fan root:     Free (estimated):#011#011 183.61GiB#011(min: 92.14GiB)
Mar 26 11:38:40 fan root:     Data ratio:#011#011#011      1.00
Mar 26 11:38:40 fan root:     Metadata ratio:#011#011      2.00
Mar 26 11:38:40 fan root:     Global reserve:#011#011 512.00MiB#011(used: 0.00B)
Mar 26 11:38:40 fan root:
Mar 26 11:38:40 fan root: Data,single: Size:79.00GiB, Used:78.33GiB
Mar 26 11:38:40 fan root:    /dev/mapper/ofanbtr#011  79.00GiB
Mar 26 11:38:40 fan root:
Mar 26 11:38:40 fan root: Metadata,DUP: Size:19.00GiB, Used:2.31GiB
Mar 26 11:38:40 fan root:    /dev/mapper/ofanbtr#011  38.00GiB
Mar 26 11:38:40 fan root:
Mar 26 11:38:40 fan root: System,DUP: Size:32.00MiB, Used:16.00KiB
Mar 26 11:38:40 fan root:    /dev/mapper/ofanbtr#011  64.00MiB
Mar 26 11:38:40 fan root:
Mar 26 11:38:40 fan root: Unallocated:
Mar 26 11:38:40 fan root:    /dev/mapper/ofanbtr#011 182.94GiB
Mar 26 11:38:40 fan mh: END btrfs-balance script
[16/515]mh@fan:~$


-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: Again, no space left on device while rebalancing and recipe doesnt work

[Marc Haber]

On Sat, Feb 27, 2016 at 10:14:50PM +0100, Marc Haber wrote:
> I have again the issue of no space left on device while rebalancing
> (with btrfs-tools 4.4.1 on kernel 4.4.2 on Debian unstable):

just for the record: The host started acting up in more and more
interesting ways, and after a call of rm during kernel build resulted
in SIGSEGV, I did the backup-format-restore routine for this system
back to ext4 just to find out whether I have bad hardware or a bad
filesystem.

And, since going back to ext4, the system is just fine again. So it's
not bad hardware.

This systems's root drive is going to stay on ext4 for a loooooong
time. If I get the btrfs phenomena I experience on other hosts get
solved at some time in the future, I might migrate /home back to
btrfs, but that's not going to happen in the next six months.

This is a really bad experience which has made me lost a lot of faith
in the new filesystem. I really feel sad about that.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421