no space left, metadata usage almost full?

no space left, metadata usage almost full?

[Tomasz Chmielewski]

I have a btrfs filesystem which has plenty of free space left, yet it's
hitting out of space regularly.

Here is how it looks like:

# btrfs fi df /home
Data, RAID1: total=2.51TiB, used=1.58TiB
System, RAID1: total=32.00MiB, used=372.00KiB
Metadata, RAID1: total=48.00GiB, used=47.23GiB


What I read from it, is we're almost full on metadata usage, and that
might be causing out of space issues.

Reading past posts on this group, I can see it's recommended to run
this if I hit out of space and the fs is low on metadata space:

# btrfs fi balance start -dusage=5 /home

Is it really the only workaround? Shouldn't the filesystem be more
intelligent and be able to grab some more metadata space if it's
running low?

I'd appreciate some clarifications on this (FYI, it was running
3.11.4, upgraded to the latest rc now).


-- 
Tomasz Chmielewski
http://wpkg.org

Re: no space left, metadata usage almost full?

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1482 bytes --]

On Wed, Dec 18, 2013 at 09:37:28PM +0900, Tomasz Chmielewski wrote:
> I have a btrfs filesystem which has plenty of free space left, yet it's
> hitting out of space regularly.
> 
> Here is how it looks like:
> 
> # btrfs fi df /home
> Data, RAID1: total=2.51TiB, used=1.58TiB
> System, RAID1: total=32.00MiB, used=372.00KiB
> Metadata, RAID1: total=48.00GiB, used=47.23GiB
> 
> 
> What I read from it, is we're almost full on metadata usage, and that
> might be causing out of space issues.

   This is highly likely.

> Reading past posts on this group, I can see it's recommended to run
> this if I hit out of space and the fs is low on metadata space:
> 
> # btrfs fi balance start -dusage=5 /home
> 
> Is it really the only workaround? Shouldn't the filesystem be more
> intelligent and be able to grab some more metadata space if it's
> running low?

   Currently, yes, it is the only approach.

   The automatic reclamation of unused chunks (or barely-used chunks)
is on the projects list. Nobody's got round to implementing it yet.

> I'd appreciate some clarifications on this (FYI, it was running
> 3.11.4, upgraded to the latest rc now).

   Hope the above helps,
   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
      --- If it's December 1941 in Casablanca,  what time is it ---      
                              in New York?                               

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: no space left, metadata usage almost full?

[Tomasz Chmielewski]

On Wed, 18 Dec 2013 12:46:52 +0000
Hugo Mills <hugo@carfax.org.uk> wrote:

> > # btrfs fi df /home
> > Data, RAID1: total=2.51TiB, used=1.58TiB
> > System, RAID1: total=32.00MiB, used=372.00KiB
> > Metadata, RAID1: total=48.00GiB, used=47.23GiB
> > # btrfs fi balance start -dusage=5 /home

>    Currently, yes, it is the only approach.
>    Hope the above helps,

So the balance finished, and metadata is still almost full:

# btrfs fi df /home
Data, RAID1: total=1.60TiB, used=1.58TiB
System, RAID1: total=32.00MiB, used=248.00KiB
Metadata, RAID1: total=49.00GiB, used=47.24GiB

Confused about the output - does it actually look any better?


# btrfs fi show /home
Label: crawler-btrfs  uuid: 60f1759c-45f6-4484-9f60-66a4e9bbf2b6
        Total devices 2 FS bytes used 1.63TiB
        devid    3 size 2.56TiB used 1.66TiB path /dev/sdb4
        devid    4 size 2.56TiB used 1.66TiB path /dev/sda4

Btrfs v3.12


Does it mean that data/system/metadata will be able to grow now,
until their size in total in 2.56TiB?

-- 
Tomasz Chmielewski
http://wpkg.org

Re: no space left, metadata usage almost full?

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1906 bytes --]

On Wed, Dec 18, 2013 at 11:54:39PM +0900, Tomasz Chmielewski wrote:
> On Wed, 18 Dec 2013 12:46:52 +0000
> Hugo Mills <hugo@carfax.org.uk> wrote:
> 
> > > # btrfs fi df /home
> > > Data, RAID1: total=2.51TiB, used=1.58TiB
> > > System, RAID1: total=32.00MiB, used=372.00KiB
> > > Metadata, RAID1: total=48.00GiB, used=47.23GiB
> > > # btrfs fi balance start -dusage=5 /home
> 
> >    Currently, yes, it is the only approach.
> >    Hope the above helps,
> 
> So the balance finished, and metadata is still almost full:
> 
> # btrfs fi df /home
> Data, RAID1: total=1.60TiB, used=1.58TiB
> System, RAID1: total=32.00MiB, used=248.00KiB
> Metadata, RAID1: total=49.00GiB, used=47.24GiB
> 
> Confused about the output - does it actually look any better?

   Yes, because...

> # btrfs fi show /home
> Label: crawler-btrfs  uuid: 60f1759c-45f6-4484-9f60-66a4e9bbf2b6
>         Total devices 2 FS bytes used 1.63TiB
>         devid    3 size 2.56TiB used 1.66TiB path /dev/sdb4
>         devid    4 size 2.56TiB used 1.66TiB path /dev/sda4

   ... you have unallocated space here, so the FS can now allocate
more metadata as it needs to.

> Does it mean that data/system/metadata will be able to grow now,
> until their size in total in 2.56TiB?

   Yes.

   Although note that where btrfs fi df reports space, that's _usable_
space (i.e. how much data you can fit in it), but where btrfs fi show
reports space, it's disk bytes (i.e. how much of the disk has useful
content on it). With RAID-1, the first figure is half the second
figure. In your case, that's simple, but with different RAID levels
for data and metadata the calculation becomes a little bit more
complicated.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
          --- What part of "gestalt" don't you understand? ---           

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]