"disk full" on a 5 GB btrfs filesystem, FAQ outdated?

"disk full" on a 5 GB btrfs filesystem, FAQ outdated?

[Marc Haber]

Hi,

I have a banana pi with a btrfs filesystem of 5 GB in size, which
frequently runs out of space (lots of snapshots). This is currently
again the case:

[27/524]mh@banana:~$ sudo btrfs balance start /
ERROR: error during balancing '/' - No space left on device
There may be more info in syslog - try dmesg | tail
[28/525]mh@banana:~$ sudo btrfs balance start / -dlimit=3
[sudo] password for mh on banana:
ERROR: error during balancing '/' - No space left on device
There may be more info in syslog - try dmesg | tail
[29/526]mh@banana:~$ sudo btrfs balance start / -dlimit=3
ERROR: error during balancing '/' - No space left on device
There may be more info in syslog - try dmesg | tail
[30/526]mh@banana:~$ sudo btrfs balance start / -dusage=0
Done, had to relocate 0 out of 8 chunks
[31/527]mh@banana:~$ sudo btrfs balance start / -dlimit=3
ERROR: error during balancing '/' - No space left on device
There may be more info in syslog - try dmesg | tail
[32/528]mh@banana:~$ sudo btrfs fi show /
Label: none  uuid: ada6b7f5-98d6-4fee-a3a3-b73bd152ff6c
        Total devices 1 FS bytes used 3.37GiB
        devid    1 size 6.89GiB used 4.22GiB path /dev/mapper/banana-root

btrfs-progs v4.3
[33/529]mh@banana:~$ sudo btrfs fi df /
Data, single: total=3.41GiB, used=3.25GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=384.00MiB, used=121.75MiB
GlobalReserve, single: total=48.00MiB, used=0.00B
[34/530]mh@banana:~$ uname -a
Linux banana 4.3.0-zgbpi-armmp-lpae+ #2 SMP Sat Nov 7 13:07:34 UTC 2015 armv7l GNU/Linux
[36/532]mh@banana:~$ df -h /
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/banana-root  6.9G  3.6G  2.9G  56% /
[37/533]mh@banana:~$

The first kernel that was ever booted on the device was 4.1, I am
therefore reasonably sure that the filesystem was also created with a
recent kernel. Is there any possibility to find out about the kernel
version that a filesystem was created with?

However, the FAQ
https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_Btrfs_claims_I.27m_out_of_space.2C_but_it_looks_like_I_should_have_lots_left.21
suggests that for small filesystems (<16 GB), the best solution would
be to upgrade to at least 2.6.37 and recreate the filesystem. 2.6.37
is ancient, from 2011, so I am pretty sure that the filesystem _was_
created at least with a kernel more recent than that.

My normal way to recover from this situation is to btrfs add a new
device, btrfs balance, btrfs --convert=single --force balance, btfs
device remove, btr balance start -mconvert=dup --force and finally
balance start again.

Is there any solution to solve this more elegantly?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: "disk full" on a 5 GB btrfs filesystem, FAQ outdated?

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 3231 bytes --]

On Sun, Nov 29, 2015 at 02:07:54PM +0100, Marc Haber wrote:
> Hi,
> 
> I have a banana pi with a btrfs filesystem of 5 GB in size, which
> frequently runs out of space (lots of snapshots). This is currently
> again the case:
> 
> [27/524]mh@banana:~$ sudo btrfs balance start /
> ERROR: error during balancing '/' - No space left on device
> There may be more info in syslog - try dmesg | tail
> [28/525]mh@banana:~$ sudo btrfs balance start / -dlimit=3
> [sudo] password for mh on banana:
> ERROR: error during balancing '/' - No space left on device
> There may be more info in syslog - try dmesg | tail
> [29/526]mh@banana:~$ sudo btrfs balance start / -dlimit=3
> ERROR: error during balancing '/' - No space left on device
> There may be more info in syslog - try dmesg | tail
> [30/526]mh@banana:~$ sudo btrfs balance start / -dusage=0
> Done, had to relocate 0 out of 8 chunks
> [31/527]mh@banana:~$ sudo btrfs balance start / -dlimit=3
> ERROR: error during balancing '/' - No space left on device
> There may be more info in syslog - try dmesg | tail
> [32/528]mh@banana:~$ sudo btrfs fi show /
> Label: none  uuid: ada6b7f5-98d6-4fee-a3a3-b73bd152ff6c
>         Total devices 1 FS bytes used 3.37GiB
>         devid    1 size 6.89GiB used 4.22GiB path /dev/mapper/banana-root
> 
> btrfs-progs v4.3
> [33/529]mh@banana:~$ sudo btrfs fi df /
> Data, single: total=3.41GiB, used=3.25GiB
> System, DUP: total=32.00MiB, used=16.00KiB
> Metadata, DUP: total=384.00MiB, used=121.75MiB
> GlobalReserve, single: total=48.00MiB, used=0.00B
> [34/530]mh@banana:~$ uname -a
> Linux banana 4.3.0-zgbpi-armmp-lpae+ #2 SMP Sat Nov 7 13:07:34 UTC 2015 armv7l GNU/Linux
> [36/532]mh@banana:~$ df -h /
> Filesystem               Size  Used Avail Use% Mounted on
> /dev/mapper/banana-root  6.9G  3.6G  2.9G  56% /
> [37/533]mh@banana:~$
> 
> The first kernel that was ever booted on the device was 4.1, I am
> therefore reasonably sure that the filesystem was also created with a
> recent kernel. Is there any possibility to find out about the kernel
> version that a filesystem was created with?
> 
> However, the FAQ
> https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_Btrfs_claims_I.27m_out_of_space.2C_but_it_looks_like_I_should_have_lots_left.21
> suggests that for small filesystems (<16 GB), the best solution would
> be to upgrade to at least 2.6.37 and recreate the filesystem. 2.6.37
> is ancient, from 2011, so I am pretty sure that the filesystem _was_
> created at least with a kernel more recent than that.

   You missed the most important thing from that paragraph: Use mixed
block groups. That's "mkfs.btrfs --mixed ..." (which I realise is
missing from the text, and I'll be adding it after I send this email).

> My normal way to recover from this situation is to btrfs add a new
> device, btrfs balance, btrfs --convert=single --force balance, btfs
> device remove, btr balance start -mconvert=dup --force and finally
> balance start again.
> 
> Is there any solution to solve this more elegantly?

   Recreate the FS with --mixed, and that should deal with it.

   Hugo.

-- 
Hugo Mills             | There isn't a noun that can't be verbed.
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: "disk full" on a 5 GB btrfs filesystem, FAQ outdated?

[Marc Haber]

Hi Hugo,

On Sun, Nov 29, 2015 at 02:18:06PM +0000, Hugo Mills wrote:
> On Sun, Nov 29, 2015 at 02:07:54PM +0100, Marc Haber wrote:
> > However, the FAQ
> > https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_Btrfs_claims_I.27m_out_of_space.2C_but_it_looks_like_I_should_have_lots_left.21
> > suggests that for small filesystems (<16 GB), the best solution would
> > be to upgrade to at least 2.6.37 and recreate the filesystem. 2.6.37
> > is ancient, from 2011, so I am pretty sure that the filesystem _was_
> > created at least with a kernel more recent than that.
> 
>    You missed the most important thing from that paragraph: Use mixed
> block groups. That's "mkfs.btrfs --mixed ..." (which I realise is
> missing from the text, and I'll be adding it after I send this email).

Yes, that was the important bit of missing information. My filesystem
now reads:

[26/512]mh@fan:/mnt/tempdisk$ df -h .
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/banana-root  6,0G  836M  5,2G  14% /mnt/tempdisk
[27/513]mh@fan:/mnt/tempdisk$ sudo btrfs fi show .
Label: none  uuid: b2906231-70a9-46d9-9830-38a13cb73171
        Total devices 1 FS bytes used 861.29MiB
        devid    1 size 6.00GiB used 6.00GiB path /dev/mapper/banana-root

btrfs-progs v4.3
[28/514]mh@fan:/mnt/tempdisk$ sudo btrfs fi df .
System, single: total=4.00MiB, used=4.00KiB
Data+Metadata, single: total=6.00GiB, used=861.29MiB
GlobalReserve, single: total=20.00MiB, used=0.00B
[29/515]mh@fan:/mnt/tempdisk$

Can I somehow get duplicate metadata back? Or is that unnecessary?

> > My normal way to recover from this situation is to btrfs add a new
> > device, btrfs balance, btrfs --convert=single --force balance, btfs
> > device remove, btr balance start -mconvert=dup --force and finally
> > balance start again.
> > 
> > Is there any solution to solve this more elegantly?
> 
>    Recreate the FS with --mixed, and that should deal with it.

Done. Thanks!

Greetings
Marc


-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: "disk full" on a 5 GB btrfs filesystem, FAQ outdated?

[Duncan]

Marc Haber posted on Sun, 29 Nov 2015 19:06:29 +0100 as excerpted:

> Hi Hugo,
> 
> On Sun, Nov 29, 2015 at 02:18:06PM +0000, Hugo Mills wrote:
>> On Sun, Nov 29, 2015 at 02:07:54PM +0100, Marc Haber wrote:

>>> However, the FAQ
>>> https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_Btrfs_claims_I.27m_out_of_space.2C_but_it_looks_like_I_should_have_lots_left.21
>>> suggests that for small filesystems (<16 GB), the best solution [...]
>> 
>> You missed the most important thing from that paragraph: Use mixed
>> block groups. That's "mkfs.btrfs --mixed ..." [...]
> 
> Yes, that was the important bit of missing information. My filesystem
> now reads:
> 
> $ df -h .
> Filesystem               Size  Used Avail Use% Mounted on
> /dev/mapper/banana-root  6,0G  836M  5,2G  14% /mnt/tempdisk
> [27/513]mh@fan:/mnt/tempdisk$ sudo btrfs fi show .
> Label: none  uuid: b2906231-70a9-46d9-9830-38a13cb73171
>     Total devices 1 FS bytes used 861.29MiB
>     devid  1 size 6.00GiB used 6.00GiB path /dev/mapper/banana-root
> 
> btrfs-progs v4.3
> $ sudo btrfs fi df .
> System, single: total=4.00MiB, used=4.00KiB
> Data+Metadata, single: total=6.00GiB, used=861.29MiB
> GlobalReserve, single: total=20.00MiB, used=0.00B
> $
> 
> Can I somehow get duplicate metadata back? Or is that unnecessary?

Define "necessary"...

Yes, you can get dup metadata back, but because data and metadata
are now combined in the same blockgroups (aka chunks), they must
both be the same replication type.

So with --mixed, if you go dup, you get dup data and metadata both,
which means halving the data you can actually store, since it's
stored twice.

Which means... 6 GiB btrfs... less than 3 GiB data storage if you go
mixed dup mode.  If say 2.5 GiB (you'll probably get a bit more but...)
is enough room to do what you need to do with it, I'd say go for
dup, as it /should/ be incrementally more reliable for you.  Obviously,
if it's not going to be enough room, you better stay single data,
which means either staying mixed and single metadata as well, or
switching back to unmixed so you can do dup metadata and still do
single data.

Dup, when it works /as/ dup, gives you two copies.  That's useful
if one copy goes bad while the other is still good, but it's
obviously not as reliable as raid1, two copies, each on different
devices, would be.  Meanwhile, various ssds do firmware level
deduplication and turn your two copies back into only one on the
physical device.  That shouldn't waste space as the device
shouldn't actually take space for deduped copies, but it will
then uselessly spend time processing that second copy, to no
benefit since the firmware second guessed you and deduped, and
if that now one copy goes bad, it's bad no matter which of the
two filesystem level pointers to it you try to read.

So dup /should/ be incrementally more reliable, /provided/ you're
actually getting dup, without the firmware or some other level
second-guessing you.

But tho I'm not personally familiar with the banana-pie, I'd guess
that the storage is emmc or the like, and likely doesn't have
firmware with that level of sophistication, so you're likely
to still have dup.

Meanwhile, in theory you could use the compress-force=zlib
mount option to squeeze a bit more space out of that dup,
letting you store somewhat more than 3 GiB of data, depending
on how zlib-compressible it is.  But I'd guess the CPU does
better on cost and performance per watt than on raw performance,
and compression, particularly zlib, is likely to slow things
down sufficiently badly that it's not going to be tolerable.
Still, you could try it, and if you're close on the data size
and performance does end up being tolerable, that just might
let you do dup, where you couldn't, otherwise.

Of course you can also try compress-force(=lzo the default
compression so the =spec isn't required), which should give
you slightly better performance than zlib, but also a bit
less efficient compression in terms of size saved.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: "disk full" on a 5 GB btrfs filesystem, FAQ outdated?

[Imran Geriskovan]

On 11/30/15, Duncan <1i5t5.duncan@cox.net> wrote:
> Of course you can also try compress-force(=lzo the default
> compression so the =spec isn't required), which should give
> you slightly better performance than zlib, but also a bit
> less efficient compression in terms of size saved.

lzo perf relative to zlib (A very very rough comparison)

Compress/Decompress times: 1/3 - 1/2 (which is significant)
Output Size: %10-20 larger

Re: "disk full" on a 5 GB btrfs filesystem, FAQ outdated?

[Marc Haber]

On Mon, Nov 30, 2015 at 05:44:23AM +0000, Duncan wrote:
> Yes, you can get dup metadata back, but because data and metadata
> are now combined in the same blockgroups (aka chunks), they must
> both be the same replication type.

Thanks for this explanation, it's perfectly clear to me now.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421