filesystem full when it's not? out of inodes? huh?

filesystem full when it's not? out of inodes? huh?

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 2435 bytes --]

I have a 5G /usr btrfs filesystem on a 3.0.0-12-generic kernel that is
returning ENOSPC when it's only 75% full:

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rootvol-mint_usr
                      5.0G  2.8G  967M  75% /usr

And yet I can't even unpack a linux-headers package on to it, which
should be nowhere near 967MB.  dpkg says it will need 10MB:

$ sudo apt-get install -f
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following extra packages will be installed:
  linux-headers-3.0.0-16-generic
The following NEW packages will be installed:
  linux-headers-3.0.0-16-generic
0 upgraded, 1 newly installed, 0 to remove and 2 not upgraded.
264 not fully installed or removed.
Need to get 0 B/851 kB of archives.
After this operation, 10.8 MB of additional disk space will be used.
Do you want to continue [Y/n]? y
(Reading database ... 180246 files and directories currently installed.)
Unpacking linux-headers-3.0.0-16-generic (from .../linux-headers-3.0.0-16-generic_3.0.0-16.28_i386.deb) ...
dpkg: error processing /var/cache/apt/archives/linux-headers-3.0.0-16-generic_3.0.0-16.28_i386.deb (--unpack):
 unable to install new version of `/usr/src/linux-headers-3.0.0-16-generic/include/config/dvb/tuner/dib0070.h': No space left on device

And indeed, using dd I am able to create a 967MB file:

$ sudo dd if=/dev/zero of=/usr/bigfile bs=1M count=10000000
dd: writing `/usr/bigfile': No space left on device
967+0 records in
966+0 records out
1012924416 bytes (1.0 GB) copied, 16.1545 s, 62.7 MB/s

strace yields this as the cause of the ENOSPC:

8213  rename("/usr/src/linux-headers-3.0.0-16-generic/include/config/dvb/tuner/dib0070.h.dpkg-new", "/usr/src/linux-headers-3.0.0-16-generic/include/config/dvb/tuner/dib0070.h" <unfinished ...>
...
8213  <... rename resumed> )            = -1 ENOSPC (No space left on device)

So this starts to feel like some kind of inode count limitation.  But I
didn't think btrfs had inode count limitations.  Here's the df stats on
the filesystem:

$ btrfs filesystem df /usr
Data: total=3.22GB, used=3.22GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=896.00MB, used=251.62MB
Metadata: total=8.00MB, used=0.00

I don't know if that's useful or not.

Any ideas?

Cheers
b.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: filesystem full when it's not? out of inodes? huh?

[Fahrzin Hemmati]

btrfs is horrible for small filesystems (like a 5GB drive). df -h says 
you have 967MB available, but btrfs (at least by default) allocates 1GB 
at a time to data/metadata. This means that your 10MB file is too big 
for the current allocation and requires a new data chunk, or another 
1GB, which you don't have.

Others might know of a way of changing the allocation size to less than 
1GB, but otherwise I recommend switching to something more stable like 
ext4/reiserfs/etc.

On 2/25/2012 5:55 PM, Brian J. Murrell wrote:
> I have a 5G /usr btrfs filesystem on a 3.0.0-12-generic kernel that is
> returning ENOSPC when it's only 75% full:
>
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/mapper/rootvol-mint_usr
>                        5.0G  2.8G  967M  75% /usr
>
> And yet I can't even unpack a linux-headers package on to it, which
> should be nowhere near 967MB.  dpkg says it will need 10MB:
>
> $ sudo apt-get install -f
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Correcting dependencies... Done
> The following extra packages will be installed:
>    linux-headers-3.0.0-16-generic
> The following NEW packages will be installed:
>    linux-headers-3.0.0-16-generic
> 0 upgraded, 1 newly installed, 0 to remove and 2 not upgraded.
> 264 not fully installed or removed.
> Need to get 0 B/851 kB of archives.
> After this operation, 10.8 MB of additional disk space will be used.
> Do you want to continue [Y/n]? y
> (Reading database ... 180246 files and directories currently installed.)
> Unpacking linux-headers-3.0.0-16-generic (from .../linux-headers-3.0.0-16-generic_3.0.0-16.28_i386.deb) ...
> dpkg: error processing /var/cache/apt/archives/linux-headers-3.0.0-16-generic_3.0.0-16.28_i386.deb (--unpack):
>   unable to install new version of `/usr/src/linux-headers-3.0.0-16-generic/include/config/dvb/tuner/dib0070.h': No space left on device
>
> And indeed, using dd I am able to create a 967MB file:
>
> $ sudo dd if=/dev/zero of=/usr/bigfile bs=1M count=10000000
> dd: writing `/usr/bigfile': No space left on device
> 967+0 records in
> 966+0 records out
> 1012924416 bytes (1.0 GB) copied, 16.1545 s, 62.7 MB/s
>
> strace yields this as the cause of the ENOSPC:
>
> 8213  rename("/usr/src/linux-headers-3.0.0-16-generic/include/config/dvb/tuner/dib0070.h.dpkg-new", "/usr/src/linux-headers-3.0.0-16-generic/include/config/dvb/tuner/dib0070.h"<unfinished ...>
> ...
> 8213<... rename resumed>  )            = -1 ENOSPC (No space left on device)
>
> So this starts to feel like some kind of inode count limitation.  But I
> didn't think btrfs had inode count limitations.  Here's the df stats on
> the filesystem:
>
> $ btrfs filesystem df /usr
> Data: total=3.22GB, used=3.22GB
> System, DUP: total=8.00MB, used=4.00KB
> System: total=4.00MB, used=0.00
> Metadata, DUP: total=896.00MB, used=251.62MB
> Metadata: total=8.00MB, used=0.00
>
> I don't know if that's useful or not.
>
> Any ideas?
>
> Cheers
> b.
>

Re: filesystem full when it's not? out of inodes? huh?

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 872 bytes --]

On 12-02-25 09:10 PM, Fahrzin Hemmati wrote:
> btrfs is horrible for small filesystems (like a 5GB drive). df -h says
> you have 967MB available,

So does dd.  It manages to write that 967MB before getting an ENOSPC.

> but btrfs (at least by default) allocates 1GB
> at a time to data/metadata.

Bleah.

> This means that your 10MB file

It's a package of kernel headers so not a 10MB file but 10MB of lots of
small files, so heavy metadata allocation.

> is too big
> for the current allocation and requires a new data chunk, or another
> 1GB, which you don't have.

I see.

> Others might know of a way of changing the allocation size to less than
> 1GB, but otherwise I recommend switching to something more stable like
> ext4/reiserfs/etc.

So btrfs is still not yet suitable to be a root/usr/var filesystem, even
in kernel 3.0.0?

b.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: filesystem full when it's not? out of inodes? huh?

[Fahrzin Hemmati]

On 2/25/2012 6:16 PM, Brian J. Murrell wrote:
>> Others might know of a way of changing the allocation size to less than
>> 1GB, but otherwise I recommend switching to something more stable like
>> ext4/reiserfs/etc.
> So btrfs is still not yet suitable to be a root/usr/var filesystem, even
> in kernel 3.0.0?
>
> b.
>
Nope, still in heavy development, though you should upgrade to 3.2. 
Also, the devs mentioned in several places it's not friendly to small 
drives, and I'm pretty sure 5GB is considered tiny.

I don't think you need to separate /usr out to it's own disk. You could 
instead create a single drive with multiple subvolumes for /, /var, 
/usr, etc. When you have Ubuntu use btrfs for /, it creates @ and @home 
for / and /home, respectively, so it's a common phenomenon if you look 
for help.

--Farz

Re: filesystem full when it's not? out of inodes? huh?

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 1395 bytes --]

On 12-02-25 09:37 PM, Fahrzin Hemmati wrote:
>
> Nope, still in heavy development, though you should upgrade to 3.2.

I recall being told I should upgrade to 2.6.36 (or was it .37 or .38) at
one time.  Seems like one should always upgrade.  :-/

> Also, the devs mentioned in several places it's not friendly to small
> drives, and I'm pretty sure 5GB is considered tiny.

But it won't ever get taken serious if it can't be used on "regular"
filesystems.  I shouldn't have to allocate an 80G filesystem for 3G of
data just so that the filesystem isn't "tiny".

> I don't think you need to separate /usr out to it's own disk. You could
> instead create a single drive with multiple subvolumes for /, /var,
> /usr, etc.

The point is to separate filesystems which can easily fill with
application data growth from filesystems that can have more fatal
effects by being filled.

That said, I don't think having /var as a subvolume in the same pool as
/ and /usr achieves that usage isolation, does it?  Isn't /var still
allowed to consume all of the space that it, / and /usr share with them
all being subvolumes in the same pool?

> When you have Ubuntu use btrfs for /, it creates @ and @home
> for / and /home, respectively,

Yes, I had noticed that.  I also didn't immediately see anything that
prevents /home from filling / as I describe above.

Cheers,
b.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: filesystem full when it's not? out of inodes? huh?

[Fahrzin Hemmati]

On 2/25/2012 7:57 PM, Brian J. Murrell wrote:
> On 12-02-25 09:37 PM, Fahrzin Hemmati wrote:
>> Nope, still in heavy development, though you should upgrade to 3.2.
> I recall being told I should upgrade to 2.6.36 (or was it .37 or .38) at
> one time.  Seems like one should always upgrade.  :-/
It's a new, in-development filesystem. Until they say "It's stable, have 
fun", you should upgrade. If you follow any other filesystem before 
they've marked it stable and you'll get the same responses.
>> Also, the devs mentioned in several places it's not friendly to small
>> drives, and I'm pretty sure 5GB is considered tiny.
> But it won't ever get taken serious if it can't be used on "regular"
> filesystems.  I shouldn't have to allocate an 80G filesystem for 3G of
> data just so that the filesystem isn't "tiny".
All filesystems have their own pros and cons; btrfs, at least while 
in-development doesn't support small filesystems. Again, nobody's 
responded to the contrary, but there may be a way to changing the 
default allocation size to less than 1GB, making your use case viable. I 
recommend Google.
>> I don't think you need to separate /usr out to it's own disk. You could
>> instead create a single drive with multiple subvolumes for /, /var,
>> /usr, etc.
> The point is to separate filesystems which can easily fill with
> application data growth from filesystems that can have more fatal
> effects by being filled.
>
> That said, I don't think having /var as a subvolume in the same pool as
> / and /usr achieves that usage isolation, does it?  Isn't /var still
> allowed to consume all of the space that it, / and /usr share with them
> all being subvolumes in the same pool?
>
>> When you have Ubuntu use btrfs for /, it creates @ and @home
>> for / and /home, respectively,
> Yes, I had noticed that.  I also didn't immediately see anything that
> prevents /home from filling / as I describe above.
>
> Cheers,
> b.
>

No, at least not yet, nor am I aware of any plans for subvolume quotas, 
though I could be wrong. If you wish to use a small space for your /usr, 
you can either wait for btrfs to support your use-case (or find out how 
to change allocation size), or use another filesystem that already does.

--Farz

Re: filesystem full when it's not? out of inodes? huh?

[Johannes Hirte]

Am Sat, 25 Feb 2012 20:05:13 -0800
schrieb Fahrzin Hemmati <fahhem2@gmail.com>:

> No, at least not yet, nor am I aware of any plans for subvolume
> quotas, though I could be wrong. 

Arne Jansen is working on it, IIRC.

regards,
  Johannes

Re: filesystem full when it's not? out of inodes? huh?

[Duncan]

Fahrzin Hemmati posted on Sat, 25 Feb 2012 18:37:24 -0800 as excerpted:

> On 2/25/2012 6:16 PM, Brian J. Murrell wrote:
>>> Others might know of a way of changing the allocation size to less
>>> than 1GB, but otherwise I recommend switching to something more stable
>>> like ext4/reiserfs/etc.

>> So btrfs is still not yet suitable to be a root/usr/var filesystem,
>> even in kernel 3.0.0?
>>
> Nope, still in heavy development, though you should upgrade to 3.2.
> Also, the devs mentioned in several places it's not friendly to small
> drives, and I'm pretty sure 5GB is considered tiny.
> 
> I don't think you need to separate /usr out to it's own disk. You could
> instead create a single drive with multiple subvolumes for /, /var,
> /usr, etc. When you have Ubuntu use btrfs for /, it creates @ and @home
> for / and /home, respectively, so it's a common phenomenon if you look
> for help.

It's astonishing to me the number of people that come in here complaining 
about problems with a filesystem the kernel option of which says 

Title:

Btrfs filesystem (EXPERIMENTAL) Unstable disk format

Description (excerpt):

Btrfs is highly experimental, and THE DISK FORMAT IS NOT YET
FINALIZED.  You should say N here unless you are interested in
testing Btrfs with non-critical data.


So, does testing with non-critical data sound like it's appropriate for 
running on your (OP's) root or home (or anything else you might be using 
it for) filesystem, the way you're doing it now?  If not, what are you 
doing on btrfs?  You better either change your choice of filesystem, or 
your update and backup strategy, to be more in line with the testing 
that's an appropriate use of btrfs at this point.

About twice a week we get people on the list asking about recovery tools 
because the filesystem won't mount, too, and they apparently had no 
backups.  WTF?  Not only should they have had backups (that's normal with 
every filesystem, you always have backups if the data's valuable to you, 
and always test them), but btrfs isn't appropriate for anything besides 
testing data that they *EXPECT* the file system to chew up and spit out 
at them!  IOW, not only should there be backups, but the btrfs copy 
should be considered the non-primary copy, essentially garbage, because 
there's a very real chance that's what it'll be if something goes wrong!

Of course, anyone who *IS* using it as intended at this point will be 
following the lists, know what bugs are being seen frequently, know the 
status of the various recovery tools, etc.  They won't HAVE to ask, as 
they'll already KNOW if they're using the filesystem as intended, for 
testing, reporting bugs, and possibly submitting patches if they have the 
skills, in coordination with the devs and other testers on the list, at 
this point.

Meanwhile, if you DO decide to continue with btrfs for testing...

There's a wiki covering the ENOSPC problem, discussing kernel and tools 
status (keep upto date, the kernel especially is in HEAVY development, 
and even running the latest stable kernel, means you're already missing 
the bug-fixes and better stability in the current development kernel!), 
etc.

Actually, there's two wikis, the kernel.org wiki which is static content 
(not updated and thus not upto date) ATM due to the kernel.org breakin a 
few months ago, and a "temporary" one that's rather more current.

Here's the link to both:

Kernel.org wiki (outdated static ATM):
https://btrfs.wiki.kernel.org/

Temporary upto-date wiki:
http://btrfs.ipv5.de/index.php?title=Main_Page

There, the second paragraph of the main page says:

Btrfs is under heavy development, but every effort is being made to keep 
the filesystem stable and fast. Because of the speed of development, you 
should run the latest kernel you can (either the latest release kernel 
from kernel.org, or the latest -rc kernel.

So YES, a 3.0 kernel is *OLD*!  Even the latest stable 3.2.x kernel won't 
have the latest btrfs fixes.  For that, you need the latest 3.3-rc if you 
aren't running git kernels and updating between the rcs!

The FAQ page there has a whole section titled "Help! I ran out of disk 
space!"  In particular, see the "If your device is small" subsection.

http://btrfs.ipv5.de/index.php?title=FAQ

Also, down further in the FAQ, see

Why does df show incorrect free space for my RAID volume?
Aaargh! My filesystem is full, and I've put almost nothing into it!
Why are there so many ways to check the amount of free space?

(That's one section covering the three.)  Plus the answers to these two 
as well:

Why is free space so complicated?

Why is there so much space overhead?

Meanwhile, what you SHOULD have read before even STARTING with btrfs, the 
getting started page:

http://btrfs.ipv5.de/index.php?title=Getting_started

Right at the top:
[begin quote]

btrfs is a fast-moving target. There are typically a great many bug fixes 
and enhancements between one kernel release and the next. Therefore:

* If you have btrfs filesystems, run the latest kernel. *

If you are running a kernel two or more versions behind the latest one 
available from kernel.org, the first thing you will be asked to when you 
report a problem is to upgrade to the latest kernel. Some distributions 
keep backports of recent kernels to earlier releases -- see the page 
below for details.

Having the latest user-space tools are also useful, as they contain 
additional features and tools which may be of use in debugging or 
recovering your filesystem if something goes wrong.

Note also that btrfs is still considered experimental. While many people 
use it reliably, there are still problems being found.

* You should keep and test backups of your data, and be prepared to use 
them. *

[end quote]

So YES, UPGRADE.  If you're not interested in keeping up with the latest 
kernels, why are you running a testing filesystem?

And here again, from the source code page:

http://btrfs.ipv5.de/index.php?title=Btrfs_source_repositories

[begin quote]
Warning, Btrfs evolves very quickly do not test it unless:

1. You have good backups and you have tested the restore capability
2. You have a backup installation that you can switch to when
       something breaks
3. You are willing to report any issues you find
4. You can apply patches and compile the latest btrfs code against
       your kernel (quite easy with git and dkms, see below)
5. You acknowledge that btrfs may eat your data
6. Backups! Backups! Backups! 

[end quote]


Meanwhile, new enough it's not on the wiki that I saw, and it wasn't in 
3.0 I don't believe but 3.2 has it, and it'll require a current btrfs-
progs (the git version, updated regularly, is best), but mkbtrfs aka 
mkfs.btrfs now has the --mixed/-M option, with this description in the 
mkfs.btrfs (8) manpage:

-M, --mixed
  Mix data and metadata chunks together for more efficient space
  utilization.   This  feature incurs  a  performance  penalty
  in  larger  filesystems.   It  is recommended for use with
  filesystems of 1 GiB or smaller.

You could try that on 5-gig filesystems as well.

Finally, the previous smallest allocation block was 1/4 GiB (256 MiB).  
Combine that with the fact that until recently each such block could be 
set to date OR metadata, NOT mixed (fixed at mkfs time with the option 
above), and btrfs really was NOT suited to small filesystems.

Very new, so new I know it's in 3.3 but am not sure it's in 3.2, that 256 
MiB lower end limitation is now gone, and btrfs can make use of "the 
small change" space it previously couldn't do anything with.

Combine those two very new changes together, and it's actually semi-
practical to use btrfs for smaller filesystems now.  But they're both 
quite new, I believe both newer than your actually relatively ancient (in 
btrfs terms) kernel 3.0.  It's exactly that sort of thing that's behind 
the emphasis on keeping up, as you'll be doing anyway if you're using the 
filesystem as intended at this point, for rapid-turn-around testing, 
reporting problems, and getting fixes with each kernel version and often 
in between (with 3.3, there was the merge-window changes before rc1, and 
just a few days ago a whole host of fixes submitted, that if Linus pulled 
them, they'd be in rc4).

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: filesystem full when it's not? out of inodes? huh?

[Helmut Hullen]

Hallo, Duncan,

Du meintest am 26.02.12:

> It's astonishing to me the number of people that come in here
> complaining about problems with a filesystem the kernel option of
> which says

> Title:

> Btrfs filesystem (EXPERIMENTAL) Unstable disk format

> Description (excerpt):

> Btrfs is highly experimental, and THE DISK FORMAT IS NOT YET
> FINALIZED.  You should say N here unless you are interested in
> testing Btrfs with non-critical data.

Just take a look at Fedora.
The maintainers had planned to use btrfs as standard filesystem for  
Fedora 16 (but haven't done so), they had planned to use btrfs for  
Fedora 17, but perhaps hesitate, see

  https://fedorahosted.org/fesco/ticket/704

There are some other distributions which also seem to (perhaps) follow  
the "bleading edge".

And therefore end users believe that using btrfs is safe.
(I've learned my lesson ...)

Just for the record: using btrfs (when it runs stable) may reduce many  
other problems on my system. I'm still hoping.

Viele Gruesse!
Helmut

Re: filesystem full when it's not? out of inodes? huh?

[Duncan]

Helmut Hullen posted on Sun, 26 Feb 2012 10:10:00 +0100 as excerpted:

> Just take a look at Fedora.
> The maintainers had planned to use btrfs as standard filesystem for
> Fedora 16 (but haven't done so), they had planned to use btrfs for
> Fedora 17, but perhaps hesitate, see
> 
>   https://fedorahosted.org/fesco/ticket/704
> 
> There are some other distributions which also seem to (perhaps) follow
> the "bleading edge".
> 
> And therefore end users believe that using btrfs is safe.
> (I've learned my lesson ...)
> 
> Just for the record: using btrfs (when it runs stable) may reduce many
> other problems on my system. I'm still hoping.

FWIW, until a few weeks ago I had the idea based on the general Linux 
community press that btrfs was safe and basically ready, as well.  
Luckily I research such things as filesystems a bit closer than general 
community press, before I jump, tho.  I ended up at the kernel wiki, and 
read a couple weeks worth of this list before starting to respond to 
posters, mostly quoting the wiki, and posting my own thread, asking some 
questions the wiki didn't deal with.

It turns out that the raid1 features I was most interested in, at this 
point, aren't true raid1 (that is, N-way mirror, with N corresponding to 
the number of disks/SSDs), but rather, two-way-mirror only.  Either 3-way 
or N-way mirror (it was covered as 3-way, but I'm not sure if that's 3+ 
because 2-way is covered already, or 3-way only, limiting it to 2-3-way) 
is on the roadmap to be introduced after (and building on) raid5/6, which 
is on the roadmap for kernel 3.4/3.5, so that'd put 3-way-raid1 at 
3.5/3.6, probably.  So I've not even bothered trying it yet (I've 
installed btrfs-progs and have the kernel option enabled, but haven't 
actually setup any btrfs filesystems) pending that feature introduction.  
But now that I'm here on-list, I'm sticking around to track progress, and 
to help with replies, mostly quoting wiki stuff, the manpages, etc, where 
I can.

So I'm (im)patiently waiting, but it's not too bad, because already in 
the few weeks I've been on-list, I've already seen a decent number of 
fixes including the ones for small-filesystems, so I'm watching the 
filesystem gain stability and eliminate bugs "live", and I know that by 
the time multi-way raid1 is available, the filesystem will only be more 
stable and less buggy than it is now, and that's a good thing! =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: filesystem full when it's not? out of inodes? huh?

[Chris Samuel]

[-- Attachment #1: Type: Text/Plain, Size: 640 bytes --]

On Sunday 26 February 2012 19:52:06 Duncan wrote:

> It's astonishing to me the number of people that come in here
> complaining  about problems with a filesystem the kernel option of
> which says
> 
> Title:
> 
> Btrfs filesystem (EXPERIMENTAL) Unstable disk format

On the other hand, if they didn't keep complaining then perhaps that 
label would go away prematurely because nobody was reporting problems? 

:-)

cheers!
Chris
-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC

This email may come with a PGP signature as a file. Do not panic.
For more info see: http://en.wikipedia.org/wiki/OpenPGP

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 482 bytes --]

Re: filesystem full when it's not? out of inodes? huh?

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 4469 bytes --]

On 12-02-25 09:10 PM, Fahrzin Hemmati wrote:
> btrfs is horrible for small filesystems (like a 5GB drive). df -h says
> you have 967MB available, but btrfs (at least by default) allocates 1GB
> at a time to data/metadata. This means that your 10MB file is too big
> for the current allocation and requires a new data chunk, or another
> 1GB, which you don't have.

So increasing the size of the filesystem should suffice then?  How much
bigger?  10G?  Nope.  still not big enough:

# lvextend -L+1G /dev/rootvol/mint_usr; btrfs fi resize max /usr; df -h /usr
  Extending logical volume mint_usr to 10.00 GiB
  Logical volume mint_usr successfully resized
Resize '/usr' of 'max'
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rootvol-mint_usr
                       10G  2.8G  6.0G  32% /usr
test ~ # apt-get install -y -f
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following extra packages will be installed:
  linux-headers-3.0.0-16-generic
The following NEW packages will be installed:
  linux-headers-3.0.0-16-generic
0 upgraded, 1 newly installed, 0 to remove and 2 not upgraded.
264 not fully installed or removed.
Need to get 0 B/851 kB of archives.
After this operation, 10.8 MB of additional disk space will be used.
(Reading database ... 180246 files and directories currently installed.)
Unpacking linux-headers-3.0.0-16-generic (from .../linux-headers-3.0.0-16-generic_3.0.0-16.28_i386.deb) ...
dpkg: error processing /var/cache/apt/archives/linux-headers-3.0.0-16-generic_3.0.0-16.28_i386.deb (--unpack):
 unable to install new version of `/usr/src/linux-headers-3.0.0-16-generic/include/config/dvb/usb.h': No space left on device

20G maybe?  Nope:

# lvextend -L20G /dev/rootvol/mint_usr; btrfs fi resize max /usr; df -h /usr
  Extending logical volume mint_usr to 20.00 GiB
  Logical volume mint_usr successfully resized
Resize '/usr' of 'max'
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rootvol-mint_usr
                       20G  2.8G   16G  15% /usr
test ~ # apt-get install -y -f
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following extra packages will be installed:
  linux-headers-3.0.0-16-generic
The following NEW packages will be installed:
  linux-headers-3.0.0-16-generic
0 upgraded, 1 newly installed, 0 to remove and 2 not upgraded.
264 not fully installed or removed.
Need to get 0 B/851 kB of archives.
After this operation, 10.8 MB of additional disk space will be used.
(Reading database ... 180246 files and directories currently installed.)
Unpacking linux-headers-3.0.0-16-generic (from .../linux-headers-3.0.0-16-generic_3.0.0-16.28_i386.deb) ...
dpkg: error processing /var/cache/apt/archives/linux-headers-3.0.0-16-generic_3.0.0-16.28_i386.deb (--unpack):
 unable to install new version of `/usr/src/linux-headers-3.0.0-16-generic/include/config/ncpfs/packet/signing.h': No space left on device

Maybe 50G?  Yup:

# apt-get install -y -f
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following extra packages will be installed:
  linux-headers-3.0.0-16-generic
The following NEW packages will be installed:
  linux-headers-3.0.0-16-generic
0 upgraded, 1 newly installed, 0 to remove and 2 not upgraded.
264 not fully installed or removed.
Need to get 0 B/851 kB of archives.
After this operation, 10.8 MB of additional disk space will be used.
(Reading database ... 180246 files and directories currently installed.)
Unpacking linux-headers-3.0.0-16-generic (from .../linux-headers-3.0.0-16-generic_3.0.0-16.28_i386.deb) ...
Setting up linux-image-3.0.0-16-generic (3.0.0-16.28) ...
...
# df -h /usr
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rootvol-mint_usr
                       50G  2.8G   43G   7% /usr

So I guess I need a 50G btrfs filesystem for 2.8G worth of data?

Does that really seem right?  I suppose to be fair it could have been
some other value between 20G and 50G since I didn't test values in
between.  So still, I need some amount more than 20G of space to store
2.8G of data?

Surely there is something going on here other than just btrfs sucks for
small filesystems.

b.





[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: filesystem full when it's not? out of inodes? huh?

[Fahrzin Hemmati]

On 2/25/2012 9:45 PM, Brian J. Murrell wrote:
> On 12-02-25 09:10 PM, Fahrzin Hemmati wrote:
>> btrfs is horrible for small filesystems (like a 5GB drive). df -h says
>> you have 967MB available, but btrfs (at least by default) allocates 1GB
>> at a time to data/metadata. This means that your 10MB file is too big
>> for the current allocation and requires a new data chunk, or another
>> 1GB, which you don't have.
> So increasing the size of the filesystem should suffice then?  How much
> bigger?  10G?  Nope.  still not big enough:
>
> # lvextend -L+1G /dev/rootvol/mint_usr; btrfs fi resize max /usr; df -h /usr
>    Extending logical volume mint_usr to 10.00 GiB
>    Logical volume mint_usr successfully resized
> Resize '/usr' of 'max'
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/mapper/rootvol-mint_usr
>                         10G  2.8G  6.0G  32% /usr
> test ~ # apt-get install -y -f
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Correcting dependencies... Done
> The following extra packages will be installed:
>    linux-headers-3.0.0-16-generic
> The following NEW packages will be installed:
>    linux-headers-3.0.0-16-generic
> 0 upgraded, 1 newly installed, 0 to remove and 2 not upgraded.
> 264 not fully installed or removed.
> Need to get 0 B/851 kB of archives.
> After this operation, 10.8 MB of additional disk space will be used.
> (Reading database ... 180246 files and directories currently installed.)
> Unpacking linux-headers-3.0.0-16-generic (from .../linux-headers-3.0.0-16-generic_3.0.0-16.28_i386.deb) ...
> dpkg: error processing /var/cache/apt/archives/linux-headers-3.0.0-16-generic_3.0.0-16.28_i386.deb (--unpack):
>   unable to install new version of `/usr/src/linux-headers-3.0.0-16-generic/include/config/dvb/usb.h': No space left on device
>
> 20G maybe?  Nope:
>
> # lvextend -L20G /dev/rootvol/mint_usr; btrfs fi resize max /usr; df -h /usr
>    Extending logical volume mint_usr to 20.00 GiB
>    Logical volume mint_usr successfully resized
> Resize '/usr' of 'max'
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/mapper/rootvol-mint_usr
>                         20G  2.8G   16G  15% /usr
> test ~ # apt-get install -y -f
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Correcting dependencies... Done
> The following extra packages will be installed:
>    linux-headers-3.0.0-16-generic
> The following NEW packages will be installed:
>    linux-headers-3.0.0-16-generic
> 0 upgraded, 1 newly installed, 0 to remove and 2 not upgraded.
> 264 not fully installed or removed.
> Need to get 0 B/851 kB of archives.
> After this operation, 10.8 MB of additional disk space will be used.
> (Reading database ... 180246 files and directories currently installed.)
> Unpacking linux-headers-3.0.0-16-generic (from .../linux-headers-3.0.0-16-generic_3.0.0-16.28_i386.deb) ...
> dpkg: error processing /var/cache/apt/archives/linux-headers-3.0.0-16-generic_3.0.0-16.28_i386.deb (--unpack):
>   unable to install new version of `/usr/src/linux-headers-3.0.0-16-generic/include/config/ncpfs/packet/signing.h': No space left on device
>
> Maybe 50G?  Yup:
>
> # apt-get install -y -f
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Correcting dependencies... Done
> The following extra packages will be installed:
>    linux-headers-3.0.0-16-generic
> The following NEW packages will be installed:
>    linux-headers-3.0.0-16-generic
> 0 upgraded, 1 newly installed, 0 to remove and 2 not upgraded.
> 264 not fully installed or removed.
> Need to get 0 B/851 kB of archives.
> After this operation, 10.8 MB of additional disk space will be used.
> (Reading database ... 180246 files and directories currently installed.)
> Unpacking linux-headers-3.0.0-16-generic (from .../linux-headers-3.0.0-16-generic_3.0.0-16.28_i386.deb) ...
> Setting up linux-image-3.0.0-16-generic (3.0.0-16.28) ...
> ...
> # df -h /usr
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/mapper/rootvol-mint_usr
>                         50G  2.8G   43G   7% /usr
>
> So I guess I need a 50G btrfs filesystem for 2.8G worth of data?
>
> Does that really seem right?  I suppose to be fair it could have been
> some other value between 20G and 50G since I didn't test values in
> between.  So still, I need some amount more than 20G of space to store
> 2.8G of data?
>
> Surely there is something going on here other than just btrfs sucks for
> small filesystems.
>
> b.
>
>
>
>
You should have been fine with adding 1GB (really only 57MB), or at 
worst 2GB in case you were on the edge of both data and metadata.

A btrfs dev might be able to debug the problem there, since your 
original problem seemed only that you couldn't allocate a new chunk. It 
might be a problem with btrfs filesystem resize?

Re: filesystem full when it's not? out of inodes? huh?

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 586 bytes --]

On 12-02-26 12:45 AM, Brian J. Murrell wrote:
> 
> So I guess I need a 50G btrfs filesystem for 2.8G worth of data?

Interestingly enough, I was able to resize the filesystem back down to
5G after all of that:

# btrfs fi resize 5G /usr; df -h /usr
Resize '/usr' of '5G'
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rootvol-mint_usr
                      5.0G  2.8G  952M  75% /usr

Of course I have not done any integrity checking on the data in the
filesystem to be sure this resize was as successful as it's making out
to be.

Cheers,
b.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: filesystem full when it's not? out of inodes? huh?

[Jérôme Poulin]

On Sun, Feb 26, 2012 at 1:14 AM, Brian J. Murrell <brian@interlinx.bc.ca> wrote:
>
> # btrfs fi resize 5G /usr; df -h /usr
> Resize '/usr' of '5G'


What would be interesting is getting an eye on btrfs fi df of your
filesystem to see what part is getting full, or maybe just do a
balance.

I have been running 3.0.0 for quite a while without any problem,
metadata grew a bit too much (1.5 TB for 2 TB of data) and balance
fixed it back to 50 GB of metadata then 20 GB after deleting some
snapshots.

Re: filesystem full when it's not? out of inodes? huh?

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 842 bytes --]

On 12-02-26 02:19 AM, Jérôme Poulin wrote:
> 
> What would be interesting is getting an eye on btrfs fi df of your
> filesystem to see what part is getting full, or maybe just do a
> balance.

I did try a balance.  As I had mentioned subsequently, I ended up having
to grow the filesystem to 10x (somewhere between 20 and 50GB) it's data
requirement in order to get that kernel headers .deb to unpack, and
after it unpacked I was successful in shrinking back down to 5G, so it
seems the problem was something worse than just "not ideal" metadata
allocation.

> I have been running 3.0.0 for quite a while without any problem,
> metadata grew a bit too much (1.5 TB for 2 TB of data) and balance
> fixed it back to 50 GB of metadata then 20 GB after deleting some
> snapshots.

Interesting data point, thanks.

b.




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: filesystem full when it's not? out of inodes? huh?

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 3794 bytes --]

On Sat, Feb 25, 2012 at 06:10:32PM -0800, Fahrzin Hemmati wrote:
> btrfs is horrible for small filesystems (like a 5GB drive). df -h
> says you have 967MB available, but btrfs (at least by default)
> allocates 1GB at a time to data/metadata. This means that your 10MB
> file is too big for the current allocation and requires a new data
> chunk, or another 1GB, which you don't have.
> 
> Others might know of a way of changing the allocation size to less
> than 1GB, but otherwise I recommend switching to something more
> stable like ext4/reiserfs/etc.

   The option that nobody's mentioned yet is to use mixed mode. This
is the -M or --mixed option when you create the filesystem. It's
designed specifically for small filesystems, and removes the
data/metadata split for more efficient packing.

> On 2/25/2012 5:55 PM, Brian J. Murrell wrote:
> >I have a 5G /usr btrfs filesystem on a 3.0.0-12-generic kernel that is
> >returning ENOSPC when it's only 75% full:

   As mentioned before, you probably need to upgrade to 3.2 or 3.3-rc5
anyway. There were quite a few fixes in the ENOSPC/allocation area
since then.

> >Filesystem            Size  Used Avail Use% Mounted on
> >/dev/mapper/rootvol-mint_usr
> >                       5.0G  2.8G  967M  75% /usr
> >
> >And yet I can't even unpack a linux-headers package on to it, which
> >should be nowhere near 967MB.  dpkg says it will need 10MB:
> >
> >So this starts to feel like some kind of inode count limitation.  But I
> >didn't think btrfs had inode count limitations.  Here's the df stats on
> >the filesystem:

   It doesn't have inode limitations. It does, however, have some
peculiar limitations on the use of space. Specifically, the
copy-on-write nature has some implications.

   When you write *anything* to the FS, it does a CoW copy of
everything involved in the write. This includes all of the related
metadata, including the path from the B-tree leaves being touched up
to the root of the tree [in each B-tree being touched]. So, if you're
near the bounds of space in metadata, you can end up in a situation
where you modify a lot of metadata, and need a lot of space to do the
CoW in, so you try to allocate more metadata block groups -- which
requires metadata to be modified -- and run out of metadata space in
the allocation operation, which is treated as ENOSPC.

   That's not necessarily what's happened here, but it's highly
plausible. The FS does actually keep a small set of metadata reserved
to deal with this situation, but sometimes it's not very good at
planning how much metadata it needs before an operation is started.
It's that code that's had a lot of work since 3.0.

> >$ btrfs filesystem df /usr
> >Data: total=3.22GB, used=3.22GB
> >System, DUP: total=8.00MB, used=4.00KB
> >System: total=4.00MB, used=0.00
> >Metadata, DUP: total=896.00MB, used=251.62MB
> >Metadata: total=8.00MB, used=0.00
> >
> >I don't know if that's useful or not.

   Not to me directly -- there appears to be enough metadata to do
pretty much anything, so the above scenario _probably_ isn't the
problem, but it's clearly trying to allocate a new data block group
(which it should be able to do -- it should just take all the
remaining space, unlike Fahrzin's hypothesis).

   There have been some issues over having very large metadata
allocations that can't apparently be reused, though. It's possible
you've hit this one -- particularly if you're trying to untar
something, which performs lots and lots of writes all in one
transaction. Again, there's been some work on this since 3.0.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
          --- What part of "gestalt" don't you understand? ---           

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: filesystem full when it's not? out of inodes? huh?

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 1477 bytes --]

On 12-02-26 06:00 AM, Hugo Mills wrote:
> 
>    The option that nobody's mentioned yet is to use mixed mode. This
> is the -M or --mixed option when you create the filesystem. It's
> designed specifically for small filesystems, and removes the
> data/metadata split for more efficient packing.

Cool.

>    As mentioned before, you probably need to upgrade to 3.2 or 3.3-rc5
> anyway. There were quite a few fixes in the ENOSPC/allocation area
> since then.

I've upgraded to the Ubuntu Precise kernel which looks to be 3.2.6 with
btrfs-tools 0.19+20100601-3ubuntu3 so that would look like a btrfs-progs
snapshot from 2010-06-01 and (unsurprisingly) I don't see the -M option
in mkfs.btrfs.

So I went digging and I just wanted to verify what I think I am seeing.

Looking at

http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=commit;h=67377734fd24c32cbdfeb697c2e2bd7fed519e75

it would appear that the mixed data+metadata code landed in the kernel
back in Sep, of 2010, is that correct?

And looking at

http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-progs.git;a=commit;h=b8802ae3fa0c70d4cfc3287ed07479925973b0ac

the userspace support for this landed in Dec. of 2010, is that right?

If my archeology is correct, then I only need to update my btrfs-tools,
yes?  Is  2010-06-01 really the last time the tools were considered
stable or are Ubuntu just being conservative and/or lazy about updating?

Cheers,
b.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: filesystem full when it's not? out of inodes? huh?

[Fajar A. Nugraha]

On Fri, Mar 2, 2012 at 6:50 PM, Brian J. Murrell <brian@interlinx.bc.ca=
> wrote:
> Is =A02010-06-01 really the last time the tools were considered
> stable or are Ubuntu just being conservative and/or lazy about updati=
ng?

The last one :)

Or probably no one has bugged them enough and point out they're
already using a git snapshot anyway and there are many new features in
the "current" git version of btrfs-tools.

I have been compiling my own kernel (just recently switched to
Precise's kernel though) and btrfs-progs for quite some time, so even
if Ubuntu doesn't provide updated package it wouldn't matter much to
me. If it's important for you, you could file a bug report in
launchpad asking for an update. Even debian testing has an updated
version (which you might be able to use:
http://packages.debian.org/btrfs-tools)

Or create your own ppa with an updated version (or at least rebuilt of
Debian's version).

--=20
=46ajar
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: filesystem full when it's not? out of inodes? huh?

[Daniel Lee]

On 02/25/2012 05:55 PM, Brian J. Murrell wrote:
> $ btrfs filesystem df /usr
> Data: total=3.22GB, used=3.22GB
> System, DUP: total=8.00MB, used=4.00KB
> System: total=4.00MB, used=0.00
> Metadata, DUP: total=896.00MB, used=251.62MB
> Metadata: total=8.00MB, used=0.00
>
> I don't know if that's useful or not.
>
> Any ideas?
>
> Cheers
> b.
3.22GB + (896MB * 2) = 5GB

There's no mystery here, you're simply out of space. The system df 
command basically doesn't understand btrfs so will erroneously report 
free space if there isn't any.

Re: filesystem full when it's not? out of inodes? huh?

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 329 bytes --]

On 12-02-26 02:37 PM, Daniel Lee wrote:
> 3.22GB + (896MB * 2) = 5GB
> 
> There's no mystery here, you're simply out of space.

Except the mystery that I had to expand the filesystem to something
between 20GB and 50GB in order to complete the operation, after which I
could reduce it back down to 5GB.

Cheers,
b.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: filesystem full when it's not? out of inodes? huh?

[Daniel Lee]

On 02/26/2012 11:48 AM, Brian J. Murrell wrote:
> On 12-02-26 02:37 PM, Daniel Lee wrote:
>> 3.22GB + (896MB * 2) = 5GB
>>
>> There's no mystery here, you're simply out of space.
> Except the mystery that I had to expand the filesystem to something
> between 20GB and 50GB in order to complete the operation, after which I
> could reduce it back down to 5GB.
>
> Cheers,
> b.
>
What's mysterious about that? When you shrink it btrfs is going to throw 
away unused data to cram it all in the requested space and you had empty 
space that was taken up by the metadata allocation. Did you compare 
btrfs fi df after you shrank it with before?

Re: filesystem full when it's not? out of inodes? huh?

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 612 bytes --]

On 12-02-26 02:52 PM, Daniel Lee wrote:
> What's mysterious about that?

What's mysterious about needing to grow the filesystem to over 20GB to
unpack 10MB of (small, so yes, many) files?

> When you shrink it btrfs is going to throw
> away unused data to cram it all in the requested space and you had empty
> space that was taken up by the metadata allocation.

The shrinking is secondary mystery.  It's the need for more than 20GB of
space for less than 3GB of files that's the major mystery.

> Did you compare
> btrfs fi df after you shrank it with before?

I didn't unfortunately.

b.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: filesystem full when it's not? out of inodes? huh?

[Daniel Lee]

On 02/26/2012 12:05 PM, Brian J. Murrell wrote:
> On 12-02-26 02:52 PM, Daniel Lee wrote:
>> What's mysterious about that?
> What's mysterious about needing to grow the filesystem to over 20GB to
> unpack 10MB of (small, so yes, many) files?
>> When you shrink it btrfs is going to throw
>> away unused data to cram it all in the requested space and you had empty
>> space that was taken up by the metadata allocation.
> The shrinking is secondary mystery.  It's the need for more than 20GB of
> space for less than 3GB of files that's the major mystery.
Several people in this list have already answered this question but here
goes.

Btrfs isn't like other more common filesystems where metadata is fixed
at filesystem creation. Rather, metadata allocations happen just like
data allocations do. Btrfs also tries to allocate metadata in big chunks
so it doesn't get fragmented and lead to slowdowns when doing something
like running du on the root folder. The downside to all of this is that
it's not very friendly to "small" filesystems, in your case it allocated
some 1.8 GB of metadata of which only 500 MB was actually in use.

In the future you can create your filesystem with metadata=single to
free up more space for regular data or look into forcing the mixed block
groups mode which is normally only enabled for 1GB or smaller
filesystems. Mixed block group mode can't be switched off so you could
make a really tiny FS, several hunder MB or so, and then just grow it to
whatever size you want. The btrfs wiki seems to define small filesystems
as anything under 16GB so might be a good lower bound for actually using
btrfs in a day to day environment.