ENOSPC after conversion [Was: Fixing Btrfs Filesystem Full Problems typo?]

ENOSPC after conversion [Was: Fixing Btrfs Filesystem Full Problems typo?]

[Patrik Lundquist]

I'll reboot the thread with a recap and my latest findings.

* Half full 3TB disk converted from ext4 to Btrfs, after first
verifying it with fsck.
* Undo subvolume deleted after being happy with the conversion.
* Recursive defrag.
* Full balance, that ended with "98 enospc errors during balance."

In that order, nothing in between. No snapshots or other subvolumes.
Loads of real free space.

Btrfs check reports a clean filesystem.

Btrfs balance -musage=100 -dusage=99 works, but not -dusage=100.

Conversion of metadata (~1.55 GiB) to DUP worked fine.

A theory, based on the error messages, is that some of the converted
files, even after defrag, still have extents larger than 1GiB and
hence don't fit in a native Btrfs extent.

Running defrag several more times and balance again doesn't help.

An error looks like:
BTRFS info (device sdc1): relocating block group 1821099687936 flags 1
BTRFS error (device sdc1): allocation failed flags 1, wanted 2013265920
BTRFS: space_info 1 has 4773171200 free, is not full
BTRFS: space_info total=1494648619008, used=1489775505408, pinned=0,
reserved=99700736, may_use=2102390784, readonly=241664


The following script returned 46 filenames (looking up the block group
in the error):
grep -B 1 "BTRFS error" /var/log/syslog | grep relocating | cut -d ' ' -f 14 | \
while read block
do
    echo "Block group: $block"
    btrfs inspect-internal logical-resolve $block /mnt
done

The files are ranging from 41KiB to 6.6GiB in size, which doesn't seem
to support the theory of too large extents.

Moving the 46 files to another disk (no errors reported) and running
balance again resulted in "64 enospc errors during balance" - down
from 98 errors.

Running the above script again gives this error for about half of the
block groups:
ioctl ret=-1, error: No such file or directory

I had no such errors the first time I looked up block groups.

What's the next step in zeroing in on the bug, before I start over?
And I will start over.

Re: ENOSPC after conversion [Was: Fixing Btrfs Filesystem Full Problems typo?]

[Robert White]

So far I don't see a "bug".

On 12/11/2014 12:18 AM, Patrik Lundquist wrote:
> I'll reboot the thread with a recap and my latest findings.
>
> * Half full 3TB disk converted from ext4 to Btrfs, after first
> verifying it with fsck.
> * Undo subvolume deleted after being happy with the conversion.
> * Recursive defrag.
> * Full balance, that ended with "98 enospc errors during balance."

This is running out of space to allocate a disk extent. Not running out 
of space to allocate a file.

> In that order, nothing in between. No snapshots or other subvolumes.
> Loads of real free space.

Space for files is not space for extents.

> Btrfs check reports a clean filesystem.

So not a "bug", just out of raw space.

> Btrfs balance -musage=100 -dusage=99 works, but not -dusage=100.

Ibid.

> Conversion of metadata (~1.55 GiB) to DUP worked fine.

More evidence that things are fine.


> A theory, based on the error messages, is that some of the converted
> files, even after defrag, still have extents larger than 1GiB and
> hence don't fit in a native Btrfs extent.

You are conflating file extents with storage extents.

Here is a clean example:

Gust t # mkfs.btrfs -f /dev/loop0
Btrfs v3.17.1
See http://btrfs.wiki.kernel.org for more information.

Performing full device TRIM (2.00GiB) ...
Turning ON incompat feature 'extref': increased hardlink limit per file 
to 65536
fs created label (null) on /dev/loop0
         nodesize 16384 leafsize 16384 sectorsize 4096 size 2.00GiB
Gust t # mount /dev/loop0 /mnt/src
Gust t # btrfs balance start /mnt/src
Done, had to relocate 5 out of 5 chunks
Gust t # btrfs fi df /mnt/src
Data, single: total=208.00MiB, used=128.00KiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=104.00MiB, used=112.00KiB
GlobalReserve, single: total=16.00MiB, used=0.00B
Gust t # dd if=/dev/urandom of=/mnt/src/scratch bs=1M count=40
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 3.91674 s, 10.7 MB/s
Gust t # btrfs fi sync /mnt/src
FSSync '/mnt/src'
Gust t # btrfs fi df /mnt/src
Data, single: total=208.00MiB, used=40.12MiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=104.00MiB, used=160.00KiB
GlobalReserve, single: total=16.00MiB, used=0.00B


Notice this sequence...
brand new clean and balanced file system
Data, single: total=208.00MiB, used=128.00KiB

Created a 40Meg non-compressable non-empty file
Gust t # dd if=/dev/urandom of=/mnt/src/scratch bs=1M count=40

Flushed the file system to update the metadata and then I have
Data, single: total=208.00MiB, used=40.12MiB

The 40Meg of random data bytes didn't change the total data extent(s) 
allocated, it only changed the total amount of the allocated data 
extents that is in use.

> Running defrag several more times and balance again doesn't help.

That sounds correct as defrag defrags files, it does not reallocate 
extents.

> An error looks like:
> BTRFS info (device sdc1): relocating block group 1821099687936 flags 1
> BTRFS error (device sdc1): allocation failed flags 1, wanted 2013265920
> BTRFS: space_info 1 has 4773171200 free, is not full
> BTRFS: space_info total=1494648619008, used=1489775505408, pinned=0,
> reserved=99700736, may_use=2102390784, readonly=241664

As explained in the other thread, the extent-tree.c extent allocator 
could not find 2013265920 contiguous bytes in order to make a new extent 
into which it would sort the old extent's file fragments.


> The following script returned 46 filenames (looking up the block group
> in the error):
> grep -B 1 "BTRFS error" /var/log/syslog | grep relocating | cut -d ' ' -f 14 | \
> while read block
> do
>      echo "Block group: $block"
>      btrfs inspect-internal logical-resolve $block /mnt
> done
>
> The files are ranging from 41KiB to 6.6GiB in size, which doesn't seem
> to support the theory of too large extents.

Sure it does, the 2013265920 _bytes_ of the extent in the error is 
bigger than some files in the extent (see 41KiB less than 2-ish gigs) 
but smaller than others files which reside only partly in the extent 
(see 6.6GiB greater than 2-ish gigs).

This is because data extents are not file specific, which is why there 
was more than one file in that extent.


> Moving the 46 files to another disk (no errors reported) and running
> balance again resulted in "64 enospc errors during balance" - down
> from 98 errors.

So by moving and deleting all the files in that extent reduced it to 
usage=0, and balance could then recover that space once it noticed it 
was empty.

There's a good chance that if you balanced again and again the number of 
no space errors might decrease. With only one 2-ish gig empty slot 
sliding around like one of those puzzles where you have to sort the 
numbers from 1 to 15 by sliding them around in the 4x4=16 element grid.


> Running the above script again gives this error for about half of the
> block groups:
> ioctl ret=-1, error: No such file or directory
>
> I had no such errors the first time I looked up block groups.

Because the first time the files were there and the second time they had 
been moved to a different disk and therefore deleted. Leaving little 
gaps with no file in them at all. Then the subsequent balance events may 
have moved other files into those locations and whatnot.


> What's the next step in zeroing in on the bug, before I start over?
> And I will start over.

The first step is admitting that you _don't_ have a problem.

You are out of raw space in which to construct new extents. This is not 
a "bug" this is a fact.

You are _not_ out of space in which to create files. (or so I presume, 
you still haven't posted the output of /bin/df or btrfs filesystem df).

You are confusing ext4 "file extents" with btrfs storage extents. A 
btrfs storage extent is analogous to the non-inode portion of an ext4 
"block group".

EXT4 allocates fixed size block groups with a ratio of inode-space to 
data-space.

BTRFS allocates data-space (for data) and metadata-space (for inodes, 
the data for very small files, chcksums, and overhead that the system 
may need because the whole file system wasn't pre-carved up 
geographically like with EXT4.)

Your next step is to either add storage in accordance with your plan of 
adding four more volumes to make a RAID (as expressed elsewhere), or 
make a clean filesystem and copy your files over.

Both options are equal give or take things like any desire to use skinny 
metadata, compression, or other advanced options that may only be 
available at file or filesystem creation time.

A note on spotting "bugs" [Was: ENOSPC after conversion]

[Robert White]

On 12/11/2014 12:18 AM, Patrik Lundquist wrote:
> * Full balance, that ended with "98 enospc errors during balance."

Assuming that quote is an actual quote from the output of the balance...

We can strongly infer that this sort of occurrence is expected since 
there is code to keep track of it and report the total times it happened.

"Bugs" are unexpected things that cause failures and/or damage.

Expected but non-optimal things that print summaries of their 
occurrences tend to be "expected unpleasantness that has been explored 
by the programmer, causes no harm, and is not worth fixing", which is 
different thing than a bug. It's a "No Useful Options".

Cant Fix and Wont Fix events lie somewhere above that on the programmers 
scale that goes from perfect execution to absolute train-wreck bug.

Were I the programmer I might have written this as "98 extents skipped 
due to space constraints (ENOSPC)".

I won't be offering a patch to that effect, however, as there may be 
other kinds of expected ENOSPC events contributing to that counter, so 
re-writing the summary text could be making untrue statements.

I've been have been chasing this with you because your statement that 
"-dusage=99 works, but not -dusage=100". But the message above tells me 
that your characterization as "not working" is somewhat overstating 
things. It _worked_ with -dusage=100 in that it didn't abort, crash, 
trash data, or hang. It just had to skip some elements due to well 
understood (by the implementor) and fully reported conditions.

So lets explore what the system "could have done" instead of just 
skipping those extents...

It could have tried to break the extent into smaller pieces. But to do 
that it would have to dissect the contents of the extent and go looking 
for ways to repack them into two or more smaller extents. Those 
candidate extents would have to be allocated based on guesses before the 
attempt because other writers might steal the space if you don't 
preallocate. This could involve repeated retries and result in taking 
one big extent and exploding it into any number of tiny extents. 
Performing this task could take unbounded time. In computer science it's 
an NP-complete function of arbitrary complexity sometimes called "the 
floppy problem" (a name that is impossible to google usefully, it seems, 
because the word floppy is search poison 8-) ).

The Floppy Problem :: so called because one of the original formulations 
was "how many floppy disks do I need to optimally pack these files 
without having to cut up the files themselves?" Indeed multi-floppy 
"Zip" programs were invented to skip that whole painful mess so people 
could just ship their software. 8-)

If you start reading here 
http://en.wikipedia.org/wiki/Cutting_stock_problem and work your way 
back through the knapsack problem you'll get a glimpse how ugly this 
sort of corner case can get.

In our case the "roll" being "cut" is the donor extent and the possible 
widths/sizes are the discernible gaps in the raw extent map and the 
constraint is that we can't break cut any of the internally allocated 
regions within the extent (we can only relocate them not break them up 
because that could lead to needing to allocate more metadata space in 
the extent tree which could invalidate our planned cuts etc till the end 
of time.)

So it is a problem that _can_ be solved programatically, but it's not a 
problem that is worth the time to solve either in programmer hours or in 
disk write hours.

So yea... It's big, It's valid, and you've got no single place to copy 
it to that is equally big, so it gets skipped.

Not a bug.

Re: ENOSPC after conversion [Was: Fixing Btrfs Filesystem Full Problems typo?]

[Patrik Lundquist]

On 11 December 2014 at 11:18, Robert White <rwhite@pobox.com> wrote:
> So far I don't see a "bug".

Fair enough, lets call it a huge problem with btrfs convert. I think
it warrants a note in the wiki.


> On 12/11/2014 12:18 AM, Patrik Lundquist wrote:
>>
>> Running defrag several more times and balance again doesn't help.
>
> That sounds correct as defrag defrags files, it does not reallocate extents.

>From https://btrfs.wiki.kernel.org/index.php/Conversion_from_Ext3

"A notable caveat is that a balance can fail with "ENOSPC" if the
defragment is skipped. This is usually due to large extents on ext
being larger than the maximum size btrfs normally operates with (1
GB). A defrag of all large files will avoid this:"

I interpreted it as breaking down large extents and reallocating them,
thus avoiding my current situation.


> There's a good chance that if you balanced again and again the number of no
> space errors might decrease. With only one 2-ish gig empty slot sliding
> around like one of those puzzles where you have to sort the numbers from 1
> to 15 by sliding them around in the 4x4=16 element grid.

I was never fond of those puzzles.


> The first step is admitting that you _don't_ have a problem.

I've got 99 problems and balance is one of them (the other are block
groups). :-)

Of course the filesystem is in a problematic state after the
conversion, even if it's not a bug. ~1.5TB of free space and yet out
of space and it can't be fixed with a balance. It might not be wrong
per se but it's very problematic from a user perspective.

Anyway, this thread has turned up lots of good information.


> You are _not_ out of space in which to create files. (or so I presume, you
> still haven't posted the output of /bin/df or btrfs filesystem df).

I'm not; creating new files works.

$ df
Filesystem      1K-blocks       Used  Available Use% Mounted on
/dev/sdc1      2930265088 1402223656 1526389096  48% /mnt

$ btrfs fi df /mnt
Data, single: total=1.41TiB, used=1.30TiB
System, DUP: total=32.00MiB, used=124.00KiB
Metadata, DUP: total=2.50GiB, used=1.49GiB
GlobalReserve, single: total=512.00MiB, used=0.00B


> Your next step is to either add storage in accordance with your plan of
> adding four more volumes to make a RAID (as expressed elsewhere), or make a
> clean filesystem and copy your files over.

I've already decided to start over with a clean filesystem to get rid
of the ext4 legacy. I'm only curious about how to solve the balance
problem, and now I know how.

Re: ENOSPC after conversion [Was: Fixing Btrfs Filesystem Full Problems typo?]

[Robert White]

On 12/11/2014 03:01 PM, Patrik Lundquist wrote:
> On 11 December 2014 at 11:18, Robert White <rwhite@pobox.com> wrote:
>> So far I don't see a "bug".
>
> Fair enough, lets call it a huge problem with btrfs convert. I think
> it warrants a note in the wiki.
>
>
>> On 12/11/2014 12:18 AM, Patrik Lundquist wrote:
>>>
>>> Running defrag several more times and balance again doesn't help.
>>
>> That sounds correct as defrag defrags files, it does not reallocate extents.
>
>>From https://btrfs.wiki.kernel.org/index.php/Conversion_from_Ext3
>
> "A notable caveat is that a balance can fail with "ENOSPC" if the
> defragment is skipped. This is usually due to large extents on ext
> being larger than the maximum size btrfs normally operates with (1
> GB). A defrag of all large files will avoid this:"
>
> I interpreted it as breaking down large extents and reallocating them,
> thus avoiding my current situation.
>
>
>> There's a good chance that if you balanced again and again the number of no
>> space errors might decrease. With only one 2-ish gig empty slot sliding
>> around like one of those puzzles where you have to sort the numbers from 1
>> to 15 by sliding them around in the 4x4=16 element grid.
>
> I was never fond of those puzzles.
>
>
>> The first step is admitting that you _don't_ have a problem.
>
> I've got 99 problems and balance is one of them (the other are block
> groups). :-)
>
> Of course the filesystem is in a problematic state after the
> conversion, even if it's not a bug. ~1.5TB of free space and yet out
> of space and it can't be fixed with a balance. It might not be wrong
> per se but it's very problematic from a user perspective.
>
> Anyway, this thread has turned up lots of good information.
>
>
>> You are _not_ out of space in which to create files. (or so I presume, you
>> still haven't posted the output of /bin/df or btrfs filesystem df).
>
> I'm not; creating new files works.

Exactly!

I know you are not, thats why the DF below doesn't say you are out of 
space. You have 1526389096 one-K blocks available on the drive for file 
storage.

You just don't have enough _contiguous_ unallocated _raw_ storage to 
copy a 2-gig-ish extent.

Until you "get" this idea you are not going to understand what is happening.

When you make an EXT4/3/2 file system it goes out and allocates _all_ 
_the_ _space_ to the file system's structures. It goes out and writes 
block groups where all the block groups go, with zero-ed out allocation 
bitmaps and zeroed-out inodes. (Actually the kernel does most of that 
work on "first mount" in the modern cases, which is way a mkfs.ext4 
takes _way_ _less_ _time_ than an mkfs.ext2 on the same expanse of disk).

BTRFS, on the other hand, does something _completely_ _different_. It 
just writes a few things. One is the superblock(s) [up to three per 
physical disk,depending on media size]. One is the metadata chunk(s) 
that contain those blocks. One is the raw space storage tree. And so on. 
It never _touches_ the bulk of the disk(s).

As you fill your metadata and data extents, the raw storage manager 
makes new ones on-demand. This requires space from the "raw storage".

In EXT4 speak it would be like if EXT4 only wrote the _first_ block 
group and then only added the second block group once the first was 
filled. Then a third when the second and first were both filled.

There will come a point when BTRFS has allocated _all_ the raw storage. 
And at that point it will fail to allocate new storage for block groups 
because there is no more raw storage. At that moment, the BTRFS 
filesystem is "just like" the freshly built EXT4 in that 100% of the raw 
storage has "initialized file system bits" written on it.

Recent kernel patches have added a feature to BTRFS that says "if this 
file system bit is completely empty, lets de-initialize it and return it 
to the raw pool". This was added because one could fill ones file system 
with "large files" (and so data extents) then delete all those and fill 
the same space wiht tiny files (and so need metadata extents). If there 
was no way to "remove the empty data extent" so that it could be 
replaced by one or more metadata extents, you could end up in the very 
same jam you can end up wiht in EXT4, which is running out of inodes 
when you've got plenty of "storage space".

So BTRFS has this give-and-take. But when it comes to "relocating" 
something, as in "balancing it", you have to be able to make the new 
space before you can move the files over and remove the old space. 
That's the core idea behind Copy On Write. In COW there is no "move" 
operation for storage. You have only "copy" and "delete".

So when the raw storage pool gets full you run out of space to make more 
BTRFS. When the allocated storage space gets full, you run out of space 
to make files.

Its unlike other filesystems. Every byte is allocated twice. First it 
has to be allocated to the file system, then the file system has to 
allocate it to the individual files.

This double layer makes it more adaptable as described above.

So Is There A Precedent?

Yes.

ITEM: A normal program running in a system has an address space (on a 
32-bit linux box this is typically 2G for code, and 2G for data). No 
matter how "big" the program is, it gets the same 4G of "space" in its 
virtual memory map. That is, the map is _capable_ _of_ containing 4G in 
tww 2G regions. This map is assembled by the kernel when it calls exec, 
and it is assembled _before_ the code of the program is even read off 
the disk.

This is very like what happens in a mkfs.btrfs. We point it at a region 
of disk and it builds the _ability_ to make the map by setting up the 
minimal necessary structures needed to maintain the map that will be built.

ITEM: The kernel then loads the core executable into the start of the 2G 
space by using mmap(). But because of dynamic linking, this load may 
bring in lots of other files, each will take up some of that 2G of code 
space. Many will take up some of the 2G of data space since most code 
requires data.

This is like how BTRFS makes the first extents for data and whtnot.

ITEM: As the program runs it may dynamically load even more code (like 
say loading the flash player extension into your web browser when you go 
to a site that has some stupid advert or movie); but what it's really 
going to start doing is using the data area. You want to look at that 
picture of that kitten, you are going to need to take a chunk out of 
that 2G of data and "realize it", that is access after doing an 
anynonymous mmap() or just make your effective amount of "ram in use" 
bigger with setbrk().

This is like how BTRFS allocates metadata (analogous to code space) and 
data space extents.

ITEM: If you go to another page with a picture of a puppy instead of a 
kitten, the space for the kitten picture has been freed up and if the 
puppy picture can fit there then no additional mmap() or setbrk() calls 
need to be made. This sort of thing is normally dealt-with/controlled at 
the deepest level by the memory allocation library. I don't have to know 
when setbrk() or mmap() is being called for this data memory, I just 
call malloc() when I need more and free() when I am done with something 
I got from malloc(). Under the hood there are linked list structures, 
and trees, and bitmaps for small allocations too small to be worth 
calling setbrk() for each one. etc.

This is normal runtime behavior for BTRFS. It serves up already 
controlled bits when it can, and it goes out and allocates and 
initialized more when the need arises.

This all happens under the hood, and normally the only way you'd ever 
see an out of space error ("ENOSPC") is if you asked for a bit and it 
didn't have the right kind of space, so it went to allocate more space 
on the disk and that failed as well.

In the same sense a program could fill up its 2G for data, and still 
have nearly 2G available for code, but its the wrong kind of space, so a 
malloc would return "out of memory".

==== SO WTF HERE THEN, AM I RIGHT? ====

EXT4 had already allocated all the raw space and decided what kind of 
storage each segment was for.

You ran btrfs-convert, and it went in and adopted all those spaces for 
all those purposes. Then it did "what it could" to free up as much of 
that allocated space as was possible.

Since your EXT4 file system had been full-to-busting at least once in 
the past, there was very little the convert program to could to free up 
those data extents because on the average the now half-empty file system 
consisted of "all the data extents, each half empty at best".

If you original EXT4 image had been less something-is-everywhere, the 
convert process, and the balance process thereafter, would have had more 
raw space to do the "sliding block puzzle" that is a balance operation.

By adding another storage volume (disk/partition/etc) you would be 
giving it that room.

Where the EXT4/3/2 raw storage map looks like a fully built brick wall 
made with fixed sized bricks, a BTRFS raw storage layout looks like a 
sliding block puzzle.

To really understand what balance is trying to do, play this game for a 
while

   http://www.agame.com/game/sliding-block-puzzle

Only imagine that each block had to be copied to the new location before 
it could be removed from the old one. And keep in mind that the little 
blocks are the metadata, the medium blocks are partly full data blocks, 
and the big blocks are the giant 2-gig-ish extents crammed full of files 
that you were asking balance to juggle around.


>
> $ df
> Filesystem      1K-blocks       Used  Available Use% Mounted on
> /dev/sdc1      2930265088 1402223656 1526389096  48% /mnt

See, there's your space.

It's not "lost" its just not on hand in 2Gig contiguous chunks that 
balance can copy around. Chances are there a good number of 1 gig chunks 
for future data, and lots more 256M chunks for future metadata (and the 
small files that fit wholly therein).

Nothing broken here. No "mising space" just a sliding block puzzle with 
no current solution that would be easily solvable if more raw space was 
available.

>
> $ btrfs fi df /mnt
> Data, single: total=1.41TiB, used=1.30TiB
> System, DUP: total=32.00MiB, used=124.00KiB
> Metadata, DUP: total=2.50GiB, used=1.49GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
>
>
>> Your next step is to either add storage in accordance with your plan of
>> adding four more volumes to make a RAID (as expressed elsewhere), or make a
>> clean filesystem and copy your files over.
>
> I've already decided to start over with a clean filesystem to get rid
> of the ext4 legacy. I'm only curious about how to solve the balance
> problem, and now I know how.

You solve the balance problem by adding enough working space for it to 
proceed, or by removing enough files that it can coalesce enough to 
delete a few of those big extents, thus providing enough working space 
to proceed.

Re: ENOSPC after conversion [Was: Fixing Btrfs Filesystem Full Problems typo?]

[Robert White]

TL;DR version

On 12/11/2014 03:01 PM, Patrik Lundquist wrote:
> Of course the filesystem is in a problematic state after the
> conversion, even if it's not a bug. ~1.5TB of free space and yet out
> of space and it can't be fixed with a balance. It might not be wrong
> per se but it's very problematic from a user perspective.

The file system is not in a problematic state, you just don't understand 
the state its in. It has ~1.5TB of space for files. Storing files is why 
we have filesystems after all.

> $ df
> Filesystem      1K-blocks       Used  Available Use% Mounted on
> /dev/sdc1      2930265088 1402223656 1526389096  48% /mnt

There it is, ~1.5TB of available space.

That space is available for files. So no problem.

So it's not "out of space" at all. Go ahead and make a couple thousand 
files of various sizes, you'll see. Lots of space.

Since it's not broken there is nothing for balance to "fix". The fact 
that balance tells you it doesn't have enough space to juggle filesystem 
bits HAS NOTHING TO DO with the file system's ability to store files.

You might as well be complaining that your hard disk is unfixable 
because you don't have enough space to add another partition.

Apples and Polar Bears. The "out of space" is only a problem in your 
head. At the same level of abstraction the EXT4 file system partition 
was "out of space" before you started the conversion because all of the 
partition was filled up with EXT4 structures.

No matter how many times you try to turn this into a "bug" it's just 
_not_ a bug. Rewording it doesn't change the facts. The _file_ _system_ 
isn't "out of space" it's just filled its partition with BTRFS extents 
enough to cause some extents to be skipped during balance.

Nothing to see here, move along. 8-)

Re: A note on spotting "bugs" [Was: ENOSPC after conversion]

[Patrik Lundquist]

On 11 December 2014 at 23:00, Robert White <rwhite@pobox.com> wrote:
> On 12/11/2014 12:18 AM, Patrik Lundquist wrote:
>>
>> * Full balance, that ended with "98 enospc errors during balance."
>
> Assuming that quote is an actual quote from the output of the balance...

It is, from dmesg.


> "Bugs" are unexpected things that cause failures and/or damage.

Not all errors are as pretty as

BTRFS info (device sdc1): relocating block group 1756675178496 flags 1
BTRFS error (device sdc1): allocation failed flags 1, wanted 1272844288
BTRFS: space_info 1 has 13703077888 free, is not full
BTRFS: space_info total=1504312295424, used=1487622750208, pinned=0,
reserved=2986196992, may_use=1308749824, readonly=270336

some are

BTRFS info (device sdc1): relocating block group 1780297498624 flags 1
------------[ cut here ]------------
WARNING: CPU: 2 PID: 11094 at
/build/linux-Y9HjRe/linux-3.16.7/fs/btrfs/extent-tree.c:7280
btrfs_alloc_free_block+0x219/0x450 [btrfs]()
BTRFS: block rsv returned -28
Modules linked in: nfsd auth_rpcgss oid_registry nfs_acl nfs lockd
fscache sunrpc btrfs xor nls_utf8 nls_cp437 vfat fat kvm_intel
raid6_pq kvm crc32_pclmul jc42 coretemp ghash_clmulni_intel iTCO_wdt
ipmi_watchdog iTCO_vendor_support aesni_intel joydev aes_x86_64
efi_pstore lrw gf128mul evdev glue_helper ast ablk_helper lpc_ich
cryptd ttm pcspkr efivars mfd_core i2c_i801 drm_kms_helper drm tpm_tis
tpm acpi_cpufreq i2c_ismt shpchp button processor thermal_sys ipmi_si
ipmi_poweroff ipmi_devintf ipmi_msghandler autofs4 ext4 crc16 mbcache
jbd2 sg sd_mod crc_t10dif crct10dif_generic hid_generic usbhid hid
ahci libahci crct10dif_pclmul crct10dif_common crc32c_intel igb libata
ehci_pci i2c_algo_bit xhci_hcd ehci_hcd i2c_core dca scsi_mod ptp
usbcore pps_core usb_common
CPU: 2 PID: 11094 Comm: btrfs Tainted: G        W     3.16.0-4-amd64
#1 Debian 3.16.7-2
Hardware name: Supermicro A1SAi/A1SAi, BIOS 1.0c 02/27/2014
 0000000000000009 ffffffff81506b43 ffff88032779f780 ffffffff81065717
 ffff88032d68a640 ffff88032779f7d0 0000000000001000 ffff8803117df480
 0000000000000000 ffffffff8106577c ffffffffa0536338 0000000000000020
Call Trace:
 [<ffffffff81506b43>] ? dump_stack+0x41/0x51
 [<ffffffff81065717>] ? warn_slowpath_common+0x77/0x90
 [<ffffffff8106577c>] ? warn_slowpath_fmt+0x4c/0x50
 [<ffffffffa04a8b09>] ? btrfs_alloc_free_block+0x219/0x450 [btrfs]
 [<ffffffff81142bf6>] ? free_hot_cold_page_list+0x46/0x90
 [<ffffffffa04dc5c8>] ? read_extent_buffer+0xc8/0x120 [btrfs]
 [<ffffffffa0492c31>] ? btrfs_copy_root+0x101/0x2e0 [btrfs]
 [<ffffffffa05032d1>] ? create_reloc_root+0x201/0x2d0 [btrfs]
 [<ffffffffa0509398>] ? btrfs_init_reloc_root+0x98/0xb0 [btrfs]
 [<ffffffffa04b9564>] ? record_root_in_trans+0xa4/0xf0 [btrfs]
 [<ffffffffa04ba95f>] ? btrfs_record_root_in_trans+0x3f/0x70 [btrfs]
 [<ffffffffa04bb940>] ? start_transaction+0x90/0x560 [btrfs]
 [<ffffffffa04c605a>] ? btrfs_evict_inode+0x33a/0x4d0 [btrfs]
 [<ffffffff811bf0ec>] ? evict+0xac/0x170
 [<ffffffffa04c0762>] ? btrfs_run_delayed_iputs+0xd2/0xf0 [btrfs]
 [<ffffffffa04bb812>] ? btrfs_commit_transaction+0x922/0x9c0 [btrfs]
 [<ffffffffa04bb940>] ? start_transaction+0x90/0x560 [btrfs]
 [<ffffffffa0504ea4>] ? prepare_to_relocate+0xf4/0x1b0 [btrfs]
 [<ffffffffa0509e72>] ? relocate_block_group+0x42/0x670 [btrfs]
 [<ffffffffa050a667>] ? btrfs_relocate_block_group+0x1c7/0x2d0 [btrfs]
 [<ffffffffa04e0432>] ? btrfs_relocate_chunk.isra.27+0x62/0x700 [btrfs]
 [<ffffffffa04928d1>] ? btrfs_set_path_blocking+0x31/0x70 [btrfs]
 [<ffffffffa0497d8d>] ? btrfs_search_slot+0x4ad/0xad0 [btrfs]
 [<ffffffffa04d1fd5>] ? btrfs_get_token_64+0x55/0xf0 [btrfs]
 [<ffffffffa04e355b>] ? btrfs_balance+0x82b/0xe80 [btrfs]
 [<ffffffffa04eaba4>] ? btrfs_ioctl_balance+0x154/0x500 [btrfs]
 [<ffffffffa04ef89c>] ? btrfs_ioctl+0x58c/0x2b10 [btrfs]
 [<ffffffff811670f1>] ? handle_mm_fault+0xa91/0x11a0
 [<ffffffff810562a1>] ? __do_page_fault+0x1d1/0x4e0
 [<ffffffff8116afc1>] ? vma_link+0xb1/0xc0
 [<ffffffff811b788f>] ? do_vfs_ioctl+0x2cf/0x4b0
 [<ffffffff811b7af1>] ? SyS_ioctl+0x81/0xa0
 [<ffffffff8150ecc8>] ? page_fault+0x28/0x30
 [<ffffffff8150cc2d>] ? system_call_fast_compare_end+0x10/0x15
---[ end trace 880987d36ae50245 ]---
BTRFS error (device sdc1): allocation failed flags 1, wanted 2013265920
BTRFS: space_info 1 has 8384299008 free, is not full
BTRFS: space_info total=1500017328128, used=1491533037568, pinned=0,
reserved=99807232, may_use=2147475456, readonly=184320

Re: A note on spotting "bugs" [Was: ENOSPC after conversion]

[Robert White]

On 12/11/2014 10:42 PM, Patrik Lundquist wrote:
> On 11 December 2014 at 23:00, Robert White <rwhite@pobox.com> wrote:
>> On 12/11/2014 12:18 AM, Patrik Lundquist wrote:
>>>
>>> * Full balance, that ended with "98 enospc errors during balance."
>>
>> Assuming that quote is an actual quote from the output of the balance...
>
> It is, from dmesg.
>
>
>> "Bugs" are unexpected things that cause failures and/or damage.
>
> Not all errors are as pretty as
>
> BTRFS info (device sdc1): relocating block group 1756675178496 flags 1
> BTRFS error (device sdc1): allocation failed flags 1, wanted 1272844288
> BTRFS: space_info 1 has 13703077888 free, is not full
> BTRFS: space_info total=1504312295424, used=1487622750208, pinned=0,
> reserved=2986196992, may_use=1308749824, readonly=270336
>
> some are
>
> BTRFS info (device sdc1): relocating block group 1780297498624 flags 1
> ------------[ cut here ]------------
> WARNING: CPU: 2 PID: 11094 at
> /build/linux-Y9HjRe/linux-3.16.7/fs/btrfs/extent-tree.c:7280
> btrfs_alloc_free_block+0x219/0x450 [btrfs]()
> BTRFS: block rsv returned -28
> Modules linked in: nfsd auth_rpcgss oid_registry nfs_acl nfs lockd
> fscache sunrpc btrfs xor nls_utf8 nls_cp437 vfat fat kvm_intel
> raid6_pq kvm crc32_pclmul jc42 coretemp ghash_clmulni_intel iTCO_wdt
> ipmi_watchdog iTCO_vendor_support aesni_intel joydev aes_x86_64
> efi_pstore lrw gf128mul evdev glue_helper ast ablk_helper lpc_ich
> cryptd ttm pcspkr efivars mfd_core i2c_i801 drm_kms_helper drm tpm_tis
> tpm acpi_cpufreq i2c_ismt shpchp button processor thermal_sys ipmi_si
> ipmi_poweroff ipmi_devintf ipmi_msghandler autofs4 ext4 crc16 mbcache
> jbd2 sg sd_mod crc_t10dif crct10dif_generic hid_generic usbhid hid
> ahci libahci crct10dif_pclmul crct10dif_common crc32c_intel igb libata
> ehci_pci i2c_algo_bit xhci_hcd ehci_hcd i2c_core dca scsi_mod ptp
> usbcore pps_core usb_common
> CPU: 2 PID: 11094 Comm: btrfs Tainted: G        W     3.16.0-4-amd64
> #1 Debian 3.16.7-2
> Hardware name: Supermicro A1SAi/A1SAi, BIOS 1.0c 02/27/2014
>   0000000000000009 ffffffff81506b43 ffff88032779f780 ffffffff81065717
>   ffff88032d68a640 ffff88032779f7d0 0000000000001000 ffff8803117df480
>   0000000000000000 ffffffff8106577c ffffffffa0536338 0000000000000020
> Call Trace:
>   [<ffffffff81506b43>] ? dump_stack+0x41/0x51
>   [<ffffffff81065717>] ? warn_slowpath_common+0x77/0x90
>   [<ffffffff8106577c>] ? warn_slowpath_fmt+0x4c/0x50
>   [<ffffffffa04a8b09>] ? btrfs_alloc_free_block+0x219/0x450 [btrfs]
>   [<ffffffff81142bf6>] ? free_hot_cold_page_list+0x46/0x90
>   [<ffffffffa04dc5c8>] ? read_extent_buffer+0xc8/0x120 [btrfs]
>   [<ffffffffa0492c31>] ? btrfs_copy_root+0x101/0x2e0 [btrfs]
>   [<ffffffffa05032d1>] ? create_reloc_root+0x201/0x2d0 [btrfs]
>   [<ffffffffa0509398>] ? btrfs_init_reloc_root+0x98/0xb0 [btrfs]
>   [<ffffffffa04b9564>] ? record_root_in_trans+0xa4/0xf0 [btrfs]
>   [<ffffffffa04ba95f>] ? btrfs_record_root_in_trans+0x3f/0x70 [btrfs]
>   [<ffffffffa04bb940>] ? start_transaction+0x90/0x560 [btrfs]
>   [<ffffffffa04c605a>] ? btrfs_evict_inode+0x33a/0x4d0 [btrfs]
>   [<ffffffff811bf0ec>] ? evict+0xac/0x170
>   [<ffffffffa04c0762>] ? btrfs_run_delayed_iputs+0xd2/0xf0 [btrfs]
>   [<ffffffffa04bb812>] ? btrfs_commit_transaction+0x922/0x9c0 [btrfs]
>   [<ffffffffa04bb940>] ? start_transaction+0x90/0x560 [btrfs]
>   [<ffffffffa0504ea4>] ? prepare_to_relocate+0xf4/0x1b0 [btrfs]
>   [<ffffffffa0509e72>] ? relocate_block_group+0x42/0x670 [btrfs]
>   [<ffffffffa050a667>] ? btrfs_relocate_block_group+0x1c7/0x2d0 [btrfs]
>   [<ffffffffa04e0432>] ? btrfs_relocate_chunk.isra.27+0x62/0x700 [btrfs]
>   [<ffffffffa04928d1>] ? btrfs_set_path_blocking+0x31/0x70 [btrfs]
>   [<ffffffffa0497d8d>] ? btrfs_search_slot+0x4ad/0xad0 [btrfs]
>   [<ffffffffa04d1fd5>] ? btrfs_get_token_64+0x55/0xf0 [btrfs]
>   [<ffffffffa04e355b>] ? btrfs_balance+0x82b/0xe80 [btrfs]
>   [<ffffffffa04eaba4>] ? btrfs_ioctl_balance+0x154/0x500 [btrfs]
>   [<ffffffffa04ef89c>] ? btrfs_ioctl+0x58c/0x2b10 [btrfs]
>   [<ffffffff811670f1>] ? handle_mm_fault+0xa91/0x11a0
>   [<ffffffff810562a1>] ? __do_page_fault+0x1d1/0x4e0
>   [<ffffffff8116afc1>] ? vma_link+0xb1/0xc0
>   [<ffffffff811b788f>] ? do_vfs_ioctl+0x2cf/0x4b0
>   [<ffffffff811b7af1>] ? SyS_ioctl+0x81/0xa0
>   [<ffffffff8150ecc8>] ? page_fault+0x28/0x30
>   [<ffffffff8150cc2d>] ? system_call_fast_compare_end+0x10/0x15
> ---[ end trace 880987d36ae50245 ]---
> BTRFS error (device sdc1): allocation failed flags 1, wanted 2013265920
> BTRFS: space_info 1 has 8384299008 free, is not full
> BTRFS: space_info total=1500017328128, used=1491533037568, pinned=0,
> reserved=99807232, may_use=2147475456, readonly=184320
>

Interesting but only fractionally so.

The function btrfs_alloc_free_block() has disappeared from the kernel 
sources in Linus' git tree for the kernel. It used to be in 
linux/fs/btrfs/extent-tree.c ... direct allocation seems to have been 
replaced by a reservation system.

This still doesnt say _anything_ is wrong with your filesystem except 
that it doesn't have enough _raw_ space to create a 2-ish gig extent.

To produce that backtrace as a _WARNING_ (check out the first line) the 
programmer explicitly had to call the function that generates that 
backtrace. That is, it's not a "oops" or other _unforeseen_ critical 
path failure.

So while it's still just a harmless out-of-space condition in terms 
balance, and its got nothing to do with being "out of space" at the 
functional level, some work is being done on the way the handling is 
taking place.

Particularly, there was some code that explicitly called WARN() or 
BUG_ON() while it was processing that out of raw space condition. This 
is a normal-ish thing for code to do when the programmer is like "hey, 
I'd like to see what the state actually is when this happens".

Since the code has literally been replaced whole-scale in 3.18 (that 
just got tagged in the development tree I'm referencing) chances are its 
been on someone's mind for a while now.

That is someone was thinking "this downright likely condition could 
happen when we don't have a big enough contiguous chunk of raw space, 
maybe we should handle it better". Then they replaced the code.

---

So as much as you seem to want to characterize this as a "huge problem" 
or a "bug" it's just a less-than-optimal but completely stable and 
foreseeable result of feeding an really chaotic and previously full EXT4 
file system into btrfs-convert.

You yourself even found the annotation in the wiki that said you should 
have e4defragged the system before conversion.

...

We are not on new, shifting, or terrible ground here.

Just because you don't know how to read a backtrace doesn't mean that 
every backtrace is cause for concern. Some are. The "warnings" usually 
not so much.

You've already found what you missed (the e4defrag) when preparing for 
the conversion.

You've already heard my rationale for why conversions tend to be less 
than optimal regardless of the systems.

You've already heard Duncan's rational for the same position.

You've already heard my argument for building a new filesystem and 
copying the contents over onto it.

You've already decided that it would have been better to start with a 
clean filesystem and then copy the files.

You've already decided to do that create and copy process.

I've written maybe a couple thousand words to guide you through the 
analysis so you can understand the difference between raw allocation at 
the partition space level versus user-level allocations for storing 
files etc.

What you are experiencing is a little vexing, but it's not a bug. It's 
not even a "huge problem". And if you'd stop banging your head against 
it it wouldn't be any sort of problem at all. Neither of us can change 
these facts.

I feel your pain man, but thats about it.

What more can I do?
What is it that you want?

Re: A note on spotting "bugs" [Was: ENOSPC after conversion]

[Patrik Lundquist]

On 12 December 2014 at 14:29, Robert White <rwhite@pobox.com> wrote:
>
> You yourself even found the annotation in the wiki that said you should have
> e4defragged the system before conversion.

There's no mention of e4defrag on the Btrfs wiki, it says to btrfs
defrag before balance to avoid ENOSPC, as the last step of conversion.


> What you are experiencing is a little vexing, but it's not a bug. It's not
> even a "huge problem". And if you'd stop banging your head against it it
> wouldn't be any sort of problem at all. Neither of us can change these
> facts.

I stopped banging my head several emails ago. I understand the problem
and I will start over.


> I feel your pain man, but thats about it.

I'm in no pain, it has been interesting. No data loss. No hurry.


> What more can I do?

The conversion wiki is lacking. It would be great if someone (maybe
you?) could expand upon the drawbacks of conversion.


> What is it that you want?

Nothing more.

Re: A note on spotting "bugs" [Was: ENOSPC after conversion]

[Duncan]

Robert White posted on Fri, 12 Dec 2014 05:29:58 -0800 as excerpted:

> This still doesnt say _anything_ is wrong with your filesystem except
> that it doesn't have enough _raw_ space to create a 2-ish gig extent.

What's wrong with the filesystem is that there shouldn't /be/ a need to 
create a 2-ish gig extent.  All btrfs native structures are 1 GiB each or 
smaller, and the completed-without-error btrfs fi defrag should have 
eliminated any > 1 GiB structures remaining from the conversion from 
ext*, such that btrfs balance only has to deal with <= 1 GiB structures.

So that balance is having to deal with a 2-ish gig extent at all is 
indicative of a bug.  Balance isn't prepared to have to allocate 2-ish 
GiB extents in the first place as that's beyond it's design specs.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: A note on spotting "bugs" [Was: ENOSPC after conversion]

[Robert White]

On 12/12/2014 05:12 PM, Duncan wrote:
> Robert White posted on Fri, 12 Dec 2014 05:29:58 -0800 as excerpted:
>
>> This still doesnt say _anything_ is wrong with your filesystem except
>> that it doesn't have enough _raw_ space to create a 2-ish gig extent.
>
> What's wrong with the filesystem is that there shouldn't /be/ a need to
> create a 2-ish gig extent.  All btrfs native structures are 1 GiB each or
> smaller, and the completed-without-error btrfs fi defrag should have
> eliminated any > 1 GiB structures remaining from the conversion from
> ext*, such that btrfs balance only has to deal with <= 1 GiB structures.
>
> So that balance is having to deal with a 2-ish gig extent at all is
> indicative of a bug.  Balance isn't prepared to have to allocate 2-ish
> GiB extents in the first place as that's beyond it's design specs.
>

Looking at the error message and the code, I would strongly disagree.

Balance has no hard-coded limit of 1-Gig.

The superblock has entries for the various sizes the filesystem uses.

Balance is _clearly_ asking for the chunk of that size and the system 
configuraion is _clearly_ considering that size legal.

The structures are only limited by their 64bit unsigned integer 
representation which is _way_ bigger than 2-ish gigs.

The actual code that controls the 1GiB extent allocation does a 
(paraphrased) :: min(requested,system_minimum_for_this_purpose)

Indeed, if it were a hard limit of 1GiB, then the message wouldn't even 
bother to say how much was requested, it would just say "Could not 
allocate data extent" because that extent would have both a minimum and 
maxium of the same known value.

Nowhere in or near the allocator does it do a test that I can find that 
would force the number to be smaller. And if it did, that wouldn't be a 
"ENOSPC" etc, it'd come back as a "too big" result such as ENOSUP.

So your statement that this is "beyond its design specs" isn't supported 
by the code. (That's why I looked it up the before I said "this is not a 
bug").

I went through the code until I accounted for _ever_ _word_ in the error 
message.

I don't see any kind of bug.

And now that we know that btrfs-convert has to make larger-than-one-GiB 
extents to encompass the block groups that are preexistent in EXT4, we 
also know that the sliding-block puzzle its trying to solve doesn't have 
nice round numbers of 1GiB and 256MiB, so the holes it leaves behind 
when it does move something are not nice and square and evenly sized etc.

So yea, he's jammed up and needs more space. As his filesystem churns 
all its new extents will be the 1GiB or 256MiB sized things and he'd 
slowly, but perhaps asymptotically, approach a clean layout of customary 
extent sizes.

It's not broken, it's just not pretty.

And _IF_ I had been saying it's "not a bug" and any of the actual code 
contributors had disagreed, they'd have jumped in here and shut me down 
(as they should in that case). e.g. If I was wrong about its non-bug 
status we'd have heard by now.