btrfs balance fails with no space errors (despite having plenty)

btrfs balance fails with no space errors (despite having plenty)

[Kline, Matthew]

Hi all,

Yesterday I converted my ext4 root and home partitions on my home machine
to btrfs using btrfs-convert. After confirming that everything went well,
I followed the wiki instructions to nuke the 'ext2_saved" subvolume,
then defraggad and rebalanced. Everything went according to plan on my root
partition, but my home partition claimed to have run out of space
when rebalancing.

I did some digging and tried the following to resolve the problem,
but to no avail. So far I have:

- Made sure I deleted the subvolume (a common cause of this problem).
  `sudo btrfs subvolume list -a /home` exits with no output.

- Made sure I defragged the newly converted btrfs partition before attempting
  to rebalance it.

- Made sure that I actually have space on the partition.
  It is only about 60% full - see sdb1 below:

  ~ % sudo btrfs fi show
  Label: none  uuid: 3a154348-9bd4-4c3f-aaf8-e9446d3797db
      Total devices 1 FS bytes used 9.49GiB
      devid    1 size 87.47GiB used 11.03GiB path /dev/sda3

  Label: none  uuid: cc90ee50-bbda-46d6-a7e6-fe1c8578d75b
      Total devices 1 FS bytes used 124.98GiB
      devid    1 size 200.00GiB used 127.03GiB path /dev/sdb1

- Made sure I have metadata space (as suggested on the problem FAQ):

  ~ % sudo btrfs fi df /home
  Data, single: total=126.00GiB, used=124.58GiB
  System, single: total=32.00MiB, used=20.00KiB
  Metadata, single: total=1.00GiB, used=404.27MiB
  GlobalReserve, single: total=136.00MiB, used=0.00B

- Ran partial rebalances using the `-dusage` flag
  (as suggested on the problem FAQ),
  which successfully balanced a handful of blocks.

- Checked the system log - nothing interesting comes up.
  btrfs happily chugs along with "found xxx extents" and
  "relocating block group xxxxx flags 1" messages before unceremoniously
  ending with "7 enospc errors during balance".

In spite of all of this, a full rebalance still fails when it's about 95% done.
I'm at a complete loss as to what could be causing it - I know that it's not
completely necessary (especially with a single drive),
and `btrfs scrub` finds no errors with the file system,
but the wiki gives the impression that it's a good idea after you convert from
ext.

Is there something I'm missing?

Best,
Matt Kline

Other obligatory info: I'm on Arch Linux using btrfs 3.17.1. uname -a is

Linux kline-arch 3.17.2-1-ARCH #1 SMP PREEMPT Thu Oct 30 20:49:39 CET 2014
x86_64 GNU/Linux

Re: btrfs balance fails with no space errors (despite having plenty)

[Oon-Ee Ng]

On Thu, Nov 13, 2014 at 2:48 AM, Kline, Matthew
<matthew.kline@flukenetworks.com> wrote:
> Hi all,
>
> Yesterday I converted my ext4 root and home partitions on my home machine
> to btrfs using btrfs-convert. After confirming that everything went well,
> I followed the wiki instructions to nuke the 'ext2_saved" subvolume,
> then defraggad and rebalanced. Everything went according to plan on my root
> partition, but my home partition claimed to have run out of space
> when rebalancing.

I had a similar problem on a btrfs partition (not a converted one) and
had to add a separate partition's space to provide enough
space/metadata space to successfully do a balance. In my case, just a
4GB thumb drive.

Re: btrfs balance fails with no space errors (despite having plenty)

[Duncan]

Kline, Matthew posted on Wed, 12 Nov 2014 18:48:47 +0000 as excerpted:

> Hi all,
> 
> Yesterday I converted my ext4 root and home partitions on my home machine
> to btrfs using btrfs-convert. After confirming that everything went well,
> I followed the wiki instructions to nuke the 'ext2_saved" subvolume,
> then defraggad and rebalanced. Everything went according to plan on my root
> partition, but my home partition claimed to have run out of space
> when rebalancing.
> 
> I did some digging and tried the following to resolve the problem,
> but to no avail. So far I have:
> 
> - Made sure I deleted the subvolume (a common cause of this problem).
>   `sudo btrfs subvolume list -a /home` exits with no output.
> 
> - Made sure I defragged the newly converted btrfs partition before
>   attempting to rebalance it.
> 
> - Made sure that I actually have space on the partition.
>   It is only about 60% full - see sdb1 below:
> 
>   ~ % sudo btrfs fi show
>   Label: none  uuid: 3a154348-9bd4-4c3f-aaf8-e9446d3797db
>       Total devices 1 FS bytes used 9.49GiB
>       devid    1 size 87.47GiB used 11.03GiB path /dev/sda3
> 
>   Label: none  uuid: cc90ee50-bbda-46d6-a7e6-fe1c8578d75b
>       Total devices 1 FS bytes used 124.98GiB
>       devid    1 size 200.00GiB used 127.03GiB path /dev/sdb1
> 
> - Made sure I have metadata space (as suggested on the problem FAQ):
> 
>   ~ % sudo btrfs fi df /home
>   Data, single: total=126.00GiB, used=124.58GiB
>   System, single: total=32.00MiB, used=20.00KiB
>   Metadata, single: total=1.00GiB, used=404.27MiB
>   GlobalReserve, single: total=136.00MiB, used=0.00B
> 
> - Ran partial rebalances using the `-dusage` flag
>   (as suggested on the problem FAQ),
>   which successfully balanced a handful of blocks.
> 
> - Checked the system log - nothing interesting comes up.
>   btrfs happily chugs along with "found xxx extents" and
>   "relocating block group xxxxx flags 1" messages before unceremoniously
>   ending with "7 enospc errors during balance".
> 
> In spite of all of this, a full rebalance still fails when it's about
> 95% done.  I'm at a complete loss as to what could be causing it - I know
> that it's not completely necessary (especially with a single drive),
> and `btrfs scrub` finds no errors with the file system,
> but the wiki gives the impression that it's a good idea after you
> convert from ext.
> 
> Is there something I'm missing?
> 
> Best,
> Matt Kline
> 
> Other obligatory info: I'm on Arch Linux using btrfs 3.17.1. uname -a is
> 
> Linux kline-arch 3.17.2-1-ARCH #1 SMP PREEMPT
> Thu Oct 30 20:49:39 CET 2014 x86_64 GNU/Linux

Hmm...  You've jumped thru all the conversion hoops, including deleting
the old ext4 saved subvolume, and a defrag after, and it's still not
working.  You're absolutely right that a balance is a good idea at this
point, because once it completes successfully, you know everything made
it thru successfully into native btrfs format, so that a balance is
failing is a good hint that /something/ didn't get fully converted to
native btrfs format yet, despite all those hoops.

I'd guess something went wrong with the defrag.  Here's the deal with it.
Hope you don't have too many files over a GiB (but have at least one,
or the problem must be something else)...

Ext3/4 filesystems can have extents far larger than those on native
btrfs, which are limited to the 1 GiB data-chunk size.  What the initial
btrfs conversion does is simply create btrfs metadata pointing at the
existing ext3/4 data, placing the new btrfs metadata chunks (which are
a quarter GiB in size) in what was the free space on the ext* fs.  The
ext* metadata remains in place, along with the data with both the ext*
and the btrfs metadata pointed at it.  The ext* data and metadata
together form the ext2_saved subvolume, and because btrfs is
copy-on-write, aka COW, any changes made to the data on the btrfs side
are automatically written elsewhere, leaving the existing data (and
ext* metadata) untouched in the ext2_saved subvolume.

So as it is when converted, data extents can still be far larger than
the btrfs-native 1 GiB data extent, and balance can't deal with that.

After the ext2_saved subvolume is removed and there's no going back
to the ext*, it's safe to break up those > 1 GiB non-native-btrfs
data extents, and that's what the defrag is SUPPOSED to do.  Btrfs
defrag rewrites files, normally combining smaller extents into larger
ones upto the 1 GiB data-chunk size, but in this case, it also splits
apart these non-native > 1 GiB super-extents, rewriting them into
native btrfs chunks.

There is, however, a catch.  At one point btrfs defrag was
snapshot aware, but the algorithm used for it matched against
that used for quotas and for snapshots proved unable to scale.
So the snapshot awareness has been disabled again for a few kernel
cycles now, while they rewrite various btrfs internals to scale
better.

So ideally, you'd have no snapshots and no subvolumes when
you did that first defrag.  You mentioned deleting subvolumes
and presumably you deleted snapshots (which are subvolume
special-case) if any as well.

But, did you do that defrag when they were still there?
If so, that /might/ be the problem.

It's also possible there's some other bug in defrag
ATM, and for whatever reason it failed to rewrite some of
those super-extents.

Either way, assuming it's the super-extent problem which I
think it is, other than trying a defrag again, one way to
for sure solve that problem is to make a list of all your
over 1 GiB files, and move them out of the filesystem and
then back in, so they get recreated in native btrfs format.
You can use a find with file-size, or a du -ah piped to
sort -h, or something like filelight, to find the > 1 GiB
files.  Just be sure to get all of them on the filesystem,
because of course if you miss one, that's going to be the
one that was triggering the problem. =:^\

If you have more spare memory than the largest of
these files, probably the fastest way to do this is to
copy them one or a few at a time, to fill memory, into
a tmpfs.  Since tmpfs is memory, unless you go into swap
this should be quite fast.

Then move them back, either to a different filename or
to a different subdir.  Once they are moved back, sync
or btrfs filesystem sync that filesystem, then delete
the old copy from the filesystem.  Doing it this way you
are sure to get rid of the old copy, creating a new
native btrfs copy, but by syncing after the new one
is there before you actually delete the old one, you
should stay safe in case the power goes out at the wrong
time and you lose the content of the tmpfs, because
you will not have actually deleted the old file until
the new one is safely written and you've synced.

Of course when you are done you can move the files back
to their original names/locations.

Another alternative, of course, would be to use either
the other btrfs (root, since it's /home with the problem,
assuming it's large enough), or a thumb-drive large enough
to hold the files.  This will probably take longer than
copies to and from memory, but if memory is too small,
or you have enough room on root or the thumb-drive to
copy everything over at once and you don't have memory
for that, it's a reasonable alternative.


If moving all > 1 GiB files off of the filesystem and
then back on, thus recreating them in native btrfs format,
doesn't help, then either you missed one and it's still
triggering the issue, or there's something else going
on.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: btrfs balance fails with no space errors (despite having plenty)

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 3582 bytes --]

On Wed, Nov 12, 2014 at 06:48:47PM +0000, Kline, Matthew wrote:
> Yesterday I converted my ext4 root and home partitions on my home machine
> to btrfs using btrfs-convert. After confirming that everything went well,
> I followed the wiki instructions to nuke the 'ext2_saved" subvolume,
> then defraggad and rebalanced. Everything went according to plan on my root
> partition, but my home partition claimed to have run out of space
> when rebalancing.
> 
> I did some digging and tried the following to resolve the problem,
> but to no avail. So far I have:
> 
> - Made sure I deleted the subvolume (a common cause of this problem).
>   `sudo btrfs subvolume list -a /home` exits with no output.
> 
> - Made sure I defragged the newly converted btrfs partition before attempting
>   to rebalance it.
> 
> - Made sure that I actually have space on the partition.
>   It is only about 60% full - see sdb1 below:
> 
>   ~ % sudo btrfs fi show
>   Label: none  uuid: 3a154348-9bd4-4c3f-aaf8-e9446d3797db
>       Total devices 1 FS bytes used 9.49GiB
>       devid    1 size 87.47GiB used 11.03GiB path /dev/sda3
> 
>   Label: none  uuid: cc90ee50-bbda-46d6-a7e6-fe1c8578d75b
>       Total devices 1 FS bytes used 124.98GiB
>       devid    1 size 200.00GiB used 127.03GiB path /dev/sdb1
> 
> - Made sure I have metadata space (as suggested on the problem FAQ):
> 
>   ~ % sudo btrfs fi df /home
>   Data, single: total=126.00GiB, used=124.58GiB
>   System, single: total=32.00MiB, used=20.00KiB
>   Metadata, single: total=1.00GiB, used=404.27MiB
>   GlobalReserve, single: total=136.00MiB, used=0.00B
> 
> - Ran partial rebalances using the `-dusage` flag
>   (as suggested on the problem FAQ),
>   which successfully balanced a handful of blocks.
> 
> - Checked the system log - nothing interesting comes up.
>   btrfs happily chugs along with "found xxx extents" and
>   "relocating block group xxxxx flags 1" messages before unceremoniously
>   ending with "7 enospc errors during balance".
> 
> In spite of all of this, a full rebalance still fails when it's about 95% done.
> I'm at a complete loss as to what could be causing it - I know that it's not
> completely necessary (especially with a single drive),
> and `btrfs scrub` finds no errors with the file system,
> but the wiki gives the impression that it's a good idea after you convert from
> ext.

   I'm fairly sure it's a bug. We've had several reports of something
like this, particularly with respect to converted filesystems. I know
that failed balances with apparently plenty of space are reasonably
high up on josef's list of things to investigate. I would recommend at
minimum putting this report on bugzilla.kernel.org as well. The other
thing you could do, which may help with debugging, is to take a copy
of the metadata:

$ btrfs-image -c9 -t4 /dev/sdb1 /path/to/image.img

and hang on to it just in case josef (or whoever else looks at it)
needs another sample to debug with.

> Is there something I'm missing?

   I don't think so. This is probably a bug -- I'd guess in the FS
(because we've seen something similar on non-converted FSes), but
maybe set up more easily by the conversion process.

   Hugo.

> Other obligatory info: I'm on Arch Linux using btrfs 3.17.1. uname -a is
> 
> Linux kline-arch 3.17.2-1-ARCH #1 SMP PREEMPT Thu Oct 30 20:49:39 CET 2014
> x86_64 GNU/Linux

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
                       --- Strive for apathy! ---                        

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: btrfs balance fails with no space errors (despite having plenty)

[Kline, Matthew]

Sorry to necrobump this, but the issue came up again.
I'm now on Linux 3.19.2 with btrfs-progs 3.19, and I converted my work laptop
from ext4 to btrfs. Same issue - the conversion goes well,
removing the ext* subvolume is fine, defragging is works,
and then when I balance, it claims there is no space left on the device
when the drive is 35% full. Dmesg is useless once again,
giving only "1 enospc errors during balance".

The last time this was an issue (the original post in this thread),
I ended up just copying everything to an external drive,
reformatting the btrfs partition, and copying things back.

This time I did what was suggested below (moving all +1GB files off-drive
to tmpfs and then back on) in case defrag missed something, and balance
still fails. I was sure to rename the files and sync as suggested.

Is there a known bug still lurking in ext4 conversion?

Duncan <1i5t5.duncan <at> cox.net> writes:

> Either way, assuming it's the super-extent problem which I
> think it is, other than trying a defrag again, one way to
> for sure solve that problem is to make a list of all your
> over 1 GiB files, and move them out of the filesystem and
> then back in, so they get recreated in native btrfs format.
> 
> If moving all > 1 GiB files off of the filesystem and
> then back on, thus recreating them in native btrfs format,
> doesn't help, then either you missed one and it's still
> triggering the issue, or there's something else going
> on.
> 

Re: btrfs balance fails with no space errors (despite having plenty)

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 1422 bytes --]

On Tue, 24 Mar 2015 22:25:30 +0000
"Kline, Matthew" <matthew.kline@flukenetworks.com> wrote:

> Sorry to necrobump this, but the issue came up again.
> I'm now on Linux 3.19.2 with btrfs-progs 3.19, and I converted my work laptop
> from ext4 to btrfs. Same issue - the conversion goes well,
> removing the ext* subvolume is fine, defragging is works,
> and then when I balance, it claims there is no space left on the device
> when the drive is 35% full. Dmesg is useless once again,
> giving only "1 enospc errors during balance".
> 
> The last time this was an issue (the original post in this thread),
> I ended up just copying everything to an external drive,
> reformatting the btrfs partition, and copying things back.
> 
> This time I did what was suggested below (moving all +1GB files off-drive
> to tmpfs and then back on) in case defrag missed something, and balance
> still fails. I was sure to rename the files and sync as suggested.
> 
> Is there a known bug still lurking in ext4 conversion?

Post your "btrfs fi df" for the filesystem.

In my experience it looks like the conversion may designate large
portions of data as "metadata" for whatever reason.

A way to clean this up is to run

  btrfs fi balance start -musage=5 -dusage=5 /path

try repeating this again and again increasing the value gradually from 5 to
50-70 in steps of 5 or 10.

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

RE: btrfs balance fails with no space errors (despite having plenty)

[Kline, Matthew]

> Post your "btrfs fi df" for the filesystem.
>
> In my experience it looks like the conversion may designate large portions of data as "metadata" for whatever reason.
>
> A way to clean this up is to run
>
>  btrfs fi balance start -musage=5 -dusage=5 /path
>
> try repeating this again and again increasing the value gradually from 5 to
> 50-70 in steps of 5 or 10.

Data, single: total=44.00GiB, used=43.37GiB
System, single: total=32.00MiB, used=12.00KiB
Metadata, single: total=2.00GiB, used=876.84MiB
GlobalReserve, single: total=296.00MiB, used=0.00B

Total partition size is 322 GB, as given by `parted -l` (I misspoke in the last email - the drive is only 14% full)

Best,
Matt

Re: btrfs balance fails with no space errors (despite having plenty)

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1392 bytes --]

On Tue, Mar 24, 2015 at 11:09:47PM +0000, Kline, Matthew wrote:
> > Post your "btrfs fi df" for the filesystem.
> >
> > In my experience it looks like the conversion may designate large portions of data as "metadata" for whatever reason.
> >
> > A way to clean this up is to run
> >
> >  btrfs fi balance start -musage=5 -dusage=5 /path
> >
> > try repeating this again and again increasing the value gradually from 5 to
> > 50-70 in steps of 5 or 10.
> 
> Data, single: total=44.00GiB, used=43.37GiB
> System, single: total=32.00MiB, used=12.00KiB
> Metadata, single: total=2.00GiB, used=876.84MiB
> GlobalReserve, single: total=296.00MiB, used=0.00B
> 
> Total partition size is 322 GB, as given by `parted -l` (I misspoke in the last email - the drive is only 14% full)

   Yeah, this smells like the known bug that's been on josef's hitlist
for months now. He keeps getting side-tracked by data-corruption bugs,
so it's not been fixed yet, unfortunately. The only known solution to
date is, as you've found, remaking the FS from scratch and restoring
from backups.

   Sorry to be the bearer of bad news.

   Hugo.

-- 
Hugo Mills             | Geek, n.:
hugo@... carfax.org.uk | Circus sideshow performer specialising in the eating
http://carfax.org.uk/  | of live animals.
PGP: 65E74AC0          |                                                   OED

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

RE: btrfs balance fails with no space errors (despite having plenty)

[Kline, Matthew]

>  Sorry to be the bearer of bad news.

>  Hugo.

Thanks for the prompt response! It's annoying, but copying to and from an external hard drive while I eat dinner isn't the end of the world.

Thanks again,
Matt

Re: btrfs balance fails with no space errors (despite having plenty)

[Russell Coker]

I've been having ongoing issues with balance failing with no space errors in spite of having plenty. Strangely it seems to most often happen from cron jobs, when a cron job fails I can count on a manual balance succeeding.

I'm running the latest Debian/Jessie kernel.
-- 
Sent from my Samsung Galaxy Note 3 with K-9 Mail.

Re: btrfs balance fails with no space errors (despite having plenty)

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 607 bytes --]

On Tue, 24 Mar 2015 23:09:47 +0000
"Kline, Matthew" <matthew.kline@flukenetworks.com> wrote:

> Data, single: total=44.00GiB, used=43.37GiB
> System, single: total=32.00MiB, used=12.00KiB
> Metadata, single: total=2.00GiB, used=876.84MiB
> GlobalReserve, single: total=296.00MiB, used=0.00B
> 
> Total partition size is 322 GB, as given by `parted -l` (I misspoke in the last email - the drive is only 14% full)

Okay that's some different issue then, doesn't look like what I had, where on
a 9 TB FS after conversion I got 2 TB of metadata (with 1.2TB free space).

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]