convert from RAID5: enospc errors during balance (500 GB free on each device)

convert from RAID5: enospc errors during balance (500 GB free on each device)

[Arjen Nienhuis]

I experimented with RAID5, but now I want to get rid of it:

$ sudo btrfs balance start -dconvert=raid1,soft -v /
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x300): converting, target=16, soft is on
ERROR: error during balancing '/' - No space left on device
There may be more info in syslog - try dmesg | tail

This is in dmesg:

[  166.799139] btrfs: relocating block group 13368320589824 flags 129
[  248.927524] btrfs: relocating block group 13338255818752 flags 129
...
[  409.184588] btrfs: relocating block group 11607454253056 flags 129
[  410.528075] btrfs: relocating block group 11605306769408 flags 129
[  411.957809] btrfs: 62 enospc errors during balance

Extra info:
Data, RAID1: total=3.26TiB, used=3.25TiB
Data, RAID5: total=124.00GiB, used=123.90GiB
System, RAID1: total=32.00MiB, used=508.00KiB
System, single: total=4.00MiB, used=0.00
Metadata, RAID1: total=9.00GiB, used=7.97GiB
Label: data  uuid: 93be47ab-af7a-4329-8a6c-c6ff220060b0
        Total devices 3 FS bytes used 3.38TiB
        devid    1 size 1.82TiB used 1.22TiB path /dev/sdd
        devid    2 size 3.64TiB used 2.63TiB path /dev/sdb
        devid    6 size 3.64TiB used 2.87TiB path /dev/sdc

Btrfs v3.12

Linux maxi 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC
2014 x86_64 x86_64 x86_64 GNU/Linux

I tried scrub (no errors) and I cleared the free space cache. Nothing worked.

Now I'm doing full balance (without 'soft') but that will take days.

Re: convert from RAID5: enospc errors during balance (500 GB free on each device)

[Duncan]

Arjen Nienhuis posted on Mon, 21 Apr 2014 08:32:56 +0200 as excerpted:

> I experimented with RAID5, but now I want to get rid of it:
> 
> $ sudo btrfs balance start -dconvert=raid1,soft -v /
> Dumping filters: flags 0x1, state 0x0, force is off
>   DATA (flags 0x300): converting, target=16, soft is on
> ERROR: error during balancing '/' - No space left on device There may be
> more info in syslog - try dmesg | tail
> 
> This is in dmesg:
> 
> [ 166.799139] btrfs: relocating block group 13368320589824 flags 129
> [ 248.927524] btrfs: relocating block group 13338255818752 flags 129 ...
> [ 409.184588] btrfs: relocating block group 11607454253056 flags 129
> [ 410.528075] btrfs: relocating block group 11605306769408 flags 129
> [ 411.957809] btrfs: 62 enospc errors during balance
> 
> Extra info:

> Data, RAID1: total=3.26TiB, used=3.25TiB
> Data, RAID5: total=124.00GiB, used=123.90GiB
> System, RAID1: total=32.00MiB, used=508.00KiB
> System, single: total=4.00MiB, used=0.00
> Metadata, RAID1: total=9.00GiB, used=7.97GiB 

> Label: data  uuid: 93be47ab-af7a-4329-8a6c-c6ff220060b0
>         Total devices 3 FS bytes used 3.38TiB
> devid    1 size 1.82TiB used 1.22TiB path /dev/sdd
> devid    2 size 3.64TiB used 2.63TiB path /dev/sdb
> devid    6 size 3.64TiB used 2.87TiB path /dev/sdc
> 
> Btrfs v3.12
> 
> Linux maxi 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014
> x86_64 x86_64 x86_64 GNU/Linux
> 
> I tried scrub (no errors) and I cleared the free space cache. Nothing
> worked.
> 
> Now I'm doing full balance (without 'soft') but that will take days.

Did you try running another balance soft?

Depending on how fully allocated you were originally, those 62 enospce 
errors during the first balance may have simply been before you cleared 
enough space to rewrite additional raid1 data.  You now have two devices 
with plenty of unallocated room on them (the btrfs fi show output), so 
rerunning the soft conversion to raid1 may well be all you needed to do.

Meanwhile... kernel v3.13, btrfs-progs v3.12.  You're just a bit behind, 
as kernel 3.14 is out (and 3.15 is rc2), and btrfs-progs is 3.14 
(possibly 3.14.1, I've not updated in a week or so).  When you run into 
problems, consider updating before reporting, as there are still very 
real fixes going into each new version, and they might just fix the 
problem you're reporting.

That said, I don't know if updating would help here, but it's worth a 
shot.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: convert from RAID5: enospc errors during balance (500 GB free on each device)

[Arjen Nienhuis]

> Did you try running another balance soft?

I tried a few. The only thing that happens is that more empty blocks
get allocated. I also tried convert=single,profiles=raid5. I ended up
with a few empty 'single' blocks. (The empty blocks could be discarded
with balance usage=0)

> Meanwhile... kernel v3.13, btrfs-progs v3.12.  You're just a bit behind,
> as kernel 3.14 is out (and 3.15 is rc2), and btrfs-progs is 3.14
> (possibly 3.14.1, I've not updated in a week or so).  When you run into
> problems, consider updating before reporting, as there are still very
> real fixes going into each new version, and they might just fix the
> problem you're reporting.

I just upgraded to Ubuntu 14.04. I can try a kernel ppa but I didn't
find anything in the about this on the wiki.

On Mon, Apr 21, 2014 at 8:32 AM, Arjen Nienhuis <a.g.nienhuis@gmail.com> wrote:
> I experimented with RAID5, but now I want to get rid of it:
>
> $ sudo btrfs balance start -dconvert=raid1,soft -v /
> Dumping filters: flags 0x1, state 0x0, force is off
>   DATA (flags 0x300): converting, target=16, soft is on
> ERROR: error during balancing '/' - No space left on device
> There may be more info in syslog - try dmesg | tail
>
> This is in dmesg:
>
> [  166.799139] btrfs: relocating block group 13368320589824 flags 129
> [  248.927524] btrfs: relocating block group 13338255818752 flags 129
> ...
> [  409.184588] btrfs: relocating block group 11607454253056 flags 129
> [  410.528075] btrfs: relocating block group 11605306769408 flags 129
> [  411.957809] btrfs: 62 enospc errors during balance
>
> Extra info:
> Data, RAID1: total=3.26TiB, used=3.25TiB
> Data, RAID5: total=124.00GiB, used=123.90GiB
> System, RAID1: total=32.00MiB, used=508.00KiB
> System, single: total=4.00MiB, used=0.00
> Metadata, RAID1: total=9.00GiB, used=7.97GiB
> Label: data  uuid: 93be47ab-af7a-4329-8a6c-c6ff220060b0
>         Total devices 3 FS bytes used 3.38TiB
>         devid    1 size 1.82TiB used 1.22TiB path /dev/sdd
>         devid    2 size 3.64TiB used 2.63TiB path /dev/sdb
>         devid    6 size 3.64TiB used 2.87TiB path /dev/sdc
>
> Btrfs v3.12
>
> Linux maxi 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC
> 2014 x86_64 x86_64 x86_64 GNU/Linux
>
> I tried scrub (no errors) and I cleared the free space cache. Nothing worked.
>
> Now I'm doing full balance (without 'soft') but that will take days.

Re: convert from RAID5: enospc errors during balance (500 GB free on each device)

[Duncan]

Arjen Nienhuis posted on Tue, 22 Apr 2014 23:52:43 +0200 as excerpted:

>> Did you try running another balance soft?
> 
> I tried a few. The only thing that happens is that more empty blocks get
> allocated. I also tried convert=single,profiles=raid5. I ended up with a
> few empty 'single' blocks. (The empty blocks could be discarded with
> balance usage=0)

Hmm... So the blocks of the new type get allocated, but nothing gets 
written into them.  That's strange.

I'm not a dev just a btrfs user and list regular, and this is the first 
time I've seen that sort of behavior reported so I can't help much 
further there, but the reason I suggested rerunning the balance is that 
I've had times when I got ENOSPC errors for some chunks on the first 
balance, but it did free enough space from the ones it could rewrite that 
a second pass went just fine, no errors, and since you didn't mention 
retrying in the original post, I thought I would.

The one similar report we did have was with someone that had originally 
done the conversion from ext3/4 following the instructions in the wiki, 
and had removed the ext3/4 subvolume after conversion, but had *NOT* then 
run the defrag and rebalance as recommended.  IIRC he was actually trying 
a later btrfs mode convert (IIRC from single to raid1, after adding a 
second device) as well, when he ran into problems.  Those problems were 
traced to the fact that btrfs data chunks are 1 GiB in size, so that's 
the largest possible extent on native btrfs, while ext3/4 allowed larger 
extents.  The problems thus occurred when the conversion process ran into 
these larger extents, which the original btrfs single mode handled, but 
which the conversion to raid1, copying to the other device in now native 
btrfs mode, couldn't manage.

He ended up having to move a bunch of files off of the btrfs (to tmpfs in 
his case, the files were multi-gig but he had the memory to handle it) 
and then back onto it, in ordered to have them recreated in btrfs-native 
1-GiB data chunks mode.  I don't recall whether they were written back in 
raid1 or single mode, but in any case, once he dealt with the problem 
files that way, the conversion finished properly.

So on the off chance... your filesystem wasn't also originally converted 
from ext3/4 to btrfs, was it?  If so, it might be that issue.  Otherwise 
I don't have a clue.  Maybe the devs can help.

>> Meanwhile... kernel v3.13, btrfs-progs v3.12.  You're just a bit
>> behind, as kernel 3.14 is out (and 3.15 is rc2), and btrfs-progs is
>> 3.14 (possibly 3.14.1, I've not updated in a week or so).  When you
>> run into problems, consider updating before reporting, as there are
>> still very real fixes going into each new version, and they might just
>> fix the problem you're reporting.
> 
> I just upgraded to Ubuntu 14.04. I can try a kernel ppa but I didn't
> find anything in the about this on the wiki.

When you do a mkfs.btrfs, the output there warns that btrfs is still 
experimental, run current kernels, keep backups, etc.  (I'm not creating 
one ATM so I unlike the wiki I don't have that right in front of me, but 
I just redid my backups with a new mkfs.btrfs a few days ago and the 
experimental warning was still there).  The wiki also strongly encourages 
both.  See the stuff in red right near the top of the getting started 
page:

https://btrfs.wiki.kernel.org/index.php/Getting_started

Further down the page there's both a distros vs shipped versions table, 
and a discussion of building from source when the distro version is 
behind.

https://btrfs.wiki.kernel.org/index.php/
Getting_started#Compiling_Btrfs_from_sources

Also see stability status, which emphasizes running the latest stable if 
not the development version, on the main page:

https://btrfs.wiki.kernel.org/index.php/Main_Page#Stability_status

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman