Problem converting data raid0 to raid1: enospc errors during balance

Problem converting data raid0 to raid1: enospc errors during balance

[Jasper Verberk]

Hello,

I'm trying to change my 4 disk btrfs data from raid0 to raid1. The metadata is allready in raid1 and now I'm trying to also make the data raid1.

I'm getting the following errors and I got told on irc to go report this as a bug.

Hoping there is anybody that can give me a hand here in solving it.


-------------------------------
Command I issued was:

  btrfs balance start -dconvert=raid1,soft /data
-------------------------------

-------------------------------
Error I got:

  ERROR: error during balancing '/data' - No space left on device
  There may be more info in syslog - try dmesg | tail
-------------------------------

-------------------------------
Last lines of dmesg:

[60098.584459] BTRFS info (device sdc): relocating block group 18282971136 flags 9
[60102.422310] BTRFS info (device sdc): relocating block group 5398069248 flags 9
[60102.776406] BTRFS info (device sdc): relocating block group 1103101952 flags 9
[60103.168554] BTRFS info (device sdc): relocating block group 4194304 flags 4
[60103.619552] BTRFS info (device sdc): relocating block group 0 flags 2
[60104.040776] BTRFS info (device sdc): 1046 enospc errors during balance
-------------------------------

-------------------------------
Further info:

root@BlackMesa:/mnt# btrfs fi show
Label: 'data'  uuid: a1e36575-0843-45ab-850e-618a5d8a4b7e
        Total devices 4 FS bytes used 2.75TiB
        devid    1 size 2.73TiB used 1.94TiB path /dev/sdb
        devid    2 size 2.73TiB used 1.94TiB path /dev/sdd
        devid    3 size 2.73TiB used 1.94TiB path /dev/sdc
        devid    4 size 2.73TiB used 1.94TiB path /dev/sde


root@BlackMesa:/mnt# btrfs fi df /data
Data, RAID1: total=2.85TiB, used=790.46GiB
Data, RAID0: total=2.04TiB, used=1.97TiB
System, RAID1: total=8.00MiB, used=612.00KiB
Metadata, RAID1: total=5.00GiB, used=3.39GiB
unknown, single: total=512.00MiB, used=0.00

root@BlackMesa:/mnt# uname -a
Linux BlackMesa 3.17-1-amd64 #1 SMP Debian 3.17-1~exp1 (2014-10-14) x86_64 GNU/Linux

root@BlackMesa:/mnt# btrfs --version
Btrfs v3.14.1
-------------------------------

-------------------------------
And this is what happens when I try to make an image:

root@BlackMesa:/mnt# btrfs-image -c9 -t4 /dev/sdb /home/jasper/dump.image
btrfs-image: disk-io.c:155: readahead_tree_block: Assertion `!(ret)' failed.
Aborted
------------------------------- 		 	   		  

Re: Problem converting data raid0 to raid1: enospc errors during balance

[Chris Murphy]

On Oct 24, 2014, at 7:32 AM, Jasper Verberk <jverberk@hotmail.com> wrote:
> 
> [60104.040776] BTRFS info (device sdc): 1046 enospc errors during balance


To get more information, remount with enospc_debug option, and try to convert again.

> root@BlackMesa:/mnt# btrfs --version
> Btrfs v3.14.1

There are many fixes since this version, I suggest going to 3.17 which also has a lot of new fsck code in case that ends up being needed. You can try btrfs-image again. Then see if btrfs check (without --repair) shows anything suspicious.


Chris Murphy

RE: Problem converting data raid0 to raid1: enospc errors during balance

[Jasper Verberk]

Did that, you can find the result in the pastebin.

http://paste.debian.net/128552/

----------------------------------------
> Subject: Re: Problem converting data raid0 to raid1: enospc errors during balance
> From: lists@colorremedies.com
> Date: Fri, 24 Oct 2014 11:11:21 -0600
> To: linux-btrfs@vger.kernel.org; jverberk@hotmail.com
>
>
> On Oct 24, 2014, at 7:32 AM, Jasper Verberk <jverberk@hotmail.com> wrote:
>>
>> [60104.040776] BTRFS info (device sdc): 1046 enospc errors during balance
>
>
> To get more information, remount with enospc_debug option, and try to convert again.
>
>> root@BlackMesa:/mnt# btrfs --version
>> Btrfs v3.14.1
>
> There are many fixes since this version, I suggest going to 3.17 which also has a lot of new fsck code in case that ends up being needed. You can try btrfs-image again. Then see if btrfs check (without --repair) shows anything suspicious.
>
>
> Chris Murphy
>
 		 	   		  

Re: Problem converting data raid0 to raid1: enospc errors during balance

[Chris Murphy]

On Oct 24, 2014, at 11:49 AM, Jasper Verberk <jverberk@hotmail.com> wrote:

> Did that, you can find the result in the pastebin.
> 
> http://paste.debian.net/128552/

Maybe Josef or David will have some idea.

I'd still take a btrfs-image and btrfs check with current btrfs-progs.

Chris Murphy

Re: Problem converting data raid0 to raid1: enospc errors during balance

[Qu Wenruo]

Hi,

Although I'm not completely sure, but it seems that, you really ran out 
of space.

[1] Your array won't hold raid1 for 1.97T data
Your array used up 1.97T raid0 data, it takes 1.97T for raid0.
But if converted to 1.97T, it will occupy 1.97T X2 = 3.94T.
Your array are only 2.73T, it is too small to contain the data.

[2]No unallocated free space
When doing convert, btrfs will *NOT* convert the data/metadata in-place, 
it allocate chunk and move data,
which means, if btrfs wants to convert it needs to allocate some raid1 
data chunk for the convert,
and then copy the raid0 data to raid1 data chunk.

However, your 'btrfs fi show' output shows that there is no unallocated 
free space for raid1 data chunk allocation.
(2.75T space all used up for chunk allocation).
Although your 'btrfs fi df' shows there is still about 800G free space, 
that's just free space inside data/medata chunk,
and chunk allocation needs completely free space, which is not allocated 
for any chunk type.
If you really want to free some space from the unused space, you can try 
balance with -d/-m options,
but it won't really do much help on your case due to [1].

[Conculsion]
So the ENOSPC error is not a false alert.
Add more devices (easiest but expensive) or free up some space and then 
use balance with -d/-m
to free up some free space(balance speed maybe slow) seems to be the 
only 2 ways to finish your RAID1
setup.

Thanks,
Qu


-------- Original Message --------
Subject: Problem converting data raid0 to raid1: enospc errors during 
balance
From: Jasper Verberk <jverberk@hotmail.com>
To: linux-btrfs@vger.kernel.org <linux-btrfs@vger.kernel.org>
Date: 2014年10月24日 21:32
> Hello,
>
> I'm trying to change my 4 disk btrfs data from raid0 to raid1. The metadata is allready in raid1 and now I'm trying to also make the data raid1.
>
> I'm getting the following errors and I got told on irc to go report this as a bug.
>
> Hoping there is anybody that can give me a hand here in solving it.
>
>
> -------------------------------
> Command I issued was:
>
>    btrfs balance start -dconvert=raid1,soft /data
> -------------------------------
>
> -------------------------------
> Error I got:
>
>    ERROR: error during balancing '/data' - No space left on device
>    There may be more info in syslog - try dmesg | tail
> -------------------------------
>
> -------------------------------
> Last lines of dmesg:
>
> [60098.584459] BTRFS info (device sdc): relocating block group 18282971136 flags 9
> [60102.422310] BTRFS info (device sdc): relocating block group 5398069248 flags 9
> [60102.776406] BTRFS info (device sdc): relocating block group 1103101952 flags 9
> [60103.168554] BTRFS info (device sdc): relocating block group 4194304 flags 4
> [60103.619552] BTRFS info (device sdc): relocating block group 0 flags 2
> [60104.040776] BTRFS info (device sdc): 1046 enospc errors during balance
> -------------------------------
>
> -------------------------------
> Further info:
>
> root@BlackMesa:/mnt# btrfs fi show
> Label: 'data'  uuid: a1e36575-0843-45ab-850e-618a5d8a4b7e
>          Total devices 4 FS bytes used 2.75TiB
>          devid    1 size 2.73TiB used 1.94TiB path /dev/sdb
>          devid    2 size 2.73TiB used 1.94TiB path /dev/sdd
>          devid    3 size 2.73TiB used 1.94TiB path /dev/sdc
>          devid    4 size 2.73TiB used 1.94TiB path /dev/sde
>
>
> root@BlackMesa:/mnt# btrfs fi df /data
> Data, RAID1: total=2.85TiB, used=790.46GiB
> Data, RAID0: total=2.04TiB, used=1.97TiB
> System, RAID1: total=8.00MiB, used=612.00KiB
> Metadata, RAID1: total=5.00GiB, used=3.39GiB
> unknown, single: total=512.00MiB, used=0.00
>
> root@BlackMesa:/mnt# uname -a
> Linux BlackMesa 3.17-1-amd64 #1 SMP Debian 3.17-1~exp1 (2014-10-14) x86_64 GNU/Linux
>
> root@BlackMesa:/mnt# btrfs --version
> Btrfs v3.14.1
> -------------------------------
>
> -------------------------------
> And this is what happens when I try to make an image:
>
> root@BlackMesa:/mnt# btrfs-image -c9 -t4 /dev/sdb /home/jasper/dump.image
> btrfs-image: disk-io.c:155: readahead_tree_block: Assertion `!(ret)' failed.
> Aborted
> ------------------------------- 		 	   		  --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem converting data raid0 to raid1: enospc errors during balance

[Chris Murphy]

On Oct 26, 2014, at 7:40 PM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:

> Hi,
> 
> Although I'm not completely sure, but it seems that, you really ran out of space.
> 
> [1] Your array won't hold raid1 for 1.97T data
> Your array used up 1.97T raid0 data, it takes 1.97T for raid0.
> But if converted to 1.97T, it will occupy 1.97T X2 = 3.94T.
> Your array are only 2.73T, it is too small to contain the data.

I'm not understanding. The btrfs fi show, shows 4x 2.73TiB devices, so that seems like it's a 10+TiB array.

There's 2.04TiB raid0 data chunks, so roughly 500GiB per device, yet 1.94TiB is reported used per device by fi show. Confusing.

Also it's still very confusing: Data, RAID1: total=2.85TiB, used=790.46GiB whether this means 2.85TiB out of 10TiB is allocated, or if it's twice that due to raid1. I can't ever remember this presentation detail, so again the secret decode ring where the UI doesn't expressly tell us what's going on is going to continue to be a source of confusion for users.


Chris Murphy

Re: Problem converting data raid0 to raid1: enospc errors during balance

[Qu Wenruo]

-------- Original Message --------
Subject: Re: Problem converting data raid0 to raid1: enospc errors 
during balance
From: Chris Murphy <lists@colorremedies.com>
To: Qu Wenruo <quwenruo@cn.fujitsu.com>
Date: 2014年10月27日 12:40
> On Oct 26, 2014, at 7:40 PM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>
>> Hi,
>>
>> Although I'm not completely sure, but it seems that, you really ran out of space.
>>
>> [1] Your array won't hold raid1 for 1.97T data
>> Your array used up 1.97T raid0 data, it takes 1.97T for raid0.
>> But if converted to 1.97T, it will occupy 1.97T X2 = 3.94T.
>> Your array are only 2.73T, it is too small to contain the data.
> I'm not understanding. The btrfs fi show, shows 4x 2.73TiB devices, so that seems like it's a 10+TiB array.
>
> There's 2.04TiB raid0 data chunks, so roughly 500GiB per device, yet 1.94TiB is reported used per device by fi show. Confusing.
>
> Also it's still very confusing: Data, RAID1: total=2.85TiB, used=790.46GiB whether this means 2.85TiB out of 10TiB is allocated, or if it's twice that due to raid1. I can't ever remember this presentation detail, so again the secret decode ring where the UI doesn't expressly tell us what's going on is going to continue to be a source of confusion for users.
>
>
> Chris Murphy
Oh, I misread the output....

That turns strange now....
BTW what's the output of 'df' command?

Thanks,
Qu

RE: Problem converting data raid0 to raid1: enospc errors during balance

[Jasper Verberk]

Hej guys!

Thanks for your input on the issue this far.

Too my knowledge raid1 in btrfs means 2 copies of each piece of data independent of the amount of disks used.

So 4 x 2,73tb would result in a totaal storage of roughly 5,5tb right?

Shouldn't this be more then enough?

btw, here is the output for df:

http://paste.debian.net/128932/


----------------------------------------
> Date: Mon, 27 Oct 2014 12:49:15 +0800
> From: quwenruo@cn.fujitsu.com
> To: lists@colorremedies.com
> CC: jverberk@hotmail.com; linux-btrfs@vger.kernel.org
> Subject: Re: Problem converting data raid0 to raid1: enospc errors during balance
>
>
> -------- Original Message --------
> Subject: Re: Problem converting data raid0 to raid1: enospc errors
> during balance
> From: Chris Murphy <lists@colorremedies.com>
> To: Qu Wenruo <quwenruo@cn.fujitsu.com>
> Date: 2014年10月27日 12:40
>> On Oct 26, 2014, at 7:40 PM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>>
>>> Hi,
>>>
>>> Although I'm not completely sure, but it seems that, you really ran out of space.
>>>
>>> [1] Your array won't hold raid1 for 1.97T data
>>> Your array used up 1.97T raid0 data, it takes 1.97T for raid0.
>>> But if converted to 1.97T, it will occupy 1.97T X2 = 3.94T.
>>> Your array are only 2.73T, it is too small to contain the data.
>> I'm not understanding. The btrfs fi show, shows 4x 2.73TiB devices, so that seems like it's a 10+TiB array.
>>
>> There's 2.04TiB raid0 data chunks, so roughly 500GiB per device, yet 1.94TiB is reported used per device by fi show. Confusing.
>>
>> Also it's still very confusing: Data, RAID1: total=2.85TiB, used=790.46GiB whether this means 2.85TiB out of 10TiB is allocated, or if it's twice that due to raid1. I can't ever remember this presentation detail, so again the secret decode ring where the UI doesn't expressly tell us what's going on is going to continue to be a source of confusion for users.
>>
>>
>> Chris Murphy
> Oh, I misread the output....
>
> That turns strange now....
> BTW what's the output of 'df' command?
>
> Thanks,
> Qu
>
 		 	   		  

Re: Problem converting data raid0 to raid1: enospc errors during balance

[Chris Murphy]

> 
> > On Oct 26, 2014, at 7:40 PM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
> 
> BTW what's the output of 'df' command?

Jasper,
What do you get for the conventional df command when this btrfs volume is mounted? Thanks.

Chris Murphy

Re: Problem converting data raid0 to raid1: enospc errors during balance

[Chris Murphy]

On Oct 27, 2014, at 9:56 AM, Jasper Verberk <jverberk@hotmail.com> wrote:

> These are the results to a normal df:
> 
> http://paste.debian.net/128932/
> 
> The mountpoint is /data.

OK so this is with the new computation in kernel 3.17 (which I think contains a bug by counting free space twice); so now it shows available blocks based on the loss due to mirroring or parity. So 1k blocks 5860533168 = 5.45TiB. If you boot an older kernel my expectation is this shows up as 10.91TiB. In any case, df says there's 1.77TiB worth of data, so there should be plenty of space.

Somewhere there's a bug. Either the 'btrfs fi df' is insufficiently communicating whether the desired operation can be done, or there's actual kernel confusion on how much space is available to do the conversion.

I wonder what happens if you go back to kernel 3.16 and try do do the conversion?


Chris Murphy

RE: Problem converting data raid0 to raid1: enospc errors during balance

[Jasper Verberk]

I actually tried going to raid1 with kernel 3.10 before....the reason I updated to 3.17 was to see if this would fix the error. 

It still remained though....


----------------------------------------
> Subject: Re: Problem converting data raid0 to raid1: enospc errors during balance
> From: lists@colorremedies.com
> Date: Mon, 27 Oct 2014 10:43:41 -0600
> CC: quwenruo@cn.fujitsu.com; linux-btrfs@vger.kernel.org
> To: jverberk@hotmail.com
>
>
> On Oct 27, 2014, at 9:56 AM, Jasper Verberk <jverberk@hotmail.com> wrote:
>
>> These are the results to a normal df:
>>
>> http://paste.debian.net/128932/
>>
>> The mountpoint is /data.
>
> OK so this is with the new computation in kernel 3.17 (which I think contains a bug by counting free space twice); so now it shows available blocks based on the loss due to mirroring or parity. So 1k blocks 5860533168 = 5.45TiB. If you boot an older kernel my expectation is this shows up as 10.91TiB. In any case, df says there's 1.77TiB worth of data, so there should be plenty of space.
>
> Somewhere there's a bug. Either the 'btrfs fi df' is insufficiently communicating whether the desired operation can be done, or there's actual kernel confusion on how much space is available to do the conversion.
>
> I wonder what happens if you go back to kernel 3.16 and try do do the conversion?
>
>
> Chris Murphy--
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
 		 	   		  

Re: Problem converting data raid0 to raid1: enospc errors during balance

[Chris Murphy]

On Oct 27, 2014, at 10:54 AM, Jasper Verberk <jverberk@hotmail.com> wrote:

> I actually tried going to raid1 with kernel 3.10 before....the reason I updated to 3.17 was to see if this would fix the error. 
> 
> It still remained though….

If you have btrfs-progs you could do a btrfs check without repair and see if that's enlightening. Can you do a normal balance successfully, without conversion? If you haven't tried it recently with a newer kernel you might try for starters:

btrfs balance start -musage=100 -dusage=50

Any data chunks less than 50% full will be balanced, and all the metadata chunks will be balanced. This line:
Data, RAID1: total=2.85TiB, used=790.46GiB
Is suspiciously high for allocated chunks compared to the data stored in it. Maybe releasing some chunks will solve the problem and permit conversion to proceed.


Chris Murphy

Re: Problem converting data raid0 to raid1: enospc errors during balance

[Jasper Verberk]

I already tried balancing with dusage=1. This removed all the allocated space.

It still doesn't allow me to do the conversion though, as it then reallocates the space.



> On 27 okt. 2014, at 18:28, Chris Murphy <lists@colorremedies.com> wrote:
> 
> 
>> On Oct 27, 2014, at 10:54 AM, Jasper Verberk <jverberk@hotmail.com> wrote:
>> 
>> I actually tried going to raid1 with kernel 3.10 before....the reason I updated to 3.17 was to see if this would fix the error. 
>> 
>> It still remained though….
> 
> If you have btrfs-progs you could do a btrfs check without repair and see if that's enlightening. Can you do a normal balance successfully, without conversion? If you haven't tried it recently with a newer kernel you might try for starters:
> 
> btrfs balance start -musage=100 -dusage=5
> 
> Any data chunks less than 50% full will be balanced, and all the metadata chunks will be balanced. This line:
> Data, RAID1: total=2.85TiB, used=790.46GiB
> Is suspiciously high for allocated chunks compared to the data stored in it. Maybe releasing some chunks will solve the problem and permit conversion to proceed.
> 
> 
> Chris Murphy--
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html