Adding 500G disk to btrfs volume... but I don't get 500G more of available space (raid0)

Adding 500G disk to btrfs volume... but I don't get 500G more of available space (raid0)

[Axelle]

Hi btrfs folks,

I'm afraid I have a newbie question... but I can't sort it out? It's
just about adding a disk to a btrfs volume and not getting the correct
amount of GB in the end...

I have a btrfs volume which already consists of two different devices
and which is mounted on /samples. Its total size is 194G.

$ df -h
Filesystem Size Used Avail Use% Mounted on ... /dev/sdc1 194G 165G 20G
90% /samples

Now, I would like to add another 500G to that volume, from another device. I did

$ sudo mkfs.btrfs -m raid0 -d raid0 /dev/sdb
$ sudo btrfs device add /dev/sdb /samples
My filesystem now correctly reports:

$ sudo btrfs filesystem show
Label: none  uuid: 545e95c6-d347-4a8c-8a49-38b9f9cb9add
    Total devices 3 FS bytes used 161.98GB
    devid    3 size 465.76GB used 0.00 path /dev/sdb
    devid    2 size 93.13GB used 84.51GB path /dev/sdc1
    devid    1 size 100.61GB used 84.53GB path /dev/sdc6
But I miss some space when I do:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
...
/dev/sdc1       660G  165G   43G  80% /samples
I added 500G! Why haven't I got more available??

To debug, I ran this command:

$ sudo btrfs filesystem df /samples
Data, RAID0: total=162.00GB, used=159.79GB
Data: total=8.00MB, used=7.48MB
System, RAID1: total=8.00MB, used=24.00KB
System: total=4.00MB, used=0.00
Metadata, RAID1: total=3.50GB, used=2.19GB
Metadata: total=8.00MB, used=0.00
My data is in RAID0, that's ok. So where have my 500G gone, and how
can I fix this?

Thanks

Re: Adding 500G disk to btrfs volume... but I don't get 500G more of available space (raid0)

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2769 bytes --]

On Fri, Jul 26, 2013 at 09:05:03AM +0200, Axelle wrote:
> Hi btrfs folks,
> 
> I'm afraid I have a newbie question... but I can't sort it out? It's
> just about adding a disk to a btrfs volume and not getting the correct
> amount of GB in the end...
> 
> I have a btrfs volume which already consists of two different devices
> and which is mounted on /samples. Its total size is 194G.
> 
> $ df -h
> Filesystem Size Used Avail Use% Mounted on ... /dev/sdc1 194G 165G 20G
> 90% /samples
> 
> Now, I would like to add another 500G to that volume, from another device. I did
> 
> $ sudo mkfs.btrfs -m raid0 -d raid0 /dev/sdb
> $ sudo btrfs device add /dev/sdb /samples
> My filesystem now correctly reports:
> 
> $ sudo btrfs filesystem show
> Label: none  uuid: 545e95c6-d347-4a8c-8a49-38b9f9cb9add
>     Total devices 3 FS bytes used 161.98GB
>     devid    3 size 465.76GB used 0.00 path /dev/sdb
>     devid    2 size 93.13GB used 84.51GB path /dev/sdc1
>     devid    1 size 100.61GB used 84.53GB path /dev/sdc6
> But I miss some space when I do:

   RAID-0 requires at least two devices. If you balance this
configuration, you'll use up the first 93.13 GiB of each device
striping across all three devices, for a total of 3*93.13 = 279.39
GiB. Then /dev/sdc1 becomes full, leaving you with two devices which
have 7.48 GiB and 372.63 GiB respectively. After another 7.48 GiB on
each device (for a total of 2*7.48 = 14.96 GiB), you have filled
/dev/sdc1, leaving only /dev/sdb to work with. Since there's only one
device, it can't be used by RAID-0.

   If you want to use the full space available, you should rebalance
to single usage, which stops the RAID-0 striping, and allocates
linearly:

# btrfs balance start -dconvert=single,soft /samples

   Hugo.

> $ df -h
> Filesystem      Size  Used Avail Use% Mounted on
> ...
> /dev/sdc1       660G  165G   43G  80% /samples
> I added 500G! Why haven't I got more available??
> 
> To debug, I ran this command:
> 
> $ sudo btrfs filesystem df /samples
> Data, RAID0: total=162.00GB, used=159.79GB
> Data: total=8.00MB, used=7.48MB
> System, RAID1: total=8.00MB, used=24.00KB
> System: total=4.00MB, used=0.00
> Metadata, RAID1: total=3.50GB, used=2.19GB
> Metadata: total=8.00MB, used=0.00
> My data is in RAID0, that's ok. So where have my 500G gone, and how
> can I fix this?
> 
> Thanks
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
           --- Sometimes, when I'm alone, I Google myself. ---           

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Adding 500G disk to btrfs volume... but I don't get 500G more of available space (raid0)

[Axelle]

Hi Hugo,
Thanks for your answer, but I'm afraid I still don't get it.

>RAID-0 requires at least two devices.

Well, I have three devices, so that's more than enough isn't it?
Or do you mean I should be adding two devices at a time?

> If you balance this
>configuration, you'll use up the first 93.13 GiB of each device
>striping across all three devices, for a total of 3*93.13 = 279.39

why 93.13? I guess you meant 84.53 which is the size I am using on
sdc1 and sdc6.

># btrfs balance start -dconvert=single,soft /samples

Nice command but I wasn't thinking of stopping RAID0 striping. I was
expecting my data to be stripped evenly on all 3 devices.

Well - evenly - until the smallest one /dev/sdc1 is filled, then,
it'll use only the last two, when /dev/sdc6 is filled, it will used
/dev/sdb only.
Is that possible/correct?


But basically, what does not make sense to me is what df reports as
available size.
Look.
Before, I had ~165G used on a total of 194G.
I added a new disk of 465G. Now, df reports I have a total of 660G
(that's right) with 165G used (that's correct too) but only 43G
available!

I was expecting to have ~495G available! Where are my 465G gone?


$ df -h
Filesystem      Size  Used Avail Use% Mounted on
...
/dev/sdc1       660G  165G   43G  80% /samples



Thanks
Axelle.

On Fri, Jul 26, 2013 at 9:45 AM, Hugo Mills <hugo@carfax.org.uk> wrote:
> On Fri, Jul 26, 2013 at 09:05:03AM +0200, Axelle wrote:
>> Hi btrfs folks,
>>
>> I'm afraid I have a newbie question... but I can't sort it out? It's
>> just about adding a disk to a btrfs volume and not getting the correct
>> amount of GB in the end...
>>
>> I have a btrfs volume which already consists of two different devices
>> and which is mounted on /samples. Its total size is 194G.
>>
>> $ df -h
>> Filesystem Size Used Avail Use% Mounted on ... /dev/sdc1 194G 165G 20G
>> 90% /samples
>>
>> Now, I would like to add another 500G to that volume, from another device. I did
>>
>> $ sudo mkfs.btrfs -m raid0 -d raid0 /dev/sdb
>> $ sudo btrfs device add /dev/sdb /samples
>> My filesystem now correctly reports:
>>
>> $ sudo btrfs filesystem show
>> Label: none  uuid: 545e95c6-d347-4a8c-8a49-38b9f9cb9add
>>     Total devices 3 FS bytes used 161.98GB
>>     devid    3 size 465.76GB used 0.00 path /dev/sdb
>>     devid    2 size 93.13GB used 84.51GB path /dev/sdc1
>>     devid    1 size 100.61GB used 84.53GB path /dev/sdc6
>> But I miss some space when I do:
>
>    RAID-0 requires at least two devices. If you balance this
> configuration, you'll use up the first 93.13 GiB of each device
> striping across all three devices, for a total of 3*93.13 = 279.39
> GiB. Then /dev/sdc1 becomes full, leaving you with two devices which
> have 7.48 GiB and 372.63 GiB respectively. After another 7.48 GiB on
> each device (for a total of 2*7.48 = 14.96 GiB), you have filled
> /dev/sdc1, leaving only /dev/sdb to work with. Since there's only one
> device, it can't be used by RAID-0.
>
>    If you want to use the full space available, you should rebalance
> to single usage, which stops the RAID-0 striping, and allocates
> linearly:
>
> # btrfs balance start -dconvert=single,soft /samples
>
>    Hugo.
>
>> $ df -h
>> Filesystem      Size  Used Avail Use% Mounted on
>> ...
>> /dev/sdc1       660G  165G   43G  80% /samples
>> I added 500G! Why haven't I got more available??
>>
>> To debug, I ran this command:
>>
>> $ sudo btrfs filesystem df /samples
>> Data, RAID0: total=162.00GB, used=159.79GB
>> Data: total=8.00MB, used=7.48MB
>> System, RAID1: total=8.00MB, used=24.00KB
>> System: total=4.00MB, used=0.00
>> Metadata, RAID1: total=3.50GB, used=2.19GB
>> Metadata: total=8.00MB, used=0.00
>> My data is in RAID0, that's ok. So where have my 500G gone, and how
>> can I fix this?
>>
>> Thanks
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> --
> === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
>   PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
>            --- Sometimes, when I'm alone, I Google myself. ---

Re: Adding 500G disk to btrfs volume... but I don't get 500G more of available space (raid0)

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 6034 bytes --]

On Fri, Jul 26, 2013 at 04:35:59PM +0200, Axelle wrote:
> Hi Hugo,
> Thanks for your answer, but I'm afraid I still don't get it.
> 
> >RAID-0 requires at least two devices.
> 
> Well, I have three devices, so that's more than enough isn't it?
> Or do you mean I should be adding two devices at a time?
> 
> > If you balance this
> >configuration, you'll use up the first 93.13 GiB of each device
> >striping across all three devices, for a total of 3*93.13 = 279.39
> 
> why 93.13? I guess you meant 84.53 which is the size I am using on
> sdc1 and sdc6.

   Sorry, a little unclear -- if you balance, and then continue
writing data to the FS. Once you hit 93.13 GiB (the size of the
smallest device), you switch to 2-device operation, and then when
that's full, you can't go any further.

> ># btrfs balance start -dconvert=single,soft /samples
> 
> Nice command but I wasn't thinking of stopping RAID0 striping. I was
> expecting my data to be stripped evenly on all 3 devices.

   It's worth noting that /dev/sdc1 and /dev/sdc6 are on the same
physical device. If that's a rotational device (i.e. traditional hard
disk), then you're going to have a serious performance decrease as a
result of that, because /dev/sdc will have to spend lots of its time
seeking between the two partitions. "single" operation really is the
better option here -- you'll get to use all your space, and you won't
suffer the performance problems of striping between two partitions on
the same disk.

> Well - evenly - until the smallest one /dev/sdc1 is filled, then,
> it'll use only the last two, when /dev/sdc6 is filled, it will used
> /dev/sdb only.
> Is that possible/correct?

   That's exactly what happens, except for the last bit. RAID-0
requires at least two devices, so it can't stripe across the one
device remaining once you have completely filled /dev/sdc1 and
/dev/sdc6.

> But basically, what does not make sense to me is what df reports as
> available size.
> Look.
> Before, I had ~165G used on a total of 194G.
> I added a new disk of 465G. Now, df reports I have a total of 660G
> (that's right) with 165G used (that's correct too) but only 43G
> available!
> 
> I was expecting to have ~495G available! Where are my 465G gone?

   It's not usable with the RAID configuration you've specified, so
it's not shown.

   Hugo.

> $ df -h
> Filesystem      Size  Used Avail Use% Mounted on
> ...
> /dev/sdc1       660G  165G   43G  80% /samples
> 
> 
> 
> Thanks
> Axelle.
> 
> On Fri, Jul 26, 2013 at 9:45 AM, Hugo Mills <hugo@carfax.org.uk> wrote:
> > On Fri, Jul 26, 2013 at 09:05:03AM +0200, Axelle wrote:
> >> Hi btrfs folks,
> >>
> >> I'm afraid I have a newbie question... but I can't sort it out? It's
> >> just about adding a disk to a btrfs volume and not getting the correct
> >> amount of GB in the end...
> >>
> >> I have a btrfs volume which already consists of two different devices
> >> and which is mounted on /samples. Its total size is 194G.
> >>
> >> $ df -h
> >> Filesystem Size Used Avail Use% Mounted on ... /dev/sdc1 194G 165G 20G
> >> 90% /samples
> >>
> >> Now, I would like to add another 500G to that volume, from another device. I did
> >>
> >> $ sudo mkfs.btrfs -m raid0 -d raid0 /dev/sdb
> >> $ sudo btrfs device add /dev/sdb /samples
> >> My filesystem now correctly reports:
> >>
> >> $ sudo btrfs filesystem show
> >> Label: none  uuid: 545e95c6-d347-4a8c-8a49-38b9f9cb9add
> >>     Total devices 3 FS bytes used 161.98GB
> >>     devid    3 size 465.76GB used 0.00 path /dev/sdb
> >>     devid    2 size 93.13GB used 84.51GB path /dev/sdc1
> >>     devid    1 size 100.61GB used 84.53GB path /dev/sdc6
> >> But I miss some space when I do:
> >
> >    RAID-0 requires at least two devices. If you balance this
> > configuration, you'll use up the first 93.13 GiB of each device
> > striping across all three devices, for a total of 3*93.13 = 279.39
> > GiB. Then /dev/sdc1 becomes full, leaving you with two devices which
> > have 7.48 GiB and 372.63 GiB respectively. After another 7.48 GiB on
> > each device (for a total of 2*7.48 = 14.96 GiB), you have filled
> > /dev/sdc1, leaving only /dev/sdb to work with. Since there's only one
> > device, it can't be used by RAID-0.
> >
> >    If you want to use the full space available, you should rebalance
> > to single usage, which stops the RAID-0 striping, and allocates
> > linearly:
> >
> > # btrfs balance start -dconvert=single,soft /samples
> >
> >    Hugo.
> >
> >> $ df -h
> >> Filesystem      Size  Used Avail Use% Mounted on
> >> ...
> >> /dev/sdc1       660G  165G   43G  80% /samples
> >> I added 500G! Why haven't I got more available??
> >>
> >> To debug, I ran this command:
> >>
> >> $ sudo btrfs filesystem df /samples
> >> Data, RAID0: total=162.00GB, used=159.79GB
> >> Data: total=8.00MB, used=7.48MB
> >> System, RAID1: total=8.00MB, used=24.00KB
> >> System: total=4.00MB, used=0.00
> >> Metadata, RAID1: total=3.50GB, used=2.19GB
> >> Metadata: total=8.00MB, used=0.00
> >> My data is in RAID0, that's ok. So where have my 500G gone, and how
> >> can I fix this?
> >>
> >> Thanks
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> > --
> > === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
> >   PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
> >            --- Sometimes, when I'm alone, I Google myself. ---
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
  --- I hate housework. You make the beds, you wash the dishes, and ---  
           six months later you have to start all over again.            

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]