Help me understand what is going on with my RAID1 FS

Help me understand what is going on with my RAID1 FS

[FLJ]

Hello all,

I have a BTRFS RAID1 volume running for the past year. I avoided all
pitfalls known to me that would mess up this volume. I never
experimented with quotas, no-COW, snapshots, defrag, nothing really.
The volume is a RAID1 from day 1 and is working reliably until now.

Until yesterday it consisted of two 3 TB drives, something along the lines:

Label: 'BigVault'  uuid: a37ad5f5-a21b-41c7-970b-13b6c4db33db
        Total devices 2 FS bytes used 2.47TiB
        devid    1 size 2.73TiB used 2.47TiB path /dev/sdb
        devid    2 size 2.73TiB used 2.47TiB path /dev/sdc

Yesterday I've added a new drive to the FS and did a full rebalance
(without filters) over night, which went through without any issues.

Now I have:
 Label: 'BigVault'  uuid: a37ad5f5-a21b-41c7-970b-13b6c4db33db
        Total devices 3 FS bytes used 2.47TiB
        devid    1 size 2.73TiB used 1.24TiB path /dev/sdb
        devid    2 size 2.73TiB used 1.24TiB path /dev/sdc
        devid    3 size 7.28TiB used 2.48TiB path /dev/sda

# btrfs fi df /mnt/BigVault/
Data, RAID1: total=2.47TiB, used=2.47TiB
System, RAID1: total=32.00MiB, used=384.00KiB
Metadata, RAID1: total=4.00GiB, used=2.74GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

But still df -h is giving me:
Filesystem           Size  Used Avail Use% Mounted on
/dev/sdb             6.4T  2.5T  1.5T  63% /mnt/BigVault

Although I've heard and read about the difficulty in reporting free
space due to the flexibility of BTRFS, snapshots and subvolumes, etc.,
but I only have a single volume, no subvolumes, no snapshots, no
quotas and both data and metadata are RAID1.

My expectation would've been that in case of BigVault Size == Used + Avail.

Actually based on http://carfax.org.uk/btrfs-usage/index.html I
would've expected 6 TB of usable space. Here I get 6.4 which is odd,
but that only 1.5 TB is available is even stranger.

Could anyone explain what I did wrong or why my expectations are wrong?

Thank you in advance

Re: Help me understand what is going on with my RAID1 FS

[Kai Krakow]

Am Sun, 10 Sep 2017 15:45:42 +0200
schrieb FLJ <feci2048@gmail.com>:

> Hello all,
> 
> I have a BTRFS RAID1 volume running for the past year. I avoided all
> pitfalls known to me that would mess up this volume. I never
> experimented with quotas, no-COW, snapshots, defrag, nothing really.
> The volume is a RAID1 from day 1 and is working reliably until now.
> 
> Until yesterday it consisted of two 3 TB drives, something along the
> lines:
> 
> Label: 'BigVault'  uuid: a37ad5f5-a21b-41c7-970b-13b6c4db33db
>         Total devices 2 FS bytes used 2.47TiB
>         devid    1 size 2.73TiB used 2.47TiB path /dev/sdb
>         devid    2 size 2.73TiB used 2.47TiB path /dev/sdc
> 
> Yesterday I've added a new drive to the FS and did a full rebalance
> (without filters) over night, which went through without any issues.
> 
> Now I have:
>  Label: 'BigVault'  uuid: a37ad5f5-a21b-41c7-970b-13b6c4db33db
>         Total devices 3 FS bytes used 2.47TiB
>         devid    1 size 2.73TiB used 1.24TiB path /dev/sdb
>         devid    2 size 2.73TiB used 1.24TiB path /dev/sdc
>         devid    3 size 7.28TiB used 2.48TiB path /dev/sda
> 
> # btrfs fi df /mnt/BigVault/
> Data, RAID1: total=2.47TiB, used=2.47TiB
> System, RAID1: total=32.00MiB, used=384.00KiB
> Metadata, RAID1: total=4.00GiB, used=2.74GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> But still df -h is giving me:
> Filesystem           Size  Used Avail Use% Mounted on
> /dev/sdb             6.4T  2.5T  1.5T  63% /mnt/BigVault
> 
> Although I've heard and read about the difficulty in reporting free
> space due to the flexibility of BTRFS, snapshots and subvolumes, etc.,
> but I only have a single volume, no subvolumes, no snapshots, no
> quotas and both data and metadata are RAID1.
> 
> My expectation would've been that in case of BigVault Size == Used +
> Avail.
> 
> Actually based on http://carfax.org.uk/btrfs-usage/index.html I
> would've expected 6 TB of usable space. Here I get 6.4 which is odd,
> but that only 1.5 TB is available is even stranger.
> 
> Could anyone explain what I did wrong or why my expectations are
> wrong?
> 
> Thank you in advance

Btrfs reports estimated free space from the free space of the smallest
member as it can only guarantee that. In your case this is 2.73 minus
1.24 free which is roughly around 1.5T. But since this free space
distributes across three disks with one having much more free space, it
probably will use up that space at half the rate of actual allocation.
But due to how btrfs allocates from free space in chunks, that may not
be possible - thus the low unexpected value. You will probably need to
run balance once in a while to evenly redistribute allocated chunks
across all disks.

It may give you better estimates if you combine sdb and sdc into one
logical device, e.g. using raid0 or jbod via md or lvm.


-- 
Regards,
Kai

Replies to list-only preferred.

Re: Help me understand what is going on with my RAID1 FS

[Ferenc-Levente Juhos]

@Kai and Dmitrii
thank you for your explanations if I understand you correctly, you're
saying that btrfs makes no attempt to "optimally" use the physical
devices it has in the FS, once a new RAID1 block group needs to be
allocated it will semi-randomly pick two devices with enough space and
allocate two equal sized chunks, one on each. This new chunk may or
may not fall onto my newly added 8 TB drive. Am I understanding this
correctly?
> You will probably need to
>run balance once in a while to evenly redistribute allocated chunks
>across all disks.

Is there some sort of balance filter that would speed up this sort of
balancing? Will balance be smart enough to make the "right" decision?
As far as I read the chunk allocator used during balance is the same
that is used during normal operation. If the allocator is already
sub-optimal during normal operations, what's the guarantee that it
will make a "better" decision during balancing?

When I say "right" and "better" I mean this:
Drive1(8) Drive2(3) Drive3(3)
X1            X1
X2                            X2
X3            X3
X4                            X4
I was convinced until now that the chunk allocator at least tries a
best possible allocation. I'm sure it's complicated to develop a
generic algorithm to fit all setups, but it should be possible.

On Sun, Sep 10, 2017 at 5:47 PM, Kai Krakow <hurikhan77@gmail.com> wrote:
> Am Sun, 10 Sep 2017 15:45:42 +0200
> schrieb FLJ <feci2048@gmail.com>:
>
>> Hello all,
>>
>> I have a BTRFS RAID1 volume running for the past year. I avoided all
>> pitfalls known to me that would mess up this volume. I never
>> experimented with quotas, no-COW, snapshots, defrag, nothing really.
>> The volume is a RAID1 from day 1 and is working reliably until now.
>>
>> Until yesterday it consisted of two 3 TB drives, something along the
>> lines:
>>
>> Label: 'BigVault'  uuid: a37ad5f5-a21b-41c7-970b-13b6c4db33db
>>         Total devices 2 FS bytes used 2.47TiB
>>         devid    1 size 2.73TiB used 2.47TiB path /dev/sdb
>>         devid    2 size 2.73TiB used 2.47TiB path /dev/sdc
>>
>> Yesterday I've added a new drive to the FS and did a full rebalance
>> (without filters) over night, which went through without any issues.
>>
>> Now I have:
>>  Label: 'BigVault'  uuid: a37ad5f5-a21b-41c7-970b-13b6c4db33db
>>         Total devices 3 FS bytes used 2.47TiB
>>         devid    1 size 2.73TiB used 1.24TiB path /dev/sdb
>>         devid    2 size 2.73TiB used 1.24TiB path /dev/sdc
>>         devid    3 size 7.28TiB used 2.48TiB path /dev/sda
>>
>> # btrfs fi df /mnt/BigVault/
>> Data, RAID1: total=2.47TiB, used=2.47TiB
>> System, RAID1: total=32.00MiB, used=384.00KiB
>> Metadata, RAID1: total=4.00GiB, used=2.74GiB
>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>
>> But still df -h is giving me:
>> Filesystem           Size  Used Avail Use% Mounted on
>> /dev/sdb             6.4T  2.5T  1.5T  63% /mnt/BigVault
>>
>> Although I've heard and read about the difficulty in reporting free
>> space due to the flexibility of BTRFS, snapshots and subvolumes, etc.,
>> but I only have a single volume, no subvolumes, no snapshots, no
>> quotas and both data and metadata are RAID1.
>>
>> My expectation would've been that in case of BigVault Size == Used +
>> Avail.
>>
>> Actually based on http://carfax.org.uk/btrfs-usage/index.html I
>> would've expected 6 TB of usable space. Here I get 6.4 which is odd,
>> but that only 1.5 TB is available is even stranger.
>>
>> Could anyone explain what I did wrong or why my expectations are
>> wrong?
>>
>> Thank you in advance
>
> Btrfs reports estimated free space from the free space of the smallest
> member as it can only guarantee that. In your case this is 2.73 minus
> 1.24 free which is roughly around 1.5T. But since this free space
> distributes across three disks with one having much more free space, it
> probably will use up that space at half the rate of actual allocation.
> But due to how btrfs allocates from free space in chunks, that may not
> be possible - thus the low unexpected value. You will probably need to
> run balance once in a while to evenly redistribute allocated chunks
> across all disks.
>
> It may give you better estimates if you combine sdb and sdc into one
> logical device, e.g. using raid0 or jbod via md or lvm.
>
>
> --
> Regards,
> Kai
>
> Replies to list-only preferred.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Help me understand what is going on with my RAID1 FS

[Dmitrii Tcvetkov]

> @Kai and Dmitrii
> thank you for your explanations if I understand you correctly, you're
> saying that btrfs makes no attempt to "optimally" use the physical
> devices it has in the FS, once a new RAID1 block group needs to be
> allocated it will semi-randomly pick two devices with enough space and
> allocate two equal sized chunks, one on each. This new chunk may or
> may not fall onto my newly added 8 TB drive. Am I understanding this
> correctly?
If I remember correctly chunk allocator allocates new chunks on device
which has the most unallocated space. 

> Is there some sort of balance filter that would speed up this sort of
> balancing? Will balance be smart enough to make the "right" decision?
> As far as I read the chunk allocator used during balance is the same
> that is used during normal operation. If the allocator is already
> sub-optimal during normal operations, what's the guarantee that it
> will make a "better" decision during balancing?

I don't really see any way that being possible in raid1 profile. How
can you fill all three devices if you can split data only twice? There
will be moment when two of three disks are full and BTRFS can't
allocate new raid1 block group because it has only one drive with
unallocated space.

> 
> When I say "right" and "better" I mean this:
> Drive1(8) Drive2(3) Drive3(3)
> X1            X1
> X2                            X2
> X3            X3
> X4                            X4
> I was convinced until now that the chunk allocator at least tries a
> best possible allocation. I'm sure it's complicated to develop a
> generic algorithm to fit all setups, but it should be possible.
 

Problem is that each raid1 block group contains two chunks on two
separate devices, it can't utilize fully three devices no matter what.
If that doesn't suit you then you need to add 4th disk. After
that FS will be able to use all unallocated space on all disks in raid1
profile. But even then you'll be able to safely lose only one disk
since BTRFS still will be storing only 2 copies of data.

This behavior is not relevant for single or raid0 profiles of
multidevice BTRFS filesystems.

Re: Help me understand what is going on with my RAID1 FS

[Ferenc-Levente Juhos]

>Problem is that each raid1 block group contains two chunks on two
>separate devices, it can't utilize fully three devices no matter what.
>If that doesn't suit you then you need to add 4th disk. After
>that FS will be able to use all unallocated space on all disks in raid1
>profile. But even then you'll be able to safely lose only one disk
>since BTRFS still will be storing only 2 copies of data.

I hope I didn't say that I want to utilize all three devices fully. It
was clear to me that there will be 2 TB of wasted space.
Also I'm not questioning the chunk allocator for RAID1 at all. It's
clear and it always has been clear that for RAID1 the chunks need to
be allocated on different physical devices.
If I understood Kai's point of view, he even suggested that I might
need to do balancing to make sure that the free space on the three
devices is being used smartly. Hence the questions about balancing.

I mean in worst case it could happen like this:

Again I have disks of sizes 3, 3, 8:
Fig.1
Drive1(8) Drive2(3) Drive3(3)
 -               X1            X1
 -               X2            X2
 -               X3            X3
Here the new drive is completely unused. Even if one X1 chunk would be
on Drive1 it would be still a sub-optimal allocation.

This is the optimal allocation. Will btrfs allocate like this?
Considering that Drive1 has the most free space.
Fig. 2
Drive1(8) Drive2(3) Drive3(3)
X1            X1            -
X2            -               X2
X3            X3            -
X4            -               X4

>From my point of view Fig.2 shows the optimal allocation, by the time
the disks Drive2 and Drive3 are full (3TB) Drive1 must have 6TB
(because it is exclusively holding the mirrors for both Drive2 and 3).
For sure now btrfs can say, since two of the drives are completely
full he can't allocate any more chunks and the remaining 2 TB of space
from Drive1 is wasted. This is clear it's even pointed out by the
btrfs size calculator.

But again if the above statements are true, then df might as well tell
the "truth" and report that I have 3.5 TB space free and not 1.5TB (as
it is reported now). Again here I fully understand Kai's explanation.
Because coming back to my first e-mail, my "problem" was that df is
reporting 1.5 TB free, whereas the whole FS holds 2.5 TB of data.

So the question still remains, is it just that df is intentionally not
smart enough to give a more accurate estimation, or is the assumption
that the allocator picks the drive with most free space mistaken?
If I continue along the lines of what Kai said, and I need to do
re-balance, because the allocation is not like shown above (Fig.2),
then my question is still legitimate. Are there any filters that one
might use to speed up or to selectively balance in my case? or will I
need to do full balance?

On Sun, Sep 10, 2017 at 7:19 PM, Dmitrii Tcvetkov <demfloro@demfloro.ru> wrote:
>> @Kai and Dmitrii
>> thank you for your explanations if I understand you correctly, you're
>> saying that btrfs makes no attempt to "optimally" use the physical
>> devices it has in the FS, once a new RAID1 block group needs to be
>> allocated it will semi-randomly pick two devices with enough space and
>> allocate two equal sized chunks, one on each. This new chunk may or
>> may not fall onto my newly added 8 TB drive. Am I understanding this
>> correctly?
> If I remember correctly chunk allocator allocates new chunks on device
> which has the most unallocated space.
>
>> Is there some sort of balance filter that would speed up this sort of
>> balancing? Will balance be smart enough to make the "right" decision?
>> As far as I read the chunk allocator used during balance is the same
>> that is used during normal operation. If the allocator is already
>> sub-optimal during normal operations, what's the guarantee that it
>> will make a "better" decision during balancing?
>
> I don't really see any way that being possible in raid1 profile. How
> can you fill all three devices if you can split data only twice? There
> will be moment when two of three disks are full and BTRFS can't
> allocate new raid1 block group because it has only one drive with
> unallocated space.
>
>>
>> When I say "right" and "better" I mean this:
>> Drive1(8) Drive2(3) Drive3(3)
>> X1            X1
>> X2                            X2
>> X3            X3
>> X4                            X4
>> I was convinced until now that the chunk allocator at least tries a
>> best possible allocation. I'm sure it's complicated to develop a
>> generic algorithm to fit all setups, but it should be possible.
>
>
> Problem is that each raid1 block group contains two chunks on two
> separate devices, it can't utilize fully three devices no matter what.
> If that doesn't suit you then you need to add 4th disk. After
> that FS will be able to use all unallocated space on all disks in raid1
> profile. But even then you'll be able to safely lose only one disk
> since BTRFS still will be storing only 2 copies of data.
>
> This behavior is not relevant for single or raid0 profiles of
> multidevice BTRFS filesystems.

Re: Help me understand what is going on with my RAID1 FS

[Kai Krakow]

Am Sun, 10 Sep 2017 20:15:52 +0200
schrieb Ferenc-Levente Juhos <feci1024@gmail.com>:

> >Problem is that each raid1 block group contains two chunks on two
> >separate devices, it can't utilize fully three devices no matter
> >what. If that doesn't suit you then you need to add 4th disk. After
> >that FS will be able to use all unallocated space on all disks in
> >raid1 profile. But even then you'll be able to safely lose only one
> >disk since BTRFS still will be storing only 2 copies of data.  
> 
> I hope I didn't say that I want to utilize all three devices fully. It
> was clear to me that there will be 2 TB of wasted space.
> Also I'm not questioning the chunk allocator for RAID1 at all. It's
> clear and it always has been clear that for RAID1 the chunks need to
> be allocated on different physical devices.
> If I understood Kai's point of view, he even suggested that I might
> need to do balancing to make sure that the free space on the three
> devices is being used smartly. Hence the questions about balancing.

It will allocate chunks from the device with the most space available.
So while you fill your disks space usage will evenly distribute.

The problem comes when you start deleting stuff, some chunks may even
be freed, and everything becomes messed up. In an aging file system you
may notice that the chunks are no longer evenly distributed. A balance
is a way to fix that because it will reallocate chunks and coalesce
data back into single chunks, making free space for new allocations. In
this process it will actually evenly distribute your data again.

You may want to use this rebalance script:
https://www.spinics.net/lists/linux-btrfs/msg52076.html

> I mean in worst case it could happen like this:
> 
> Again I have disks of sizes 3, 3, 8:
> Fig.1
> Drive1(8) Drive2(3) Drive3(3)
>  -               X1            X1
>  -               X2            X2
>  -               X3            X3
> Here the new drive is completely unused. Even if one X1 chunk would be
> on Drive1 it would be still a sub-optimal allocation.

This won't happen while filling a fresh btrfs. Chunks are always
allocated from a device with most free space (within the raid1
constraints). This it will allocate space alternating between disk1+2
and disk1+3.

> This is the optimal allocation. Will btrfs allocate like this?
> Considering that Drive1 has the most free space.
> Fig. 2
> Drive1(8) Drive2(3) Drive3(3)
> X1            X1            -
> X2            -               X2
> X3            X3            -
> X4            -               X4

Yes.

> From my point of view Fig.2 shows the optimal allocation, by the time
> the disks Drive2 and Drive3 are full (3TB) Drive1 must have 6TB
> (because it is exclusively holding the mirrors for both Drive2 and 3).
> For sure now btrfs can say, since two of the drives are completely
> full he can't allocate any more chunks and the remaining 2 TB of space
> from Drive1 is wasted. This is clear it's even pointed out by the
> btrfs size calculator.

Yes.


> But again if the above statements are true, then df might as well tell
> the "truth" and report that I have 3.5 TB space free and not 1.5TB (as
> it is reported now). Again here I fully understand Kai's explanation.
> Because coming back to my first e-mail, my "problem" was that df is
> reporting 1.5 TB free, whereas the whole FS holds 2.5 TB of data.

The size calculator has undergone some revisions. I think it currently
estimates the free space from net data to raw data ratio across all
devices, taking the current raid constraints into account.

Calculating free space in btrfs is difficult because in the future
btrfs may even support different raid levels for different sub volumes.
It's probably best to calculate for the worst case scenario then.

Even today it's already difficult if you use different raid levels for
meta data and content data: The filesystem cannot predict the future of
allocations. It can only give an educated guess. And the calculation
was revised a few times to not "overshoot".


> So the question still remains, is it just that df is intentionally not
> smart enough to give a more accurate estimation,

The df utility doesn't now anything about btrfs allocations. The value
is estimated by btrfs itself. To get more detailed info for capacity
planning, you should use "btrfs fi df" and its various siblings.

> or is the assumption
> that the allocator picks the drive with most free space mistaken?
> If I continue along the lines of what Kai said, and I need to do
> re-balance, because the allocation is not like shown above (Fig.2),
> then my question is still legitimate. Are there any filters that one
> might use to speed up or to selectively balance in my case? or will I
> need to do full balance?

Your assumption is misguided. The total free space estimation is a
totally different thing than what the allocator bases its decision on.
See "btrfs dev usage". The allocator uses space from the biggest
unallocated space within the raid constraints.

Plus, the raid constraints are what forces you to 1.5 TB free space
(df) as you already pointed out above.


> On Sun, Sep 10, 2017 at 7:19 PM, Dmitrii Tcvetkov
> <demfloro@demfloro.ru> wrote:
> >> @Kai and Dmitrii
> >> thank you for your explanations if I understand you correctly,
> >> you're saying that btrfs makes no attempt to "optimally" use the
> >> physical devices it has in the FS, once a new RAID1 block group
> >> needs to be allocated it will semi-randomly pick two devices with
> >> enough space and allocate two equal sized chunks, one on each.
> >> This new chunk may or may not fall onto my newly added 8 TB drive.
> >> Am I understanding this correctly?  
> > If I remember correctly chunk allocator allocates new chunks on
> > device which has the most unallocated space.
> >  
> >> Is there some sort of balance filter that would speed up this sort
> >> of balancing? Will balance be smart enough to make the "right"
> >> decision? As far as I read the chunk allocator used during balance
> >> is the same that is used during normal operation. If the allocator
> >> is already sub-optimal during normal operations, what's the
> >> guarantee that it will make a "better" decision during balancing?  
> >
> > I don't really see any way that being possible in raid1 profile. How
> > can you fill all three devices if you can split data only twice?
> > There will be moment when two of three disks are full and BTRFS
> > can't allocate new raid1 block group because it has only one drive
> > with unallocated space.
> >  
> >>
> >> When I say "right" and "better" I mean this:
> >> Drive1(8) Drive2(3) Drive3(3)
> >> X1            X1
> >> X2                            X2
> >> X3            X3
> >> X4                            X4
> >> I was convinced until now that the chunk allocator at least tries a
> >> best possible allocation. I'm sure it's complicated to develop a
> >> generic algorithm to fit all setups, but it should be possible.  
> >
> >
> > Problem is that each raid1 block group contains two chunks on two
> > separate devices, it can't utilize fully three devices no matter
> > what. If that doesn't suit you then you need to add 4th disk. After
> > that FS will be able to use all unallocated space on all disks in
> > raid1 profile. But even then you'll be able to safely lose only one
> > disk since BTRFS still will be storing only 2 copies of data.
> >
> > This behavior is not relevant for single or raid0 profiles of
> > multidevice BTRFS filesystems.  
> --
> To unsubscribe from this list: send the line "unsubscribe
> linux-btrfs" in the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 



-- 
Regards,
Kai

Replies to list-only preferred.

Re: Help me understand what is going on with my RAID1 FS

[Andrei Borzenkov]

10.09.2017 18:47, Kai Krakow пишет:
> Am Sun, 10 Sep 2017 15:45:42 +0200
> schrieb FLJ <feci2048@gmail.com>:
> 
>> Hello all,
>>
>> I have a BTRFS RAID1 volume running for the past year. I avoided all
>> pitfalls known to me that would mess up this volume. I never
>> experimented with quotas, no-COW, snapshots, defrag, nothing really.
>> The volume is a RAID1 from day 1 and is working reliably until now.
>>
>> Until yesterday it consisted of two 3 TB drives, something along the
>> lines:
>>
>> Label: 'BigVault'  uuid: a37ad5f5-a21b-41c7-970b-13b6c4db33db
>>         Total devices 2 FS bytes used 2.47TiB
>>         devid    1 size 2.73TiB used 2.47TiB path /dev/sdb
>>         devid    2 size 2.73TiB used 2.47TiB path /dev/sdc
>>
>> Yesterday I've added a new drive to the FS and did a full rebalance
>> (without filters) over night, which went through without any issues.
>>
>> Now I have:
>>  Label: 'BigVault'  uuid: a37ad5f5-a21b-41c7-970b-13b6c4db33db
>>         Total devices 3 FS bytes used 2.47TiB
>>         devid    1 size 2.73TiB used 1.24TiB path /dev/sdb
>>         devid    2 size 2.73TiB used 1.24TiB path /dev/sdc
>>         devid    3 size 7.28TiB used 2.48TiB path /dev/sda
>>
>> # btrfs fi df /mnt/BigVault/
>> Data, RAID1: total=2.47TiB, used=2.47TiB
>> System, RAID1: total=32.00MiB, used=384.00KiB
>> Metadata, RAID1: total=4.00GiB, used=2.74GiB
>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>
>> But still df -h is giving me:
>> Filesystem           Size  Used Avail Use% Mounted on
>> /dev/sdb             6.4T  2.5T  1.5T  63% /mnt/BigVault
>>
>> Although I've heard and read about the difficulty in reporting free
>> space due to the flexibility of BTRFS, snapshots and subvolumes, etc.,
>> but I only have a single volume, no subvolumes, no snapshots, no
>> quotas and both data and metadata are RAID1.
>>
>> My expectation would've been that in case of BigVault Size == Used +
>> Avail.
>>
>> Actually based on http://carfax.org.uk/btrfs-usage/index.html I
>> would've expected 6 TB of usable space. Here I get 6.4 which is odd,

Total size is estimation which in this case is computed as (sum of
device sizes)/2 which is approximately 6.4TiB.

>> but that only 1.5 TB is available is even stranger.
>>
>> Could anyone explain what I did wrong or why my expectations are
>> wrong?
>>
>> Thank you in advance
> 
> Btrfs reports estimated free space from the free space of the smallest
> member as it can only guarantee that.

It's not exactly true. For three devices with free space of 1TiB, 2TiB
and 3TiB it would return 2TiB as available space. But it is not
sophisticated enough to notice that it actually has 3TiB available.

I wonder if this is only free space calculation or actual allocation
algorithm behaves similar (effectively ignoring part of available space).

Re: Help me understand what is going on with my RAID1 FS

[Dmitrii Tcvetkov]

>Actually based on http://carfax.org.uk/btrfs-usage/index.html I
>would've expected 6 TB of usable space. Here I get 6.4 which is odd,
>but that only 1.5 TB is available is even stranger.
>
>Could anyone explain what I did wrong or why my expectations are wrong?
>
>Thank you in advance

I'd say df and the website calculate different things. In btrfs raid1 profile stores exactly 2 copies of data, each copy is on separate device. 
So by adding third drive, no matter how big, effective free space didn't expand because btrfs still needs space on any one of other two drives to store second half of each raid1 chunk stored on that third drive. 

Basically:

Drive1      Drive2    Drive3
X               X
X                               X
                  X             X

Where X is a chunk of raid1 block group.

Re: Help me understand what is going on with my RAID1 FS

[Andrei Borzenkov]

10.09.2017 19:11, Dmitrii Tcvetkov пишет:
>> Actually based on http://carfax.org.uk/btrfs-usage/index.html I
>> would've expected 6 TB of usable space. Here I get 6.4 which is odd,
>> but that only 1.5 TB is available is even stranger.
>>
>> Could anyone explain what I did wrong or why my expectations are wrong?
>>
>> Thank you in advance
> 
> I'd say df and the website calculate different things. In btrfs raid1 profile stores exactly 2 copies of data, each copy is on separate device. 
> So by adding third drive, no matter how big, effective free space didn't expand because btrfs still needs space on any one of other two drives to store second half of each raid1 chunk stored on that third drive. 
> 
> Basically:
> 
> Drive1      Drive2    Drive3
> X           X
> X                         X
>             X             X
> 
> Where X is a chunk of raid1 block group.

But this table clearly shows that adding third drive increases free
space by 50%. You need to reallocate data to actually make use of it,
but it was done in this case.

Re: Help me understand what is going on with my RAID1 FS

[Dmitrii Tcvetkov]

> > Drive1      Drive2    Drive3
> > X           X
> > X                         X
> >             X             X
> > 
> > Where X is a chunk of raid1 block group.  
> 
> But this table clearly shows that adding third drive increases free
> space by 50%. You need to reallocate data to actually make use of it,
> but it was done in this case.

It increases it but I don't see how this space is in any way useful
unless data is in single profile. After full balance chunks will be
spread over 3 devices, how it helps in raid1 data profile case?

Re: Help me understand what is going on with my RAID1 FS

[Andrei Borzenkov]

10.09.2017 23:17, Dmitrii Tcvetkov пишет:
>>> Drive1      Drive2    Drive3
>>> X           X
>>> X                         X
>>>             X             X
>>>
>>> Where X is a chunk of raid1 block group.  
>>
>> But this table clearly shows that adding third drive increases free
>> space by 50%. You need to reallocate data to actually make use of it,
>> but it was done in this case.
> 
> It increases it but I don't see how this space is in any way useful
> unless data is in single profile. After full balance chunks will be
> spread over 3 devices, how it helps in raid1 data profile case?
> 
A1 A2  => A1 A2 - => A1 A2 B1 => A1 A2 B1
B1 B2     B1 B2 -    -  B2  -    C1 B2 C2

It is raid1 profile on three disks fully utilizing them (assuming equal
sizes of course). Where "raid1" means - each data block has two copies
on different devices.

Re: Help me understand what is going on with my RAID1 FS

[FLJ]

Thanks everyone for the helpful and detailed responses.
Now that you confirmed that everything is fine with my FS, I'm all
relaxed because I can for sure live with the output of df.



On Mon, Sep 11, 2017 at 5:29 AM, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
> 10.09.2017 23:17, Dmitrii Tcvetkov пишет:
>>>> Drive1      Drive2    Drive3
>>>> X           X
>>>> X                         X
>>>>             X             X
>>>>
>>>> Where X is a chunk of raid1 block group.
>>>
>>> But this table clearly shows that adding third drive increases free
>>> space by 50%. You need to reallocate data to actually make use of it,
>>> but it was done in this case.
>>
>> It increases it but I don't see how this space is in any way useful
>> unless data is in single profile. After full balance chunks will be
>> spread over 3 devices, how it helps in raid1 data profile case?
>>
> A1 A2  => A1 A2 - => A1 A2 B1 => A1 A2 B1
> B1 B2     B1 B2 -    -  B2  -    C1 B2 C2
>
> It is raid1 profile on three disks fully utilizing them (assuming equal
> sizes of course). Where "raid1" means - each data block has two copies
> on different devices.

Re: Help me understand what is going on with my RAID1 FS

[Duncan]

FLJ posted on Sun, 10 Sep 2017 15:45:42 +0200 as excerpted:

> I have a BTRFS RAID1 volume running for the past year. I avoided all
> pitfalls known to me that would mess up this volume. I never
> experimented with quotas, no-COW, snapshots, defrag, nothing really.
> The volume is a RAID1 from day 1 and is working reliably until now.
> 
> Until yesterday it consisted of two 3 TB drives, something along the
> lines:
> 
> Label: 'BigVault'  uuid: a37ad5f5-a21b-41c7-970b-13b6c4db33db
>         Total devices 2 FS bytes used 2.47TiB
>         devid    1 size 2.73TiB used 2.47TiB path /dev/sdb
>         devid    2 size 2.73TiB used 2.47TiB path /dev/sdc

I'm going to try a different approach than I see in the two existing 
subthreads, so I started from scratch with my own subthread...

So the above looks reasonable so far...

> 
> Yesterday I've added a new drive to the FS and did a full rebalance
> (without filters) over night, which went through without any issues.
> 
> Now I have:
>  Label: 'BigVault'  uuid: a37ad5f5-a21b-41c7-970b-13b6c4db33db
>         Total devices 3 FS bytes used 2.47TiB
>         devid    1 size 2.73TiB used 1.24TiB path /dev/sdb
>         devid    2 size 2.73TiB used 1.24TiB path /dev/sdc
>         devid    3 size 7.28TiB used 2.48TiB path /dev/sda

That's exactly as expected, after a balance.

Note the size, 2.73 TiB (twos-power) for the smaller two, not 3 (tho it's 
probably 3 TB, tens-power), 7.28 TiB, not 8, for the larger one.

The most-free-space chunk allocation, with raid1-paired chunks, means the 
first chunk of every pair will get allocated to the largest, 7.28 TiB 
device.  The other two devices are equal in size, 2.73 TiB each, and the 
second chunk can't get allocated to the largest device as only one chunk 
of the pair can go there, so the allocator will in general alternate 
allocations from the smaller two, for the second chunk of each pair.  (I 
say in general, because metadata chunks are smaller than data chunks, so 
it's possible that two chunks in a row, a metadata chunk and a data 
chunk, will be allocated from the same device, before it switches to the 
other.)

Because the larger device is larger than the other two combined, it'll 
always get one copy, while the others fill up evenly at half the usage of 
the larger device, until both smaller devices are full, at which point 
you won't be able to allocate further raid1 chunks and you'll ENOSPC.

> # btrfs fi df /mnt/BigVault/
> Data, RAID1: total=2.47TiB, used=2.47TiB
> System, RAID1: total=32.00MiB, used=384.00KiB
> Metadata, RAID1: total=4.00GiB, used=2.74GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B

Still looks reasonable.

Note that assuming you're using a reasonably current btrfs-progs, there's 
also the btrfs fi usage and btrfs dev usage commands.  Btrfs fi df is an 
older form that has much less information than the fi and dev usage 
commands, tho between btrfs fi show and btrfs fi df, /most/ of the 
filesystem-level information in btrfs fi usage can be deduced, tho not 
necessarily the device-level detail.  Btrfs fi usage is thus preferred, 
assuming it's available to you.  (In addition to btrfs fi usage being 
newer, both it and btrfs fi df require a mounted btrfs.  If the 
filesystem refuses to mount, btrfs fi show may be all that's available.)

While I'm digressing, I'm guessing you know this already, but for others, 
global reserve is reserved from and comes out of metadata, so you can add 
global reserve total to metadata used.  Normally, btrfs won't use 
anything from the global reserve, so usage there will be zero.  If it's 
not, that's a very strong indication that your filesystem believes it is 
very short on space (even if data and metadata say they both have lots of 
unused space left, for some reason, very likely a bug in that case, the 
filesystem believes otherwise) and you need to take corrective action 
immediately, or risk the filesystem effectively going read-only when 
nothing else can be written.
 
> But still df -h is giving me:
> Filesystem           Size  Used Avail Use% Mounted on
> /dev/sdb             6.4T  2.5T  1.5T  63% /mnt/BigVault
> 
> Although I've heard and read about the difficulty in reporting free
> space due to the flexibility of BTRFS, snapshots and subvolumes, etc.,
> but I only have a single volume, no subvolumes, no snapshots, no quotas
> and both data and metadata are RAID1.

The most practical advice I've seen regarding "normal" df (that is, the 
one from coreutils, not btrfs fi df) in the case of uneven device sizes 
in particular, is simply ignore its numbers -- they're not reliable.  The 
only thing you need to be sure of is that it says you have enough space 
for whatever you're actually doing ATM, since various applications will 
trust its numbers and may refuse to do whatever filesystem operation at 
all, if it says there's not enough space.

The algorithm reasonably new coreutils df (and the kernel calls it 
depends on) uses is much better than it used to be for btrfs, but it 
remains too simplistic to get it correct in "complex" cases such as 
uneven device sizes with raid1, because it makes use of an older 
interface that simply does not and cannot for backward compatibility 
reasons, provide enough information to actually calculate accurate 
numbers.

Tho as you use space, the accuracy of what df sees as remaining should 
improve, so that by the time you're counting in 10s to a couple hundred 
GiB reported left by df, it should be accurate to within several GiB, and 
by the time you're counting in MiB, it should be accurate to that level.

Knowing the fact that your two smaller devices combined are still smaller 
than the largest device, and given the numbers provided by the btrfs fi 
show and btrfs fi df commands above, we can reasonably easily manually 
calculate the total usable and unused space, but don't expect coreutils' 
df to do it, because it simply doesn't have the information available to 
it that it would need to be accurate.

Again, btrfs fi usage should be quite helpful here.  But let's just 
calculate given the above.

* Given that the two smaller devices will fill up evenly, and when 
they're full, no more raid1 chunks can be allocated, we can sum their 
sizes to get the total usable:

>From the btrfs fi show output:

2.73 TiB * 2 ~= 5.5 TiB total usable space.

(Note again that we're working in TiB, twos-power, not TB, tens-power, so 
it's not 6 TiB usable, tho it may be 6 TB tens-power usable.)

Of that ~ 5.5 TiB usable, ~ 1.25 * 2 TiB ~= 2.5 TiB is used (that is, 
allocated to chunks).

You should thus have ~ 5.5 TiB - 2.5 TiB = 3 TiB usable-as-raid1-space to 
be allocated.

In addition to that, you can look at btrfs fi df (or usage, which will 
provide a more practically usable without additional math output) to see 
how much space is remaining within already allocated chunks.  As it 
happens, since you just did a full balance, there's not significant 
already allocated chunk-space that's not yet actually used by files, but 
after some months of normal usage without further balances, you'll likely 
have tens to hundreds of GiB of chunk-allocated but not yet used space 
available, enough so it'd show in the hundredths-TiB figures reported.

> My expectation would've been that in case of BigVault Size == Used +
> Avail.
> 
> Actually based on http://carfax.org.uk/btrfs-usage/index.html I would've
> expected 6 TB of usable space. Here I get 6.4 which is odd,
> but that only 1.5 TB is available is even stranger.

If by that you mean you'd expect it to say 6T, instead of the 6.4T it 
lists, you'd be failing to account for the fact that df -h reports in 
powers-of-two, not powers-of-10 (despite it not using the standardized 
TiB, as df's output likely predates the TiB standard significantly, and 
again, that'd be changing the interface that many scripts have 
standardized on over the years).  If you wanted powers-of-10, you'd use
-H instead. See the manpage.

But of course 6.4 TiB is even further from the expected ~ 5.5 TiB than 
from the powers-mixed-up 6T you mention...

Of course you could dig into the specific df code and see where it gets 
its numbers if you wanted.  But in practice, it doesn't matter.  What 
matters in practice is that (coreutils') df's numbers simply aren't 
reliable in complex btrfs cases such as yours.  After the changes a few 
versions ago, they're /somewhat/ accurate in less complex cases like your 
previous setup, two devices of identical size in raid1.  

Meanwhile, as it happens two identically sized devices in btrfs raid1 
happens to be what I'm running here for all my btrfs except the /boot and 
its backups (which are single device dup mode), so coreutils' df happens 
to be relatively accurate for me, too, but I still don't rely on it, 
because I've simply learned not to.  FWIW, I actually don't tend to run 
normal df much at all these days, but do see the same numbers reported as 
total and free in my file managers (generally mc for admin hat work, 
sometimes kde's dolphin or gwenview or the like when I'm wearing my user 
hat).  As I said, mostly all I worry about is whether they show enough 
room for my current operations.  If they look way out of whack, I'll run 
the appropriate btrfs commands in a terminal to see what's up, but I 
don't trust the df/fileman numbers, because I know on btrfs, they really 
/cannot/ be trusted.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman