Re: df free space not correct with raid1 pools with an odd number of devices

[Andrei Borzenkov <arvidjaar@gmail.com>]

25.07.2020 10:30, Andrei Borzenkov пишет:
> 24.07.2020 23:46, Chris Murphy пишет:
>> On Fri, Jul 24, 2020 at 2:16 AM Jorge Bastos <jorge.mrbastos@gmail.com> wrote:
>>>
>>>>> Filesystem      Size  Used Avail Use% Mounted on
>>>>> /dev/sdd1       1.2T  3.4M  931G   1% /mnt/cache
>>
>> Oh yeah Avail is clearly goofy.
>>
>>
>>> Filesystem      Size  Used Avail Use% Mounted on
>>> /dev/sdd1       699G  3.4M  466G   1% /mnt/cache
>>
>>
>> Anybody know what's up?
>>
>>
> 
> df "Used" and "Avail" are totally independent values.
> 
> "Used" is computed as (total - free), both of which are reported by
> statfs. By default df does not show "Free", you need to use --output=
> option (at least using coreutils df).
> 
> "Avail" is computed by filesystem. Originally the difference comes from
> "available to root" and "available to user" .
> 
> btrfs computes "Avail" by simulating chunk allocations on devices. See
> super.c:btrfs_calc_avail_data_space(), the final chunk:
> 
>         btrfs_descending_sort_devices(devices_info, nr_devices);
> 
>         i = nr_devices - 1;
>         avail_space = 0;
>         while (nr_devices >= rattr->devs_min) {
>                 num_stripes = min(num_stripes, nr_devices);
> 
>                 if (devices_info[i].max_avail >= min_stripe_size) {
>                         int j;
>                         u64 alloc_size;
> 
>                         avail_space += devices_info[i].max_avail *
> num_stripes;
>                         alloc_size = devices_info[i].max_avail;
>                         for (j = i + 1 - num_stripes; j <= i; j++)
>                                 devices_info[j].max_avail -= alloc_size;
>                 }
>                 i--;
>                 nr_devices--;
>         }
> 
>         kfree(devices_info);
>         *free_bytes = avail_space;
> 
> devices_info holds device list sorted by unallocated space. We start
> with device with smallest available space and add its full available
> space (adjusted by allocation profile), then move to the previous device
> with more free space.
> 
> The problem is that if we have three equal sized devices and RAID1
> profile, the first iteration consumes two full devices, thus third
> device cannot be used anymore (we need two of them for raid1). Real
> allocator will evaluate free space every time and so alternate between
> all three devices.
> 
> 

OTOH, this is the correct if the most pessimistic estimation either. If
you have three 250G RAID1 devices and you allocate 250G data in one file
you consume two full devices and won't be able to allocate new data at
all (or for that matter no new metadata either).


So whatever value btrfs returns will be wrong for some allocation pattern.