Re: df free space not correct with raid1 pools with an odd number of devices

[Andrei Borzenkov <arvidjaar@gmail.com>]

24.07.2020 23:46, Chris Murphy пишет:
> On Fri, Jul 24, 2020 at 2:16 AM Jorge Bastos <jorge.mrbastos@gmail.com> wrote:
>>
>>>> Filesystem      Size  Used Avail Use% Mounted on
>>>> /dev/sdd1       1.2T  3.4M  931G   1% /mnt/cache
> 
> Oh yeah Avail is clearly goofy.
> 
> 
>> Filesystem      Size  Used Avail Use% Mounted on
>> /dev/sdd1       699G  3.4M  466G   1% /mnt/cache
> 
> 
> Anybody know what's up?
> 
> 

df "Used" and "Avail" are totally independent values.

"Used" is computed as (total - free), both of which are reported by
statfs. By default df does not show "Free", you need to use --output=
option (at least using coreutils df).

"Avail" is computed by filesystem. Originally the difference comes from
"available to root" and "available to user" .

btrfs computes "Avail" by simulating chunk allocations on devices. See
super.c:btrfs_calc_avail_data_space(), the final chunk:

        btrfs_descending_sort_devices(devices_info, nr_devices);

        i = nr_devices - 1;
        avail_space = 0;
        while (nr_devices >= rattr->devs_min) {
                num_stripes = min(num_stripes, nr_devices);

                if (devices_info[i].max_avail >= min_stripe_size) {
                        int j;
                        u64 alloc_size;

                        avail_space += devices_info[i].max_avail *
num_stripes;
                        alloc_size = devices_info[i].max_avail;
                        for (j = i + 1 - num_stripes; j <= i; j++)
                                devices_info[j].max_avail -= alloc_size;
                }
                i--;
                nr_devices--;
        }

        kfree(devices_info);
        *free_bytes = avail_space;

devices_info holds device list sorted by unallocated space. We start
with device with smallest available space and add its full available
space (adjusted by allocation profile), then move to the previous device
with more free space.

The problem is that if we have three equal sized devices and RAID1
profile, the first iteration consumes two full devices, thus third
device cannot be used anymore (we need two of them for raid1). Real
allocator will evaluate free space every time and so alternate between
all three devices.