Re: [PATCH RFC 00/10] btrfs: new performance-based chunk allocation using device roles

[Anand Jain <anand.jain@oracle.com>]

On 23/5/25 02:19, waxhead wrote:
> Anand Jain wrote:
>> In host hardware, devices can have different speeds. Generally, faster
>> devices come with lesser capacity while slower devices come with larger
>> capacity. A typical configuration would expect that:
>>
>>   - A filesystem's read/write performance is evenly distributed on 
>> average
>>   across the entire filesystem. This is not achievable with the current
>>   allocation method because chunks are allocated based only on device 
>> free
>>   space.
>>
>>   - Typically, faster devices are assigned to metadata chunk allocations
>>   while slower devices are assigned to data chunk allocations.
>>
>> Introducing Device Roles:
>>
>>   Here I define 5 device roles in a specific order for metadata and in 
>> the
>>   reverse order for data: metadata_only, metadata, none, data, data_only.
>>   One or more devices may have the same role.
>>
>>   The metadata and data roles indicate preference but not exclusivity for
>>   that role, whereas data_only and metadata_only are exclusive roles.
> 
> As a BTRFS user I would like to comment a bit on this. I have earlier 
> mentioned that I think that BTRFS should allow for device groups. E.g. 
> assigning a storage device to one or more groups (or vice versa).
> 
> I really like what is being introduced here, but I would like to suggest 
> to take this a step further. Instead of assigning a role to the storage 
> device itself then maybe it would have been wiser to follow a scheme 
> like this:
> 
> DeviceID -> Group(s) -> Group properties
> 
> In this case what is being introduced here could easily be dealt with as 
> a simple group property like (meta)data_weight=0...128 for example.
> 
> Personally I think that would have been a much cleaner interface.
> 
> Setting a metadata/data roles as originally suggested here would be fine 
> on a low number of devices, but on larger storage arrays with many 
> devices it sounds (to me) like it would quickly become difficult to keep 
> track of.
> 
> With the scheme I suggest you would simply list the properties of a 
> group and see what DeviceID's that belong in that group... perhaps even 
> in a nice table if you where lucky.
> 
> (And just for the record: other properties I can from the top of my head 
> imagine that would be useful would be read/write weight that could 
> (automatically) be set higher and higher if a device starts to throw 
> errors, or group_exclusive=1|0 (to prevent other groups owning that 
> DeviceID etc... etc...)
> 
> And this would of course require another step after mkfs, but personally 
> I do not understand why setting these roles (or the scheme I suggest) 
> would be very useful at mkfs time. It might as well be done at first 
> mount before the filesystem gets put to use.
> 
> Great to see progress for BTRFS for things like this , but please do 
> consider another scheme for setting the roles.


Thanks for the feedback.

The question is: which approach handles large numbers of devices
better, Mode Groups or Direct Modes?

Let’s try to break it down.

Both approaches need to manage the following:

Five role types (preferences):
    metadata_only, metadata, none (any), data, data_only

Fault tolerance (FT) groups:
    2 to n device groups

Four allocation strategies:
    linear-devid, linear-priority, round-robin, free-space

Pros and Cons:
Direct Modes are simpler and work well for small setups. As things
scale, complexity grows, but scripts or tooling can manage that.

Mode Groups are better organized for large setups, but may be overkill
for small ones. They also require managing an extra btrfs key, which
adds some overhead.

Did I miss anything?

So far, I'm leaning toward Direct Modes. But if there's enough interest
in Mode Groups, we can explore that too. Alternatively, we could start
with Direct Modes and add Mode Groups later if needed.
Does that sound reasonable?

I’ve put up a draft work in progress version of the proposal here:

   https://asj.github.io/chunk-alloc-enhancement.html

Thanks, Anand