Re: [PATCH RFC 00/10] btrfs: new performance-based chunk allocation using device roles

[Anand Jain <anand.jain@oracle.com>]

On 23/5/25 04:39, Ferry Toth wrote:
> Hi,
> 
> Op 12-05-2025 om 20:07 schreef Anand Jain:
>> In host hardware, devices can have different speeds. Generally, faster
>> devices come with lesser capacity while slower devices come with larger
>> capacity. A typical configuration would expect that:
>>
>>   - A filesystem's read/write performance is evenly distributed on 
>> average
>>   across the entire filesystem. This is not achievable with the current
>>   allocation method because chunks are allocated based only on device 
>> free
>>   space.
>>
>>   - Typically, faster devices are assigned to metadata chunk allocations
>>   while slower devices are assigned to data chunk allocations.
> 
> Finally a new effort in this direction.
> 
>> Introducing Device Roles:
>>
>>   Here I define 5 device roles in a specific order for metadata and in 
>> the
>>   reverse order for data: metadata_only, metadata, none, data, data_only.
>>   One or more devices may have the same role.
>>
>>   The metadata and data roles indicate preference but not exclusivity for
>>   that role, whereas data_only and metadata_only are exclusive roles.
>>
>> Introducing Role-then-Space allocation method:
>>
>>   Metadata allocation can happen on devices with the roles metadata_only,
>>   metadata, none, and data in that order. If multiple devices share a 
>> role,
>>   they are arranged based on device free space.
>>
>>   Similarly, data allocation can happen on devices with the roles 
>> data_only,
>>   data, none, and metadata in that order. If multiple devices share a 
>> role,
>>   they are arranged based on device free space.
> 
> I can see the use case for large pools of disks used in server 
> environments where disks get assigned a role.
> 
> For desktop use I would like it a lot better with no roles, just a 
> performance-based chunk allocation to select between a ssd and a hdd. 
> And then used more like a hint to the allocator. Really nothing should 
> go wrong if a data or meta-data gets allocated on the wrong / sub- 
> optimal disk.
> 
> This could then bring back the old hot relocation idea, finally.
> 
>> Finding device speed automatically:
>>
>>   Measuring device read/write latency for the allocaiton is not good 
>> idea,
>>   as the historical readings and may be misleading, as they could include
>>   iostat data from periods with issues that have since been fixed. 
>> Testing
>>   to determine relative latency and arranging in ascending order for 
>> metadata
>>   and descending for data is possible, but is better handled by an 
>> external
>>   tool that can still set device roles.
>>
>> On-Disk Format changes:
>>
>>   The following items are defined but are unused on-disk format:
>>
>>     btrfs_dev_item::
>>      __le64 type; // unused
>>      __le64 start_offset; // unused
>>      __le32 dev_group; // unused
>>      __u8 seek_speed; // unused
>>      __u8 bandwidth; // unused
>>
>>   The device roles is using the dev_item::type 8-bit field to store each
>>   device's role.
> 
> I think filling the fields with either measured or user entered data 
> should be fine, as long as when the disk behavior changes you can re- 
> measure or re-enter.
> 
> The difference between a ssd and a hdd will be so huge small changes 
> will have no real effect.


Yeah, for desktop setups with SSDs and HDDs, the distinction is clear
and stable, so assigning data or metadata based on device type makes
sense. It’s straightforward to handle statically, and a
--set-roles-by-type mkfs option will make it automatic.

Even if the SSD temporarily slows down during a balance, we’d still
prefer to keep metadata on it, assuming the slowdown is short-lived.
SSD performance typically recovers, so there's no need to overreact
to transient dips.

For virtual devices, mkfs --set-roles-by-iostat should also work well.
And later if performance characteristics change permanently, a
balance-time option like --recalibrate-role-by-iostat could
re-evaluate based on I/O stats, confirm with the user, and relocate
chunks accordingly.

Also, I'm trying not to introduce too many options or configuration
paths, just enough to keep Btrfs simple to use.

Does that sound reasonable?

Thanks, Anand

>> Anand Jain (10):
>>    btrfs: fix thresh scope in should_alloc_chunk()
>>    btrfs: refactor should_alloc_chunk() arg type
>>    btrfs: introduce btrfs_split_sysfs_arg() for argument parsing
>>    btrfs: introduce device allocation method
>>    btrfs: sysfs: show device allocation method
>>    btrfs: skip device sorting when only one device is present
>>    btrfs: refactor chunk allocation device handling to use list_head
>>    btrfs: introduce explicit device roles for block groups
>>    btrfs: introduce ROLE_THEN_SPACE device allocation method
>>    btrfs: pass device roles through device add ioctl
>>
>>   fs/btrfs/block-group.c |  11 +-
>>   fs/btrfs/ioctl.c       |  12 +-
>>   fs/btrfs/sysfs.c       | 130 ++++++++++++++++++++--
>>   fs/btrfs/volumes.c     | 242 +++++++++++++++++++++++++++++++++--------
>>   fs/btrfs/volumes.h     |  35 +++++-
>>   5 files changed, 366 insertions(+), 64 deletions(-)
>>
>