Re: [PATCH RFC 00/10] btrfs: new performance-based chunk allocation using device roles

[Ferry Toth <fntoth@gmail.com>]

Hi,

Op 02-06-2025 om 06:24 schreef Anand Jain:
> On 23/5/25 04:39, Ferry Toth wrote:
>> Hi,
>>
>> Op 12-05-2025 om 20:07 schreef Anand Jain:
>>> In host hardware, devices can have different speeds. Generally, faster
>>> devices come with lesser capacity while slower devices come with larger
>>> capacity. A typical configuration would expect that:
>>>
>>>   - A filesystem's read/write performance is evenly distributed on 
>>> average
>>>   across the entire filesystem. This is not achievable with the current
>>>   allocation method because chunks are allocated based only on device 
>>> free
>>>   space.
>>>
>>>   - Typically, faster devices are assigned to metadata chunk allocations
>>>   while slower devices are assigned to data chunk allocations.
>>
>> Finally a new effort in this direction.
>>
>>> Introducing Device Roles:
>>>
>>>   Here I define 5 device roles in a specific order for metadata and 
>>> in the
>>>   reverse order for data: metadata_only, metadata, none, data, 
>>> data_only.
>>>   One or more devices may have the same role.
>>>
>>>   The metadata and data roles indicate preference but not exclusivity 
>>> for
>>>   that role, whereas data_only and metadata_only are exclusive roles.
>>>
>>> Introducing Role-then-Space allocation method:
>>>
>>>   Metadata allocation can happen on devices with the roles 
>>> metadata_only,
>>>   metadata, none, and data in that order. If multiple devices share a 
>>> role,
>>>   they are arranged based on device free space.
>>>
>>>   Similarly, data allocation can happen on devices with the roles 
>>> data_only,
>>>   data, none, and metadata in that order. If multiple devices share a 
>>> role,
>>>   they are arranged based on device free space.
>>
>> I can see the use case for large pools of disks used in server 
>> environments where disks get assigned a role.
>>
>> For desktop use I would like it a lot better with no roles, just a 
>> performance-based chunk allocation to select between a ssd and a hdd. 
>> And then used more like a hint to the allocator. Really nothing should 
>> go wrong if a data or meta-data gets allocated on the wrong / sub- 
>> optimal disk.
>>
>> This could then bring back the old hot relocation idea, finally.
>>
>>> Finding device speed automatically:
>>>
>>>   Measuring device read/write latency for the allocaiton is not good 
>>> idea,
>>>   as the historical readings and may be misleading, as they could 
>>> include
>>>   iostat data from periods with issues that have since been fixed. 
>>> Testing
>>>   to determine relative latency and arranging in ascending order for 
>>> metadata
>>>   and descending for data is possible, but is better handled by an 
>>> external
>>>   tool that can still set device roles.
>>>
>>> On-Disk Format changes:
>>>
>>>   The following items are defined but are unused on-disk format:
>>>
>>>     btrfs_dev_item::
>>>      __le64 type; // unused
>>>      __le64 start_offset; // unused
>>>      __le32 dev_group; // unused
>>>      __u8 seek_speed; // unused
>>>      __u8 bandwidth; // unused
>>>
>>>   The device roles is using the dev_item::type 8-bit field to store each
>>>   device's role.
>>
>> I think filling the fields with either measured or user entered data 
>> should be fine, as long as when the disk behavior changes you can re- 
>> measure or re-enter.
>>
>> The difference between a ssd and a hdd will be so huge small changes 
>> will have no real effect.
> 
> 
> Yeah, for desktop setups with SSDs and HDDs, the distinction is clear
> and stable, so assigning data or metadata based on device type makes
> sense. It’s straightforward to handle statically, and a
> --set-roles-by-type mkfs option will make it automatic.
> 
> Even if the SSD temporarily slows down during a balance, we’d still
> prefer to keep metadata on it, assuming the slowdown is short-lived.
> SSD performance typically recovers, so there's no need to overreact
> to transient dips.
> 
> For virtual devices, mkfs --set-roles-by-iostat should also work well.
> And later if performance characteristics change permanently, a
> balance-time option like --recalibrate-role-by-iostat could
> re-evaluate based on I/O stats, confirm with the user, and relocate
> chunks accordingly.
> 
> Also, I'm trying not to introduce too many options or configuration
> paths, just enough to keep Btrfs simple to use.
> 
> Does that sound reasonable?

That sounds very good.

I  am curious what happens when the fast device fills up, what will the 
allocator do? I guess it will fall back to allocating to the slow device?

If so, we're going to need some periodic or just in time "move files 
that have for a long time not been written / read" to the slow disk.

While that file my be referenced from multiple subvolumes, and you 
wouldn't want those duped (like happens with defragmenting).

> Thanks, Anand
> 
>>> Anand Jain (10):
>>>    btrfs: fix thresh scope in should_alloc_chunk()
>>>    btrfs: refactor should_alloc_chunk() arg type
>>>    btrfs: introduce btrfs_split_sysfs_arg() for argument parsing
>>>    btrfs: introduce device allocation method
>>>    btrfs: sysfs: show device allocation method
>>>    btrfs: skip device sorting when only one device is present
>>>    btrfs: refactor chunk allocation device handling to use list_head
>>>    btrfs: introduce explicit device roles for block groups
>>>    btrfs: introduce ROLE_THEN_SPACE device allocation method
>>>    btrfs: pass device roles through device add ioctl
>>>
>>>   fs/btrfs/block-group.c |  11 +-
>>>   fs/btrfs/ioctl.c       |  12 +-
>>>   fs/btrfs/sysfs.c       | 130 ++++++++++++++++++++--
>>>   fs/btrfs/volumes.c     | 242 +++++++++++++++++++++++++++++++++--------
>>>   fs/btrfs/volumes.h     |  35 +++++-
>>>   5 files changed, 366 insertions(+), 64 deletions(-)
>>>
>>
>