Re: [PATCH RFC 00/10] btrfs: new performance-based chunk allocation using device roles

[Anand Jain <anand.jain@oracle.com>]

On 20/5/25 17:19, Forza wrote:
> Hi,
> 
> On 2025-05-12 20:07, Anand Jain wrote:
>> In host hardware, devices can have different speeds. Generally, faster
>> devices come with lesser capacity while slower devices come with larger
>> capacity. A typical configuration would expect that:
>>
>>   - A filesystem's read/write performance is evenly distributed on 
>> average
>>   across the entire filesystem. This is not achievable with the current
>>   allocation method because chunks are allocated based only on device 
>> free
>>   space.
>>
>>   - Typically, faster devices are assigned to metadata chunk allocations
>>   while slower devices are assigned to data chunk allocations.
>>
>> Introducing Device Roles:
>>
>>   Here I define 5 device roles in a specific order for metadata and in 
>> the
>>   reverse order for data: metadata_only, metadata, none, data, data_only.
>>   One or more devices may have the same role.
>>
>>   The metadata and data roles indicate preference but not exclusivity for
>>   that role, whereas data_only and metadata_only are exclusive roles.
> 
> This sounds like the old preferred_metadata (Allocator Hints) patch 
> series from Goffredo Baroncelli[1] back in the 2020, now being 
> maintained and improved by Kai Krakow[2] and others. Is this an updated/ 
> enhanced version of those patches?
> 

Thanks for the comments.

I haven't reviewed the implementation details of [1], so I can't make a
direct comparison. The goal here is to define a generic device priority
range from 1 to 255, which can be externally assigned and stored.

In one of the current modes under development, ROLE_THEN_SPACE, devices
are first grouped by three priority levels, then sorted by available
free space at the time of allocation.

I’m calling them generic device priorities because even when all devices
have similar performance—as is common in most general-purpose setups—we
can still use priorities to enable simple, linear allocation for the
single profile.

>>
>> Introducing Role-then-Space allocation method:
>>
>>   Metadata allocation can happen on devices with the roles metadata_only,
>>   metadata, none, and data in that order. If multiple devices share a 
>> role,
>>   they are arranged based on device free space.
>>
>>   Similarly, data allocation can happen on devices with the roles 
>> data_only,
>>   data, none, and metadata in that order. If multiple devices share a 
>> role,
>>   they are arranged based on device free space.
> 
> The Allocator Hints patch series show that this is a good method. We are 
> several users that use those, also in production environments to good 
> effect. Some argue that having more tiers would be beneficial, it could 
> be combined with defrag or balance operation to place data on slow or 
> fast storage.
> 
>>
>> Finding device speed automatically:
>>
>>   Measuring device read/write latency for the allocaiton is not good 
>> idea,
>>   as the historical readings and may be misleading, as they could include
>>   iostat data from periods with issues that have since been fixed. 
>> Testing
>>   to determine relative latency and arranging in ascending order for 
>> metadata
>>   and descending for data is possible, but is better handled by an 
>> external
>>   tool that can still set device roles.
> 
> Benchmarks using round-robin, latency and latency-round-robin and queue 
> based scheduling show that latency based allocation can be particularly 
> useful for some workloads and device types. It is difficult to 
> generalise, but based on benchmarks we see that a good all-rounder is a 
> queue based approach. See [3] for a complete set of raw data from these 
> benchmarks.
> 

I'm not commenting on the implementation details. My point is that
dynamic latency-based allocation was previously rejected because
temporary latency spikes can mislead the allocator and cause data to
land on the wrong device.

That said, for reads, there are indeed patches that support a latency-
based read_policy.

> 
> |  # | Storage    | Jobs | Test                | Policy      |   IOPS  |
> | -: | :--------- | ---: | :------------------ | :---------- | ------: |
> |  1 | HDD RAID1  |    1 | RandRead 4 KiB QD1  | pid         |      81 |
> |  2 | HDD RAID1  |    1 | RandRead 4 KiB QD1  | round-robin |      93 |
> |  3 | HDD RAID1  |    1 | RandRead 4 KiB QD1  | latency     |      89 |
> |  4 | HDD RAID1  |    1 | RandRead 4 KiB QD1  | latency-rr  |      87 |
> |  5 | HDD RAID1  |    1 | RandRead 4 KiB QD1  | queue       |     102 |
> |  6 | SSD RAID10 |    1 | RandRead 4 KiB QD32 | pid         |  68 800 |
> |  7 | SSD RAID10 |    1 | RandRead 4 KiB QD32 | round-robin | 143 000 |
> |  8 | SSD RAID10 |    1 | RandRead 4 KiB QD32 | latency     | 142 000 |
> |  9 | SSD RAID10 |    1 | RandRead 4 KiB QD32 | latency-rr  | 137 000 |
> | 10 | SSD RAID10 |    1 | RandRead 4 KiB QD32 | queue       | 143 000 |
> 
> (table wraps)
> 
> |  # | Policy      | BW (KiB/s) | Avg Lat (ms) | 99 % Lat | 99.9 % Lat |
> | -: | :---------- | ---------: | -----------: | -------: | ---------: |
> |  1 | pid         |        328 |        0.310 |   30.016 |    242.222 |
> |  2 | round-robin |        374 |        0.091 |   26.084 |     60.031 |
> |  3 | latency     |        358 |        0.041 |   26.608 |     32.900 |
> |  4 | latency-rr  |        348 |        0.041 |   28.181 |     33.817 |
> |  5 | queue       |        409 |        0.050 |   24.511 |     35.390 |
> |  6 | pid         |    275 456 |        0.458 |    8.029 |     10.290 |
> |  7 | round-robin |    572 416 |        0.217 |    0.338 |      0.627 |
> |  8 | latency     |    569 344 |        0.219 |    0.306 |      0.400 |
> |  9 | latency-rr  |    547 840 |        0.227 |    0.326 |      0.449 |
> | 10 | queue       |    571 392 |        0.218 |    0.457 |      0.594 |
> 
> I think md uses a mix of queue based and sector-distance based approach 
> depending on device type[4].
> 
>>
>> On-Disk Format changes:
>>
>>   The following items are defined but are unused on-disk format:
>>
>>     btrfs_dev_item::
>>      __le64 type; // unused
>>      __le64 start_offset; // unused
>>      __le32 dev_group; // unused
>>      __u8 seek_speed; // unused
>>      __u8 bandwidth; // unused
>>
>>   The device roles is using the dev_item::type 8-bit field to store each
>>   device's role.
>>
>> Anand Jain (10):
>>   btrfs: fix thresh scope in should_alloc_chunk()
>>   btrfs: refactor should_alloc_chunk() arg type
>>   btrfs: introduce btrfs_split_sysfs_arg() for argument parsing
>>   btrfs: introduce device allocation method
>>   btrfs: sysfs: show device allocation method
>>   btrfs: skip device sorting when only one device is present
>>   btrfs: refactor chunk allocation device handling to use list_head
>>   btrfs: introduce explicit device roles for block groups
>>   btrfs: introduce ROLE_THEN_SPACE device allocation method
>>   btrfs: pass device roles through device add ioctl
> 
> 
> 
> Have you considered how to deal with `df` and disk free calculation? Are 
> device roles preserved during `btrfs device replace`?
>

This is the foundational framework; the remaining features will be added
progressively.

Thanks!
Anand

> Thank you!
> 
> [1] https://lore.kernel.org/linux- 
> btrfs/20210116002533.GE31381@hungrycats.org/T/
> [2] https://github.com/kakra/linux/pull/36
> [3] https://gist.github.com/kakra/ce99896e5915f9b26d13c5637f56ff37
> [4] https://github.com/torvalds/linux/blob/ 
> a5806cd506af5a7c19bcd596e4708b5c464bfd21/drivers/md/raid1.c#L832-L843
> 
> 
> 
>