Re: [PATCH RFC 00/10] btrfs: new performance-based chunk allocation using device roles

[Anand Jain <anand.jain@oracle.com>]

Thanks for the detailed proposal, more below..

On 22/5/25 12:07, Zygo Blaxell wrote:
> On Tue, May 13, 2025 at 02:07:06AM +0800, Anand Jain wrote:
>> In host hardware, devices can have different speeds. Generally, faster
>> devices come with lesser capacity while slower devices come with larger
>> capacity. A typical configuration would expect that:
>>
>>   - A filesystem's read/write performance is evenly distributed on average
>>   across the entire filesystem. This is not achievable with the current
>>   allocation method because chunks are allocated based only on device free
>>   space.
>>
>>   - Typically, faster devices are assigned to metadata chunk allocations
>>   while slower devices are assigned to data chunk allocations.
>>
>> Introducing Device Roles:
>>
>>   Here I define 5 device roles in a specific order for metadata and in the
>>   reverse order for data: metadata_only, metadata, none, data, data_only.
>>   One or more devices may have the same role.
>>
>>   The metadata and data roles indicate preference but not exclusivity for
>>   that role, whereas data_only and metadata_only are exclusive roles.
> 
> Using role-based names like these presents three problems:
> 
> 1. **Stripe incompatibility** -- These roles imply a hierarchy that breaks
> in some multi-device arrays. e.g. with 5 devices of equal size and mixed
> roles ("data_only" vs "data"), it's impossible to form a 5-device-wide
> data chunk.
> 
Thanks for the feedback.

Details about the current proposal are here:

[1] https://asj.github.io/chunk-alloc-enhancement.html

Some allocation modes aren't compatible with certain block group
profiles. We'll need to check this at mkfs time and fail the command if
the number of devices is below the minimum required.

The role hierarchy (exclusive-> none-> non-exclusive) only applies when
there are more devices than required for a given block group profile and
the allocator has a choice of which devices to use.

The use case for non-exclusive roles with striped profiles isn't very
practical, but the design allows for future extensions if needed.

> 2. **Poor extensibility** -- The role system doesn't scale when
> introducing additional allocation types. Any new category (e.g. PPL or
> journal) would require duplicating preference permutations like "data,
> then journal, then metadata" vs "journal, then data, then metadata",
> resulting in combinatorial explosion.

Special devices like journal or write-cache are different; they are
separate from the data and metadata storage devices. We will still hit
ENOSPC even if the journal device is empty.

That said, it is still possible to specify write-cache as a role. For
example:

	mkfs.btrfs /dev/sdx:write-cache ...

I'm not sure I understood what you meant by "not extensible"?

Also, allocation modes (for example, FREE_SPACE, ROLE, LINEAR,
ROUND_ROBIN) are designed to be composable as needed.

If roles do not cover a specific use case, the existing alloc_priority
(1 to 255) and alloc_mode can be extended to support new logic.

Note: LINEAR and ROUND_ROBIN are not implemented yet.

> 3. **Misleading terminology** -- The name "none" is used in a misleading
> way.  That name should be reserved for the case that prohibits all new
> chunk allocations--a critical use case for array reshaping. A clearer
> term would be "default," but the scheme would be even clearer if all
> the legacy role names were discarded.
> 

Got it, I'll rename none to default.

"None" is internal to the kernel and means no particular role
preference. It currently falls into the middle tier (41 to 80) of
alloc_priority, but we could adjust that to something more meaningful if
needed.

> I suggest replacing roles with a pair of orthogonal properties per device
> for each allocation type:
> 
> * Per-type tier level -- A simple u8 tier number that expresses allocation
> preference. Allocators attempt to satisfy allocation using devices at
> the lowest available tier, expanding the set to higher tiers as needed
> until the minimum number of devices is reached.

This is the same as alloc_priority stored in dev_item::type:8.

> * Per-type enable bit -- Indicates whether the device allows allocations
> of that type at all. This can be stored explicitly, or encoded using a
> reserved tier value (e.g. 0xFF = disabled).

The device type can refer to a special device (like write-cache) or a
regular data/metadata device. Within a data/metadata device, the role,
whether for data or metadata, can still be represented using the current
*_only, *, or default/any roles. So this approach remains compatible.

> Encoding this way makes "0" a reasonable default value for each field.
> 
> Then you get all of the required combinations, e.g.
> 

Added below the current proposal.

> * metadata 0, data 0 - what btrfs does now, equal preference

  role=< > no role | default

> 
> * metadata 2, data 1 - metadata preferred, data allowed
> 
  role=metadata

> * metadata 1, data 2 - data preferred, metadata allowed

  role=data

> * metadata 0, data 255 - metadata only, no data

  role=metadata_only

> * metadata 255, data 0 - data only, no metadata

  role=data_only

> * metadata 255, data 255 - no new chunk allocations

  Flag it read-only.

> This model offers cleaner semantics and more robust scaling:
> 
> * It eliminates unintended allocation spillover. A device either allows
> data/metadata, or it doesn't.
> * It expresses preference via explicit tiering rather than role overlap.
> * It generalizes easily to future allocation types without rewriting
> role logic.
> 

> "Allow nothing" is an important case for reshaping arrays.  If you are
> upgrading 4 out of 12 disks in a striped raid filesystem, you don't
> want to rewrite all the data in the filesystem 4 times.  Instead, set
> the devices you want to remove to "allow nothing", run a balance with a
> `devid` filter targeting each device to evacuate the data, and then run
> device delete on the 4 empty drives.

We can do the same by setting the device read-only.


I actually started with the idea of using bitmap flags, since it's more
straightforward. But I eventually leaned toward using an Allocation
Priority list to allow for a manual priority order within roles or
tiers, if needed in the future. That flexibility pushed me in that
direction.

You can find more details about the current Allocation Priority list
here:

	https://asj.github.io/chunk-alloc-enhancement.html

That said, we could store the mode in a separate btrfs-key and keep the
manual priority in dev_item::type, which would give us both.
But as always, we try to avoid new on-disk new keys unless absolute
necessary.

>> Introducing Role-then-Space allocation method:
>>
>>   Metadata allocation can happen on devices with the roles metadata_only,
>>   metadata, none, and data in that order. If multiple devices share a role,
>>   they are arranged based on device free space.
>>
>>   Similarly, data allocation can happen on devices with the roles data_only,
>>   data, none, and metadata in that order. If multiple devices share a role,
>>   they are arranged based on device free space.
>>
>> Finding device speed automatically:
>>
>>   Measuring device read/write latency for the allocaiton is not good idea,
>>   as the historical readings and may be misleading, as they could include
>>   iostat data from periods with issues that have since been fixed. Testing
>>   to determine relative latency and arranging in ascending order for metadata
>>   and descending for data is possible, but is better handled by an external
>>   tool that can still set device roles.
>>
>> On-Disk Format changes:
>>
>>   The following items are defined but are unused on-disk format:
>>
>> 	btrfs_dev_item::
>> 	 __le64 type; // unused
>> 	 __le64 start_offset; // unused
>> 	 __le32 dev_group; // unused
>> 	 __u8 seek_speed; // unused
>> 	 __u8 bandwidth; // unused
>>
>>   The device roles is using the dev_item::type 8-bit field to store each
>>   device's role.
> 
> In the other implementations of this idea, allocation roles are stored in
> `dev_item::type`, a single `u8` field, for simplicity; however, it would
> be better to store these roles in the filesystem tree--e.g. using a
> `BTRFS_PERSISTENT_ITEM_KEY` with a dedicated objectid for allocation
> roles, and offset values corresponding to device IDs. This would enable
> versioning of the schema and flexible extension (e.g., to add migration
> policies, size-based allocation preferences, or other enhancements).
> 
> Since btrfs loads the trees before allocation can occur, tree-based
> role data will be available in time for allocation, and we don't need
> to store roles in the superblocks.
> 
> A longer version of this with use cases and some discussion is available
> here:
> 
> 	https://github.com/kakra/linux/pull/36#issuecomment-2784251968
> 
> 	https://github.com/kakra/linux/pull/36#issuecomment-2784434490
> 

dev_item::dev_type (u64) comes from the reserved field list, so there's
no additional space overhead in using it. I considered whether using a
btrfs_key for roles would offer any advantage over dev_item::dev_type,
but I couldn't find a clear benefit.

Also, with alloc_priority + alloc_mode, we can support a manual device
order with the same cost.

Let me still consider what you proposed again to see if there’s any
advantage to doing it that way.

Good discussion, thanks a lot.

-Anand

>> Anand Jain (10):
>>    btrfs: fix thresh scope in should_alloc_chunk()
>>    btrfs: refactor should_alloc_chunk() arg type
>>    btrfs: introduce btrfs_split_sysfs_arg() for argument parsing
>>    btrfs: introduce device allocation method
>>    btrfs: sysfs: show device allocation method
>>    btrfs: skip device sorting when only one device is present
>>    btrfs: refactor chunk allocation device handling to use list_head
>>    btrfs: introduce explicit device roles for block groups
>>    btrfs: introduce ROLE_THEN_SPACE device allocation method
>>    btrfs: pass device roles through device add ioctl
>>
>>   fs/btrfs/block-group.c |  11 +-
>>   fs/btrfs/ioctl.c       |  12 +-
>>   fs/btrfs/sysfs.c       | 130 ++++++++++++++++++++--
>>   fs/btrfs/volumes.c     | 242 +++++++++++++++++++++++++++++++++--------
>>   fs/btrfs/volumes.h     |  35 +++++-
>>   5 files changed, 366 insertions(+), 64 deletions(-)
>>
>> -- 
>> 2.49.0
>>
>>
>