Re: [RFC] fs/resctrl: Generic schema description

Reinette Chatre
2025-12-16T22:26:33Z

Hi Babu and Fenghua,

Could you please consider how the new AMD and MPAM features [2] may benefit
from the new interfaces proposed here? More below ...

On 10/24/25 4:12 AM, Dave Martin wrote:
> Hi all,
>
> Going forward, a single resctrl resource (such as memory bandwidth) is
> likely to require multiple schemata, either because we want to add new
> schemata that provide finer control, or because the hardware has
> multiple controls, covering different aspects of resource allocation.
>
> The fit between MPAM's memory bandwidth controls and the resctrl MB
> schema is already awkward, and later Intel RDT features such as Region
> Aware Memory Bandwidth Allocation are already pushing past what the MB
> schema can describe. Both of these can involve multiple control
> values and finer resolution than the 100 steps offered by the current
> "MB" schema.
>
> The previous discussion went off in a few different directions [1], so
> I want to focus back onto defining an extended schema description that
> aims to cover the use cases that we know about or anticipate today, and
> allows for future extension as needed.
>
> (A separate discussion is needed on how new schemata interact with
> previously-defined schemata (such as the MB percentage schema).
> suggest we pause that discussion for now, in the interests of getting
> the schema description nailed down.)
>
>
> Following on from the previous mail thread, I've tried to refine and
> flesh out the proposal for schema descriptions a bit, as follows.
>
> Proposal:
>
> * Split resource names and schema names in resctrlfs.
>
> Resources will be named for the unique, existing schema for each
> resource.
>
> The existing schema will keep its name (the same as the resource
> name), and new schemata defined for a resource will include that
> name as a prefix (at least, by default).
>
> So, for example, we will have an MB resource with a schema called
> MB (the schema that we have already). But we may go on to define
> additional schemata for the MB resource, with names such MB_MAX,
> etc.
>
> * Stop adding new schema description information in the top-level
> info/<resource>/ directory in resctrlfs.
>
> For backwards compatibilty, we can keep the existing property
> files under the resource info directory to describe the previously
> defined resource, but we seem to need something richer going
> forward.
>
> * Add a hierarchy to list all the schemata for each resource, along
> with their properties. So far, the proposal looks like this,
> taking the MB resource as an example:
>
> info/
> └─ MB/
> └─ resource_schemata/
> ├─ MB/
> ├─ MB_MIN/
> ├─ MB_MAX/
> ┆
>
> Here, MB, MB_MIN and MB_MAX are all schemata for the "MB" resource.
> In this proposal, what these just dummy schema names for
> illustration purposes. The important thing is that they all
> control aspects of the "MB" resource, and that there can be more
> than one of them.
>
> It may be appropriate to have a nested hierarchy, where some
> schemata are presented as children of other schemata if they
> affect the same hardware controls. For now, let's put this issue
> on one side, and consider what properties should be advertsed for
> each schema.
>
> * Current properties that I think we might want are:
>
> info/
> └─ SOME_RESOURCE/
> └─ resource_schemata/
> ├─ SOME_SCHEMA/
> ┆ ├─ type
> ├─ min
> ├─ max
> ├─ tolerance
> ├─ resolution
> ├─ scale
> └─ unit
>
> (I've tweaked the properties a bit since previous postings.
> "type" replaces "map"; "scale" is now the unit multiplier;
> "resolution" is now a scaling divisor -- details below.)
>
> I assume that we expose the properties in individual files, but we
> could also combine them into a single description file per schema,
> per resource or (possibly) a single global file.
> (I don't have a strong view on the best option.)
>
>
> Either way, the following set of properties may be a reasonable
> place to start:
>
>
> type: the schema type, followed by optional flag specifiers:
>
> - "scalar": a single-valued numeric control
>
> A mandatory flag indicates how the control value written to
> the schemata file is converted to an amount of resource for
> hardware regulation.
>
> The flag "linear" indicates a linear mapping.
>
> In this case, the amount of resource E that is actually
> allocated is derived from the control value C written to the
> schemata file as follows:
>
> E = C * scale * unit / resolution
>
> Other flags values could be defined later, if we encounter
> hardware with non-linear controls.
>
> - "bitmap": a bitmap control
>
> The optional flag "sparse" is present if the control accepts
> sparse bitmaps.
>
> In this case, E = bitmap_weight(C) * scale * unit / resolution.
>
> As before, each bit controls access to a specific chunk of
> resource in the hardware, such as a group of cache lines. All
> chunks are equally sized.
>
> (Different CTRL_MON groups may still contend within the
> allocation E, when they have bits in common between their
> bitmaps.)
>
> min:
>
> - For a scalar schema, the minimum value that can be written to
> the control when writing the schemata file.
>
> - For a bitmap schema, a bitmap of the minimum weight that the
> schema accepts: if an empty bitmap is accepted, this can be 0.
> Otherwise, if bitmaps with a single bit set are acceptable,
> this can just have the lowest-order bit set.
>
> Most commonly, the value will probably be "1".
>
> For bitmap schemata, we might report this in hex. In the
> interest of generic parsing, we could include a "0x" prefix if
> so.
>
> max:
>
> - For a scalar schema, the maximum value that can be written to
> the control when writing the schemata file.
>
> - For a bitmap schema, the mask with all bits set.
>
> Possibly reported in hex for bitmap schemata (as for "min").
>
> tolerance:
>
> (See below for discussion on this.)
>
> - "0": the control is exact
>
> - "1": the effective control value is within ±1 of the control
> value written to the schemata file. (Similary, positive "n" ->
> ±n.)
>
> A negative value could be used to indicate that the tolerance
> is unknown. (Possibly we could also just omit the property,
> though it seems better to warn userspace explicitly if we
> don't know.)
>
> Tests might make use of this parameter in order to determine
> how picky to be about exact measurement results.
>
> resolution:
>
> - For a proportional scalar schema: the number of divisions that
> the whole resource is divided into. (See below for
> "proportional scalar schema.)
>
> Typically, this will be the same as the "max" value.
>
> - For an absolute scalar schema: the divisor applied to the
> control value.
>
> - For a bitmap schema: the size of the bitmap in bits.
>
> scale:
>
> - For a scalar schema: the scale-up multiplier applied to
> "unit".
>
> - For a bitmap schema: probably "1".
>
> unit:
>
> - The base unit of the quantity measured by the control value.
>
> The special unit "all" denotes a proportional schema. In this
> case, the resource is a finite, physical thing such as a cache
> or maxed-out data throughput of a memory controller. The
> entire physical resource is available for allocation, and the
> control value indicates what proportion of it is allocated.
>
> Bitmap schemata will probably all be proportional and use the
> unit "all". (This applies to cache bitmaps, at least.)
>
> Absolute schemata will require specification of the base unit
> here, say, "MBps". The "scale" parameter can be used to avoid
> proliferation of unit strings:
>
> For example, {scale=1000, unit="MBps"} would be equivalent to
> {scale=1, unit="GBps"}.
>
>
> Note on the "tolerance" parameter:
>
> This is a new addition. On the MPAM side, the hardware has a choice
> about how to interpret the control value in some edge-case situations.
> We may not reasonably be able to probe for this, so it may be useful
> to warn software that there is an uncertainty margin.
>
> We might also be able to use the "tolerance" parameter to accommodate
> the rounding behaviour of the existing "MB" schema (otherwise, we
> might want a special "type" for this schema, if it doesn't comply
> closely enough).
>
>
> If we want to deploy resctrl under virtualisation, resctrl on the host
> could dynamically affect the actual amount of resource that is
> available for allocation inside a VM.
>
> Whether or not we ever want to do that, it might be useful to have a
> way to warn software that the effective control values hitting the
> hardware may not be entirely predictable.
>
> Thoughts?
>
> Cheers
> ---Dave

One thing I was pondering is that resctrl currently uses L3 interchangeably
as a scope and a resource but if instead that is separated then it should be
easier to support interactions with resource at a different scope.

I am concerned that, for example, support for Global Memory Bandwidth Allocation
(GMBA) is planned to be done with a new resource. resctrl already has a
"memory bandwidth allocation" resource and introducing a new resource to essentially
manage the same resource, but at a different scope, sounds like a risk of fragmentation
and duplication to me.

What if the "resource control" instead gains a new property, for example, "scope" that
essentially communicates to user space what a domain ID in the schemata file means.

It is not clear to me what a "domain ID" of GMBA means so I will use the MPAM CPU-less
MBM as example that I expect will build on SMBA that supports CXL.mem. Consider, an interface
like below:

info
└── SMBA
└── resource_schemata
├── SMBA
│   ├── max
│   ├── min
│   ├── resolution
│   ├── scale
│   ├── scope <== contains "L3"
│   ├── tolerance
│   ├── type
│   └── unit
└── SMBA_NODE
├── max
├── min
├── resolution
├── scale
├── scope <== contains "NODE"
├── tolerance
├── type
└── unit

With an interface like above there is a single resource and allocating it at a different
scope is just another control. This correlates to how other parts of resctrl is managed.
For example, it can become explicit that the monitor groups' mon_data directory contains
sub-directories organized by scope. For example:

mon_data
├── mon_L3_00 <== monitoring data at scope L3
│   ├── llc_occupancy
│   ├── mbm_local_bytes
│   └── mbm_total_bytes
├── mon_L3_01 <== monitoring data at scope L3
│ ├── llc_occupancy
│ ├── mbm_local_bytes
│ └── mbm_total_bytes
├── mon_NODE_00 <== monitoring data at scope NODE
│ └── mbm_total_bytes
└── mon_NODE_01 <== monitoring data at scope NODE
└── mbm_total_bytes

What do you think?

Reinette

> [1] Re: [PATCH] fs/resctrl,x86/resctrl: Factor mba rounding to be per-arch
> https://lore.kernel.org/lkml/aNFliMZTTUiXyZzd@e133380.arm.com/

[2] https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf