Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept

[Fenghua Yu <fenghuay@nvidia.com>]

Hi, Ben,

On 7/21/26 06:23, Ben Horgan wrote:
> Hi Reinette,
> 
> On 7/20/26 23:54, Reinette Chatre wrote:
>> Hi Ben,
>>
>> On 7/20/26 6:30 AM, Ben Horgan wrote:
>>> Hi Reinette,
>>>
>>> On 7/17/26 17:00, Reinette Chatre wrote:
>>>> Hi Ben,
>>>>
>>>> On 7/17/26 5:20 AM, Ben Horgan wrote:
>>>>> On 7/16/26 18:07, Reinette Chatre wrote:
>>>>>> On 7/16/26 9:44 AM, Ben Horgan wrote:
>>>>>>> On 7/16/26 17:04, Reinette Chatre wrote:
>>>>>>>> Hi Ben, Chenyu, and Tony,
>>>>>>>>
>>>>>>>> On 7/16/26 7:59 AM, Ben Horgan wrote:
>>>>>>>>> On 7/15/26 16:41, Reinette Chatre wrote:
>>>>>>>>>> On 7/15/26 1:34 AM, Ben Horgan wrote:
>>>>>>>>>>> On 7/14/26 23:06, Reinette Chatre wrote:
>>>>
>>>> ...
>>>>
>>>>>>>>>>> Alternatively, could the "mode" file be used to switch between "MB" and "MB_MAX" and the two never
>>>>>>>>>>> need be shown at the same time. The user opts in to using the new interface, "MB_MAX" by setting
>>>>>>>>>>> "mode" and can just toggle back if they want to use "MB" directly again.
>>>>>>>>>> Interesting. So far the "mode" options have been "legacy" and "native" where "legacy" would show the
>>>>>>>>>> legacy as well as emulated controls in the schemata file and "native" will only show the new controls.
>>>>>>>>>> resctrl could make the default of "legacy" mean that *only* the legacy control is shown without insight
>>>>>>>>>> into the controls it is being emulated with. The only insight to this would continue to be via the
>>>>>>>>>> hierarchy in the info directory, which would show relationship but not the actual control values.
>>>>>>>>>> Do you think there could be a need for users (excluding validation?) that may want to see the underlying
>>>>>>>>>> control values used to emulate a legacy control?
>>>>>>>>>
>>>>>>>>> They may want to see the underlying values to be able to migrate there legacy configuration to the
>>>>>>>>> native configuration but they can just write their legacy configuration and then toggle the mode to
>>>>>>>>> see the values in the native mode.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I assume for backward compatibility that "legacy" would remain the default so a possible inconvenience
>>>>>>>>>> here would be that users familiar with the new controls would forever need to switch the mode before being
>>>>>>>>>> able to use them.
>>>>>>>>>
>>>>>>>>> This does also make the new controls slightly harder to discover.
>>>>>>>>>
>>>>>>>> What if resctrl combines the two suggestions? Specifically, for backward compatibility the mode will be
>>>>>>>> "legacy" and if there are emulated controls then resctrl displays them in schemata file with "#" prefix
>>>>>>>> but not(*) support any changes from user space to the underlying hardware controls. This will make new controls
>>>>>>>> easier to discover and let user space see the underlying values, but not break a user space that
>>>>>>>> may read schemata file, change a few values, and write entire file back.
>>>>>>>
>>>>>>> My initial impression is that this would work but is probably unnecessary.
>>>>>>
>>>>>> ok. My goal was to present ideas to address the issues raised so far. Ideally this would result in discussion of
>>>>>> pros/cons. If you find this unnecessary, could you please expand with some insight into which parts you find
>>>>>> unnecessary or how you would prefer this solved? I am finding it difficult to interpret above response.
>>>>>
>>>>> Sorry, yes. I was a bit lazy in my reply yesterday. If the mode is easy and non-destructive to
>>>>> change then doesn't the user get the same information with just the burden of toggling the mode and
>>>>> then reading the schemata a second time. This does rely on the user being aware of the new behaviour
>>>>> though.
>>>>>
>>>>> If the emulation isn't a, one to one, bijection then is changing mode expected to be destructive?
>>>>> More on this below.
>>>>
>>>> There may be some corner cases (region aware may have some as you highlight below), but in general I do not expect
>>>> changing mode to be destructive. The idea behind giving insight into underlying control values while legacy interface
>>>> is in use is to address your earlier point that not doing so will make new controls slightly harder to discover.
>>>> (more below)
>>>>
>>>>>>>> How the "enabled" vs "disabled" state of a control works with this needs some confirmation since region-aware
>>>>>>>> MBA support has this extra caveat of supporting MSR and ACPI interfaces which results in the relationship
>>>>>>>> between "mode" and "status" of underlying controls not being consistent. This may be ok but please consider
>>>>>>>> example below.
>>>>>>>>
>>>>>>>> Thinking through this with examples as I understand MPAM and RDT region aware so far. Could this work for
>>>>>>>> MPAM and region aware MBA?
>>>>>>>>
>>>>>>>> (*) Should resctrl allow user space to change underlying control value of an emulated control in "legacy" mode?
>>>>>>>
>>>>>>> I don't think this causes problems for MPAM but for RDT region aware controls couldn't you end up
>>>>>>> with a control state that isn't reachable just by configuring the legacy schema.
>>>>>>
>>>>>> Could you please highlight the scenario you refer to?
>>>>>
>>>>> My expectation here was that for any value of MB, X, the value for each of the regions would be the
>>>>> same, Y. (Not sure if this is actually the case.)
>>>>>
>>>>> MB: X
>>>>> # MB_REGION0_MAX:Y
>>>>> # MB_REGION1_MAX:Y
>>>>> # MB_REGION2_MAX:Y
>>>>> # MB_REGION3_MAX:Y
>>>>>
>>>>> So, if one of the regions was to be changed individually then it would be in a state not reachable
>>>>> by just changing the legacy control, MB. Potentially this complicated switching mode as well as
>>>>> uncommenting and writing a schemata.
>>>>
>>>> Indeed. This scenario was highlighted in slide 8 of
>>>> https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf
>>>> and discussed between Tony and Dave Martin in the thread starting at
>>>> https://lore.kernel.org/lkml/aPf0OKwDZ4XbmVRB@agluck-desk3/
>>>>
>>>> Tony and Dave discussed a few scenarios and how resctrl could behave with different user interactions.
>>>> On a high level I understood that the underlying controls will always, as you state below, "show the
>>>> full story" and if user space interacts with them (instead of using the legacy control) then the
>>>> control values associated with the legacy control cannot be relied on to represent accurate state to
>>>> the point that it may even be better to not be visible.
>>>> You are right, this is a good motivation to not allow user to modify underlying controls when in
>>>> "legacy" mode.
>>>>
>>>> The remaining open is whether resctrl should display the (read-only) underlying control
>>>> values in schemata file when in "legacy" mode. By default this does not work for region-aware since
>>>> it will at least initially use different hardware interfaces for the different controls, essentially
>>>> this means that there is no actual emulation and the legacy and region aware control are both
>>>> legitimate hardware controls. Specifically, considering your example, there is no mapping from "X" to "Y".
>>>>
>>>> The way region-aware is planning to address this is to use the "enabled" vs "disabled" status of a
>>>> control to start with underlying controls disabled so that they are not displayed in schemata file
>>>> when legacy mode is enabled.
>>>>
>>>> If resctrl instead only displays legacy controls when in "legacy" mode then the "status" may no longer
>>>> be needed for region-aware. RISC-V also considered using this "status" that I think may also be
>>>> solved with this approach, I am not sure though.
>>>>
>>>>> I was thinking that in legacy mode that the region values would always be kept the same but perhaps
>>>>> the legacy (MB) value could just be the 'best estimate' and the native (MB_REGION0) values show the
>>>>> full story. 'best estimate' could be difficult to choose but allows switching mode to be
>>>>> non-destructive.
>>>>>
>>>>> For MPAM, at least until Fenghua sent his series today on MBA control emulation [1], I was expecting
>>>>> that MPAM would only use emulation for exposing finer grade controls to the user and not allowing
>>>>> different underlying hardware to be used for the emulation. To me, it seems reasonable that a new
>>>>> control would involve a new schema but I do appreciate the benefit of being able to continue to use
>>>>> the old interface without stopping a new interface being used. The danger is that there are
>>>>> unexpected user visible side effects, e.g. domain lifetime, and that control behaviour is different
>>>>> from the user expectations.
>>>>
>>>> I have not looked at Fenghua's series in detail but from earlier discussion I understood the high level
>>>> problem to be that user space may have expectation that for every "resource" represented by a directory in
>>>> info/ there is a matching entry in the schemata file. Whether this is an actual expectation from user
>>>> space tools is not obvious to me though: there is already an exception since there is the
>>>> "L3_MON" "resource" that does not have a schemata file entry.
>>>>
>>>> To ensure backward compatibility the safest would be for resctrl to always provide a "MB" control via
>>>> an entry in the schemata file when the "MB" resource is exposed via info/. If on the other hand resctrl
>>>> is not expected to provide a "MB" entry in schemata file when there is a "MB" resource described in info/MB
>>>> then resctrl should not be forced to always provide such emulation.
>>>>
>>>> I would appreciate your thoughts here.
>>>
>>> Ok, thanks for pointing this out, I hadn't previously properly understood the motivation for
>>> resource emulation.
>>>
>>> As emulation adds complication and has the potential to break other user expectations I hope we can
>>> manage to keep it to only simple cases. For instance as I mentioned before, the lifetime of a
>>> resctrl domain lifetime is necessary different (when there are multiple NUMA nodes) for domains
>>> backed by MSC at the memory to MSC at the cache. At the memory they would be inaccessible when the
>>> NUMA node is powered off. Also, any partitioning is happening at a different point in the topology.
>>>
>>> Can we get around this by by choosing a different organization of the info/ directory? We already
>>> have separate resource for L2 and L3. Perhaps, the same for MB, MB_NODE, (not sure about MB_REGION)
>>> and move the scope to be a property at info/<resource>/scope rather than
>>> info/<resource>/resource_schemata/ctrl/scope.
>>
>> Good point that L2 and L3, which is the same resource at different scope, are treated as different
>> resources. With that view it seems reasonable to also consider MB and MB_NODE as different resources.
>> The point here is that it is at a different scope so more specifically what a "domain ID" in the
>> schemata file represents. With this view, MB_REGION is L3 scope that matches to the MB resource.
> 
> Ok. This makes sense to me. I had confused myself about what MB_REGION is.
> 
>>
>> You will notice I get a bit lost in the discussion below so getting back here I would like to
>> clarify if you propose that info/ contains a directory for each allocation scope of each resource with that
>> directory containing descriptions of all the controls for that resource at the indicated scope or do
>> you propose that info/ contains a directory for each control?
> 
> The former, info/ contains a directory for each allocation scope of each resource.
> 
> 
> 
> info
> ├── L2
> │   ├── resource_schemata
> │   │   ├── L2
> │   │   ├── L2_CMAX
> │   │   └── L2_CMIN
> │   └── scope : L2
> ├── L3
> │   ├── resource_schemata
> │   │   ├── L3
> │   │   ├── L3_CMAX
> │   │   └── L3_CMIN
> │   └── scope : L3
> ├── MB
> │   ├── resource_schemata
> │   │   ├── MB
> │   │   │   └── MB_MAX
> │   │   ├── MB_MIN
> │   │   ├── MB_PBM
> │   │   └── MB_PROP
> │   └── scope : L3
> └── MB_NODE
>      ├── resource_schemata
>      │   ├── MB_NODE_MAX
>      │   ├── MB_NODE_MIN
>      │   ├── MB_NODE_PBM
>      │   └── MB_NODE_PROP
>      └── scope : NUMA NODE
> 
>>
>>>
>>> Can MB_REGION be considered an orthogonal new control or does using it require that the traditional
>>> intel MB (delay) not be configured?
>>
>> The Intel systems that support both MSR ("traditional") and ACPI (region aware MBA) cannot use both
>> concurrently. Not sure if this answers your question.
> 
> Thanks, this answers my question.
> 
>>
>> Just to clarify, there is no single "MB_REGION" control. When considering the "region aware MBA"
>> feature I am currently aware of the following possible controls:
>> MB_REGION0_MIN
>> MB_REGION0_MAX
>> MB_REGION0_OPT
>> MB_REGION1_MIN
>> MB_REGION1_MAX
>> MB_REGION1_OPT
>> MB_REGION2_MIN
>> MB_REGION2_MAX
>> MB_REGION2_OPT
>> MB_REGION3_MIN
>> MB_REGION3_MAX
>> MB_REGION3_OPT
>>
>>
>>> On an MPAM system:
>>>
>>> info
>>> ├── MB
>>> │   ├── resource_schemata
>>> │   │   ├── MB
>>> │   │   └── MB_MAX
>>> │   └── scope
>>> └── MB_NODE
>>>      ├── resource_schemata
>>>      │   └── MB_NODE
>>>      └── scope
>>
>> Above MB and MB_MAX are represented on the same level. My understanding is that MB_MAX is
>> used by driver as the underlying hardware control for the percentage based MB exposed to
>> user space. Thus, when user space changes the "MB" control via schemata file it is expected
>> to also impact the underlying "MB_MAX" control. By representing them as above this relationship
>> is lost. So far I understood that the "MB_MAX" control will be shown as a child of the "MB" control
>> to show this relationship. Looks like you plan to change this but from what I understand this
>> emulation is still relevant?
> 
> Please just consider this a mistake. MB_MAX should be a child of MB.
> 
> info/
> ├── MB
> │   ├── resource_schemata
> │   │   └── MB
> │   │       └── MB_MAX
> │   └── scope
> └── MB_NODE
>      ├── resource_schemata
>      │   └── MB_NODE_MAX (Changed from MB_NODE)
>      └── scope
> 
>>
>> When thinking about MPAM, what would the underlying hardware control of "MB_NODE" be? It looks
>> from above that it would either start out by itself having the properties of the underlying
>> "MAX" control or is the plan to have it be a percentage based control backed by the
>> underlying "MAX" hardware control?
> 
> The underlying hardware of MB_NODE would be essentially the same hardware as that backing MB_MAX,
> but at a different location in the SoC, at the memory controller rather than in the L3.
> 
> To correct myself slightly, I don't think we should have a control called MB_NODE, rather, it should
> be MB_NODE_MAX.
> 
> My understanding of previous discussions is that <resource_type>_<scope>_<control_type> it the
> pattern for control names and the pattern for resource names being <resource_type>_<scope>.
> (allowing for <scope> or <control_type> being missing to match existing naming.)
> 
> I don't think we should introduce more percent based controls and for new controls we can introduce
> a new format to describe them. Perhaps just the positive integer with a resolution supplied in info/
> as discussed previously. Although, I have been pondering on whether we can do a bit better.
> 
> We could use hexadecimal point based format for controls which are a proportion of a resource and
> have a resolution which is a power of 2. The advantage of this is that the meaning of the value is
> independent of the granularity of the control (number of parts).
> 
> 0 is represented as 0x0
> 1 as                0x1
> 1/2 as              0x0.8
> 7/256 as            0x0.07
> 1/2**28             0x0.00000001
> etc
> 
> This maps well to the MPAM fixed-point fraction point format without having the weirdness of having
> values forced to 1 or 0 not being really 0. These MPAM h/w oddities can be hidden just by using the
> mbw_min mbw_max of a control. In MPAM this could be used in CMIN, CMAX, MB_MAX, MB_MIN and I would
> hope this would be useful for other architectures too. I am preparing some RFC patches on top of
> your PoC for consideration of this idea and to explore some of the proposals discussed relating to
> generic schemata and how they land in practice from the MPAM side.
> 
>>
>> I also understand MPAM to support more memory bandwidth controls ("MIN", "HARDMAX"/"HARDLIM", etc.).
>> Do you envision them to exist within info/MB/resource_schemata/<control> as well as within
>> info/MB_NODE/resource_schemata/<control>?
> 
> Yes, at least for MIN, see the info/ tree above. For HARDLIM, perhaps, but HARDLIM has the added
> complications that it is a property of the MBW_MAX control and that it may be configurable for each
> PARTID or a fixed property of the h/w. When HARDLIM is configurable the control name could be of the
> form <resource_type>_<scope>_<control>_<control_param> where <control_param> is HARDLIM and the full
> name for the HARDLIM configuration on the MB_NODE resource is MB_NODE_MAX_HARDLIM. There can also be
> an info/<resource>/resource_schemata/<control>/lim file which has values, soft, hard, configurable.
> 
> For CMAX, maximum cache capacity, there is an equivalent control SOFTLIM, which behaves as HARDLIM
> except the meaning of the bit is reversed. We can just use a consistent name in s/w though.
> 

Is it possible to view MBW_MAX hard limit, CMAX soft limit, or future 
similar things as a "configuration" for a control, instead of a 
"control" itself?

A "configurations" is different from a "control" in that:
1. The configuration configures the control, e.g. toggle hard limit on 
MB control.
2. The configuration doesn't have the control's properties (e.g. gran).
3. The configuration is similar to "event_configs" in MB_MON, that can 
also configure MB_MON's event_filter.

MBW_MAX hard limit configures MB/MB_MAX control's hard limit. CMAX soft 
limit configures L3 control's soft limit.

Is it OK to implement configurations in a control?

Then in struct resctrl_ctrl, add "list_head configs" to track all 
configurations in this control.

$ cat schemata
         MB_NODE:0=100;1=100;2=100;10=100;18=100;26=100;34=100;35=100
# MB_MAXHLIM_NODE is a configuration of MB_NODE control
MB_MAXHLIM_NODE:0=1;1=0;2=0;10=0;18=0;26=0;34=0;35=0
              MB:0=100;1=100;2=100;10=100;18=100;26=100;34=100;35=100
# MB_MAXHLIM is a configuration of MB control
      MB_MAXHLIM:0=1;1=0;2=0;10=0;18=0;26=0;34=0;35=0
              L3:1=ffff;2=ffff

info/
│   └── resource_schemata
│       ├── MB (simulated by MB_NODE)
│       │   ├── configs          # all configs in MB control
│       │   │   └── MB_MAXHLIM   # MB_MAXHLIM config in MB control
│       │   │       └── type     # boolean
│       │   ├── max
│       │   ├── MB_NODE          # simulated MB
│       │   │   ├── configs             # all configs in MB_NODE control
│       │   │   │   └── MB_MAXHLIM_NODE # MB_MAXHLIM config in MB_NODE
│       │   │   │       └── type
│       │   │   ├── max
│       │   │   ├── min
│       │   │   ├── resolution
│       │   │   ├── scale
│       │   │   ├── scope
│       │   │   ├── status
│       │   │   ├── tolerance
│       │   │   ├── type
│       │   │   └── unit
│       │   ├── min
│       │   ├── resolution
│       │   ├── scale
│       │   ├── scope
│       │   ├── status
│       │   ├── tolerance
│       │   ├── type
│       │   └── unit
│       └── mode


>>
>>>
>>> On an x86 system:
>>>
>>> info
>>> ├── MB
>>> │   ├── resource_schemata
>>> │   │   ├── MB
>>> │   │   └── MB_MAX (Finer grained MB, more below)
>>> │   └── scope
>>> └── MB_REGION
>>>      ├── resource_schemata
>>>      │   └── MB_REGION
>>>      └── scope
>>>
>>>
>>> Do you think this helps?
>>
>> "REGION" is not a new scope but instead region-aware MBA is controlled and manages bandwidth at L3 scope.
>> Combine that with up to (currently) four regions each with three controls I find an interface like above
>> potentially confusing to document in an intuitive way. Unless you are perhaps saying that we should introduce
>> a new separate  "MB_REGION" L3 scope resource (so let resctrl support multiple "MB" resources at the same
>> scope?) and then *it* contains the twelve new controls within its resource_schemata directory?
>>
>> Since the region-aware controls are orthogonal to the MSR based legacy control resctrl would still need a
>> way for user space to switch from one to the other which implies a dependency between "MB" and "MB_REGION"
>> that is not presented in above hierarchy.
> 
> OK. The /sys/fs/resctrl/info/MB/schemata/mode and the MB_REGION controls a child of MB you described
> previously seem s better fit than what I suggested.
> 
>>
>> I think I am missing quite a bit here as I try to navigate an interface so different from what we have
>> discussed so far.
>> I would like to explore with more detail how this interface can handle the different scenarios we have
>> discussed so far.
> 
> Certainly, I don't think we have got to the bottom of this yet.
> 
>>
>> The other x86 feature to consider is AMD's upcoming "Global" MBA/SMBA that exposes memory bandwidth allocation
>> in "groups of L3" that I understand could usually be mapped to NODE scope (but it remains controlled at L3 scope),
>> except for one configuration where it is "SYSTEM"(?) scope.
>> Ref.: https://lore.kernel.org/lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@amd.com/
> 
> Hmmm, I'm not sure that the scope can be considered to be NODE scope for GMBA. To me it seems to be
> accidental that it maps to the NUMA node but really the scope is just a grouping of L3 instances.
> For a control to NUMA scope I would expect the resctrl domains to go offline and online in sync with
> the NUMA nodes. For GMBA it looks like it would just going offline/online based on whether any of
> the CPUs and so L3 instances in the group are online. Am I correct here?
> 
> Assuming the domains are on L3 groups rather than NUMA also changes which end of the link the
> traffic is regulated and so how cross-NUMA traffic behaves differently. If the domain is an L3 group
> then a task running on a CPU affine to that L3 group won't be throttled unless that particular
> domain is throttled but with NUMA node domains it may be throttled if it has traffic going to that
> domain.
>>
>>>
>>> This also brings another question. On MPAM systems the 'MB_MAX' is backed by the same MSC h/w as MB
>>> but it exposed a different interface to the user. If I understand correctly intel have an option to
>>> have finer grained control of MB (delay) as well and so it would make sense to use a common name
>>> rather than just going for the MPAM centric name of MB_MAX.
>>
>> Apologies but I was not able to parse above.
> 
> Ok, let me try to explain again (although it's probably not what we want to do). My intent here was
> to try and explore whether we can reuse naming and controls across architectures in the same way we
> already have for the L2/L3 cache portion bitmap and the existing MB control.
> 
> To quote from a previous mail of yours:
> https://lore.kernel.org/lkml/a84af037-6439-4362-be07-d45143e06309@intel.com/
> """
>    For example, on an MPAM system (if I understand correctly) the user may see:
>    info/
>    └── MB/
>        └── resource_schemata/
>            ├── MB/
>            │   └── MB_MAX/
>            └── MB_MIN/
> 
>    Compared with a possible implementation on Intel that looks like:
>    info/
>    └── MB/
>        └── resource_schemata/
>            ├── MB/
>            │   └── MB_OPT/
>            ├── MB_MAX/
>            └── MB_MIN/
> """
> 
> In the two setups MPAM MB_MAX and intel MB_OPT play the same role, a finer grained control of the
> legacy MB control. I was thinking these could share a name (MB-PRECISE), but it probably doesn't
> make sense as MB_OPT and MB_MAX have different relationships to MB_MIN.
> 
> Thanks,
> 
> Ben
> 
>>
>> Reinette
>>
> 
> 
Thanks.

-Fenghua