Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept

[Babu Moger <babu.moger@amd.com>]

Hi Ben/Reinette,

On 7/21/26 12:30, Reinette Chatre wrote:
> Hi Ben,
> 
> On 7/21/26 6:23 AM, Ben Horgan wrote:
>> On 7/20/26 23:54, Reinette Chatre wrote:
>>> On 7/20/26 6:30 AM, Ben Horgan wrote:
> ...>
>> The former, info/ contains a directory for each allocation scope of each resource.
>>
>>
>>
>> info
>> ├── L2
>> │   ├── resource_schemata
>> │   │   ├── L2
>> │   │   ├── L2_CMAX
>> │   │   └── L2_CMIN
>> │   └── scope : L2
>> ├── L3
>> │   ├── resource_schemata
>> │   │   ├── L3
>> │   │   ├── L3_CMAX
>> │   │   └── L3_CMIN
>> │   └── scope : L3
>> ├── MB
>> │   ├── resource_schemata
>> │   │   ├── MB
>> │   │   │   └── MB_MAX
>> │   │   ├── MB_MIN
>> │   │   ├── MB_PBM
>> │   │   └── MB_PROP
>> │   └── scope : L3
>> └── MB_NODE
>>      ├── resource_schemata
>>      │   ├── MB_NODE_MAX
>>      │   ├── MB_NODE_MIN
>>      │   ├── MB_NODE_PBM
>>      │   └── MB_NODE_PROP
>>      └── scope : NUMA NODE
>>
> 
> At first glance this looks good to me. As you highlight below the nuances of how "Global S/MBA" can fit
> in here still needs to be worked out. One thing that resctrl may need to highlight when documenting this
> new capability is that while historically a resource had a matching schemata entry, this is no longer the
> case. As you show above some systems may have MB_NODE resource but no MB_NODE control while I expect that
> "Global MBA" (if it adopts this) may indeed have a MB_NODE resource with a MB_NODE control.
> 
>   
>> Please just consider this a mistake. MB_MAX should be a child of MB.
>>
>> info/
>> ├── MB
>> │   ├── resource_schemata
>> │   │   └── MB
>> │   │       └── MB_MAX
>> │   └── scope
>> └── MB_NODE
>>      ├── resource_schemata
>>      │   └── MB_NODE_MAX (Changed from MB_NODE)
>>      └── scope
> 
> ack.
> 
>>
>>>
>>> When thinking about MPAM, what would the underlying hardware control of "MB_NODE" be? It looks
>>> from above that it would either start out by itself having the properties of the underlying
>>> "MAX" control or is the plan to have it be a percentage based control backed by the
>>> underlying "MAX" hardware control?
>>
>> The underlying hardware of MB_NODE would be essentially the same hardware as that backing MB_MAX,
>> but at a different location in the SoC, at the memory controller rather than in the L3.
>>
>> To correct myself slightly, I don't think we should have a control called MB_NODE, rather, it should
>> be MB_NODE_MAX.
> 
> ack.
> 
>>
>> My understanding of previous discussions is that <resource_type>_<scope>_<control_type> it the
>> pattern for control names and the pattern for resource names being <resource_type>_<scope>.
>> (allowing for <scope> or <control_type> being missing to match existing naming.)
> 
> This is where discussions are from my view also.
> 
>>
>> I don't think we should introduce more percent based controls and for new controls we can introduce
>> a new format to describe them. Perhaps just the positive integer with a resolution supplied in info/
> 
> No, we should not introduce more percentage based controls per se. With the new schema format a
> percentage based control is just a variant of a proportional scalar control.
> 
>> as discussed previously. Although, I have been pondering on whether we can do a bit better.
>>
>> We could use hexadecimal point based format for controls which are a proportion of a resource and
>> have a resolution which is a power of 2. The advantage of this is that the meaning of the value is
>> independent of the granularity of the control (number of parts).
>>
>> 0 is represented as 0x0
>> 1 as                0x1
>> 1/2 as              0x0.8
>> 7/256 as            0x0.07
>> 1/2**28             0x0.00000001
>> etc
>>
>> This maps well to the MPAM fixed-point fraction point format without having the weirdness of having
>> values forced to 1 or 0 not being really 0. These MPAM h/w oddities can be hidden just by using the
>> mbw_min mbw_max of a control. In MPAM this could be used in CMIN, CMAX, MB_MAX, MB_MIN and I would
>> hope this would be useful for other architectures too. I am preparing some RFC patches on top of
>> your PoC for consideration of this idea and to explore some of the proposals discussed relating to
>> generic schemata and how they land in practice from the MPAM side.
> 
> I'm going to stand with Dave Martin [1] on this point with a preference to avoid floating point in
> resctrl input and output.
> Could you please elaborate where values are forced to 1 or 0? The new schema format was intentionally
> created to *avoid* rounding errors (and parsing complexity).
> 
> [1] https://lore.kernel.org/lkml/aNFliMZTTUiXyZzd@e133380.arm.com/
> 
>>> I also understand MPAM to support more memory bandwidth controls ("MIN", "HARDMAX"/"HARDLIM", etc.).
>>> Do you envision them to exist within info/MB/resource_schemata/<control> as well as within
>>> info/MB_NODE/resource_schemata/<control>?
>>
>> Yes, at least for MIN, see the info/ tree above. For HARDLIM, perhaps, but HARDLIM has the added
>> complications that it is a property of the MBW_MAX control and that it may be configurable for each
>> PARTID or a fixed property of the h/w. When HARDLIM is configurable the control name could be of the
>> form <resource_type>_<scope>_<control>_<control_param> where <control_param> is HARDLIM and the full
>> name for the HARDLIM configuration on the MB_NODE resource is MB_NODE_MAX_HARDLIM. There can also be
>> an info/<resource>/resource_schemata/<control>/lim file which has values, soft, hard, configurable.
> 
> ack. HARDLIM sounds like it would be a new control type. Perhaps a "boolean" type for which new control
> files need to be decided on? Sounds like you are headed in this direction and already have one control file
> in mind for this new type.
> 
> With this in mind the control could be built on top of what is being developed at the moment, possibly
> be presented to user space following Dave Martin's suggestion in
> https://lore.kernel.org/lkml/aO0Oazuxt54hQFbx@e133380.arm.com/:
> 
>   | MB_HARDMAX: 0=0, 1=1, 2=1, 3=0 [...]
> 
>   or
> 
>   | MB_HARDMAX: 0=off, 1=on, 2=on, 3=off [...]
> 
>>
>> For CMAX, maximum cache capacity, there is an equivalent control SOFTLIM, which behaves as HARDLIM
>> except the meaning of the bit is reversed. We can just use a consistent name in s/w though.
>>
>>>
>>>>
>>>> On an x86 system:
>>>>
>>>> info
>>>> ├── MB
>>>> │   ├── resource_schemata
>>>> │   │   ├── MB
>>>> │   │   └── MB_MAX (Finer grained MB, more below)
>>>> │   └── scope
>>>> └── MB_REGION
>>>>      ├── resource_schemata
>>>>      │   └── MB_REGION
>>>>      └── scope
>>>>
>>>>
>>>> Do you think this helps?
>>>
>>> "REGION" is not a new scope but instead region-aware MBA is controlled and manages bandwidth at L3 scope.
>>> Combine that with up to (currently) four regions each with three controls I find an interface like above
>>> potentially confusing to document in an intuitive way. Unless you are perhaps saying that we should introduce
>>> a new separate  "MB_REGION" L3 scope resource (so let resctrl support multiple "MB" resources at the same
>>> scope?) and then *it* contains the twelve new controls within its resource_schemata directory?
>>>
>>> Since the region-aware controls are orthogonal to the MSR based legacy control resctrl would still need a
>>> way for user space to switch from one to the other which implies a dependency between "MB" and "MB_REGION"
>>> that is not presented in above hierarchy.
>>
>> OK. The /sys/fs/resctrl/info/MB/schemata/mode and the MB_REGION controls a child of MB you described
>> previously seem s better fit than what I suggested.
> 
> ok, I'll keep following that.
> 
>>
>>>
>>> I think I am missing quite a bit here as I try to navigate an interface so different from what we have
>>> discussed so far.
>>> I would like to explore with more detail how this interface can handle the different scenarios we have
>>> discussed so far.
>>
>> Certainly, I don't think we have got to the bottom of this yet.
>>
>>>
>>> The other x86 feature to consider is AMD's upcoming "Global" MBA/SMBA that exposes memory bandwidth allocation
>>> in "groups of L3" that I understand could usually be mapped to NODE scope (but it remains controlled at L3 scope),
>>> except for one configuration where it is "SYSTEM"(?) scope.
>>> Ref.: https://lore.kernel.org/lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@amd.com/
>>
>> Hmmm, I'm not sure that the scope can be considered to be NODE scope for GMBA. To me it seems to be
>> accidental that it maps to the NUMA node but really the scope is just a grouping of L3 instances.
>> For a control to NUMA scope I would expect the resctrl domains to go offline and online in sync with
>> the NUMA nodes. For GMBA it looks like it would just going offline/online based on whether any of
>> the CPUs and so L3 instances in the group are online. Am I correct here?
>>
>> Assuming the domains are on L3 groups rather than NUMA also changes which end of the link the
>> traffic is regulated and so how cross-NUMA traffic behaves differently. If the domain is an L3 group
>> then a task running on a CPU affine to that L3 group won't be throttled unless that particular
>> domain is throttled but with NUMA node domains it may be throttled if it has traffic going to that
>> domain.
> 
> I'll defer to Babu for accurate answers about this hardware capability.
> 

To me, Global MBA should be considered a NODE-scoped resource. In some 
configurations it may appear as SYSTEM-scoped, but that is effectively 
equivalent to a single-node encompassing the entire system. In such 
cases, there is only one schemata entry controlling the whole system.

Yes, multiple L3 instances are grouped together to form a NODE. 
Internally, programming is still performed at the L3 level, but that 
implementation detail can be hidden from users and does not need to be 
exposed through the interface.

Thanks
Babu

>>>
>>>>
>>>> This also brings another question. On MPAM systems the 'MB_MAX' is backed by the same MSC h/w as MB
>>>> but it exposed a different interface to the user. If I understand correctly intel have an option to
>>>> have finer grained control of MB (delay) as well and so it would make sense to use a common name
>>>> rather than just going for the MPAM centric name of MB_MAX.
>>>
>>> Apologies but I was not able to parse above.
>>
>> Ok, let me try to explain again (although it's probably not what we want to do). My intent here was
>> to try and explore whether we can reuse naming and controls across architectures in the same way we
>> already have for the L2/L3 cache portion bitmap and the existing MB control.
>>
>> To quote from a previous mail of yours:
>> https://lore.kernel.org/lkml/a84af037-6439-4362-be07-d45143e06309@intel.com/
>> """
>>    For example, on an MPAM system (if I understand correctly) the user may see:
>>    info/
>>    └── MB/
>>        └── resource_schemata/
>>            ├── MB/
>>            │   └── MB_MAX/
>>            └── MB_MIN/
>>
>>    Compared with a possible implementation on Intel that looks like:
>>    info/
>>    └── MB/
>>        └── resource_schemata/
>>            ├── MB/
>>            │   └── MB_OPT/
>>            ├── MB_MAX/
>>            └── MB_MIN/
>> """
>>
>> In the two setups MPAM MB_MAX and intel MB_OPT play the same role, a finer grained control of the
>> legacy MB control. I was thinking these could share a name (MB-PRECISE), but it probably doesn't
>> make sense as MB_OPT and MB_MAX have different relationships to MB_MIN.
>>
> Apologies, I neglected to follow up on this after I clarified internally which RDT control should
> actually be used to emulate the percentage based control. My original thoughts that you highlight
> above is not correct the RDT also plans to use the "MAX" underlying hardware control for the
> percentage based MB control. With this RDT and MPAM should look more similar with RDT simplified
> (by dropping the region-aware terms) as:
> 
>     info/
>     └── MB/
>         └── resource_schemata/
>             ├── MB/
>             │   └── MB_MAX/
>             ├── MB_OPT/
>             └── MB_MIN/
> 
> 
> Reinette
>