* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-05-29 18:06 [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept Reinette Chatre
@ 2026-06-02 20:23 ` Babu Moger
2026-06-02 22:56 ` Reinette Chatre
2026-06-02 23:32 ` Chen, Yu C
` (5 subsequent siblings)
6 siblings, 1 reply; 66+ messages in thread
From: Babu Moger @ 2026-06-02 20:23 UTC (permalink / raw)
To: Reinette Chatre, Tony Luck, Ben Horgan, James Morse, Dave Martin,
Drew Fustini, Fenghua Yu, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi Reinette,
For some reason, I couldn’t find your patch on lore.kernel.org:
https://lore.kernel.org/lkml/?q=Reinette+Chatre
I eventually located it here:
https://sashiko.dev/#/message/aab804b9-e8b5-40ad-a85b-af7033391243%40intel.com
Thanks for sharing the patches. I’m still reviewing them.
I was able to build and boot the kernel and can see the MIN and MAX
controls. After moving your test code to __rdt_get_mem_config_amd().
On 5/29/26 13:06, Reinette Chatre wrote:
> Hi Everybody,
>
> It has been a while since we discussed the resctrl changes required to support
> hardware that has controls with fine granularity or hardware that has multiple
> controls per resource. For reference, the most recent email discussion can
> be found at [1] with a summary of discussions in last year's plumbers slides [2].
>
> I created a PoC that I believe supports what folks have agreed to so far. I
> hope this can help us to restart the discussion with the goal that resctrl gains
> support for upcoming hardware that require these features.
>
> Request regarding this PoC
> ==========================
>
> Please consider this PoC as a "direction check" on the schema description and multiple
> control discussions held thus far.
>
> Could folks working on enabling new hardware requiring this capability please consider
> if this is something you can build on and how it should be improved to support these
> upcoming capabilities?
>
> Opens
> =====
>
> While the PoC aims to support what folks agreed on some opens remain:
> - I attempted to make some MPAM supporting changes but these are all just compile
> tested. While MPAM should benefit from the new control properties I did not
> initialize them on MPAM and did not attempt refactor to separate out
> the architecture specific control properties (more on what this means later).
> I did attempt some MPAM refactoring that duplicates the MPAM domain to the
> control domain and monitoring domain lists in support of there being multiple
> controls each with its own list of control domains but it is definitely not good
> design.
> - No support for emulated controls (yet). The PoC is quite large already
> but I think it can be used as a base for emulated controls for which the software
> controller could be a potential first customer. In this PoC mounting with
> software controller will still display the original controller's properties.
> - One open that needs to be addressed as part of support for emulated controls is
> how best to display emulation relationship via resctrl hierarchy.
> - No support for "read-modify-write" usage of schemata file. This is where we
> discussed (without agreement) on possibly introducing the "#" prefix to schemata
> file entries. This PoC does not support this prefix and the current assumption/expectation
> is that when user space changes a configuration only the new control values are
> written to schemata file. I thus do not have a plan to support this so please
> share opinions in this regard if you have some.
> - Controls are independent for now. This means that, for example, if a resource
> supports a "MIN" and "MAX" control then this implementation would allow user to
> set the "maximum" control values to be less than the "minimum" control values.
> - PoC supports the "bitmap" control but does not (yet) expose properties of a bitmap
> control to the new info/<resource>/resource_schemata directory.
>
> Accessing PoC
> =============
>
> Please consider the PoC as a rough draft. It has only been compile tested for Arm
> and known to be incomplete in Arm support. To help with experimenting I only
> fully adapted the Intel MBA resource to demo two dummy additional MBA controls.
> All architectures should immediately benefit from the new schema descriptions
> and new info/MB/resource_schemata hierarchy.
>
> I considered the patches self too many for email. Instead, the PoC can be found at:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git branch resctrl/controls_rfc_v1
>
> The work is based on v7.1-rc2 that also includes the following series (two of which has
> since been queued) included:
>
> "selftests/resctrl: Fixes and improvements focused on Intel platforms"
> https://lore.kernel.org/lkml/cover.1775266384.git.reinette.chatre@intel.com/
>
> "x86,fs/resctrl: Improve resctrl quality and consistency"
> https://lore.kernel.org/lkml/cover.1777419024.git.reinette.chatre@intel.com/
>
> "x86,fs/resctrl: Pave the way for MPAM counter assignment"
> https://lore.kernel.org/lkml/20260506082855.3694761-1-ben.horgan@arm.com/
>
>
> Primary resctrl fs data structure changes
> =========================================
>
> Introduces a control represented by struct resctrl_ctrl that looks as below. To make
> the changes easier to follow I kept some of the original names to help communicate
> where familiar data structures land.
>
> What to notice about a control is that it has some common properties required
> from all controls (scope, type, etc.) and then depending on the type of control
> (RESCTRL_CTRL_BITMAP or RESCTRL_CTRL_SCALAR) there are type specific properties.
>
> /**
> * struct resctrl_ctrl - A resource control
> * @entry: List entry of rdt_resource::controls
> * @scope: Scope of the resource that this control allocates
> * @domains: RCU list of all control domains
> * @type: The control type that determines the properties of the control,
> * format string for displaying control values to user space, and
> * parser of control values provided by user space.
> * @name: Name of the control. Appended to final resource name
> * (rdt_resource_final::name) to create final schema entry.
> * Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
> * For example, with resource name "MB" and control name "MAX" the
> * schema entry will be "MB_MAX".
> * @cache: Cache allocation control properties.
> * @membw: Bandwidth control properties.
> */
> struct resctrl_ctrl {
> struct list_head entry;
> enum resctrl_scope scope;
> struct list_head domains;
> enum resctrl_ctrl_type type;
> enum resctrl_ctrl_name name;
> union {
> struct resctrl_cache cache;
> struct resctrl_membw membw;
> };
> };
>
> Two members summarize how this new structure fits into the rest of resctrl:
> a) resctrl_ctrl::entry
> Since a resource can support multiple controls there is a new list
> in struct rdt_resource named "controls" that contains the list of all
> controls supported by the resource.
> b) resctrl_ctrl::domains
> Instead of the list of control domains belonging to a resource they
> now belong to the control self. By doing so resctrl can support resource
> controls at different scope for the same resource. This is intended to
> support some upcoming MPAM and RISC-V usages.
>
I like the idea of supporting multiple controls for each resource.
With these patches, now we have one list containing all the controls.
However, in case of RDT_RESOURCE_L3, we have two lists "mon_domains" and
"controls". mon_domains list deals with monitoring and control deals
with control parts(multiple).
Have you thought about making the list("control") generic so that the
control can be monitoring also. It will just one list containing
multiple controls or monitor.
> Example architectural data structure changes
> ============================================
>
> An architecture can use the new control by following a similar pattern to
> resource and domain use by architectures. Consider the following for x86
> where a new architecture specific struct resctrl_hw_ctrl includes
> struct resctrl_ctrl and any architecture private data needed to support
> the control:
>
> /*
> * struct resctrl_hw_ctrl - Arch private properties of a resource control
> * @r_ctrl: Control properties exposed to resctrl file system
> * @msr_base: Base MSR address where control values should be programmed
> * @msr_update: Function pointer to update control values
> */
> struct resctrl_hw_ctrl {
> struct resctrl_ctrl r_ctrl;
> unsigned int msr_base;
> void (*msr_update)(struct msr_param *m);
> };
>
> Structure of patch series
> =========================
>
> As a PoC the series is not perfectly structured but to help navigate this work
> on a high level the changes can be categorized as follows:
>
> Patch 1 to 11:
> With a vision of what a "control" is, remove unused/unnecessary
> members, make clear what is a *resource* property vs a *control*
> property, do some renaming to help with the PoC.
>
> Patch 12:
> Introduce struct resctrl_ctrl and re-arrange existing struct rdt_resource
> members to form part of new rdt_resource::ctrl
>
> Patch 13 to 44:
> A lot of wrangling to introduce struct resctrl_ctrl to all code that needs
> to work with a control and/or domain without assuming that the control is
> the one and only control embedded in the resource it belongs to. Essentially,
> a lot of changes passing the control around in addition to the resource/domain.
>
> Patch 45:
> Switch the single struct resctrl_ctrl member of struct rdt_resource to be
> a list of struct resctrl_ctrl.
>
> Patch 47 to 49:
> Introduce new info/<resource>/resource_schemata hierarchy to first only
> consist of properties already known to resctrl fs.
>
> Patch 50 to 52:
> Introduce the new control properties per [1], initialize them for x86,
> and expose them via info/<resource>/resource_schemata
>
> Patch 53:
> Let the new struct resctrl_hw_ctrl contain architecture's control properties.
>
> Patch 54:
> Teach resctrl fs about "MIN" and "MAX" controls.
>
> Patch 55:
> Sample of "MIN" and "MAX" memory bandwidth controls for x86.
My assumption is that the MIN and MAX controls here are just examples,
correct?
You only mentioned patch 55 as "NOT_FOR_INCLUSION". I assume patch 54
should also be marked as "NOT_FOR_INCLUSION"?
Thanks
Babu
>
> Example interactions
> ====================
>
> This series can be used on an x86 system where it will show two new dummy controls
> where it is possible to interact with the new controls.
> For example:
>
> # cat schemata
> MB_MAX:0=100;1=100
> MB_MIN:0=100;1=100
> MB:0=100;1=100
> L3:0=fff;1=fff
> # echo 'MB_MIN:0=50' > schemata
> # cat schemata
> MB_MAX:0=100;1=100
> MB_MIN:0=50;1=100
> MB:0=100;1=100
> L3:0=fff;1=fff
>
> Writing to the dummy control will call a dummy callback that just prints to the
> kernel log:
> "resctrl: Updata temporary MIN control on domain 0 with user value 50"
>
>
> Example output of info/MB/:
> /sys/fs/resctrl/info/MB/thread_throttle_mode:max
> /sys/fs/resctrl/info/MB/num_closids:15
> /sys/fs/resctrl/info/MB/delay_linear:1
> /sys/fs/resctrl/info/MB/min_bandwidth:10
> /sys/fs/resctrl/info/MB/resource_schemata/MB/resolution:100
> /sys/fs/resctrl/info/MB/resource_schemata/MB/tolerance:5
> /sys/fs/resctrl/info/MB/resource_schemata/MB/type:scalar
> /sys/fs/resctrl/info/MB/resource_schemata/MB/min:10
> /sys/fs/resctrl/info/MB/resource_schemata/MB/scale:1
> /sys/fs/resctrl/info/MB/resource_schemata/MB/scope:L3
> /sys/fs/resctrl/info/MB/resource_schemata/MB/unit:all
> /sys/fs/resctrl/info/MB/resource_schemata/MB/max:100
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/resolution:100
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/tolerance:5
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/type:scalar
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/min:10
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scale:1
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scope:L3
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/unit:all
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/max:100
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/resolution:100
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/tolerance:5
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/type:scalar
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/min:10
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scale:1
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scope:L3
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/unit:all
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/max:100
> /sys/fs/resctrl/info/MB/bandwidth_gran:10
>
> Any feedback is appreciated.
>
> Reinette
>
> [1] https://lore.kernel.org/lkml/aPtfMFfLV1l%2FRB0L@e133380.arm.com/
> [2] https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf
>
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-02 20:23 ` Babu Moger
@ 2026-06-02 22:56 ` Reinette Chatre
2026-06-03 1:14 ` Moger, Babu
0 siblings, 1 reply; 66+ messages in thread
From: Reinette Chatre @ 2026-06-02 22:56 UTC (permalink / raw)
To: Babu Moger, Tony Luck, Ben Horgan, James Morse, Dave Martin,
Drew Fustini, Fenghua Yu, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi Babu,
On 6/2/26 1:23 PM, Babu Moger wrote:
> Hi Reinette,
>
> For some reason, I couldn’t find your patch on lore.kernel.org:
> https://lore.kernel.org/lkml/?q=Reinette+Chatre
How about:
https://lore.kernel.org/lkml/aab804b9-e8b5-40ad-a85b-af7033391243@intel.com/
>
> I eventually located it here:
> https://sashiko.dev/#/message/aab804b9-e8b5-40ad-a85b-af7033391243%40intel.com
>
> Thanks for sharing the patches. I’m still reviewing them.
>
> I was able to build and boot the kernel and can see the MIN and MAX controls. After moving your test code to __rdt_get_mem_config_amd().
Thank you very much for trying it out.
> On 5/29/26 13:06, Reinette Chatre wrote:
...
>> Primary resctrl fs data structure changes
>> =========================================
>>
>> Introduces a control represented by struct resctrl_ctrl that looks as below. To make
>> the changes easier to follow I kept some of the original names to help communicate
>> where familiar data structures land.
>>
>> What to notice about a control is that it has some common properties required
>> from all controls (scope, type, etc.) and then depending on the type of control
>> (RESCTRL_CTRL_BITMAP or RESCTRL_CTRL_SCALAR) there are type specific properties.
>>
>> /**
>> * struct resctrl_ctrl - A resource control
>> * @entry: List entry of rdt_resource::controls
>> * @scope: Scope of the resource that this control allocates
>> * @domains: RCU list of all control domains
>> * @type: The control type that determines the properties of the control,
>> * format string for displaying control values to user space, and
>> * parser of control values provided by user space.
>> * @name: Name of the control. Appended to final resource name
>> * (rdt_resource_final::name) to create final schema entry.
>> * Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
>> * For example, with resource name "MB" and control name "MAX" the
>> * schema entry will be "MB_MAX".
>> * @cache: Cache allocation control properties.
>> * @membw: Bandwidth control properties.
>> */
>> struct resctrl_ctrl {
>> struct list_head entry;
>> enum resctrl_scope scope;
>> struct list_head domains;
>> enum resctrl_ctrl_type type;
>> enum resctrl_ctrl_name name;
>> union {
>> struct resctrl_cache cache;
>> struct resctrl_membw membw;
>> };
>> };
>>
>> Two members summarize how this new structure fits into the rest of resctrl:
>> a) resctrl_ctrl::entry
>> Since a resource can support multiple controls there is a new list
>> in struct rdt_resource named "controls" that contains the list of all
>> controls supported by the resource.
>> b) resctrl_ctrl::domains
>> Instead of the list of control domains belonging to a resource they
>> now belong to the control self. By doing so resctrl can support resource
>> controls at different scope for the same resource. This is intended to
>> support some upcoming MPAM and RISC-V usages.
>>
>
> I like the idea of supporting multiple controls for each resource.
>
> With these patches, now we have one list containing all the controls.
>
> However, in case of RDT_RESOURCE_L3, we have two lists "mon_domains" and "controls". mon_domains list deals with monitoring and control deals with control parts(multiple).
>
> Have you thought about making the list("control") generic so that the control can be monitoring also. It will just one list containing multiple controls or monitor.
The control list adds an additional layer of abstraction just for control management,
independent from monitoring. The mon_domains list is unchanged while each control now
has its own ctrl_domains list.
Here is an attempt to visualize how a resource with two monitoring domains, and two controls,
each with two control domains end up being managed:
+-------------------------+
| struct rdt_resource |
+-------------------------+
| ... |
| controls (list_head) |---------+
| mon_domains (list_head) |---+ |
| ... | | |
+-------------------------+ | |
| |
+---------------------------+ |
| |
v v
+-----------------------------+ +-------------------------+
| struct rdt_l3_mon_domain #1 | | struct resctrl_ctrl #A |
+-----------------------------+ +-------------------------+
| rdt_domain_hdr |-+ | entry (list_head) | +----------------------------+
| ... | | | domains (list_head) |------>| struct rdt_ctrl_domain #A1 |
+-----------------------------+ | | ... | +----------------------------+
| +-------------------------+ | rdt_domain_hdr |---+
+-----------------------------+ | | ... | |
| (next) | (next) +----------------------------+ |
v v |
+-----------------------------+ +-------------------------+ +----------------------------+<--+
| struct rdt_l3_mon_domain #2 | | struct resctrl_ctrl #B | | struct rdt_ctrl_domain #A2 |
+-----------------------------+ +-------------------------+ +----------------------------+
| rdt_domain_hdr | | entry (list_head) | | rdt_domain_hdr |
| ... | | domains (list_head) |----n | ... |
+-----------------------------+ | ... | | +----------------------------+
+-------------------------+ |
|
+----------------------------+
|
v
+----------------------------+
| struct rdt_ctrl_domain #B1 |
+----------------------------+
| rdt_domain_hdr |---+
| ... | |
+----------------------------+ |
|
+----------------------------+<--+
| struct rdt_ctrl_domain #B2 |
+----------------------------+
| rdt_domain_hdr |
| ... |
+----------------------------+
resctrl used to manage single domains that are capable of both monitoring and control
but this was split to support scenario where monitoring and control of a resource are
done at different scope. See
cd84f72b6a5c ("x86/resctrl: Prepare for different scope for control/monitor operations")
This feature further expands the difference between monitoring and control since now there
can be multiple instances of a control domain (one per control) associated with a resource
while monitoring still just supports one monitoring domain per resource.
I thus cannot see how this can be accomplished with a single list. Could you sketch out
what you have in mind?
...
>>
>> Patch 54:
>> Teach resctrl fs about "MIN" and "MAX" controls.
>>
>> Patch 55:
>> Sample of "MIN" and "MAX" memory bandwidth controls for x86.
>
> My assumption is that the MIN and MAX controls here are just examples, correct?
>
> You only mentioned patch 55 as "NOT_FOR_INCLUSION". I assume patch 54 should also be marked as "NOT_FOR_INCLUSION"?
While patch 55 is the first and only "user" of patch 54 I believe that patch 54 (or some variant
of it) will stay since we already know that both MPAM and Intel need to support min and max controls.
Reinette
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-02 22:56 ` Reinette Chatre
@ 2026-06-03 1:14 ` Moger, Babu
2026-06-03 3:55 ` Reinette Chatre
0 siblings, 1 reply; 66+ messages in thread
From: Moger, Babu @ 2026-06-03 1:14 UTC (permalink / raw)
To: Reinette Chatre, Babu Moger, Tony Luck, Ben Horgan, James Morse,
Dave Martin, Drew Fustini, Fenghua Yu, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi Reinette,
On 6/2/2026 5:56 PM, Reinette Chatre wrote:
> Hi Babu,
>
> On 6/2/26 1:23 PM, Babu Moger wrote:
>> Hi Reinette,
>>
>> For some reason, I couldn’t find your patch on lore.kernel.org:
>> https://lore.kernel.org/lkml/?q=Reinette+Chatre
>
> How about:
> https://lore.kernel.org/lkml/aab804b9-e8b5-40ad-a85b-af7033391243@intel.com/
Yes. It works. But, I used to find patches using the names. It did not
work this time.
>
>>
>> I eventually located it here:
>> https://sashiko.dev/#/message/aab804b9-e8b5-40ad-a85b-af7033391243%40intel.com
>>
>> Thanks for sharing the patches. I’m still reviewing them.
>>
>> I was able to build and boot the kernel and can see the MIN and MAX controls. After moving your test code to __rdt_get_mem_config_amd().
>
> Thank you very much for trying it out.
>
>> On 5/29/26 13:06, Reinette Chatre wrote:
>
> ...
>
>>> Primary resctrl fs data structure changes
>>> =========================================
>>>
>>> Introduces a control represented by struct resctrl_ctrl that looks as below. To make
>>> the changes easier to follow I kept some of the original names to help communicate
>>> where familiar data structures land.
>>>
>>> What to notice about a control is that it has some common properties required
>>> from all controls (scope, type, etc.) and then depending on the type of control
>>> (RESCTRL_CTRL_BITMAP or RESCTRL_CTRL_SCALAR) there are type specific properties.
>>>
>>> /**
>>> * struct resctrl_ctrl - A resource control
>>> * @entry: List entry of rdt_resource::controls
>>> * @scope: Scope of the resource that this control allocates
>>> * @domains: RCU list of all control domains
>>> * @type: The control type that determines the properties of the control,
>>> * format string for displaying control values to user space, and
>>> * parser of control values provided by user space.
>>> * @name: Name of the control. Appended to final resource name
>>> * (rdt_resource_final::name) to create final schema entry.
>>> * Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
>>> * For example, with resource name "MB" and control name "MAX" the
>>> * schema entry will be "MB_MAX".
>>> * @cache: Cache allocation control properties.
>>> * @membw: Bandwidth control properties.
>>> */
>>> struct resctrl_ctrl {
>>> struct list_head entry;
>>> enum resctrl_scope scope;
>>> struct list_head domains;
>>> enum resctrl_ctrl_type type;
>>> enum resctrl_ctrl_name name;
>>> union {
>>> struct resctrl_cache cache;
>>> struct resctrl_membw membw;
>>> };
>>> };
>>>
>>> Two members summarize how this new structure fits into the rest of resctrl:
>>> a) resctrl_ctrl::entry
>>> Since a resource can support multiple controls there is a new list
>>> in struct rdt_resource named "controls" that contains the list of all
>>> controls supported by the resource.
>>> b) resctrl_ctrl::domains
>>> Instead of the list of control domains belonging to a resource they
>>> now belong to the control self. By doing so resctrl can support resource
>>> controls at different scope for the same resource. This is intended to
>>> support some upcoming MPAM and RISC-V usages.
>>>
>>
>> I like the idea of supporting multiple controls for each resource.
>>
>> With these patches, now we have one list containing all the controls.
>>
>> However, in case of RDT_RESOURCE_L3, we have two lists "mon_domains" and "controls". mon_domains list deals with monitoring and control deals with control parts(multiple).
>>
>> Have you thought about making the list("control") generic so that the control can be monitoring also. It will just one list containing multiple controls or monitor.
>
> The control list adds an additional layer of abstraction just for control management,
> independent from monitoring. The mon_domains list is unchanged while each control now
> has its own ctrl_domains list.
>
> Here is an attempt to visualize how a resource with two monitoring domains, and two controls,
> each with two control domains end up being managed:
>
> +-------------------------+
> | struct rdt_resource |
> +-------------------------+
> | ... |
> | controls (list_head) |---------+
> | mon_domains (list_head) |---+ |
> | ... | | |
> +-------------------------+ | |
> | |
> +---------------------------+ |
> | |
> v v
> +-----------------------------+ +-------------------------+
> | struct rdt_l3_mon_domain #1 | | struct resctrl_ctrl #A |
> +-----------------------------+ +-------------------------+
> | rdt_domain_hdr |-+ | entry (list_head) | +----------------------------+
> | ... | | | domains (list_head) |------>| struct rdt_ctrl_domain #A1 |
> +-----------------------------+ | | ... | +----------------------------+
> | +-------------------------+ | rdt_domain_hdr |---+
> +-----------------------------+ | | ... | |
> | (next) | (next) +----------------------------+ |
> v v |
> +-----------------------------+ +-------------------------+ +----------------------------+<--+
> | struct rdt_l3_mon_domain #2 | | struct resctrl_ctrl #B | | struct rdt_ctrl_domain #A2 |
> +-----------------------------+ +-------------------------+ +----------------------------+
> | rdt_domain_hdr | | entry (list_head) | | rdt_domain_hdr |
> | ... | | domains (list_head) |----n | ... |
> +-----------------------------+ | ... | | +----------------------------+
> +-------------------------+ |
> |
> +----------------------------+
> |
> v
> +----------------------------+
> | struct rdt_ctrl_domain #B1 |
> +----------------------------+
> | rdt_domain_hdr |---+
> | ... | |
> +----------------------------+ |
> |
> +----------------------------+<--+
> | struct rdt_ctrl_domain #B2 |
> +----------------------------+
> | rdt_domain_hdr |
> | ... |
> +----------------------------+
>
> resctrl used to manage single domains that are capable of both monitoring and control
> but this was split to support scenario where monitoring and control of a resource are
> done at different scope. See
> cd84f72b6a5c ("x86/resctrl: Prepare for different scope for control/monitor operations")
>
> This feature further expands the difference between monitoring and control since now there
> can be multiple instances of a control domain (one per control) associated with a resource
> while monitoring still just supports one monitoring domain per resource.
>
> I thus cannot see how this can be accomplished with a single list. Could you sketch out
> what you have in mind?
>
Thanks for the sketch. It makes it much more clear.
mon_domains directly maintains a list of monitoring domains, whereas
controls maintains a list of control objects, each of which owns its own
domain list.
I was thinking, one possibility would be to make mon_domains follow a
structure similar to controls, allowing both to be represented through a
common top-level list.
However, the resulting abstraction may become fairly large and complex,
with limited overlap between the monitoring and control data structures.
> ...
>
>>>
>>> Patch 54:
>>> Teach resctrl fs about "MIN" and "MAX" controls.
>>>
>>> Patch 55:
>>> Sample of "MIN" and "MAX" memory bandwidth controls for x86.
>>
>> My assumption is that the MIN and MAX controls here are just examples, correct?
>>
>> You only mentioned patch 55 as "NOT_FOR_INCLUSION". I assume patch 54 should also be marked as "NOT_FOR_INCLUSION"?
>
> While patch 55 is the first and only "user" of patch 54 I believe that patch 54 (or some variant
> of it) will stay since we already know that both MPAM and Intel need to support min and max controls.
That is good to know.
Thanks
Babu
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-03 1:14 ` Moger, Babu
@ 2026-06-03 3:55 ` Reinette Chatre
2026-06-03 14:40 ` Babu Moger
0 siblings, 1 reply; 66+ messages in thread
From: Reinette Chatre @ 2026-06-03 3:55 UTC (permalink / raw)
To: Moger, Babu, Babu Moger, Tony Luck, Ben Horgan, James Morse,
Dave Martin, Drew Fustini, Fenghua Yu, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi Babu,
On 6/2/26 6:14 PM, Moger, Babu wrote:
> Hi Reinette,
>
> On 6/2/2026 5:56 PM, Reinette Chatre wrote:
>> Hi Babu,
>>
>> On 6/2/26 1:23 PM, Babu Moger wrote:
>>> Hi Reinette,
>>>
>>> For some reason, I couldn’t find your patch on lore.kernel.org:
>>> https://lore.kernel.org/lkml/?q=Reinette+Chatre
>>
>> How about:
>> https://lore.kernel.org/lkml/aab804b9-e8b5-40ad-a85b-af7033391243@intel.com/
>
> Yes. It works. But, I used to find patches using the names. It did not work this time.
Interesting. When I run your lore query the first email from this thread is among the
results.
>
>>
>>>
>>> I eventually located it here:
>>> https://sashiko.dev/#/message/aab804b9-e8b5-40ad-a85b-af7033391243%40intel.com
>>>
>>> Thanks for sharing the patches. I’m still reviewing them.
>>>
>>> I was able to build and boot the kernel and can see the MIN and MAX controls. After moving your test code to __rdt_get_mem_config_amd().
>>
>> Thank you very much for trying it out.
>>
>>> On 5/29/26 13:06, Reinette Chatre wrote:
>>
>> ...
>>
>>>> Primary resctrl fs data structure changes
>>>> =========================================
>>>>
>>>> Introduces a control represented by struct resctrl_ctrl that looks as below. To make
>>>> the changes easier to follow I kept some of the original names to help communicate
>>>> where familiar data structures land.
>>>>
>>>> What to notice about a control is that it has some common properties required
>>>> from all controls (scope, type, etc.) and then depending on the type of control
>>>> (RESCTRL_CTRL_BITMAP or RESCTRL_CTRL_SCALAR) there are type specific properties.
>>>>
>>>> /**
>>>> * struct resctrl_ctrl - A resource control
>>>> * @entry: List entry of rdt_resource::controls
>>>> * @scope: Scope of the resource that this control allocates
>>>> * @domains: RCU list of all control domains
>>>> * @type: The control type that determines the properties of the control,
>>>> * format string for displaying control values to user space, and
>>>> * parser of control values provided by user space.
>>>> * @name: Name of the control. Appended to final resource name
>>>> * (rdt_resource_final::name) to create final schema entry.
>>>> * Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
>>>> * For example, with resource name "MB" and control name "MAX" the
>>>> * schema entry will be "MB_MAX".
>>>> * @cache: Cache allocation control properties.
>>>> * @membw: Bandwidth control properties.
>>>> */
>>>> struct resctrl_ctrl {
>>>> struct list_head entry;
>>>> enum resctrl_scope scope;
>>>> struct list_head domains;
>>>> enum resctrl_ctrl_type type;
>>>> enum resctrl_ctrl_name name;
>>>> union {
>>>> struct resctrl_cache cache;
>>>> struct resctrl_membw membw;
>>>> };
>>>> };
>>>>
>>>> Two members summarize how this new structure fits into the rest of resctrl:
>>>> a) resctrl_ctrl::entry
>>>> Since a resource can support multiple controls there is a new list
>>>> in struct rdt_resource named "controls" that contains the list of all
>>>> controls supported by the resource.
>>>> b) resctrl_ctrl::domains
>>>> Instead of the list of control domains belonging to a resource they
>>>> now belong to the control self. By doing so resctrl can support resource
>>>> controls at different scope for the same resource. This is intended to
>>>> support some upcoming MPAM and RISC-V usages.
>>>>
>>>
>>> I like the idea of supporting multiple controls for each resource.
>>>
>>> With these patches, now we have one list containing all the controls.
>>>
>>> However, in case of RDT_RESOURCE_L3, we have two lists "mon_domains" and "controls". mon_domains list deals with monitoring and control deals with control parts(multiple).
>>>
>>> Have you thought about making the list("control") generic so that the control can be monitoring also. It will just one list containing multiple controls or monitor.
>>
>> The control list adds an additional layer of abstraction just for control management,
>> independent from monitoring. The mon_domains list is unchanged while each control now
>> has its own ctrl_domains list.
>>
>> Here is an attempt to visualize how a resource with two monitoring domains, and two controls,
>> each with two control domains end up being managed:
>>
>> +-------------------------+
>> | struct rdt_resource |
>> +-------------------------+
>> | ... |
>> | controls (list_head) |---------+
>> | mon_domains (list_head) |---+ |
>> | ... | | |
>> +-------------------------+ | |
>> | |
>> +---------------------------+ |
>> | |
>> v v
>> +-----------------------------+ +-------------------------+
>> | struct rdt_l3_mon_domain #1 | | struct resctrl_ctrl #A |
>> +-----------------------------+ +-------------------------+
>> | rdt_domain_hdr |-+ | entry (list_head) | +----------------------------+
>> | ... | | | domains (list_head) |------>| struct rdt_ctrl_domain #A1 |
>> +-----------------------------+ | | ... | +----------------------------+
>> | +-------------------------+ | rdt_domain_hdr |---+
>> +-----------------------------+ | | ... | |
>> | (next) | (next) +----------------------------+ |
>> v v |
>> +-----------------------------+ +-------------------------+ +----------------------------+<--+
>> | struct rdt_l3_mon_domain #2 | | struct resctrl_ctrl #B | | struct rdt_ctrl_domain #A2 |
>> +-----------------------------+ +-------------------------+ +----------------------------+
>> | rdt_domain_hdr | | entry (list_head) | | rdt_domain_hdr |
>> | ... | | domains (list_head) |----n | ... |
>> +-----------------------------+ | ... | | +----------------------------+
>> +-------------------------+ |
>> |
>> +----------------------------+
>> |
>> v
>> +----------------------------+
>> | struct rdt_ctrl_domain #B1 |
>> +----------------------------+
>> | rdt_domain_hdr |---+
>> | ... | |
>> +----------------------------+ |
>> |
>> +----------------------------+<--+
>> | struct rdt_ctrl_domain #B2 |
>> +----------------------------+
>> | rdt_domain_hdr |
>> | ... |
>> +----------------------------+
>>
>> resctrl used to manage single domains that are capable of both monitoring and control
>> but this was split to support scenario where monitoring and control of a resource are
>> done at different scope. See
>> cd84f72b6a5c ("x86/resctrl: Prepare for different scope for control/monitor operations")
>>
>> This feature further expands the difference between monitoring and control since now there
>> can be multiple instances of a control domain (one per control) associated with a resource
>> while monitoring still just supports one monitoring domain per resource.
>>
>> I thus cannot see how this can be accomplished with a single list. Could you sketch out
>> what you have in mind?
>>
>
> Thanks for the sketch. It makes it much more clear.
>
> mon_domains directly maintains a list of monitoring domains, whereas controls maintains a list of control objects, each of which owns its own domain list.
>
> I was thinking, one possibility would be to make mon_domains follow a structure similar to controls, allowing both to be represented through a common top-level list.
>
> However, the resulting abstraction may become fairly large and complex, with limited overlap between the monitoring and control data structures.
>
I am trying to envision this but it sounds to be as though each control has its own monitoring
which is not clear to me. resctrl ultimately needs to support hardware features. Could you please
share more about the hardware feature(s) this design intends to support?
Reinette
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-03 3:55 ` Reinette Chatre
@ 2026-06-03 14:40 ` Babu Moger
0 siblings, 0 replies; 66+ messages in thread
From: Babu Moger @ 2026-06-03 14:40 UTC (permalink / raw)
To: Reinette Chatre, Moger, Babu, Tony Luck, Ben Horgan, James Morse,
Dave Martin, Drew Fustini, Fenghua Yu, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi Reinette,
On 6/2/26 22:55, Reinette Chatre wrote:
> Hi Babu,
>
> On 6/2/26 6:14 PM, Moger, Babu wrote:
>> Hi Reinette,
>>
>> On 6/2/2026 5:56 PM, Reinette Chatre wrote:
>>> Hi Babu,
>>>
>>> On 6/2/26 1:23 PM, Babu Moger wrote:
>>>> Hi Reinette,
>>>>
>>>> For some reason, I couldn’t find your patch on lore.kernel.org:
>>>> https://lore.kernel.org/lkml/?q=Reinette+Chatre
>>>
>>> How about:
>>> https://lore.kernel.org/lkml/aab804b9-e8b5-40ad-a85b-af7033391243@intel.com/
>>
>> Yes. It works. But, I used to find patches using the names. It did not work this time.
>
> Interesting. When I run your lore query the first email from this thread is among the
> results.
>
>>
>>>
>>>>
>>>> I eventually located it here:
>>>> https://sashiko.dev/#/message/aab804b9-e8b5-40ad-a85b-af7033391243%40intel.com
>>>>
>>>> Thanks for sharing the patches. I’m still reviewing them.
>>>>
>>>> I was able to build and boot the kernel and can see the MIN and MAX controls. After moving your test code to __rdt_get_mem_config_amd().
>>>
>>> Thank you very much for trying it out.
>>>
>>>> On 5/29/26 13:06, Reinette Chatre wrote:
>>>
>>> ...
>>>
>>>>> Primary resctrl fs data structure changes
>>>>> =========================================
>>>>>
>>>>> Introduces a control represented by struct resctrl_ctrl that looks as below. To make
>>>>> the changes easier to follow I kept some of the original names to help communicate
>>>>> where familiar data structures land.
>>>>>
>>>>> What to notice about a control is that it has some common properties required
>>>>> from all controls (scope, type, etc.) and then depending on the type of control
>>>>> (RESCTRL_CTRL_BITMAP or RESCTRL_CTRL_SCALAR) there are type specific properties.
>>>>>
>>>>> /**
>>>>> * struct resctrl_ctrl - A resource control
>>>>> * @entry: List entry of rdt_resource::controls
>>>>> * @scope: Scope of the resource that this control allocates
>>>>> * @domains: RCU list of all control domains
>>>>> * @type: The control type that determines the properties of the control,
>>>>> * format string for displaying control values to user space, and
>>>>> * parser of control values provided by user space.
>>>>> * @name: Name of the control. Appended to final resource name
>>>>> * (rdt_resource_final::name) to create final schema entry.
>>>>> * Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
>>>>> * For example, with resource name "MB" and control name "MAX" the
>>>>> * schema entry will be "MB_MAX".
>>>>> * @cache: Cache allocation control properties.
>>>>> * @membw: Bandwidth control properties.
>>>>> */
>>>>> struct resctrl_ctrl {
>>>>> struct list_head entry;
>>>>> enum resctrl_scope scope;
>>>>> struct list_head domains;
>>>>> enum resctrl_ctrl_type type;
>>>>> enum resctrl_ctrl_name name;
>>>>> union {
>>>>> struct resctrl_cache cache;
>>>>> struct resctrl_membw membw;
>>>>> };
>>>>> };
>>>>>
>>>>> Two members summarize how this new structure fits into the rest of resctrl:
>>>>> a) resctrl_ctrl::entry
>>>>> Since a resource can support multiple controls there is a new list
>>>>> in struct rdt_resource named "controls" that contains the list of all
>>>>> controls supported by the resource.
>>>>> b) resctrl_ctrl::domains
>>>>> Instead of the list of control domains belonging to a resource they
>>>>> now belong to the control self. By doing so resctrl can support resource
>>>>> controls at different scope for the same resource. This is intended to
>>>>> support some upcoming MPAM and RISC-V usages.
>>>>>
>>>>
>>>> I like the idea of supporting multiple controls for each resource.
>>>>
>>>> With these patches, now we have one list containing all the controls.
>>>>
>>>> However, in case of RDT_RESOURCE_L3, we have two lists "mon_domains" and "controls". mon_domains list deals with monitoring and control deals with control parts(multiple).
>>>>
>>>> Have you thought about making the list("control") generic so that the control can be monitoring also. It will just one list containing multiple controls or monitor.
>>>
>>> The control list adds an additional layer of abstraction just for control management,
>>> independent from monitoring. The mon_domains list is unchanged while each control now
>>> has its own ctrl_domains list.
>>>
>>> Here is an attempt to visualize how a resource with two monitoring domains, and two controls,
>>> each with two control domains end up being managed:
>>>
>>> +-------------------------+
>>> | struct rdt_resource |
>>> +-------------------------+
>>> | ... |
>>> | controls (list_head) |---------+
>>> | mon_domains (list_head) |---+ |
>>> | ... | | |
>>> +-------------------------+ | |
>>> | |
>>> +---------------------------+ |
>>> | |
>>> v v
>>> +-----------------------------+ +-------------------------+
>>> | struct rdt_l3_mon_domain #1 | | struct resctrl_ctrl #A |
>>> +-----------------------------+ +-------------------------+
>>> | rdt_domain_hdr |-+ | entry (list_head) | +----------------------------+
>>> | ... | | | domains (list_head) |------>| struct rdt_ctrl_domain #A1 |
>>> +-----------------------------+ | | ... | +----------------------------+
>>> | +-------------------------+ | rdt_domain_hdr |---+
>>> +-----------------------------+ | | ... | |
>>> | (next) | (next) +----------------------------+ |
>>> v v |
>>> +-----------------------------+ +-------------------------+ +----------------------------+<--+
>>> | struct rdt_l3_mon_domain #2 | | struct resctrl_ctrl #B | | struct rdt_ctrl_domain #A2 |
>>> +-----------------------------+ +-------------------------+ +----------------------------+
>>> | rdt_domain_hdr | | entry (list_head) | | rdt_domain_hdr |
>>> | ... | | domains (list_head) |----n | ... |
>>> +-----------------------------+ | ... | | +----------------------------+
>>> +-------------------------+ |
>>> |
>>> +----------------------------+
>>> |
>>> v
>>> +----------------------------+
>>> | struct rdt_ctrl_domain #B1 |
>>> +----------------------------+
>>> | rdt_domain_hdr |---+
>>> | ... | |
>>> +----------------------------+ |
>>> |
>>> +----------------------------+<--+
>>> | struct rdt_ctrl_domain #B2 |
>>> +----------------------------+
>>> | rdt_domain_hdr |
>>> | ... |
>>> +----------------------------+
>>>
>>> resctrl used to manage single domains that are capable of both monitoring and control
>>> but this was split to support scenario where monitoring and control of a resource are
>>> done at different scope. See
>>> cd84f72b6a5c ("x86/resctrl: Prepare for different scope for control/monitor operations")
>>>
>>> This feature further expands the difference between monitoring and control since now there
>>> can be multiple instances of a control domain (one per control) associated with a resource
>>> while monitoring still just supports one monitoring domain per resource.
>>>
>>> I thus cannot see how this can be accomplished with a single list. Could you sketch out
>>> what you have in mind?
>>>
>>
>> Thanks for the sketch. It makes it much more clear.
>>
>> mon_domains directly maintains a list of monitoring domains, whereas controls maintains a list of control objects, each of which owns its own domain list.
>>
>> I was thinking, one possibility would be to make mon_domains follow a structure similar to controls, allowing both to be represented through a common top-level list.
>>
>> However, the resulting abstraction may become fairly large and complex, with limited overlap between the monitoring and control data structures.
>>
>
> I am trying to envision this but it sounds to be as though each control has its own monitoring
> which is not clear to me. resctrl ultimately needs to support hardware features. Could you please
> share more about the hardware feature(s) this design intends to support?
>
This is mostly my thinking. There is no new hardware feature that
required this change. We can safely ignore this idea.
Thanks
Babu
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-05-29 18:06 [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept Reinette Chatre
2026-06-02 20:23 ` Babu Moger
@ 2026-06-02 23:32 ` Chen, Yu C
2026-06-03 3:45 ` Reinette Chatre
2026-06-03 15:15 ` Ben Horgan
` (4 subsequent siblings)
6 siblings, 1 reply; 66+ messages in thread
From: Chen, Yu C @ 2026-06-02 23:32 UTC (permalink / raw)
To: Reinette Chatre
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org, Tony Luck,
Dave Martin, James Morse, Ben Horgan, Babu Moger, Drew Fustini,
Fenghua Yu
Hi Reinette,
On 5/30/2026 2:06 AM, Reinette Chatre wrote:
[ ... ]
> /**
> * struct resctrl_ctrl - A resource control
> * @entry: List entry of rdt_resource::controls
> * @scope: Scope of the resource that this control allocates
> * @domains: RCU list of all control domains
> * @type: The control type that determines the properties of the control,
> * format string for displaying control values to user space, and
> * parser of control values provided by user space.
> * @name: Name of the control. Appended to final resource name
> * (rdt_resource_final::name) to create final schema entry.
> * Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
> * For example, with resource name "MB" and control name "MAX" the
> * schema entry will be "MB_MAX".
> * @cache: Cache allocation control properties.
> * @membw: Bandwidth control properties.
> */
> struct resctrl_ctrl {
> struct list_head entry;
> enum resctrl_scope scope;
> struct list_head domains;
> enum resctrl_ctrl_type type;
> enum resctrl_ctrl_name name;
> union {
> struct resctrl_cache cache;
> struct resctrl_membw membw;
> };
> };
>
Thanks for re-spinning this patch set.
Looking at commit (fs/resctrl: Introduce additional schema properties),
the newly added properties appear to be implemented for the
MBA resctrl_membw controller.
Would it make sense for these properties to be generic across all resctrl
controllers, CAT included? Should they consequently be relocated into
struct resctrl_ctrl if that approach is appropriate?
thanks,
Chenyu
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-02 23:32 ` Chen, Yu C
@ 2026-06-03 3:45 ` Reinette Chatre
2026-06-03 11:53 ` Chen, Yu C
0 siblings, 1 reply; 66+ messages in thread
From: Reinette Chatre @ 2026-06-03 3:45 UTC (permalink / raw)
To: Chen, Yu C
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org, Tony Luck,
Dave Martin, James Morse, Ben Horgan, Babu Moger, Drew Fustini,
Fenghua Yu
Hi Chenyu,
On 6/2/26 4:32 PM, Chen, Yu C wrote:
> Hi Reinette,
> On 5/30/2026 2:06 AM, Reinette Chatre wrote:
>
> [ ... ]
>
>> /**
>> * struct resctrl_ctrl - A resource control
>> * @entry: List entry of rdt_resource::controls
>> * @scope: Scope of the resource that this control allocates
>> * @domains: RCU list of all control domains
>> * @type: The control type that determines the properties of the control,
>> * format string for displaying control values to user space, and
>> * parser of control values provided by user space.
>> * @name: Name of the control. Appended to final resource name
>> * (rdt_resource_final::name) to create final schema entry.
>> * Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
>> * For example, with resource name "MB" and control name "MAX" the
>> * schema entry will be "MB_MAX".
>> * @cache: Cache allocation control properties.
>> * @membw: Bandwidth control properties.
>> */
>> struct resctrl_ctrl {
>> struct list_head entry;
>> enum resctrl_scope scope;
>> struct list_head domains;
>> enum resctrl_ctrl_type type;
>> enum resctrl_ctrl_name name;
>> union {
>> struct resctrl_cache cache;
>> struct resctrl_membw membw;
>> };
>> };
>>
>
> Thanks for re-spinning this patch set.
> Looking at commit (fs/resctrl: Introduce additional schema properties),
> the newly added properties appear to be implemented for the
> MBA resctrl_membw controller.
Please do note that the "struct resctrl_membw" naming is temporary. I only kept
current naming to help folks find where existing known structures land in this new
design but the "struct resctrl_membw" name is not appropriate moving forward.
When user space interacts with the new controls the intention is that each control
has a "type" that the user can find in the new "type" file and depending on that value
the user will know what other files/properties can be expected.
In this PoC there are two types: "scalar" and "bitmap" and each is associated with
a struct that contains the properties associated with that type.
Considering this, these structures could, for example, be renamed as (very long):
struct resctrl_membw -> struct resctrl_ctrl_scalar
struct resctrl_cache -> struct resctrl_ctrl_bitmap
> Would it make sense for these properties to be generic across all resctrl
> controllers, CAT included? Should they consequently be relocated into
> struct resctrl_ctrl if that approach is appropriate?
If there are properties that are common across all controls of all types
then they should be moved to struct resctrl_ctrl. Current examples are "type"
and "scope" that user space can expect from every control. It is not obvious to
me at this time what other properties we can designate as common in this way.
For bitmaps there is already the properties that are available in the top-level
directory. These can just be replicated as properties into the resource_schemata,
but perhaps using new names as suggested by Dave in
https://lore.kernel.org/lkml/aQOUAeVP9oc7RIn%2F@e133380.arm.com/
Reinette
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-03 3:45 ` Reinette Chatre
@ 2026-06-03 11:53 ` Chen, Yu C
2026-06-04 16:37 ` Reinette Chatre
0 siblings, 1 reply; 66+ messages in thread
From: Chen, Yu C @ 2026-06-03 11:53 UTC (permalink / raw)
To: Reinette Chatre
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org, Tony Luck,
Dave Martin, James Morse, Ben Horgan, Babu Moger, Drew Fustini,
Fenghua Yu
Hi Reinette,
On 6/3/2026 11:45 AM, Reinette Chatre wrote:
> Hi Chenyu,
>
[ ... ]
>>
>> Thanks for re-spinning this patch set.
>> Looking at commit (fs/resctrl: Introduce additional schema properties),
>> the newly added properties appear to be implemented for the
>> MBA resctrl_membw controller.
>
> Please do note that the "struct resctrl_membw" naming is temporary. I only kept
> current naming to help folks find where existing known structures land in this new
> design but the "struct resctrl_membw" name is not appropriate moving forward.
>
> When user space interacts with the new controls the intention is that each control
> has a "type" that the user can find in the new "type" file and depending on that value
> the user will know what other files/properties can be expected.
>
> In this PoC there are two types: "scalar" and "bitmap" and each is associated with
> a struct that contains the properties associated with that type.
>
> Considering this, these structures could, for example, be renamed as (very long):
> struct resctrl_membw -> struct resctrl_ctrl_scalar
> struct resctrl_cache -> struct resctrl_ctrl_bitmap
>
OK, the type field serves as the primary key for querying other properties.
Alongside the "type" entry, there also exists a "flag" property. I haven't
spotted this field within resctrl_membw, though it appears in resctrl_cache
via arch_has_sparse_bitmasks. Would it make sense to introduce a dedicated
enum for this flag, or alternatively reuse the existing bw_delay_linear
for MBA?
>> Would it make sense for these properties to be generic across all resctrl
>> controllers, CAT included? Should they consequently be relocated into
>> struct resctrl_ctrl if that approach is appropriate?
>
> If there are properties that are common across all controls of all types
> then they should be moved to struct resctrl_ctrl. Current examples are "type"
> and "scope" that user space can expect from every control. It is not obvious to
> me at this time what other properties we can designate as common in this way.
>
> For bitmaps there is already the properties that are available in the top-level
> directory. These can just be replicated as properties into the resource_schemata,
> but perhaps using new names as suggested by Dave in
> https://lore.kernel.org/lkml/aQOUAeVP9oc7RIn%2F@e133380.arm.com/
>
Got it. Will further look into the code.
thanks,
Chenyu
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-03 11:53 ` Chen, Yu C
@ 2026-06-04 16:37 ` Reinette Chatre
2026-06-05 15:43 ` Chen, Yu C
0 siblings, 1 reply; 66+ messages in thread
From: Reinette Chatre @ 2026-06-04 16:37 UTC (permalink / raw)
To: Chen, Yu C
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org, Tony Luck,
Dave Martin, James Morse, Ben Horgan, Babu Moger, Drew Fustini,
Fenghua Yu
Hi Chenyu,
On 6/3/26 4:53 AM, Chen, Yu C wrote:
> Hi Reinette,
>
> On 6/3/2026 11:45 AM, Reinette Chatre wrote:
>> Hi Chenyu,
>>
>
> [ ... ]
>
>>>
>>> Thanks for re-spinning this patch set.
>>> Looking at commit (fs/resctrl: Introduce additional schema properties),
>>> the newly added properties appear to be implemented for the
>>> MBA resctrl_membw controller.
>>
>> Please do note that the "struct resctrl_membw" naming is temporary. I only kept
>> current naming to help folks find where existing known structures land in this new
>> design but the "struct resctrl_membw" name is not appropriate moving forward.
>>
>> When user space interacts with the new controls the intention is that each control
>> has a "type" that the user can find in the new "type" file and depending on that value
>> the user will know what other files/properties can be expected.
>>
>> In this PoC there are two types: "scalar" and "bitmap" and each is associated with
>> a struct that contains the properties associated with that type.
>>
>> Considering this, these structures could, for example, be renamed as (very long):
>> struct resctrl_membw -> struct resctrl_ctrl_scalar
>> struct resctrl_cache -> struct resctrl_ctrl_bitmap
>>
>
> OK, the type field serves as the primary key for querying other properties.
> Alongside the "type" entry, there also exists a "flag" property. I haven't
> spotted this field within resctrl_membw, though it appears in resctrl_cache
> via arch_has_sparse_bitmasks. Would it make sense to introduce a dedicated
> enum for this flag, or alternatively reuse the existing bw_delay_linear for MBA?
I realize now my previous answer to you was incomplete. You are right that there
is a plan to let the resctrl "type" file contain the schema type as well as
optional flags. This plan is unchanged but at this time it is not obvious to me
what flags this implementation should start with.
In the original proposal [1] "linear" was provided as example of a flag for the MB
resource but as I worked through this PoC whether a control is linear or not seemed
to fit better as a resource property. This is how I ended up with commit
bdcd8ac6e946 ("mpam,x86,fs/resctrl: Make memory bandwidth delay a resource property")
For this specific property I expect that all controls would have the same value. This
worked out well since resctrl already has the per-resource top level "delay_linear".
We can surely move this back to be a control property and have it be the first
"flag" but at this time it seems to me that all MB controls would just have the same
flag value?
Perhaps the safest alternative would be to keep it a resource property and just duplicate
this as a flag value among all the controls? I think this is what you are suggesting to
reuse the bw_delay_linear for MBA. This would not result in dedicated flags associated
with controls but it may set the user interface up for most flexibility.
Reinette
[1] https://lore.kernel.org/lkml/aPtfMFfLV1l%2FRB0L@e133380.arm.com/
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-04 16:37 ` Reinette Chatre
@ 2026-06-05 15:43 ` Chen, Yu C
2026-06-05 16:20 ` Reinette Chatre
0 siblings, 1 reply; 66+ messages in thread
From: Chen, Yu C @ 2026-06-05 15:43 UTC (permalink / raw)
To: Reinette Chatre
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org, Tony Luck,
Dave Martin, James Morse, Ben Horgan, Babu Moger, Drew Fustini,
Fenghua Yu
Hi Reinette,
On 6/5/2026 12:37 AM, Reinette Chatre wrote:
[ ... ]
>>
>> OK, the type field serves as the primary key for querying other properties.
>> Alongside the "type" entry, there also exists a "flag" property. I haven't
>> spotted this field within resctrl_membw, though it appears in resctrl_cache
>> via arch_has_sparse_bitmasks. Would it make sense to introduce a dedicated
>> enum for this flag, or alternatively reuse the existing bw_delay_linear for MBA?
>
> I realize now my previous answer to you was incomplete. You are right that there
> is a plan to let the resctrl "type" file contain the schema type as well as
> optional flags. This plan is unchanged but at this time it is not obvious to me
> what flags this implementation should start with.
>
> In the original proposal [1] "linear" was provided as example of a flag for the MB
> resource but as I worked through this PoC whether a control is linear or not seemed
> to fit better as a resource property. This is how I ended up with commit
> bdcd8ac6e946 ("mpam,x86,fs/resctrl: Make memory bandwidth delay a resource property")
>
> For this specific property I expect that all controls would have the same value. This
> worked out well since resctrl already has the per-resource top level "delay_linear".
>
> We can surely move this back to be a control property and have it be the first
> "flag" but at this time it seems to me that all MB controls would just have the same
> flag value?
>
> Perhaps the safest alternative would be to keep it a resource property and just duplicate
> this as a flag value among all the controls? I think this is what you are suggesting to
> reuse the bw_delay_linear for MBA. This would not result in dedicated flags associated
> with controls but it may set the user interface up for most flexibility.
>
Yes I think we can simply reuse the value of bw_delay_linear when
displaying the
"flag" field for each controller under the info directory.
BTW, would the L3 controller have similar case that, all L3 controllers
of the same
resource share the same value of "flag"? If yes, should we also move
arch_has_sparse_bitmasks
to rdt_resource? If yes, I'm not sure why we need "flag" in the controller.
thanks,
Chenyu
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-05 15:43 ` Chen, Yu C
@ 2026-06-05 16:20 ` Reinette Chatre
0 siblings, 0 replies; 66+ messages in thread
From: Reinette Chatre @ 2026-06-05 16:20 UTC (permalink / raw)
To: Chen, Yu C
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org, Tony Luck,
Dave Martin, James Morse, Ben Horgan, Babu Moger, Drew Fustini,
Fenghua Yu
Hi Chenyu,
On 6/5/26 8:43 AM, Chen, Yu C wrote:
> Hi Reinette,
>
> On 6/5/2026 12:37 AM, Reinette Chatre wrote:
>
> [ ... ]
>
>>>
>>> OK, the type field serves as the primary key for querying other properties.
>>> Alongside the "type" entry, there also exists a "flag" property. I haven't
>>> spotted this field within resctrl_membw, though it appears in resctrl_cache
>>> via arch_has_sparse_bitmasks. Would it make sense to introduce a dedicated
>>> enum for this flag, or alternatively reuse the existing bw_delay_linear for MBA?
>>
>> I realize now my previous answer to you was incomplete. You are right that there
>> is a plan to let the resctrl "type" file contain the schema type as well as
>> optional flags. This plan is unchanged but at this time it is not obvious to me
>> what flags this implementation should start with.
>>
>> In the original proposal [1] "linear" was provided as example of a flag for the MB
>> resource but as I worked through this PoC whether a control is linear or not seemed
>> to fit better as a resource property. This is how I ended up with commit
>> bdcd8ac6e946 ("mpam,x86,fs/resctrl: Make memory bandwidth delay a resource property")
>>
>> For this specific property I expect that all controls would have the same value. This
>> worked out well since resctrl already has the per-resource top level "delay_linear".
>>
>> We can surely move this back to be a control property and have it be the first
>> "flag" but at this time it seems to me that all MB controls would just have the same
>> flag value?
>>
>> Perhaps the safest alternative would be to keep it a resource property and just duplicate
>> this as a flag value among all the controls? I think this is what you are suggesting to
>> reuse the bw_delay_linear for MBA. This would not result in dedicated flags associated
>> with controls but it may set the user interface up for most flexibility.
>>
>
> Yes I think we can simply reuse the value of bw_delay_linear when displaying the
> "flag" field for each controller under the info directory.
> BTW, would the L3 controller have similar case that, all L3 controllers of the same
> resource share the same value of "flag"? If yes, should we also move arch_has_sparse_bitmasks
> to rdt_resource? If yes, I'm not sure why we need "flag" in the controller.
Whether a control of type bitmap allows sparse values or not does seem to be a valid
property of the bitmap control to me. It is difficult to predict how this will land since
I am not familiar with a resource that has multiple bitmap controllers.
I do think having a controller type support different flags in the interface exposed to the
user does give resctrl most flexibility for the future. With the user interface having that
capability the internal implementation can always adapt.
Reinette
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-05-29 18:06 [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept Reinette Chatre
2026-06-02 20:23 ` Babu Moger
2026-06-02 23:32 ` Chen, Yu C
@ 2026-06-03 15:15 ` Ben Horgan
2026-06-03 19:34 ` Drew Fustini
2026-06-04 17:43 ` Reinette Chatre
2026-06-03 18:46 ` Luck, Tony
` (3 subsequent siblings)
6 siblings, 2 replies; 66+ messages in thread
From: Ben Horgan @ 2026-06-03 15:15 UTC (permalink / raw)
To: Reinette Chatre, Tony Luck, James Morse, Dave Martin, Babu Moger,
Drew Fustini, Fenghua Yu, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi Reinette,
On 5/29/26 19:06, Reinette Chatre wrote:
> Hi Everybody,
>
> It has been a while since we discussed the resctrl changes required to support
> hardware that has controls with fine granularity or hardware that has multiple
> controls per resource. For reference, the most recent email discussion can
> be found at [1] with a summary of discussions in last year's plumbers slides [2].
>
> I created a PoC that I believe supports what folks have agreed to so far. I
> hope this can help us to restart the discussion with the goal that resctrl gains
> support for upcoming hardware that require these features.
Thank you very much for doing this work. I believe this will be very useful for
MPAM and other architectures.
>
> Request regarding this PoC
> ==========================
>
> Please consider this PoC as a "direction check" on the schema description and multiple
> control discussions held thus far.
>
> Could folks working on enabling new hardware requiring this capability please consider
> if this is something you can build on and how it should be improved to support these
> upcoming capabilities?
>
> Opens
> =====
>
> While the PoC aims to support what folks agreed on some opens remain:
> - I attempted to make some MPAM supporting changes but these are all just compile
> tested. While MPAM should benefit from the new control properties I did not
> initialize them on MPAM and did not attempt refactor to separate out
> the architecture specific control properties (more on what this means later).
> I did attempt some MPAM refactoring that duplicates the MPAM domain to the
> control domain and monitoring domain lists in support of there being multiple
> controls each with its own list of control domains but it is definitely not good
> design.
I appreciate you including MPAM in this PoC. With this one line change I was
able to boot an MPAM system and mount resctrl which appears to behave correctly.
We can consider how best to do the code design later.
--- a/drivers/resctrl/mpam_resctrl.c
+++ b/drivers/resctrl/mpam_resctrl.c
@@ -1697,6 +1697,8 @@ int mpam_resctrl_setup(void)
/* Initialise the resctrl structures from the classes */
for_each_mpam_resctrl_control(res, rid) {
+ INIT_LIST_HEAD(&res->resctrl_res.controls); // list_empty needs
to work
+
if (!res->class)
continue; // dummy resource
I plumbed in support for the MB_MIN resource schema which also works under light
testing. The only fs resctrl code change I needed was:
--- a/include/linux/resctrl.h
+++ b/include/linux/resctrl.h
@@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct
resctrl_ctrl *ctrl)
case RESCTRL_CTRL_BITMAP:
return BIT_MASK(ctrl->cache.cbm_len) - 1;
case RESCTRL_CTRL_SCALAR:
+ if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
+ return ctrl->membw.min_bw;
+
return ctrl->membw.max_bw;
}
At least on MPAM systems, we use a default of 0 for minimum bandwidth controls
as the maximum bandwidth controls only take effect if their value is higher than
the minimum bandwidth value. I have specialised this on the ctrl->name which
breaks your ctrl->type based classification but that's fixable by just adding a
default field to membw.
> - No support for emulated controls (yet). The PoC is quite large already
> but I think it can be used as a base for emulated controls for which the software
> controller could be a potential first customer. In this PoC mounting with
> software controller will still display the original controller's properties.
> - One open that needs to be addressed as part of support for emulated controls is
> how best to display emulation relationship via resctrl hierarchy.
What does emulated controls mean here? Is there some previous discussion you
could point me at?
> - No support for "read-modify-write" usage of schemata file. This is where we
> discussed (without agreement) on possibly introducing the "#" prefix to schemata
> file entries. This PoC does not support this prefix and the current assumption/expectation
> is that when user space changes a configuration only the new control values are
> written to schemata file. I thus do not have a plan to support this so please
> share opinions in this regard if you have some.
There is now less motivation from the MPAM side for this than when this was
initially discussed. In pre-upstream versions of the MPAM patches a change in
the MB resource control value would change both the mpam h/w mbw_min and mbw_max
values but now (on non-broken h/w) we just change the mbw_max. (mbw_min kept at 0).
However, it would be useful not to be limited by percentages. In my quick
experimentation with your patches I used a percentage value for MB_MIN but it
would be best to move away from this. For new controls I think we can mandate
that user space has to discover the resolution from the info directly but how
can we retrofit this. For MPAM, MB and MB_MAX, would control the same things.
Could we just add MB_MAX with a h/w friendly scale and then reflect changes in
MB_MAX in MB and vica versa with MB taking precedent if both are set? Old
software can continue setting MB can move to using MB_MAX and take advantage of
the improved control. (I don't think we should expose the MPAM hardware value
directly as it has confusion over whether all 1s is 100% or not and we'd like to
have something generic and friendly to the user.)
> - Controls are independent for now. This means that, for example, if a resource
> supports a "MIN" and "MAX" control then this implementation would allow user to
> set the "maximum" control values to be less than the "minimum" control values.
I think this is ok as long as adding support for new controls in resctrl doesn't
change the existing behaviour. In MPAM we dodged this by introducing MB as only
affecting the h/w mbw_max and not mbw_min (as mentioned above).
> - PoC supports the "bitmap" control but does not (yet) expose properties of a bitmap
> control to the new info/<resource>/resource_schemata directory.
>
> Accessing PoC
> =============
>
> Please consider the PoC as a rough draft. It has only been compile tested for Arm
> and known to be incomplete in Arm support. To help with experimenting I only
> fully adapted the Intel MBA resource to demo two dummy additional MBA controls.
> All architectures should immediately benefit from the new schema descriptions
> and new info/MB/resource_schemata hierarchy.
>
> I considered the patches self too many for email. Instead, the PoC can be found at:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git branch resctrl/controls_rfc_v1
>
> The work is based on v7.1-rc2 that also includes the following series (two of which has
> since been queued) included:
>
> "selftests/resctrl: Fixes and improvements focused on Intel platforms"
> https://lore.kernel.org/lkml/cover.1775266384.git.reinette.chatre@intel.com/
>
> "x86,fs/resctrl: Improve resctrl quality and consistency"
> https://lore.kernel.org/lkml/cover.1777419024.git.reinette.chatre@intel.com/
>
> "x86,fs/resctrl: Pave the way for MPAM counter assignment"
> https://lore.kernel.org/lkml/20260506082855.3694761-1-ben.horgan@arm.com/
>
>
> Primary resctrl fs data structure changes
> =========================================
>
> Introduces a control represented by struct resctrl_ctrl that looks as below. To make
> the changes easier to follow I kept some of the original names to help communicate
> where familiar data structures land.
>
> What to notice about a control is that it has some common properties required
> from all controls (scope, type, etc.) and then depending on the type of control
> (RESCTRL_CTRL_BITMAP or RESCTRL_CTRL_SCALAR) there are type specific properties.
>
> /**
> * struct resctrl_ctrl - A resource control
> * @entry: List entry of rdt_resource::controls
> * @scope: Scope of the resource that this control allocates
> * @domains: RCU list of all control domains
> * @type: The control type that determines the properties of the control,
> * format string for displaying control values to user space, and
> * parser of control values provided by user space.
> * @name: Name of the control. Appended to final resource name
> * (rdt_resource_final::name) to create final schema entry.
> * Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
> * For example, with resource name "MB" and control name "MAX" the
> * schema entry will be "MB_MAX".
> * @cache: Cache allocation control properties.
> * @membw: Bandwidth control properties.
> */
> struct resctrl_ctrl {
> struct list_head entry;
> enum resctrl_scope scope;
> struct list_head domains;
> enum resctrl_ctrl_type type;
> enum resctrl_ctrl_name name;
> union {
> struct resctrl_cache cache;
> struct resctrl_membw membw;
> };
> };
>
> Two members summarize how this new structure fits into the rest of resctrl:
> a) resctrl_ctrl::entry
> Since a resource can support multiple controls there is a new list
> in struct rdt_resource named "controls" that contains the list of all
> controls supported by the resource.
> b) resctrl_ctrl::domains
> Instead of the list of control domains belonging to a resource they
> now belong to the control self. By doing so resctrl can support resource
> controls at different scope for the same resource. This is intended to
> support some upcoming MPAM and RISC-V usages.
Please can you expand a bit on part b).
In an MPAM system we consider 3 resctrl resources, RDT_RESOURCE_L3,
RDT_RESOURCE_L2 and RDT_RESOURCE_MBA which correspond to the L3 caches, L2
caches and memory bandwidth on egress from the L3 caches. The domain for each of
these corresponds to the instance of the resource. That is, for RDT_RESOURCE_L2
there is a resource for each L2 instance, similarly for L3, and for
RDT_RESOURCE_MBA there is a domain for each L3 cache. If we were to add suport
for controls on a new cache level, say the L4, then I'd expect to add a new
resource. For memory bandwidth, we'd like to be able to control b/w on the L2
egress (e.g. in a DSU). Wouldn't this too be a separate resource or would this
be a new set of controls on the same resource?
New controls on the same resource
MB_MIN2
MB_MAX2
MB_PROP2
...
or
MB2_MIN
MB2_MAX
MB2_PROP
AFAIK, the DSU h/w just supports proportional bandwidth controls at the moment
but we should consider what to do about the potential naming.
In the MPAM driver, we collect MSC into components (based on instances) and
those into classes (components of the same type). Currently, a resource is
mapped to a single class. (Two resources may map to the same class.)
I expect it is useful in the memory region and sub numa cases but I'd still
expect the common case to be that the domains are the same within a control. Or
am I missing something?
>
> Example architectural data structure changes
> ============================================
>
> An architecture can use the new control by following a similar pattern to
> resource and domain use by architectures. Consider the following for x86
> where a new architecture specific struct resctrl_hw_ctrl includes
> struct resctrl_ctrl and any architecture private data needed to support
> the control:
>
> /*
> * struct resctrl_hw_ctrl - Arch private properties of a resource control
> * @r_ctrl: Control properties exposed to resctrl file system
> * @msr_base: Base MSR address where control values should be programmed
> * @msr_update: Function pointer to update control values
> */
> struct resctrl_hw_ctrl {
> struct resctrl_ctrl r_ctrl;
> unsigned int msr_base;
> void (*msr_update)(struct msr_param *m);
> };
>
> Structure of patch series
> =========================
>
> As a PoC the series is not perfectly structured but to help navigate this work
> on a high level the changes can be categorized as follows:
>
> Patch 1 to 11:
> With a vision of what a "control" is, remove unused/unnecessary
> members, make clear what is a *resource* property vs a *control*
> property, do some renaming to help with the PoC.
A few of the changes are generic cleanup and could hopefully be dealt with
before decisions on the larger PoC are made. I see:
fs/resctrl: Remove unused resctrl_membw::mb_map
x86,fs/resctrl: Remove "arch_needs_linear"
Perhaps a few more.
>
> Patch 12:
> Introduce struct resctrl_ctrl and re-arrange existing struct rdt_resource
> members to form part of new rdt_resource::ctrl
>
> Patch 13 to 44:
> A lot of wrangling to introduce struct resctrl_ctrl to all code that needs
> to work with a control and/or domain without assuming that the control is
> the one and only control embedded in the resource it belongs to. Essentially,
> a lot of changes passing the control around in addition to the resource/domain.
You mention a few times in the commit message that you expect the cache
resources to only have one control. On MPAM we have CMAX (and there looks to be
a RISC-V equivalent) where the total number of bytes in the cache for a given
closid is limited. The allocation must still respect the CPBM bitmap though.
Looking at the code though I don't see much problem in adding this as an
additional control. The assumption that these patches is making is not that
there is only one control for cache resources but rather that cache portions are
managed by the default cache resource control. Am I missing something or does
that assessment make sense to you?
I have been looking at adding CMAX control to resctrl and will have a go at
basing what I have so far on top of this series.
>
> Patch 45:
> Switch the single struct resctrl_ctrl member of struct rdt_resource to be
> a list of struct resctrl_ctrl.
>
> Patch 47 to 49:
> Introduce new info/<resource>/resource_schemata hierarchy to first only
> consist of properties already known to resctrl fs.
>
> Patch 50 to 52:
> Introduce the new control properties per [1], initialize them for x86,
> and expose them via info/<resource>/resource_schemata
>
> Patch 53:
> Let the new struct resctrl_hw_ctrl contain architecture's control properties.
>
> Patch 54:
> Teach resctrl fs about "MIN" and "MAX" controls.
>
> Patch 55:
> Sample of "MIN" and "MAX" memory bandwidth controls for x86.
> > Example interactions
> ====================
>
> This series can be used on an x86 system where it will show two new dummy controls
> where it is possible to interact with the new controls.
> For example:
>
> # cat schemata
> MB_MAX:0=100;1=100
> MB_MIN:0=100;1=100
> MB:0=100;1=100
> L3:0=fff;1=fff
> # echo 'MB_MIN:0=50' > schemata
> # cat schemata
> MB_MAX:0=100;1=100
> MB_MIN:0=50;1=100
> MB:0=100;1=100
> L3:0=fff;1=fff
>
> Writing to the dummy control will call a dummy callback that just prints to the
> kernel log:
> "resctrl: Updata temporary MIN control on domain 0 with user value 50"
>
>
> Example output of info/MB/:
> /sys/fs/resctrl/info/MB/thread_throttle_mode:max
> /sys/fs/resctrl/info/MB/num_closids:15
> /sys/fs/resctrl/info/MB/delay_linear:1
> /sys/fs/resctrl/info/MB/min_bandwidth:10
> /sys/fs/resctrl/info/MB/resource_schemata/MB/resolution:100
> /sys/fs/resctrl/info/MB/resource_schemata/MB/tolerance:5
> /sys/fs/resctrl/info/MB/resource_schemata/MB/type:scalar
> /sys/fs/resctrl/info/MB/resource_schemata/MB/min:10
> /sys/fs/resctrl/info/MB/resource_schemata/MB/scale:1
> /sys/fs/resctrl/info/MB/resource_schemata/MB/scope:L3
> /sys/fs/resctrl/info/MB/resource_schemata/MB/unit:all
> /sys/fs/resctrl/info/MB/resource_schemata/MB/max:100
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/resolution:100
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/tolerance:5
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/type:scalar
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/min:10
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scale:1
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scope:L3
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/unit:all
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/max:100
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/resolution:100
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/tolerance:5
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/type:scalar
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/min:10
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scale:1
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scope:L3
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/unit:all
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/max:100
> /sys/fs/resctrl/info/MB/bandwidth_gran:10
> > Any feedback is appreciated.
Overall, this looks to be a big step in the right direction.
Thanks,
Ben
>
> Reinette
>
> [1] https://lore.kernel.org/lkml/aPtfMFfLV1l%2FRB0L@e133380.arm.com/
> [2] https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-03 15:15 ` Ben Horgan
@ 2026-06-03 19:34 ` Drew Fustini
2026-06-04 11:24 ` Ben Horgan
2026-06-04 21:05 ` Reinette Chatre
2026-06-04 17:43 ` Reinette Chatre
1 sibling, 2 replies; 66+ messages in thread
From: Drew Fustini @ 2026-06-03 19:34 UTC (permalink / raw)
To: Ben Horgan
Cc: Reinette Chatre, Tony Luck, James Morse, Dave Martin, Babu Moger,
Fenghua Yu, Chen Yu, Borislav Petkov, Thomas Gleixner,
Dave Hansen, Peter Newman, x86@kernel.org,
linux-kernel@vger.kernel.org
On Wed, Jun 03, 2026 at 04:15:51PM +0100, Ben Horgan wrote:
> Hi Reinette,
>
> On 5/29/26 19:06, Reinette Chatre wrote:
> > Hi Everybody,
> >
> > It has been a while since we discussed the resctrl changes required to support
> > hardware that has controls with fine granularity or hardware that has multiple
> > controls per resource. For reference, the most recent email discussion can
> > be found at [1] with a summary of discussions in last year's plumbers slides [2].
> >
> > I created a PoC that I believe supports what folks have agreed to so far. I
> > hope this can help us to restart the discussion with the goal that resctrl gains
> > support for upcoming hardware that require these features.
>
> Thank you very much for doing this work. I believe this will be very useful for
> MPAM and other architectures.
Yes, thanks to Reinette for working on the generic schema proof of
concept. This will be helpful for supporting the RISC-V CBQRI (capacity
and bandwidth QoS) spec.
> I plumbed in support for the MB_MIN resource schema which also works under light
> testing. The only fs resctrl code change I needed was:
>
> --- a/include/linux/resctrl.h
> +++ b/include/linux/resctrl.h
> @@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct
> resctrl_ctrl *ctrl)
> case RESCTRL_CTRL_BITMAP:
> return BIT_MASK(ctrl->cache.cbm_len) - 1;
> case RESCTRL_CTRL_SCALAR:
> + if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
> + return ctrl->membw.min_bw;
> +
> return ctrl->membw.max_bw;
> }
>
>
> At least on MPAM systems, we use a default of 0 for minimum bandwidth controls
> as the maximum bandwidth controls only take effect if their value is higher than
> the minimum bandwidth value. I have specialised this on the ctrl->name which
> breaks your ctrl->type based classification but that's fixable by just adding a
> default field to membw.
This should be useful for RISC-V.
RESCTRL_CTRL_NAME_MIN maps well to CBQRI Rbwb (reserved bandwidth
blocks). The sum of Rbwb across all control groups must be less than
MRBWB (maximum number of reserved bandwidth blocks). As a result, MB_MIN
needs to default to 1 so that the sum does not violate that rule. In my
RFC series, I added default_to_min to resctrl_membw [1] but this
solution looks cleaner.
> > - No support for "read-modify-write" usage of schemata file. This is where we
> > discussed (without agreement) on possibly introducing the "#" prefix to schemata
> > file entries. This PoC does not support this prefix and the current assumption/expectation
> > is that when user space changes a configuration only the new control values are
> > written to schemata file. I thus do not have a plan to support this so please
> > share opinions in this regard if you have some.
>
> There is now less motivation from the MPAM side for this than when this was
> initially discussed. In pre-upstream versions of the MPAM patches a change in
> the MB resource control value would change both the mpam h/w mbw_min and mbw_max
> values but now (on non-broken h/w) we just change the mbw_max. (mbw_min kept at 0).
>
> However, it would be useful not to be limited by percentages. In my quick
> experimentation with your patches I used a percentage value for MB_MIN but it
> would be best to move away from this. For new controls I think we can mandate
> that user space has to discover the resolution from the info directly but how
> can we retrofit this. For MPAM, MB and MB_MAX, would control the same things.
> Could we just add MB_MAX with a h/w friendly scale and then reflect changes in
> MB_MAX in MB and vica versa with MB taking precedent if both are set? Old
> software can continue setting MB can move to using MB_MAX and take advantage of
> the improved control. (I don't think we should expose the MPAM hardware value
> directly as it has confusion over whether all 1s is 100% or not and we'd like to
> have something generic and friendly to the user.)
The facility for non-percentage value is import for RISC-V as CBQRI does
not include percentage throttle. It has two controls for bandwidth:
- Rbwb: number of reserved bandwidth blocks [1, 2^13]
- Mweight: weighted share of the remaining bandwidth [0, 255]
- 0: disables work-conserving sharing
- 1..255: compete for the leftover pool
- It makes for it to default to max (255) so that there won't be
any unused bandwidth
I think Mweight could be aligned with MPAM's proportional stride.
Here is the patch I created to add Mweight support:
diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c
index d95ab8ad36e2..3537071e3ab0 100644
--- a/fs/resctrl/ctrlmondata.c
+++ b/fs/resctrl/ctrlmondata.c
@@ -304,6 +304,7 @@ static const char * const resctrl_ctrl_name[] = {
[RESCTRL_CTRL_NAME_DEF] = "",
[RESCTRL_CTRL_NAME_MIN] = "MIN",
[RESCTRL_CTRL_NAME_MAX] = "MAX",
+ [RESCTRL_CTRL_NAME_WGHT] = "WGHT",
};
const char *resctrl_ctrl_name_str(enum resctrl_ctrl_name name)
diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h
index 72fb7256270e..09efcef9ce66 100644
--- a/include/linux/resctrl.h
+++ b/include/linux/resctrl.h
@@ -348,12 +348,14 @@ struct resctrl_mon {
* has the same name as the resource.
* @RESCTRL_CTRL_NAME_MIN: "MIN"
* @RESCTRL_CTRL_NAME_MAX: "MAX"
+ * @RESCTRL_CTRL_NAME_WGHT: "WGHT"
*/
enum resctrl_ctrl_name {
RESCTRL_CTRL_NAME_DEF,
RESCTRL_CTRL_NAME_MIN,
RESCTRL_CTRL_NAME_MAX,
- RESCTRL_CTRL_NAME_LAST = RESCTRL_CTRL_NAME_MAX
+ RESCTRL_CTRL_NAME_WGHT,
+ RESCTRL_CTRL_NAME_LAST = RESCTRL_CTRL_NAME_WGHT
};
> > - Controls are independent for now. This means that, for example, if a resource
> > supports a "MIN" and "MAX" control then this implementation would allow user to
> > set the "maximum" control values to be less than the "minimum" control values.
>
> I think this is ok as long as adding support for new controls in resctrl doesn't
> change the existing behaviour. In MPAM we dodged this by introducing MB as only
> affecting the h/w mbw_max and not mbw_min (as mentioned above).
There is no equivalent to MB (percentage throttle) in RISC-V so I would
want it to be valid to have MB_MIN (minimum reservation) without MB.
I rebased my RISC-V CBQRI v6 series on top of this proof of concept and
was able to validate it works okay in Qemu:
MB_WGHT:72=255
MB_MIN:72=756
L2:64=fff;65=fff
L3:75=ffff
Thanks,
Drew
[1] https://lore.kernel.org/all/20260601-ssqosid-cbqri-rqsc-v7-0-v6-6-baf00f50028a@kernel.org/
^ permalink raw reply related [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-03 19:34 ` Drew Fustini
@ 2026-06-04 11:24 ` Ben Horgan
2026-06-04 17:38 ` Drew Fustini
2026-06-04 21:05 ` Reinette Chatre
1 sibling, 1 reply; 66+ messages in thread
From: Ben Horgan @ 2026-06-04 11:24 UTC (permalink / raw)
To: Drew Fustini
Cc: Reinette Chatre, Tony Luck, James Morse, Dave Martin, Babu Moger,
Fenghua Yu, Chen Yu, Borislav Petkov, Thomas Gleixner,
Dave Hansen, Peter Newman, x86@kernel.org,
linux-kernel@vger.kernel.org
Hi Drew,
On 6/3/26 20:34, Drew Fustini wrote:
> On Wed, Jun 03, 2026 at 04:15:51PM +0100, Ben Horgan wrote:
>> Hi Reinette,
>>
>> On 5/29/26 19:06, Reinette Chatre wrote:
>>> Hi Everybody,
>>>
>>> It has been a while since we discussed the resctrl changes required to support
>>> hardware that has controls with fine granularity or hardware that has multiple
>>> controls per resource. For reference, the most recent email discussion can
>>> be found at [1] with a summary of discussions in last year's plumbers slides [2].
>>>
>>> I created a PoC that I believe supports what folks have agreed to so far. I
>>> hope this can help us to restart the discussion with the goal that resctrl gains
>>> support for upcoming hardware that require these features.
>>
>> Thank you very much for doing this work. I believe this will be very useful for
>> MPAM and other architectures.
>
> Yes, thanks to Reinette for working on the generic schema proof of
> concept. This will be helpful for supporting the RISC-V CBQRI (capacity
> and bandwidth QoS) spec.
>
>> I plumbed in support for the MB_MIN resource schema which also works under light
>> testing. The only fs resctrl code change I needed was:
>>
>> --- a/include/linux/resctrl.h
>> +++ b/include/linux/resctrl.h
>> @@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct
>> resctrl_ctrl *ctrl)
>> case RESCTRL_CTRL_BITMAP:
>> return BIT_MASK(ctrl->cache.cbm_len) - 1;
>> case RESCTRL_CTRL_SCALAR:
>> + if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
>> + return ctrl->membw.min_bw;
>> +
>> return ctrl->membw.max_bw;
>> }
>>
>>
>> At least on MPAM systems, we use a default of 0 for minimum bandwidth controls
>> as the maximum bandwidth controls only take effect if their value is higher than
>> the minimum bandwidth value. I have specialised this on the ctrl->name which
>> breaks your ctrl->type based classification but that's fixable by just adding a
>> default field to membw.
>
> This should be useful for RISC-V.
>
> RESCTRL_CTRL_NAME_MIN maps well to CBQRI Rbwb (reserved bandwidth
> blocks). The sum of Rbwb across all control groups must be less than
> MRBWB (maximum number of reserved bandwidth blocks). As a result, MB_MIN
> needs to default to 1 so that the sum does not violate that rule. In my
> RFC series, I added default_to_min to resctrl_membw [1] but this
> solution looks cleaner.
>
>>> - No support for "read-modify-write" usage of schemata file. This is where we
>>> discussed (without agreement) on possibly introducing the "#" prefix to schemata
>>> file entries. This PoC does not support this prefix and the current assumption/expectation
>>> is that when user space changes a configuration only the new control values are
>>> written to schemata file. I thus do not have a plan to support this so please
>>> share opinions in this regard if you have some.
>>
>> There is now less motivation from the MPAM side for this than when this was
>> initially discussed. In pre-upstream versions of the MPAM patches a change in
>> the MB resource control value would change both the mpam h/w mbw_min and mbw_max
>> values but now (on non-broken h/w) we just change the mbw_max. (mbw_min kept at 0).
>>
>> However, it would be useful not to be limited by percentages. In my quick
>> experimentation with your patches I used a percentage value for MB_MIN but it
>> would be best to move away from this. For new controls I think we can mandate
>> that user space has to discover the resolution from the info directly but how
>> can we retrofit this. For MPAM, MB and MB_MAX, would control the same things.
>> Could we just add MB_MAX with a h/w friendly scale and then reflect changes in
>> MB_MAX in MB and vica versa with MB taking precedent if both are set? Old
>> software can continue setting MB can move to using MB_MAX and take advantage of
>> the improved control. (I don't think we should expose the MPAM hardware value
>> directly as it has confusion over whether all 1s is 100% or not and we'd like to
>> have something generic and friendly to the user.)
>
> The facility for non-percentage value is import for RISC-V as CBQRI does
> not include percentage throttle. It has two controls for bandwidth:
>
> - Rbwb: number of reserved bandwidth blocks [1, 2^13]
> - Mweight: weighted share of the remaining bandwidth [0, 255]
> - 0: disables work-conserving sharing
> - 1..255: compete for the leftover pool
> - It makes for it to default to max (255) so that there won't be
> any unused bandwidth
>
> I think Mweight could be aligned with MPAM's proportional stride.
Yes, I hope so. There a few differences which would have to be considered.
MPAM doesn't have a concept of only applying the weights once reserved min
bandwidth is consumed. The interaction with min bandwidth is currently
unspecified. I don't think there are any designs where proportional bandwidth
and min b/w are on the same component and so it's only a theoretical/future problem.
For MPAM proportional stride, the higher the stride the lower the weight. We'll
have to make sure that whatever user configuration scale we provide works well
for both. If two PARTIDs have stride 2x and a third x then the 2 PARTIDS with
stride 2x together get the same bandidth as the third. Whereas, to get the same
in RISC-V the two partid would have weight y and the third 2y.
It's not specified for MPAM exactly what happens when you disable proportional
stride for a given PARTID.
The MPAM proportional control is work-conserving (the table in B.b RWKZBJ has
been confirmed as a spec mistake) and only corresponds to the current contenders
for bandwidth. From my reading of the CBQRI spec this is the same for RISC-V.
Thanks,
Ben
>
> Here is the patch I created to add Mweight support:
>
> diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c
> index d95ab8ad36e2..3537071e3ab0 100644
> --- a/fs/resctrl/ctrlmondata.c
> +++ b/fs/resctrl/ctrlmondata.c
> @@ -304,6 +304,7 @@ static const char * const resctrl_ctrl_name[] = {
> [RESCTRL_CTRL_NAME_DEF] = "",
> [RESCTRL_CTRL_NAME_MIN] = "MIN",
> [RESCTRL_CTRL_NAME_MAX] = "MAX",
> + [RESCTRL_CTRL_NAME_WGHT] = "WGHT",
> };
>
> const char *resctrl_ctrl_name_str(enum resctrl_ctrl_name name)
> diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h
> index 72fb7256270e..09efcef9ce66 100644
> --- a/include/linux/resctrl.h
> +++ b/include/linux/resctrl.h
> @@ -348,12 +348,14 @@ struct resctrl_mon {
> * has the same name as the resource.
> * @RESCTRL_CTRL_NAME_MIN: "MIN"
> * @RESCTRL_CTRL_NAME_MAX: "MAX"
> + * @RESCTRL_CTRL_NAME_WGHT: "WGHT"
> */
> enum resctrl_ctrl_name {
> RESCTRL_CTRL_NAME_DEF,
> RESCTRL_CTRL_NAME_MIN,
> RESCTRL_CTRL_NAME_MAX,
> - RESCTRL_CTRL_NAME_LAST = RESCTRL_CTRL_NAME_MAX
> + RESCTRL_CTRL_NAME_WGHT,
> + RESCTRL_CTRL_NAME_LAST = RESCTRL_CTRL_NAME_WGHT
> };
>
>>> - Controls are independent for now. This means that, for example, if a resource
>>> supports a "MIN" and "MAX" control then this implementation would allow user to
>>> set the "maximum" control values to be less than the "minimum" control values.
>>
>> I think this is ok as long as adding support for new controls in resctrl doesn't
>> change the existing behaviour. In MPAM we dodged this by introducing MB as only
>> affecting the h/w mbw_max and not mbw_min (as mentioned above).
>
> There is no equivalent to MB (percentage throttle) in RISC-V so I would
> want it to be valid to have MB_MIN (minimum reservation) without MB.
>
> I rebased my RISC-V CBQRI v6 series on top of this proof of concept and
> was able to validate it works okay in Qemu:
>
> MB_WGHT:72=255
> MB_MIN:72=756
> L2:64=fff;65=fff
> L3:75=ffff
>
> Thanks,
> Drew
>
> [1] https://lore.kernel.org/all/20260601-ssqosid-cbqri-rqsc-v7-0-v6-6-baf00f50028a@kernel.org/
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-04 11:24 ` Ben Horgan
@ 2026-06-04 17:38 ` Drew Fustini
2026-06-12 1:30 ` Shaopeng Tan (Fujitsu)
0 siblings, 1 reply; 66+ messages in thread
From: Drew Fustini @ 2026-06-04 17:38 UTC (permalink / raw)
To: Ben Horgan
Cc: Reinette Chatre, Tony Luck, James Morse, Dave Martin, Babu Moger,
Fenghua Yu, Chen Yu, Borislav Petkov, Thomas Gleixner,
Dave Hansen, Peter Newman, x86@kernel.org,
linux-kernel@vger.kernel.org
On Thu, Jun 04, 2026 at 12:24:57PM +0100, Ben Horgan wrote:
> > The facility for non-percentage value is import for RISC-V as CBQRI does
> > not include percentage throttle. It has two controls for bandwidth:
> >
> > - Rbwb: number of reserved bandwidth blocks [1, 2^13]
> > - Mweight: weighted share of the remaining bandwidth [0, 255]
> > - 0: disables work-conserving sharing
> > - 1..255: compete for the leftover pool
> > - It makes for it to default to max (255) so that there won't be
> > any unused bandwidth
> >
> > I think Mweight could be aligned with MPAM's proportional stride.
>
> Yes, I hope so. There a few differences which would have to be considered.
>
> MPAM doesn't have a concept of only applying the weights once reserved min
> bandwidth is consumed. The interaction with min bandwidth is currently
> unspecified. I don't think there are any designs where proportional bandwidth
> and min b/w are on the same component and so it's only a theoretical/future problem.
>
> For MPAM proportional stride, the higher the stride the lower the weight. We'll
> have to make sure that whatever user configuration scale we provide works well
> for both. If two PARTIDs have stride 2x and a third x then the 2 PARTIDS with
> stride 2x together get the same bandidth as the third. Whereas, to get the same
> in RISC-V the two partid would have weight y and the third 2y.
Thanks for explaining. I had been thinking it would be best to try to
share a control for stride and weight, but now I am wondering if those
should just be considered separate controls.
It seems like they are similar enough to be able to convert to a common
scale but maybe that would be too confusing for the user. I guess it is
a matter of how much userspace needs or wants to be aware of the
difference between systems with MPAM and CBQRI.
Alternatively, it just occurred to me that Mweight could be mapped to MB.
I think Mweight could be thought of in the context of throttling: all
groups start with the max of 255 which can be represented as 100%.
> It's not specified for MPAM exactly what happens when you disable proportional
> stride for a given PARTID.
>
> The MPAM proportional control is work-conserving (the table in B.b RWKZBJ has
> been confirmed as a spec mistake) and only corresponds to the current contenders
> for bandwidth. From my reading of the CBQRI spec this is the same for RISC-V.
Yes, I think they are the same. The only exception is that Mweight of 0
means no shared badwidth and that the group is restricted to just its
reserved bandwidth blocks (e.g. MB_MIN).
Thanks,
Drew
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-04 17:38 ` Drew Fustini
@ 2026-06-12 1:30 ` Shaopeng Tan (Fujitsu)
2026-06-17 15:29 ` Reinette Chatre
0 siblings, 1 reply; 66+ messages in thread
From: Shaopeng Tan (Fujitsu) @ 2026-06-12 1:30 UTC (permalink / raw)
To: Reinette Chatre, Ben Horgan, Drew Fustini
Cc: Tony Luck, James Morse, Dave Martin, Babu Moger, Fenghua Yu,
Chen Yu, Borislav Petkov, Thomas Gleixner, Dave Hansen,
Peter Newman, x86@kernel.org, linux-kernel@vger.kernel.org
Hello Reinette, Ben, Drew,
> On Thu, Jun 04, 2026 at 02:47:39PM -0700, Reinette Chatre wrote:
> > > The ability to change scope is much needed for RISC-V. There are
> > > compromises in my RFC [1] as a result of trying to map everything to
> > > either L2 or L3 scope.
> > >
> > > I would also like to see a non-cpu cache scope for monitoring too, but
> > > would that be better discussed outside the context of this proof of
> > > concept?
> >
> > I also think it would be good for it to be clear that monitoring is based on
> > scope, not a resource. With the MB controls supporting different scope I do think
> > that this would be a good next step. A previous musing from me on this topic can
> > be found (at the end of ) https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@intel.com/
> >
> > I have not yet considered how this can be built on top of this PoC though.
>
> Thanks for explaining. I like how how you show an example of
> mon_data/mon_NODE_00/mbm_total_bytes in that thread. I believe that sort
> of scheme would work well for RISc-V as a bandwidth controller
> implementing the CBQRI spec can be located anywhere within the system.
I have a few questions regarding the scope parameter and the name "mbm_total_bytes".
First, concerning the scope parameter, does the "NODE" specified in "scope" refer to a NUMA node?
If so, wouldn't using "NUMA" directly be more explicit and user-friendly?
Could you please explain why "NODE" is used instead of "NUMA"?
Second, regarding the naming of "mbm_total_bytes",
the meaning of this name seems to differ from what is typically found under mon_data/mon_L3_<id>/mbm_total_bytes.
To avoid confusion and better reflect its different nature,
how about considering an alternative name such as mbm_global_bytes or another more appropriate identifier.
Best regards,
Shaopeng TAN
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-12 1:30 ` Shaopeng Tan (Fujitsu)
@ 2026-06-17 15:29 ` Reinette Chatre
2026-06-19 1:42 ` Shaopeng Tan (Fujitsu)
0 siblings, 1 reply; 66+ messages in thread
From: Reinette Chatre @ 2026-06-17 15:29 UTC (permalink / raw)
To: Shaopeng Tan (Fujitsu), Ben Horgan, Drew Fustini
Cc: Tony Luck, James Morse, Dave Martin, Babu Moger, Fenghua Yu,
Chen Yu, Borislav Petkov, Thomas Gleixner, Dave Hansen,
Peter Newman, x86@kernel.org, linux-kernel@vger.kernel.org
Hi Shaopeng,
On 6/11/26 6:30 PM, Shaopeng Tan (Fujitsu) wrote:
> Hello Reinette, Ben, Drew,
>
>> On Thu, Jun 04, 2026 at 02:47:39PM -0700, Reinette Chatre wrote:
>>>> The ability to change scope is much needed for RISC-V. There are
>>>> compromises in my RFC [1] as a result of trying to map everything to
>>>> either L2 or L3 scope.
>>>>
>>>> I would also like to see a non-cpu cache scope for monitoring too, but
>>>> would that be better discussed outside the context of this proof of
>>>> concept?
>>>
>>> I also think it would be good for it to be clear that monitoring is based on
>>> scope, not a resource. With the MB controls supporting different scope I do think
>>> that this would be a good next step. A previous musing from me on this topic can
>>> be found (at the end of ) https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@intel.com/
>>>
>>> I have not yet considered how this can be built on top of this PoC though.
>>
>> Thanks for explaining. I like how how you show an example of
>> mon_data/mon_NODE_00/mbm_total_bytes in that thread. I believe that sort
>> of scheme would work well for RISc-V as a bandwidth controller
>> implementing the CBQRI spec can be located anywhere within the system.
>
> I have a few questions regarding the scope parameter and the name "mbm_total_bytes".
>
> First, concerning the scope parameter, does the "NODE" specified in "scope" refer to a NUMA node?
Yes.
> If so, wouldn't using "NUMA" directly be more explicit and user-friendly?
> Could you please explain why "NODE" is used instead of "NUMA"?
By using "node" the idea is to create an interface that is familiar to user space. Since
/sys/devices/system/node already exists to expose the NUMA layout to user space I expect that
using similar terminology would make the resctrl interface easier to use since it would be clear
that a "node" in /sys/devices/system/node would have a matching "node" in resctrl.
>
> Second, regarding the naming of "mbm_total_bytes",
> the meaning of this name seems to differ from what is typically found under mon_data/mon_L3_<id>/mbm_total_bytes.
Could you please elaborate how the meaning is different? The only information I have about
the systems needing this is in Nvidia's portion of
https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf
> To avoid confusion and better reflect its different nature,
> how about considering an alternative name such as mbm_global_bytes or another more appropriate identifier.
Reinette
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-17 15:29 ` Reinette Chatre
@ 2026-06-19 1:42 ` Shaopeng Tan (Fujitsu)
2026-06-22 16:10 ` Reinette Chatre
0 siblings, 1 reply; 66+ messages in thread
From: Shaopeng Tan (Fujitsu) @ 2026-06-19 1:42 UTC (permalink / raw)
To: Reinette Chatre, Ben Horgan, Drew Fustini
Cc: Tony Luck, James Morse, Dave Martin, Babu Moger, Fenghua Yu,
Chen Yu, Borislav Petkov, Thomas Gleixner, Dave Hansen,
Peter Newman, x86@kernel.org, linux-kernel@vger.kernel.org
Hello Reinette,
>On 6/11/26 6:30 PM, Shaopeng Tan (Fujitsu) wrote:
>> Hello Reinette, Ben, Drew,
>>
>>> On Thu, Jun 04, 2026 at 02:47:39PM -0700, Reinette Chatre wrote:
>>>>> The ability to change scope is much needed for RISC-V. There are
>>>>> compromises in my RFC [1] as a result of trying to map everything to
>>>>> either L2 or L3 scope.
>>>>>
>>>>> I would also like to see a non-cpu cache scope for monitoring too, but
>>>>> would that be better discussed outside the context of this proof of
>>>>> concept?
>>>>
>>>> I also think it would be good for it to be clear that monitoring is based on
>>>> scope, not a resource. With the MB controls supporting different scope I do think
>>>> that this would be a good next step. A previous musing from me on this topic can
>>>> be found (at the end of ) https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@intel.com/
>>>>
>>>> I have not yet considered how this can be built on top of this PoC though.
>>>
>>> Thanks for explaining. I like how how you show an example of
>>> mon_data/mon_NODE_00/mbm_total_bytes in that thread. I believe that sort
>>> of scheme would work well for RISc-V as a bandwidth controller
>>> implementing the CBQRI spec can be located anywhere within the system.
>>
>> I have a few questions regarding the scope parameter and the name "mbm_total_bytes".
>>
>> First, concerning the scope parameter, does the "NODE" specified in "scope" refer to a NUMA node?
>
>Yes.
>
>> If so, wouldn't using "NUMA" directly be more explicit and user-friendly?
>> Could you please explain why "NODE" is used instead of "NUMA"?
>
>By using "node" the idea is to create an interface that is familiar to user space. Since
>/sys/devices/system/node already exists to expose the NUMA layout to user space I expect that
>using similar terminology would make the resctrl interface easier to use since it would be clear
>that a "node" in /sys/devices/system/node would have a matching "node" in resctrl.
>
>>
>> Second, regarding the naming of "mbm_total_bytes",
>> the meaning of this name seems to differ from what is typically found under mon_data/mon_L3_<id>/mbm_total_bytes.
>
>Could you please elaborate how the meaning is different? The only information I have about
>the systems needing this is in Nvidia's portion of
>https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf
>
>> To avoid confusion and better reflect its different nature,
>> how about considering an alternative name such as mbm_global_bytes or another more appropriate identifier.
Current MBM events in resctrl are per L3 to monitor MB between L3 and memory,
and mbm_total_bytes monitors the L3 total external bandwidth to the next level of the memory hierarchy.
Is my understanding of this correct?
Based on NVIDIA's information, they even assume that CPU-less nodes will not feature an L3 cache.
I believe NVIDIA's objective is to collect traffic data for each NODE/NUMA,
without needing to distinguish which specific L3 cache the traffic originated from.
This aligns with the fact that, in ARM architectures (e.g., on NVIDIA's platforms or Fujitsu's MONAKA),
the MPAM MSC (Memory System Component) can also be located on the memory controller.
In such cases,it becomes impossible to distinguish which L3 cache the traffic came from.
Therefore, the proposed 'mbm_global_bytes' refers to account for traffic from all L3 cache within the global system.
Best regards
Shaopeng TAN
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-19 1:42 ` Shaopeng Tan (Fujitsu)
@ 2026-06-22 16:10 ` Reinette Chatre
2026-06-23 5:04 ` Shaopeng Tan (Fujitsu)
0 siblings, 1 reply; 66+ messages in thread
From: Reinette Chatre @ 2026-06-22 16:10 UTC (permalink / raw)
To: Shaopeng Tan (Fujitsu), Ben Horgan, Drew Fustini
Cc: Tony Luck, James Morse, Dave Martin, Babu Moger, Fenghua Yu,
Chen Yu, Borislav Petkov, Thomas Gleixner, Dave Hansen,
Peter Newman, x86@kernel.org, linux-kernel@vger.kernel.org
Hi Shaopeng Tan,
On 6/18/26 6:42 PM, Shaopeng Tan (Fujitsu) wrote:
>
>> On 6/11/26 6:30 PM, Shaopeng Tan (Fujitsu) wrote:
>>> Second, regarding the naming of "mbm_total_bytes",
>>> the meaning of this name seems to differ from what is typically found under mon_data/mon_L3_<id>/mbm_total_bytes.
>>
>> Could you please elaborate how the meaning is different? The only information I have about
>> the systems needing this is in Nvidia's portion of
>> https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf
>>
>>> To avoid confusion and better reflect its different nature,
>>> how about considering an alternative name such as mbm_global_bytes or another more appropriate identifier.
>
> Current MBM events in resctrl are per L3 to monitor MB between L3 and memory,
> and mbm_total_bytes monitors the L3 total external bandwidth to the next level of the memory hierarchy.
> Is my understanding of this correct?
It is my understanding also.
>
> Based on NVIDIA's information, they even assume that CPU-less nodes will not feature an L3 cache.
> I believe NVIDIA's objective is to collect traffic data for each NODE/NUMA,
> without needing to distinguish which specific L3 cache the traffic originated from.
That is what associating monitoring with scope (instead of L3 resource), and supporting node scope, aims
to address.
>
> This aligns with the fact that, in ARM architectures (e.g., on NVIDIA's platforms or Fujitsu's MONAKA),
> the MPAM MSC (Memory System Component) can also be located on the memory controller.
> In such cases,it becomes impossible to distinguish which L3 cache the traffic came from.
This is not clear to me. Is the goal to expose the bandwidth at each memory controller?
>
> Therefore, the proposed 'mbm_global_bytes' refers to account for traffic from all L3 cache within the global system.
>
As I see it the directory name would make the monitoring scope clear, whether it is
at L3 scope or NUMA node scope. Within the directory there would be the individual files
that should be interpreted based on the scope of the directory they are in. I thus
see a "mbm_total_bytes" in a "L3 scope" directory to have different meaning
from a "mbm_total_bytes" in a "NUMA node scope" directory.
Reinette
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-22 16:10 ` Reinette Chatre
@ 2026-06-23 5:04 ` Shaopeng Tan (Fujitsu)
0 siblings, 0 replies; 66+ messages in thread
From: Shaopeng Tan (Fujitsu) @ 2026-06-23 5:04 UTC (permalink / raw)
To: Reinette Chatre, Ben Horgan, Drew Fustini
Cc: Tony Luck, James Morse, Dave Martin, Babu Moger, Fenghua Yu,
Chen Yu, Borislav Petkov, Thomas Gleixner, Dave Hansen,
Peter Newman, x86@kernel.org, linux-kernel@vger.kernel.org
Hello Reinette,
>>>> Second, regarding the naming of "mbm_total_bytes",
>>>> the meaning of this name seems to differ from what is typically found under mon_data/mon_L3_<id>/mbm_total_bytes.
>>>
>>> Could you please elaborate how the meaning is different? The only information I have about
>>> the systems needing this is in Nvidia's portion of
>>> https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf
>>>
>>>> To avoid confusion and better reflect its different nature,
>>>> how about considering an alternative name such as mbm_global_bytes or another more appropriate identifier.
>>
>> Current MBM events in resctrl are per L3 to monitor MB between L3 and memory,
>> and mbm_total_bytes monitors the L3 total external bandwidth to the next level of the memory hierarchy.
>> Is my understanding of this correct?
>
>It is my understanding also.
>
>>
>> Based on NVIDIA's information, they even assume that CPU-less nodes will not feature an L3 cache.
>> I believe NVIDIA's objective is to collect traffic data for each NODE/NUMA,
>> without needing to distinguish which specific L3 cache the traffic originated from.
>
>That is what associating monitoring with scope (instead of L3 resource), and supporting node scope, aims
>to address.
>
>>
>> This aligns with the fact that, in ARM architectures (e.g., on NVIDIA's platforms or Fujitsu's MONAKA),
>> the MPAM MSC (Memory System Component) can also be located on the memory controller.
>> In such cases,it becomes impossible to distinguish which L3 cache the traffic came from.
>
>This is not clear to me. Is the goal to expose the bandwidth at each memory controller?
The goal is to expose the bandwidth of each NUMA/NODE.
In this context, my intention was to explain why it is impossible to distinguish which L3 cache the traffic came from.
>>
>> Therefore, the proposed 'mbm_global_bytes' refers to account for traffic from all L3 cache within the global system.
>>
>
>As I see it the directory name would make the monitoring scope clear, whether it is
>at L3 scope or NUMA node scope. Within the directory there would be the individual files
>that should be interpreted based on the scope of the directory they are in. I thus
>see a "mbm_total_bytes" in a "L3 scope" directory to have different meaning
>from a "mbm_total_bytes" in a "NUMA node scope" directory.
I understand.
Your suggestion is to clarify the monitoring scope (L3 or NUMA node) by the directory name,
and that individual files within that directory would be interpreted based on its scope.
If the scope is "L3", then "mbm_total_bytes" would mean collecting traffic data for each L3,
regardless of which NUMA node the traffic is destined for.
If the scope is "NUMA node", then "mbm_total_bytes" would mean collecting traffic data for each node/NUMA,
regardless of which L3 cache the traffic originates from.
I believe this approach will be clear as long as it's clearly documented. Thanks.
Also, regarding the scope for NUMA nodes,
since it's not possible to distinguish which L3 cache the traffic came from,
it also appears that support for `event_configs` is not necessary.
(There is no need to configure "non-local NUMA" or "local NUMA".)
Best regards,
Shaopeng TAN
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-03 19:34 ` Drew Fustini
2026-06-04 11:24 ` Ben Horgan
@ 2026-06-04 21:05 ` Reinette Chatre
2026-06-05 19:35 ` Drew Fustini
1 sibling, 1 reply; 66+ messages in thread
From: Reinette Chatre @ 2026-06-04 21:05 UTC (permalink / raw)
To: Drew Fustini, Ben Horgan
Cc: Tony Luck, James Morse, Dave Martin, Babu Moger, Fenghua Yu,
Chen Yu, Borislav Petkov, Thomas Gleixner, Dave Hansen,
Peter Newman, x86@kernel.org, linux-kernel@vger.kernel.org
Hi Drew,
On 6/3/26 12:34 PM, Drew Fustini wrote:
> On Wed, Jun 03, 2026 at 04:15:51PM +0100, Ben Horgan wrote:
>> Hi Reinette,
>>
>> On 5/29/26 19:06, Reinette Chatre wrote:
>>> Hi Everybody,
>>>
>>> It has been a while since we discussed the resctrl changes required to support
>>> hardware that has controls with fine granularity or hardware that has multiple
>>> controls per resource. For reference, the most recent email discussion can
>>> be found at [1] with a summary of discussions in last year's plumbers slides [2].
>>>
>>> I created a PoC that I believe supports what folks have agreed to so far. I
>>> hope this can help us to restart the discussion with the goal that resctrl gains
>>> support for upcoming hardware that require these features.
>>
>> Thank you very much for doing this work. I believe this will be very useful for
>> MPAM and other architectures.
>
> Yes, thanks to Reinette for working on the generic schema proof of
> concept. This will be helpful for supporting the RISC-V CBQRI (capacity
> and bandwidth QoS) spec.
Thank you very much for considering this work.
>
>> I plumbed in support for the MB_MIN resource schema which also works under light
>> testing. The only fs resctrl code change I needed was:
>>
>> --- a/include/linux/resctrl.h
>> +++ b/include/linux/resctrl.h
>> @@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct
>> resctrl_ctrl *ctrl)
>> case RESCTRL_CTRL_BITMAP:
>> return BIT_MASK(ctrl->cache.cbm_len) - 1;
>> case RESCTRL_CTRL_SCALAR:
>> + if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
>> + return ctrl->membw.min_bw;
>> +
>> return ctrl->membw.max_bw;
>> }
>>
>>
>> At least on MPAM systems, we use a default of 0 for minimum bandwidth controls
>> as the maximum bandwidth controls only take effect if their value is higher than
>> the minimum bandwidth value. I have specialised this on the ctrl->name which
>> breaks your ctrl->type based classification but that's fixable by just adding a
>> default field to membw.
>
> This should be useful for RISC-V.
>
> RESCTRL_CTRL_NAME_MIN maps well to CBQRI Rbwb (reserved bandwidth
> blocks). The sum of Rbwb across all control groups must be less than
> MRBWB (maximum number of reserved bandwidth blocks). As a result, MB_MIN
> needs to default to 1 so that the sum does not violate that rule. In my
> RFC series, I added default_to_min to resctrl_membw [1] but this
> solution looks cleaner.
As I mentioned in response to Ben [2] there seems to be a mismatch between
architecture requirements here. resctrl uses the value returned by
resctrl_get_default_ctrlval() as the control value that means "no throttling".
For Intel this means min == max but this does not seem to be the case for MPAM
and CBQRI. I am not familiar enough with either to have an alternative proposal here
so I need to become familiar now. There is a bit of backlog on other resctl
work right now so this will take me some time to sort out.
>
>>> - No support for "read-modify-write" usage of schemata file. This is where we
>>> discussed (without agreement) on possibly introducing the "#" prefix to schemata
>>> file entries. This PoC does not support this prefix and the current assumption/expectation
>>> is that when user space changes a configuration only the new control values are
>>> written to schemata file. I thus do not have a plan to support this so please
>>> share opinions in this regard if you have some.
>>
>> There is now less motivation from the MPAM side for this than when this was
>> initially discussed. In pre-upstream versions of the MPAM patches a change in
>> the MB resource control value would change both the mpam h/w mbw_min and mbw_max
>> values but now (on non-broken h/w) we just change the mbw_max. (mbw_min kept at 0).
>>
>> However, it would be useful not to be limited by percentages. In my quick
>> experimentation with your patches I used a percentage value for MB_MIN but it
>> would be best to move away from this. For new controls I think we can mandate
>> that user space has to discover the resolution from the info directly but how
>> can we retrofit this. For MPAM, MB and MB_MAX, would control the same things.
>> Could we just add MB_MAX with a h/w friendly scale and then reflect changes in
>> MB_MAX in MB and vica versa with MB taking precedent if both are set? Old
>> software can continue setting MB can move to using MB_MAX and take advantage of
>> the improved control. (I don't think we should expose the MPAM hardware value
>> directly as it has confusion over whether all 1s is 100% or not and we'd like to
>> have something generic and friendly to the user.)
>
> The facility for non-percentage value is import for RISC-V as CBQRI does
> not include percentage throttle. It has two controls for bandwidth:
>
> - Rbwb: number of reserved bandwidth blocks [1, 2^13]
> - Mweight: weighted share of the remaining bandwidth [0, 255]
> - 0: disables work-conserving sharing
> - 1..255: compete for the leftover pool
> - It makes for it to default to max (255) so that there won't be
> any unused bandwidth
>
> I think Mweight could be aligned with MPAM's proportional stride.
>
> Here is the patch I created to add Mweight support:
>
> diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c
> index d95ab8ad36e2..3537071e3ab0 100644
> --- a/fs/resctrl/ctrlmondata.c
> +++ b/fs/resctrl/ctrlmondata.c
> @@ -304,6 +304,7 @@ static const char * const resctrl_ctrl_name[] = {
> [RESCTRL_CTRL_NAME_DEF] = "",
> [RESCTRL_CTRL_NAME_MIN] = "MIN",
> [RESCTRL_CTRL_NAME_MAX] = "MAX",
> + [RESCTRL_CTRL_NAME_WGHT] = "WGHT",
> };
>
> const char *resctrl_ctrl_name_str(enum resctrl_ctrl_name name)
> diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h
> index 72fb7256270e..09efcef9ce66 100644
> --- a/include/linux/resctrl.h
> +++ b/include/linux/resctrl.h
> @@ -348,12 +348,14 @@ struct resctrl_mon {
> * has the same name as the resource.
> * @RESCTRL_CTRL_NAME_MIN: "MIN"
> * @RESCTRL_CTRL_NAME_MAX: "MAX"
> + * @RESCTRL_CTRL_NAME_WGHT: "WGHT"
> */
> enum resctrl_ctrl_name {
> RESCTRL_CTRL_NAME_DEF,
> RESCTRL_CTRL_NAME_MIN,
> RESCTRL_CTRL_NAME_MAX,
> - RESCTRL_CTRL_NAME_LAST = RESCTRL_CTRL_NAME_MAX
> + RESCTRL_CTRL_NAME_WGHT,
> + RESCTRL_CTRL_NAME_LAST = RESCTRL_CTRL_NAME_WGHT
> };
>
>>> - Controls are independent for now. This means that, for example, if a resource
>>> supports a "MIN" and "MAX" control then this implementation would allow user to
>>> set the "maximum" control values to be less than the "minimum" control values.
>>
>> I think this is ok as long as adding support for new controls in resctrl doesn't
>> change the existing behaviour. In MPAM we dodged this by introducing MB as only
>> affecting the h/w mbw_max and not mbw_min (as mentioned above).
>
> There is no equivalent to MB (percentage throttle) in RISC-V so I would
> want it to be valid to have MB_MIN (minimum reservation) without MB.
>
> I rebased my RISC-V CBQRI v6 series on top of this proof of concept and
> was able to validate it works okay in Qemu:
>
> MB_WGHT:72=255
> MB_MIN:72=756
> L2:64=fff;65=fff
> L3:75=ffff
Ideally any new support should not break existing user space and the existing
user interface expects a MB entry in the schemata file when the MB resource exists.
Is it possible to emulate the percentage based MB control with MB_WGHT or MB_MIN?
This sounds similar as what is/was planned for MPAM [2].
Something that may be of interest is a proposal that Chenyu is refining to address an
issue with the region-aware MBA support where there is no intuitive backward compatible
interface. This was highlighted in the plumbers slides (see slide titled "Open: maintaining
backward compatibility when region aware"). The current idea to deal with this is to
introduce a "mode" associated with the resource controls. For example,
# cat /sys/fs/resctrl/info/MB/resource_schemata/mode
[legacy] native
By default the "legacy" mode will be enabled and exposes the "MB" default control to user
space via the schemata file. In support of this each new control has a new property file
named "status" that can have value "enabled" or "disabled". Only "enabled" controls are
present in the schemata file but all controls are always present in the resource_schemata
directory. By writing to the "mode" file user space acknowledges familiarity with the new
"resource_schemata" based interface and can change the status of a control and
thus manage its visibility in the schemata file.
Could something like this work for CBQRI?
Reinette
> [1] https://lore.kernel.org/all/20260601-ssqosid-cbqri-rqsc-v7-0-v6-6-baf00f50028a@kernel.org/
[2] https://lore.kernel.org/lkml/c78169bc-e2d6-4583-96ec-09fa6dd6653a@intel.com/
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-04 21:05 ` Reinette Chatre
@ 2026-06-05 19:35 ` Drew Fustini
2026-06-06 5:10 ` Drew Fustini
0 siblings, 1 reply; 66+ messages in thread
From: Drew Fustini @ 2026-06-05 19:35 UTC (permalink / raw)
To: Reinette Chatre
Cc: Ben Horgan, Tony Luck, James Morse, Dave Martin, Babu Moger,
Fenghua Yu, Chen Yu, Borislav Petkov, Thomas Gleixner,
Dave Hansen, Peter Newman, x86@kernel.org,
linux-kernel@vger.kernel.org
On Thu, Jun 04, 2026 at 02:05:08PM -0700, Reinette Chatre wrote:
> >> I plumbed in support for the MB_MIN resource schema which also works under light
> >> testing. The only fs resctrl code change I needed was:
> >>
> >> --- a/include/linux/resctrl.h
> >> +++ b/include/linux/resctrl.h
> >> @@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct
> >> resctrl_ctrl *ctrl)
> >> case RESCTRL_CTRL_BITMAP:
> >> return BIT_MASK(ctrl->cache.cbm_len) - 1;
> >> case RESCTRL_CTRL_SCALAR:
> >> + if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
> >> + return ctrl->membw.min_bw;
> >> +
> >> return ctrl->membw.max_bw;
> >> }
> >>
> >>
> >> At least on MPAM systems, we use a default of 0 for minimum bandwidth controls
> >> as the maximum bandwidth controls only take effect if their value is higher than
> >> the minimum bandwidth value. I have specialised this on the ctrl->name which
> >> breaks your ctrl->type based classification but that's fixable by just adding a
> >> default field to membw.
> >
> > This should be useful for RISC-V.
> >
> > RESCTRL_CTRL_NAME_MIN maps well to CBQRI Rbwb (reserved bandwidth
> > blocks). The sum of Rbwb across all control groups must be less than
> > MRBWB (maximum number of reserved bandwidth blocks). As a result, MB_MIN
> > needs to default to 1 so that the sum does not violate that rule. In my
> > RFC series, I added default_to_min to resctrl_membw [1] but this
> > solution looks cleaner.
>
> As I mentioned in response to Ben [2] there seems to be a mismatch between
> architecture requirements here. resctrl uses the value returned by
> resctrl_get_default_ctrlval() as the control value that means "no throttling".
> For Intel this means min == max but this does not seem to be the case for MPAM
> and CBQRI. I am not familiar enough with either to have an alternative proposal here
> so I need to become familiar now. There is a bit of backlog on other resctl
> work right now so this will take me some time to sort out.
Thanks for pointing this out. In that case, it doesn't seem to match
what I was thinking of for MB_MIN. The CBQRI reserved bandwidth blocks
Rbwb) control can be thought of as a minimum amount of guranteed
bandwidth for a control group. Each RCID (e.g. CLOSID) must be assigned
at least 1 bandwidth block per the spec. Therefore, the membw.min_bw
would need to be 1.
There is also a max bandwidth reservation across all control groups
(RCIDs / CLOSIDs) so that there will be some amount of unreserved
bandwidth. Mweight (1-255) controls how much of that unreserved
bandwidth pool that a group can use. Mweight of 0 means no shared
bandwidth. I think the membw.min_bw would need to 255 so that all groups
get equal share of the unreserved pool.
It seems like that would be incorrect use of membw.min_bw in both cases?
> > There is no equivalent to MB (percentage throttle) in RISC-V so I would
> > want it to be valid to have MB_MIN (minimum reservation) without MB.
> >
> > I rebased my RISC-V CBQRI v6 series on top of this proof of concept and
> > was able to validate it works okay in Qemu:
> >
> > MB_WGHT:72=255
> > MB_MIN:72=756
> > L2:64=fff;65=fff
> > L3:75=ffff
>
> Ideally any new support should not break existing user space and the existing
> user interface expects a MB entry in the schemata file when the MB resource exists.
> Is it possible to emulate the percentage based MB control with MB_WGHT or MB_MIN?
> This sounds similar as what is/was planned for MPAM [2].
Yes, I think that Mweight could be mapped to the MB concept of
throttling. All groups could start with the max Mweight of 255 which
could can be represented as 100%.
However, I'm not sure what to do about membw.min_bw. Mweight = 0 means
it can not use any of the shared unreserved bandwidth pool. If
resctrl_get_default_ctrlval() is designed to mean "no throttling", then
it seems like the membw.min_bw would need to be 255. But that feels
weird for the min_bw value to be equal to the max weight for unreserved
bandwidth.
> Something that may be of interest is a proposal that Chenyu is refining to address an
> issue with the region-aware MBA support where there is no intuitive backward compatible
> interface. This was highlighted in the plumbers slides (see slide titled "Open: maintaining
> backward compatibility when region aware"). The current idea to deal with this is to
> introduce a "mode" associated with the resource controls. For example,
>
> # cat /sys/fs/resctrl/info/MB/resource_schemata/mode
> [legacy] native
>
> By default the "legacy" mode will be enabled and exposes the "MB" default control to user
> space via the schemata file. In support of this each new control has a new property file
> named "status" that can have value "enabled" or "disabled". Only "enabled" controls are
> present in the schemata file but all controls are always present in the resource_schemata
> directory. By writing to the "mode" file user space acknowledges familiarity with the new
> "resource_schemata" based interface and can change the status of a control and
> thus manage its visibility in the schemata file.
> Could something like this work for CBQRI?
Yes, I think that would work. There are no existing users of resctrl on
RISC-V so I think having users opt into this resource_schemata interface
would work, especially if that allows a truer represenation of the
controls in the CBQRI spec.
Thanks,
Drew
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-05 19:35 ` Drew Fustini
@ 2026-06-06 5:10 ` Drew Fustini
2026-06-06 5:23 ` Drew Fustini
0 siblings, 1 reply; 66+ messages in thread
From: Drew Fustini @ 2026-06-06 5:10 UTC (permalink / raw)
To: Reinette Chatre
Cc: Ben Horgan, Tony Luck, James Morse, Dave Martin, Babu Moger,
Fenghua Yu, Chen Yu, Borislav Petkov, Thomas Gleixner,
Dave Hansen, Peter Newman, x86@kernel.org,
linux-kernel@vger.kernel.org
On Fri, Jun 05, 2026 at 12:35:51PM -0700, Drew Fustini wrote:
> > As I mentioned in response to Ben [2] there seems to be a mismatch between
> > architecture requirements here. resctrl uses the value returned by
> > resctrl_get_default_ctrlval() as the control value that means "no throttling".
> > For Intel this means min == max but this does not seem to be the case for MPAM
> > and CBQRI. I am not familiar enough with either to have an alternative proposal here
> > so I need to become familiar now. There is a bit of backlog on other resctl
> > work right now so this will take me some time to sort out.
>
> Thanks for pointing this out. In that case, it doesn't seem to match
> what I was thinking of for MB_MIN. The CBQRI reserved bandwidth blocks
> Rbwb) control can be thought of as a minimum amount of guranteed
> bandwidth for a control group. Each RCID (e.g. CLOSID) must be assigned
> at least 1 bandwidth block per the spec. Therefore, the membw.min_bw
> would need to be 1.
>
> There is also a max bandwidth reservation across all control groups
> (RCIDs / CLOSIDs) so that there will be some amount of unreserved
> bandwidth. Mweight (1-255) controls how much of that unreserved
> bandwidth pool that a group can use. Mweight of 0 means no shared
> bandwidth. I think the membw.min_bw would need to 255 so that all groups
> get equal share of the unreserved pool.
Sorry, I wasn't thinking about this right. If Mweight is used for MB,
then membw.max_bw would be 100 (MAX_MBA_BW) and membw.min_bw would be 0
which means no shared bandwidth.
> It seems like that would be incorrect use of membw.min_bw in both cases?
The issue is really just for Rbwb (reserved bandwidth) as that needs to
default to the minimum of 1. What about introducing membw.reset_val
which would be returned by resctrl_get_default_ctrl()?
MB could set membw.reset_val to be the same value as membw.max_bw.
> > > There is no equivalent to MB (percentage throttle) in RISC-V so I would
> > > want it to be valid to have MB_MIN (minimum reservation) without MB.
> > >
> > > I rebased my RISC-V CBQRI v6 series on top of this proof of concept and
> > > was able to validate it works okay in Qemu:
> > >
> > > MB_WGHT:72=255
> > > MB_MIN:72=756
> > > L2:64=fff;65=fff
> > > L3:75=ffff
> >
> > Ideally any new support should not break existing user space and the existing
> > user interface expects a MB entry in the schemata file when the MB resource exists.
> > Is it possible to emulate the percentage based MB control with MB_WGHT or MB_MIN?
> > This sounds similar as what is/was planned for MPAM [2].
>
> Yes, I think that Mweight could be mapped to the MB concept of
> throttling. All groups could start with the max Mweight of 255 which
> could can be represented as 100%.
>
> However, I'm not sure what to do about membw.min_bw. Mweight = 0 means
> it can not use any of the shared unreserved bandwidth pool. If
> resctrl_get_default_ctrlval() is designed to mean "no throttling", then
> it seems like the membw.min_bw would need to be 255. But that feels
> weird for the min_bw value to be equal to the max weight for unreserved
> bandwidth.
MB would have the typical membw.min_bw = 100, and
resctrl_get_default_ctrl() would return 100. The controller would be
programmed with Mweight 255 for 100%.
Thanks,
Drew
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-06 5:10 ` Drew Fustini
@ 2026-06-06 5:23 ` Drew Fustini
0 siblings, 0 replies; 66+ messages in thread
From: Drew Fustini @ 2026-06-06 5:23 UTC (permalink / raw)
To: Reinette Chatre
Cc: Ben Horgan, Tony Luck, James Morse, Dave Martin, Babu Moger,
Fenghua Yu, Chen Yu, Borislav Petkov, Thomas Gleixner,
Dave Hansen, Peter Newman, x86@kernel.org,
linux-kernel@vger.kernel.org
On Fri, Jun 05, 2026 at 10:10:38PM -0700, Drew Fustini wrote:
> On Fri, Jun 05, 2026 at 12:35:51PM -0700, Drew Fustini wrote:
> > > As I mentioned in response to Ben [2] there seems to be a mismatch between
> > > architecture requirements here. resctrl uses the value returned by
> > > resctrl_get_default_ctrlval() as the control value that means "no throttling".
> > > For Intel this means min == max but this does not seem to be the case for MPAM
> > > and CBQRI. I am not familiar enough with either to have an alternative proposal here
> > > so I need to become familiar now. There is a bit of backlog on other resctl
> > > work right now so this will take me some time to sort out.
> >
> > Thanks for pointing this out. In that case, it doesn't seem to match
> > what I was thinking of for MB_MIN. The CBQRI reserved bandwidth blocks
> > Rbwb) control can be thought of as a minimum amount of guranteed
> > bandwidth for a control group. Each RCID (e.g. CLOSID) must be assigned
> > at least 1 bandwidth block per the spec. Therefore, the membw.min_bw
> > would need to be 1.
> >
> > There is also a max bandwidth reservation across all control groups
> > (RCIDs / CLOSIDs) so that there will be some amount of unreserved
> > bandwidth. Mweight (1-255) controls how much of that unreserved
> > bandwidth pool that a group can use. Mweight of 0 means no shared
> > bandwidth. I think the membw.min_bw would need to 255 so that all groups
> > get equal share of the unreserved pool.
>
> Sorry, I wasn't thinking about this right. If Mweight is used for MB,
> then membw.max_bw would be 100 (MAX_MBA_BW) and membw.min_bw would be 0
> which means no shared bandwidth.
>
> > It seems like that would be incorrect use of membw.min_bw in both cases?
>
> The issue is really just for Rbwb (reserved bandwidth) as that needs to
> default to the minimum of 1. What about introducing membw.reset_val
> which would be returned by resctrl_get_default_ctrl()?
>
> MB could set membw.reset_val to be the same value as membw.max_bw.
>
> > > > There is no equivalent to MB (percentage throttle) in RISC-V so I would
> > > > want it to be valid to have MB_MIN (minimum reservation) without MB.
> > > >
> > > > I rebased my RISC-V CBQRI v6 series on top of this proof of concept and
> > > > was able to validate it works okay in Qemu:
> > > >
> > > > MB_WGHT:72=255
> > > > MB_MIN:72=756
> > > > L2:64=fff;65=fff
> > > > L3:75=ffff
> > >
> > > Ideally any new support should not break existing user space and the existing
> > > user interface expects a MB entry in the schemata file when the MB resource exists.
> > > Is it possible to emulate the percentage based MB control with MB_WGHT or MB_MIN?
> > > This sounds similar as what is/was planned for MPAM [2].
> >
> > Yes, I think that Mweight could be mapped to the MB concept of
> > throttling. All groups could start with the max Mweight of 255 which
> > could can be represented as 100%.
> >
> > However, I'm not sure what to do about membw.min_bw. Mweight = 0 means
> > it can not use any of the shared unreserved bandwidth pool. If
> > resctrl_get_default_ctrlval() is designed to mean "no throttling", then
> > it seems like the membw.min_bw would need to be 255. But that feels
> > weird for the min_bw value to be equal to the max weight for unreserved
> > bandwidth.
>
> MB would have the typical membw.min_bw = 100, and
> resctrl_get_default_ctrl() would return 100. The controller would be
> programmed with Mweight 255 for 100%.
Sorry - I meant MB would have membw.max_bw = 100 which would cause CBQRI
bandwidth controller to be programmed with Mweight = 255.
Drew
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-03 15:15 ` Ben Horgan
2026-06-03 19:34 ` Drew Fustini
@ 2026-06-04 17:43 ` Reinette Chatre
2026-06-05 14:53 ` Ben Horgan
1 sibling, 1 reply; 66+ messages in thread
From: Reinette Chatre @ 2026-06-04 17:43 UTC (permalink / raw)
To: Ben Horgan, Tony Luck, James Morse, Dave Martin, Babu Moger,
Drew Fustini, Fenghua Yu, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi Ben,
On 6/3/26 8:15 AM, Ben Horgan wrote:
> Hi Reinette,
>
> On 5/29/26 19:06, Reinette Chatre wrote:
>> Hi Everybody,
>>
>> It has been a while since we discussed the resctrl changes required to support
>> hardware that has controls with fine granularity or hardware that has multiple
>> controls per resource. For reference, the most recent email discussion can
>> be found at [1] with a summary of discussions in last year's plumbers slides [2].
>>
>> I created a PoC that I believe supports what folks have agreed to so far. I
>> hope this can help us to restart the discussion with the goal that resctrl gains
>> support for upcoming hardware that require these features.
>
> Thank you very much for doing this work. I believe this will be very useful for
> MPAM and other architectures.
Thank you very much for reviewing this.
>
>>
>> Request regarding this PoC
>> ==========================
>>
>> Please consider this PoC as a "direction check" on the schema description and multiple
>> control discussions held thus far.
>>
>> Could folks working on enabling new hardware requiring this capability please consider
>> if this is something you can build on and how it should be improved to support these
>> upcoming capabilities?
>>
>> Opens
>> =====
>>
>> While the PoC aims to support what folks agreed on some opens remain:
>> - I attempted to make some MPAM supporting changes but these are all just compile
>> tested. While MPAM should benefit from the new control properties I did not
>> initialize them on MPAM and did not attempt refactor to separate out
>> the architecture specific control properties (more on what this means later).
>> I did attempt some MPAM refactoring that duplicates the MPAM domain to the
>> control domain and monitoring domain lists in support of there being multiple
>> controls each with its own list of control domains but it is definitely not good
>> design.
>
> I appreciate you including MPAM in this PoC. With this one line change I was
> able to boot an MPAM system and mount resctrl which appears to behave correctly.
> We can consider how best to do the code design later.
>
> --- a/drivers/resctrl/mpam_resctrl.c
> +++ b/drivers/resctrl/mpam_resctrl.c
> @@ -1697,6 +1697,8 @@ int mpam_resctrl_setup(void)
>
> /* Initialise the resctrl structures from the classes */
> for_each_mpam_resctrl_control(res, rid) {
> + INIT_LIST_HEAD(&res->resctrl_res.controls); // list_empty needs
> to work
> +
> if (!res->class)
> continue; // dummy resource
>
Thank you very much. I picked this up but ended up moving it earlier to be next
to the mon_domains list initialization. Doing so made it easier for me to follow
the initialization. That ok?
> I plumbed in support for the MB_MIN resource schema which also works under light
> testing. The only fs resctrl code change I needed was:
>
> --- a/include/linux/resctrl.h
> +++ b/include/linux/resctrl.h
> @@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct
> resctrl_ctrl *ctrl)
> case RESCTRL_CTRL_BITMAP:
> return BIT_MASK(ctrl->cache.cbm_len) - 1;
> case RESCTRL_CTRL_SCALAR:
> + if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
> + return ctrl->membw.min_bw;
> +
> return ctrl->membw.max_bw;
> }
>
>
> At least on MPAM systems, we use a default of 0 for minimum bandwidth controls
> as the maximum bandwidth controls only take effect if their value is higher than
> the minimum bandwidth value. I have specialised this on the ctrl->name which
> breaks your ctrl->type based classification but that's fixable by just adding a
> default field to membw.
This I am not sure about. In my understanding a typical "default" value means
"no throttling" and, at least on Intel, this default hardware state has been
summarized as "min" == "max" == "optimal".
Are you saying that on MPAM systems if "min" == "max" then max bandwidth controls
do not take effect? Could you please elaborate what happens if "min" == "max"?
>> - No support for emulated controls (yet). The PoC is quite large already
>> but I think it can be used as a base for emulated controls for which the software
>> controller could be a potential first customer. In this PoC mounting with
>> software controller will still display the original controller's properties.
>> - One open that needs to be addressed as part of support for emulated controls is
>> how best to display emulation relationship via resctrl hierarchy.
>
> What does emulated controls mean here? Is there some previous discussion you
> could point me at?
For emulated controls in context of MPAM I think the best reference is
https://lore.kernel.org/lkml/aPJP52jXJvRYAjjV@e133380.arm.com/
Above is the email discussion that I attempted to visualize in the middle example in slide 6
("resctrl controls vs. hardware controls") of
https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf
When comparing the slide to Dave's text, please replace "MB_HW" from Dave's example
with "MB_OPT" in the slide. I changed the name since I found the "HW" in an
emulated control to be potentially confusing.
>> - No support for "read-modify-write" usage of schemata file. This is where we
>> discussed (without agreement) on possibly introducing the "#" prefix to schemata
>> file entries. This PoC does not support this prefix and the current assumption/expectation
>> is that when user space changes a configuration only the new control values are
>> written to schemata file. I thus do not have a plan to support this so please
>> share opinions in this regard if you have some.
>
> There is now less motivation from the MPAM side for this than when this was
> initially discussed. In pre-upstream versions of the MPAM patches a change in
> the MB resource control value would change both the mpam h/w mbw_min and mbw_max
> values but now (on non-broken h/w) we just change the mbw_max. (mbw_min kept at 0).
Ah, thanks for the correction. The email I linked above indeed refers to changing
both min and max.
>
> However, it would be useful not to be limited by percentages. In my quick
Indeed. Not being limited by percentages while still needing to have a backward
compatible user interface is how we ended up with "emulated controls".
> experimentation with your patches I used a percentage value for MB_MIN but it
> would be best to move away from this. For new controls I think we can mandate
> that user space has to discover the resolution from the info directly but how
> can we retrofit this. For MPAM, MB and MB_MAX, would control the same things.
> Could we just add MB_MAX with a h/w friendly scale and then reflect changes in
> MB_MAX in MB and vica versa with MB taking precedent if both are set? Old
> software can continue setting MB can move to using MB_MAX and take advantage of
> the improved control. (I don't think we should expose the MPAM hardware value
> directly as it has confusion over whether all 1s is 100% or not and we'd like to
> have something generic and friendly to the user.)
Sounds to me as though you are describing emulated controls. Exposing two
controls in schemata file that essentially controls the same thing is what the
emulated controls aim to solve and the resctrl hierarchies presented in slide #6
of that presentation (and discussed in the email thread) is how we contemplated how
to represent the relationship among these controls to user space. So, considering
your example resctrl may display something like:
info//
└── MB/
└── resource_schemata/
└── MB/
└── MB_MAX/
Above hierarchy describes the relationship to user space that if MB is changed it
will impact MB_MAX and vice-versa.
The one open I am aware of surrounding emulated controls is how to present some
semblance of consistency to user space when considering all the possibilities
the different architectures (and even within architectures) may have.
>> - Controls are independent for now. This means that, for example, if a resource
>> supports a "MIN" and "MAX" control then this implementation would allow user to
>> set the "maximum" control values to be less than the "minimum" control values.
>
> I think this is ok as long as adding support for new controls in resctrl doesn't
> change the existing behaviour. In MPAM we dodged this by introducing MB as only
> affecting the h/w mbw_max and not mbw_min (as mentioned above).
I understand this to be a requirement for Intel where the spec contains "The Maximum Cap
should be programmed to be greater than or equal to the Minimum and Optimal caps.
Undesirable and undefined performance effects may result if cap programming guidelines
are not followed."
I am currently thinking that resctrl should not try to be too smart here and if user
space wants to make dramatic changes to min and max values then it should just ensure
the ordering is appropriate. For example, attempting to set a new min to be larger than
the old max would fail and user space should first increase the old max and then set
a new min.
>
>> - PoC supports the "bitmap" control but does not (yet) expose properties of a bitmap
>> control to the new info/<resource>/resource_schemata directory.
>>
>> Accessing PoC
>> =============
>>
>> Please consider the PoC as a rough draft. It has only been compile tested for Arm
>> and known to be incomplete in Arm support. To help with experimenting I only
>> fully adapted the Intel MBA resource to demo two dummy additional MBA controls.
>> All architectures should immediately benefit from the new schema descriptions
>> and new info/MB/resource_schemata hierarchy.
>>
>> I considered the patches self too many for email. Instead, the PoC can be found at:
>>
>> git://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git branch resctrl/controls_rfc_v1
>>
>> The work is based on v7.1-rc2 that also includes the following series (two of which has
>> since been queued) included:
>>
>> "selftests/resctrl: Fixes and improvements focused on Intel platforms"
>> https://lore.kernel.org/lkml/cover.1775266384.git.reinette.chatre@intel.com/
>>
>> "x86,fs/resctrl: Improve resctrl quality and consistency"
>> https://lore.kernel.org/lkml/cover.1777419024.git.reinette.chatre@intel.com/
>>
>> "x86,fs/resctrl: Pave the way for MPAM counter assignment"
>> https://lore.kernel.org/lkml/20260506082855.3694761-1-ben.horgan@arm.com/
>>
>>
>> Primary resctrl fs data structure changes
>> =========================================
>>
>> Introduces a control represented by struct resctrl_ctrl that looks as below. To make
>> the changes easier to follow I kept some of the original names to help communicate
>> where familiar data structures land.
>>
>> What to notice about a control is that it has some common properties required
>> from all controls (scope, type, etc.) and then depending on the type of control
>> (RESCTRL_CTRL_BITMAP or RESCTRL_CTRL_SCALAR) there are type specific properties.
>>
>> /**
>> * struct resctrl_ctrl - A resource control
>> * @entry: List entry of rdt_resource::controls
>> * @scope: Scope of the resource that this control allocates
>> * @domains: RCU list of all control domains
>> * @type: The control type that determines the properties of the control,
>> * format string for displaying control values to user space, and
>> * parser of control values provided by user space.
>> * @name: Name of the control. Appended to final resource name
>> * (rdt_resource_final::name) to create final schema entry.
>> * Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
>> * For example, with resource name "MB" and control name "MAX" the
>> * schema entry will be "MB_MAX".
>> * @cache: Cache allocation control properties.
>> * @membw: Bandwidth control properties.
>> */
>> struct resctrl_ctrl {
>> struct list_head entry;
>> enum resctrl_scope scope;
>> struct list_head domains;
>> enum resctrl_ctrl_type type;
>> enum resctrl_ctrl_name name;
>> union {
>> struct resctrl_cache cache;
>> struct resctrl_membw membw;
>> };
>> };
>>
>> Two members summarize how this new structure fits into the rest of resctrl:
>> a) resctrl_ctrl::entry
>> Since a resource can support multiple controls there is a new list
>> in struct rdt_resource named "controls" that contains the list of all
>> controls supported by the resource.
>> b) resctrl_ctrl::domains
>> Instead of the list of control domains belonging to a resource they
>> now belong to the control self. By doing so resctrl can support resource
>> controls at different scope for the same resource. This is intended to
>> support some upcoming MPAM and RISC-V usages.
>
> Please can you expand a bit on part b).
>
> In an MPAM system we consider 3 resctrl resources, RDT_RESOURCE_L3,
> RDT_RESOURCE_L2 and RDT_RESOURCE_MBA which correspond to the L3 caches, L2
> caches and memory bandwidth on egress from the L3 caches. The domain for each of
> these corresponds to the instance of the resource. That is, for RDT_RESOURCE_L2
> there is a resource for each L2 instance, similarly for L3, and for
(I'm assuming above is typo and it is "there is a domain for each L2 instance"?)
> RDT_RESOURCE_MBA there is a domain for each L3 cache. If we were to add suport
> for controls on a new cache level, say the L4, then I'd expect to add a new
> resource. For memory bandwidth, we'd like to be able to control b/w on the L2
> egress (e.g. in a DSU). Wouldn't this too be a separate resource or would this
> be a new set of controls on the same resource?
>
> New controls on the same resource
> MB_MIN2
> MB_MAX2
> MB_PROP2
> ...
>
> or
> MB2_MIN
> MB2_MAX
> MB2_PROP
The way I currently see it is that controlling bandwidth at a different scope would
be a new set of controls associated with the MB resource. There are more scenarios
coming this way with AMD's "Global MBA" that is memory bandwidth allocation at
NUMA node scope. If I understand correctly the "CPU-less Memory Node" that Nvidia
shared at plumbers would need this also and control memory bandwidth allocation
at the NUMA node scope. A related technology is Intel's region-aware MBA, which is
still at L3 scope.
I fully agree that we need to figure out how to represent all of this to user space
without turning the interface into something unintelligible. In the end this is
required for user space to know what a domain ID represents.
Would it help to make the scope part of the control name? The ship has sailed for
MB being associated with L3 scope but this could mean the "default" scope of MB
resource is L3 (which user space can still confirm by looking at the control's
"scope" file) and the others include scope in the name? Consider for example:
https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@intel.com/
>
> AFAIK, the DSU h/w just supports proportional bandwidth controls at the moment
> but we should consider what to do about the potential naming.
ack.
>
> In the MPAM driver, we collect MSC into components (based on instances) and
> those into classes (components of the same type). Currently, a resource is
> mapped to a single class. (Two resources may map to the same class.)
>
> I expect it is useful in the memory region and sub numa cases but I'd still
> expect the common case to be that the domains are the same within a control. Or
> am I missing something?
Domains of a control should all be at the same scope. Since the schemata file
exposes the control with the different IDs representing the instances of the
resource needing to be controlled it has to be clear to user space what the
domain ID represents.
>
>>
>> Example architectural data structure changes
>> ============================================
>>
>> An architecture can use the new control by following a similar pattern to
>> resource and domain use by architectures. Consider the following for x86
>> where a new architecture specific struct resctrl_hw_ctrl includes
>> struct resctrl_ctrl and any architecture private data needed to support
>> the control:
>>
>> /*
>> * struct resctrl_hw_ctrl - Arch private properties of a resource control
>> * @r_ctrl: Control properties exposed to resctrl file system
>> * @msr_base: Base MSR address where control values should be programmed
>> * @msr_update: Function pointer to update control values
>> */
>> struct resctrl_hw_ctrl {
>> struct resctrl_ctrl r_ctrl;
>> unsigned int msr_base;
>> void (*msr_update)(struct msr_param *m);
>> };
>>
>> Structure of patch series
>> =========================
>>
>> As a PoC the series is not perfectly structured but to help navigate this work
>> on a high level the changes can be categorized as follows:
>>
>> Patch 1 to 11:
>> With a vision of what a "control" is, remove unused/unnecessary
>> members, make clear what is a *resource* property vs a *control*
>> property, do some renaming to help with the PoC.
>
> A few of the changes are generic cleanup and could hopefully be dealt with
> before decisions on the larger PoC are made. I see:
> fs/resctrl: Remove unused resctrl_membw::mb_map
> x86,fs/resctrl: Remove "arch_needs_linear"
> Perhaps a few more.
ack.
>>
>> Patch 12:
>> Introduce struct resctrl_ctrl and re-arrange existing struct rdt_resource
>> members to form part of new rdt_resource::ctrl
>>
>> Patch 13 to 44:
>> A lot of wrangling to introduce struct resctrl_ctrl to all code that needs
>> to work with a control and/or domain without assuming that the control is
>> the one and only control embedded in the resource it belongs to. Essentially,
>> a lot of changes passing the control around in addition to the resource/domain.
>
> You mention a few times in the commit message that you expect the cache
> resources to only have one control. On MPAM we have CMAX (and there looks to be
> a RISC-V equivalent) where the total number of bytes in the cache for a given
> closid is limited. The allocation must still respect the CPBM bitmap though.
> Looking at the code though I don't see much problem in adding this as an
> additional control. The assumption that these patches is making is not that
> there is only one control for cache resources but rather that cache portions are
> managed by the default cache resource control. Am I missing something or does
> that assessment make sense to you?
Your assessment is correct. There are still a few assumptions built into resctrl
about there only being a single cache control and it being a bitmap control.
Since I am not familiar with the other possible cache controls I instead focused
on isolating the existing cache control. When I/we have better understanding about
how additional cache controls behave this implementation can be adapted to support
it.
>
> I have been looking at adding CMAX control to resctrl and will have a go at
> basing what I have so far on top of this series.
Thank you!
...
>>> Any feedback is appreciated.
>
> Overall, this looks to be a big step in the right direction.
Glad to hear this.
Thank you very much.
Reinette
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-04 17:43 ` Reinette Chatre
@ 2026-06-05 14:53 ` Ben Horgan
2026-06-05 15:39 ` Reinette Chatre
0 siblings, 1 reply; 66+ messages in thread
From: Ben Horgan @ 2026-06-05 14:53 UTC (permalink / raw)
To: Reinette Chatre, Tony Luck, James Morse, Dave Martin, Babu Moger,
Drew Fustini, Fenghua Yu, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi Reinette,
On 6/4/26 18:43, Reinette Chatre wrote:
> Hi Ben,
>
> On 6/3/26 8:15 AM, Ben Horgan wrote:
>> Hi Reinette,
>>
>> On 5/29/26 19:06, Reinette Chatre wrote:
>>> Hi Everybody,
>>>
>>> It has been a while since we discussed the resctrl changes required to support
>>> hardware that has controls with fine granularity or hardware that has multiple
>>> controls per resource. For reference, the most recent email discussion can
>>> be found at [1] with a summary of discussions in last year's plumbers slides [2].
>>>
>>> I created a PoC that I believe supports what folks have agreed to so far. I
>>> hope this can help us to restart the discussion with the goal that resctrl gains
>>> support for upcoming hardware that require these features.
>>
>> Thank you very much for doing this work. I believe this will be very useful for
>> MPAM and other architectures.
>
> Thank you very much for reviewing this.
>
>>
>>>
>>> Request regarding this PoC
>>> ==========================
>>>
>>> Please consider this PoC as a "direction check" on the schema description and multiple
>>> control discussions held thus far.
>>>
>>> Could folks working on enabling new hardware requiring this capability please consider
>>> if this is something you can build on and how it should be improved to support these
>>> upcoming capabilities?
>>>
>>> Opens
>>> =====
>>>
>>> While the PoC aims to support what folks agreed on some opens remain:
>>> - I attempted to make some MPAM supporting changes but these are all just compile
>>> tested. While MPAM should benefit from the new control properties I did not
>>> initialize them on MPAM and did not attempt refactor to separate out
>>> the architecture specific control properties (more on what this means later).
>>> I did attempt some MPAM refactoring that duplicates the MPAM domain to the
>>> control domain and monitoring domain lists in support of there being multiple
>>> controls each with its own list of control domains but it is definitely not good
>>> design.
>>
>> I appreciate you including MPAM in this PoC. With this one line change I was
>> able to boot an MPAM system and mount resctrl which appears to behave correctly.
>> We can consider how best to do the code design later.
>>
>> --- a/drivers/resctrl/mpam_resctrl.c
>> +++ b/drivers/resctrl/mpam_resctrl.c
>> @@ -1697,6 +1697,8 @@ int mpam_resctrl_setup(void)
>>
>> /* Initialise the resctrl structures from the classes */
>> for_each_mpam_resctrl_control(res, rid) {
>> + INIT_LIST_HEAD(&res->resctrl_res.controls); // list_empty needs
>> to work
>> +
>> if (!res->class)
>> continue; // dummy resource
>>
>
> Thank you very much. I picked this up but ended up moving it earlier to be next
> to the mon_domains list initialization. Doing so made it easier for me to follow
> the initialization. That ok?
Yes, that makes sense.
>
>> I plumbed in support for the MB_MIN resource schema which also works under light
>> testing. The only fs resctrl code change I needed was:
>>
>> --- a/include/linux/resctrl.h
>> +++ b/include/linux/resctrl.h
>> @@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct
>> resctrl_ctrl *ctrl)
>> case RESCTRL_CTRL_BITMAP:
>> return BIT_MASK(ctrl->cache.cbm_len) - 1;
>> case RESCTRL_CTRL_SCALAR:
>> + if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
>> + return ctrl->membw.min_bw;
>> +
>> return ctrl->membw.max_bw;
>> }
>>
>>
>> At least on MPAM systems, we use a default of 0 for minimum bandwidth controls
>> as the maximum bandwidth controls only take effect if their value is higher than
>> the minimum bandwidth value. I have specialised this on the ctrl->name which
>> breaks your ctrl->type based classification but that's fixable by just adding a
>> default field to membw.
>
> This I am not sure about. In my understanding a typical "default" value means
> "no throttling" and, at least on Intel, this default hardware state has been
> summarized as "min" == "max" == "optimal".
Ok, this sounds odd to me but that is probably because I don't know what Intel
systems do. On MPAM systems a MIN control is a boost rather than a throttling
control. Although, you can always think of that as throttling the traffic with
the other PARTIDs.
>
> Are you saying that on MPAM systems if "min" == "max" then max bandwidth controls
> do not take effect? Could you please elaborate what happens if "min" == "max"?
Table 5-4 from section 5.2.8 of the IHI0099B.b shows the interaction between the
min and maximum controls.
If used bandwidth is The preference is Description
Below the minimum High Only high requests compete with this
request.
Above the minimum:
Below the maximum Medium High requests are serviced first then
this request competes with other
medium requests.
Above the maximum, Low Requests are not serviced if any high
when HARDLIM is 0 or medium requests are available.
Above the maximum, None Requests are not serviced
when HARDLIM is 1
So if we keep the minimum and the maximum controls values always the same then
all traffic will be given "high" preference until the target bandwidth is
reached. For some MPAM systems it is recommended to set the minimum value as 5%
less than the maximum value to get a reliable target bandwidth. As 5% seems
implementation specific and some systems don't have min controls it seemed
better to just match the MB control with a maximum bandwidth control and let the
user have freedom to choose the minimum bandwidth control when MB_MIN support is
added.
If a default for the minimum of the maximum possible bandwidth is used (100%)
then any change of the maximum won't have any effect as it's always less than
minimum (if that's unchanged) and so all traffic is high preference. I now see
from your reply below that you are planning on not allowing this kind of
configuration.
If the minimum always tracks the maximum then we lose the distinction between
medium and high preference traffic and so to reserve some high preference
bandwidth for one control group we'd have to change the configuration in the
other controls groups so that they're bandwidth preference is medium (minimum
value at 0).
>
>>> - No support for emulated controls (yet). The PoC is quite large already
>>> but I think it can be used as a base for emulated controls for which the software
>>> controller could be a potential first customer. In this PoC mounting with
>>> software controller will still display the original controller's properties.
>>> - One open that needs to be addressed as part of support for emulated controls is
>>> how best to display emulation relationship via resctrl hierarchy.
>>
>> What does emulated controls mean here? Is there some previous discussion you
>> could point me at?
>
> For emulated controls in context of MPAM I think the best reference is
> https://lore.kernel.org/lkml/aPJP52jXJvRYAjjV@e133380.arm.com/
>
> Above is the email discussion that I attempted to visualize in the middle example in slide 6
> ("resctrl controls vs. hardware controls") of
> https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf
>
> When comparing the slide to Dave's text, please replace "MB_HW" from Dave's example
> with "MB_OPT" in the slide. I changed the name since I found the "HW" in an
> emulated control to be potentially confusing.
Ah I see, thanks for links and descriptions.
>
>
>>> - No support for "read-modify-write" usage of schemata file. This is where we
>>> discussed (without agreement) on possibly introducing the "#" prefix to schemata
>>> file entries. This PoC does not support this prefix and the current assumption/expectation
>>> is that when user space changes a configuration only the new control values are
>>> written to schemata file. I thus do not have a plan to support this so please
>>> share opinions in this regard if you have some.
>>
>> There is now less motivation from the MPAM side for this than when this was
>> initially discussed. In pre-upstream versions of the MPAM patches a change in
>> the MB resource control value would change both the mpam h/w mbw_min and mbw_max
>> values but now (on non-broken h/w) we just change the mbw_max. (mbw_min kept at 0).
>
> Ah, thanks for the correction. The email I linked above indeed refers to changing
> both min and max.
>
>>
>> However, it would be useful not to be limited by percentages. In my quick
>
> Indeed. Not being limited by percentages while still needing to have a backward
> compatible user interface is how we ended up with "emulated controls".
>
>> experimentation with your patches I used a percentage value for MB_MIN but it
>> would be best to move away from this. For new controls I think we can mandate
>> that user space has to discover the resolution from the info directly but how
>> can we retrofit this. For MPAM, MB and MB_MAX, would control the same things.
>> Could we just add MB_MAX with a h/w friendly scale and then reflect changes in
>> MB_MAX in MB and vica versa with MB taking precedent if both are set? Old
>> software can continue setting MB can move to using MB_MAX and take advantage of
>> the improved control. (I don't think we should expose the MPAM hardware value
>> directly as it has confusion over whether all 1s is 100% or not and we'd like to
>> have something generic and friendly to the user.)
>
> Sounds to me as though you are describing emulated controls. Exposing two
> controls in schemata file that essentially controls the same thing is what the
> emulated controls aim to solve and the resctrl hierarchies presented in slide #6
> of that presentation (and discussed in the email thread) is how we contemplated how
> to represent the relationship among these controls to user space. So, considering
> your example resctrl may display something like:
>
> info//
> └── MB/
> └── resource_schemata/
> └── MB/
> └── MB_MAX/
>
> Above hierarchy describes the relationship to user space that if MB is changed it
> will impact MB_MAX and vice-versa.
>
> The one open I am aware of surrounding emulated controls is how to present some
> semblance of consistency to user space when considering all the possibilities
> the different architectures (and even within architectures) may have.
What other use cases do we have apart from MB and MB_MAX? I was wondering if
this could be limited to a default control (L2, L3, MB..) with a single new
style control (L2_*, L3_*, MB_ ...) under it.
>
>
>>> - Controls are independent for now. This means that, for example, if a resource
>>> supports a "MIN" and "MAX" control then this implementation would allow user to
>>> set the "maximum" control values to be less than the "minimum" control values.
>>
>> I think this is ok as long as adding support for new controls in resctrl doesn't
>> change the existing behaviour. In MPAM we dodged this by introducing MB as only
>> affecting the h/w mbw_max and not mbw_min (as mentioned above).
>
> I understand this to be a requirement for Intel where the spec contains "The Maximum Cap
> should be programmed to be greater than or equal to the Minimum and Optimal caps.
> Undesirable and undefined performance effects may result if cap programming guidelines
> are not followed."
>
> I am currently thinking that resctrl should not try to be too smart here and if user
> space wants to make dramatic changes to min and max values then it should just ensure
> the ordering is appropriate. For example, attempting to set a new min to be larger than
> the old max would fail and user space should first increase the old max and then set
> a new min.
Ok with me.
>
>>
>>> - PoC supports the "bitmap" control but does not (yet) expose properties of a bitmap
>>> control to the new info/<resource>/resource_schemata directory.
>>>
>>> Accessing PoC
>>> =============
>>>
>>> Please consider the PoC as a rough draft. It has only been compile tested for Arm
>>> and known to be incomplete in Arm support. To help with experimenting I only
>>> fully adapted the Intel MBA resource to demo two dummy additional MBA controls.
>>> All architectures should immediately benefit from the new schema descriptions
>>> and new info/MB/resource_schemata hierarchy.
>>>
>>> I considered the patches self too many for email. Instead, the PoC can be found at:
>>>
>>> git://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git branch resctrl/controls_rfc_v1
>>>
>>> The work is based on v7.1-rc2 that also includes the following series (two of which has
>>> since been queued) included:
>>>
>>> "selftests/resctrl: Fixes and improvements focused on Intel platforms"
>>> https://lore.kernel.org/lkml/cover.1775266384.git.reinette.chatre@intel.com/
>>>
>>> "x86,fs/resctrl: Improve resctrl quality and consistency"
>>> https://lore.kernel.org/lkml/cover.1777419024.git.reinette.chatre@intel.com/
>>>
>>> "x86,fs/resctrl: Pave the way for MPAM counter assignment"
>>> https://lore.kernel.org/lkml/20260506082855.3694761-1-ben.horgan@arm.com/
>>>
>>>
>>> Primary resctrl fs data structure changes
>>> =========================================
>>>
>>> Introduces a control represented by struct resctrl_ctrl that looks as below. To make
>>> the changes easier to follow I kept some of the original names to help communicate
>>> where familiar data structures land.
>>>
>>> What to notice about a control is that it has some common properties required
>>> from all controls (scope, type, etc.) and then depending on the type of control
>>> (RESCTRL_CTRL_BITMAP or RESCTRL_CTRL_SCALAR) there are type specific properties.
>>>
>>> /**
>>> * struct resctrl_ctrl - A resource control
>>> * @entry: List entry of rdt_resource::controls
>>> * @scope: Scope of the resource that this control allocates
>>> * @domains: RCU list of all control domains
>>> * @type: The control type that determines the properties of the control,
>>> * format string for displaying control values to user space, and
>>> * parser of control values provided by user space.
>>> * @name: Name of the control. Appended to final resource name
>>> * (rdt_resource_final::name) to create final schema entry.
>>> * Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
>>> * For example, with resource name "MB" and control name "MAX" the
>>> * schema entry will be "MB_MAX".
>>> * @cache: Cache allocation control properties.
>>> * @membw: Bandwidth control properties.
>>> */
>>> struct resctrl_ctrl {
>>> struct list_head entry;
>>> enum resctrl_scope scope;
>>> struct list_head domains;
>>> enum resctrl_ctrl_type type;
>>> enum resctrl_ctrl_name name;
>>> union {
>>> struct resctrl_cache cache;
>>> struct resctrl_membw membw;
>>> };
>>> };
>>>
>>> Two members summarize how this new structure fits into the rest of resctrl:
>>> a) resctrl_ctrl::entry
>>> Since a resource can support multiple controls there is a new list
>>> in struct rdt_resource named "controls" that contains the list of all
>>> controls supported by the resource.
>>> b) resctrl_ctrl::domains
>>> Instead of the list of control domains belonging to a resource they
>>> now belong to the control self. By doing so resctrl can support resource
>>> controls at different scope for the same resource. This is intended to
>>> support some upcoming MPAM and RISC-V usages.
>>
>> Please can you expand a bit on part b).
>>
>> In an MPAM system we consider 3 resctrl resources, RDT_RESOURCE_L3,
>> RDT_RESOURCE_L2 and RDT_RESOURCE_MBA which correspond to the L3 caches, L2
>> caches and memory bandwidth on egress from the L3 caches. The domain for each of
>> these corresponds to the instance of the resource. That is, for RDT_RESOURCE_L2
>> there is a resource for each L2 instance, similarly for L3, and for
>
> (I'm assuming above is typo and it is "there is a domain for each L2 instance"?)
yes, a mistake
>
>> RDT_RESOURCE_MBA there is a domain for each L3 cache. If we were to add suport
>> for controls on a new cache level, say the L4, then I'd expect to add a new
>> resource. For memory bandwidth, we'd like to be able to control b/w on the L2
>> egress (e.g. in a DSU). Wouldn't this too be a separate resource or would this
>> be a new set of controls on the same resource?
>>
>> New controls on the same resource
>> MB_MIN2
>> MB_MAX2
>> MB_PROP2
>> ...
>>
>> or
>> MB2_MIN
>> MB2_MAX
>> MB2_PROP
>
>
> The way I currently see it is that controlling bandwidth at a different scope would
> be a new set of controls associated with the MB resource. There are more scenarios
> coming this way with AMD's "Global MBA" that is memory bandwidth allocation at
> NUMA node scope. If I understand correctly the "CPU-less Memory Node" that Nvidia
> shared at plumbers would need this also and control memory bandwidth allocation
> at the NUMA node scope.
Yes, in general for MSC at the memory controlers it would be good to scope these
by NUMA node whether or not they are CPU-less or not.
> A related technology is Intel's region-aware MBA, which is
> still at L3 scope.
>
> I fully agree that we need to figure out how to represent all of this to user space
> without turning the interface into something unintelligible. In the end this is
> required for user space to know what a domain ID represents.
>
> Would it help to make the scope part of the control name? The ship has sailed for
> MB being associated with L3 scope but this could mean the "default" scope of MB
> resource is L3 (which user space can still confirm by looking at the control's
> "scope" file) and the others include scope in the name? Consider for example:
> https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@intel.com/
This certainly helps with the naming.
The scope does have an effect on what causes a domain to be present or not. For
existing scopes, such as L3 scope, that whether a domain is online or not is
dependent on whether or not a set cpu is online and the cpu_read_lock is taken.
However, for NUMA scope in MPAM (maybe not GMBA?) then whether or not the domain
is online would need to depend on whether the memory is online or not and the
memory hotplug lock will be needed to be taken. I am wondering if this sort of
configuration means it's better to have the NUMA scoped memory bandwidth on a
different resource or we just say ok and always take the memory hotplug lock ,
get_online_mems(), where we take the cpu_read_lock.
>
>
>>
>> AFAIK, the DSU h/w just supports proportional bandwidth controls at the moment
>> but we should consider what to do about the potential naming.
>
> ack.
>
>>
>> In the MPAM driver, we collect MSC into components (based on instances) and
>> those into classes (components of the same type). Currently, a resource is
>> mapped to a single class. (Two resources may map to the same class.)
>>
>> I expect it is useful in the memory region and sub numa cases but I'd still
>> expect the common case to be that the domains are the same within a control. Or
>> am I missing something?
>
> Domains of a control should all be at the same scope. Since the schemata file
> exposes the control with the different IDs representing the instances of the
> resource needing to be controlled it has to be clear to user space what the
> domain ID represents.
Agreed. (I meant to say the domains within a resource are likely to be the same
for each control within the same resource.)
Thanks,
Ben
>
>>
>>>
>>> Example architectural data structure changes
>>> ============================================
>>>
>>> An architecture can use the new control by following a similar pattern to
>>> resource and domain use by architectures. Consider the following for x86
>>> where a new architecture specific struct resctrl_hw_ctrl includes
>>> struct resctrl_ctrl and any architecture private data needed to support
>>> the control:
>>>
>>> /*
>>> * struct resctrl_hw_ctrl - Arch private properties of a resource control
>>> * @r_ctrl: Control properties exposed to resctrl file system
>>> * @msr_base: Base MSR address where control values should be programmed
>>> * @msr_update: Function pointer to update control values
>>> */
>>> struct resctrl_hw_ctrl {
>>> struct resctrl_ctrl r_ctrl;
>>> unsigned int msr_base;
>>> void (*msr_update)(struct msr_param *m);
>>> };
>>>
>>> Structure of patch series
>>> =========================
>>>
>>> As a PoC the series is not perfectly structured but to help navigate this work
>>> on a high level the changes can be categorized as follows:
>>>
>>> Patch 1 to 11:
>>> With a vision of what a "control" is, remove unused/unnecessary
>>> members, make clear what is a *resource* property vs a *control*
>>> property, do some renaming to help with the PoC.
>>
>> A few of the changes are generic cleanup and could hopefully be dealt with
>> before decisions on the larger PoC are made. I see:
>> fs/resctrl: Remove unused resctrl_membw::mb_map
>> x86,fs/resctrl: Remove "arch_needs_linear"
>> Perhaps a few more.
>
> ack.
>
>>>
>>> Patch 12:
>>> Introduce struct resctrl_ctrl and re-arrange existing struct rdt_resource
>>> members to form part of new rdt_resource::ctrl
>>>
>>> Patch 13 to 44:
>>> A lot of wrangling to introduce struct resctrl_ctrl to all code that needs
>>> to work with a control and/or domain without assuming that the control is
>>> the one and only control embedded in the resource it belongs to. Essentially,
>>> a lot of changes passing the control around in addition to the resource/domain.
>>
>> You mention a few times in the commit message that you expect the cache
>> resources to only have one control. On MPAM we have CMAX (and there looks to be
>> a RISC-V equivalent) where the total number of bytes in the cache for a given
>> closid is limited. The allocation must still respect the CPBM bitmap though.
>> Looking at the code though I don't see much problem in adding this as an
>> additional control. The assumption that these patches is making is not that
>> there is only one control for cache resources but rather that cache portions are
>> managed by the default cache resource control. Am I missing something or does
>> that assessment make sense to you?
>
> Your assessment is correct. There are still a few assumptions built into resctrl
> about there only being a single cache control and it being a bitmap control.
> Since I am not familiar with the other possible cache controls I instead focused
> on isolating the existing cache control. When I/we have better understanding about
> how additional cache controls behave this implementation can be adapted to support
> it.
>
>>
>> I have been looking at adding CMAX control to resctrl and will have a go at
>> basing what I have so far on top of this series.
>
> Thank you!
>
>
> ...
>
>>>> Any feedback is appreciated.
>>
>> Overall, this looks to be a big step in the right direction.
>
> Glad to hear this.
>
> Thank you very much.
>
> Reinette
>
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-05 14:53 ` Ben Horgan
@ 2026-06-05 15:39 ` Reinette Chatre
2026-06-05 16:37 ` Ben Horgan
0 siblings, 1 reply; 66+ messages in thread
From: Reinette Chatre @ 2026-06-05 15:39 UTC (permalink / raw)
To: Ben Horgan, Tony Luck, James Morse, Dave Martin, Babu Moger,
Drew Fustini, Fenghua Yu, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi Ben,
On 6/5/26 7:53 AM, Ben Horgan wrote:
> On 6/4/26 18:43, Reinette Chatre wrote:
>> On 6/3/26 8:15 AM, Ben Horgan wrote:
>>> On 5/29/26 19:06, Reinette Chatre wrote:
...
>>
>>> I plumbed in support for the MB_MIN resource schema which also works under light
>>> testing. The only fs resctrl code change I needed was:
>>>
>>> --- a/include/linux/resctrl.h
>>> +++ b/include/linux/resctrl.h
>>> @@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct
>>> resctrl_ctrl *ctrl)
>>> case RESCTRL_CTRL_BITMAP:
>>> return BIT_MASK(ctrl->cache.cbm_len) - 1;
>>> case RESCTRL_CTRL_SCALAR:
>>> + if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
>>> + return ctrl->membw.min_bw;
>>> +
>>> return ctrl->membw.max_bw;
>>> }
>>>
>>>
>>> At least on MPAM systems, we use a default of 0 for minimum bandwidth controls
>>> as the maximum bandwidth controls only take effect if their value is higher than
>>> the minimum bandwidth value. I have specialised this on the ctrl->name which
>>> breaks your ctrl->type based classification but that's fixable by just adding a
>>> default field to membw.
>>
>> This I am not sure about. In my understanding a typical "default" value means
>> "no throttling" and, at least on Intel, this default hardware state has been
>> summarized as "min" == "max" == "optimal".
>
> Ok, this sounds odd to me but that is probably because I don't know what Intel
> systems do. On MPAM systems a MIN control is a boost rather than a throttling
> control. Although, you can always think of that as throttling the traffic with
> the other PARTIDs.
>
>>
>> Are you saying that on MPAM systems if "min" == "max" then max bandwidth controls
>> do not take effect? Could you please elaborate what happens if "min" == "max"?
>
> Table 5-4 from section 5.2.8 of the IHI0099B.b shows the interaction between the
> min and maximum controls.
>
> If used bandwidth is The preference is Description
> Below the minimum High Only high requests compete with this
> request.
> Above the minimum:
> Below the maximum Medium High requests are serviced first then
>
> this request competes with other
> medium requests.
>
> Above the maximum, Low Requests are not serviced if any high
> when HARDLIM is 0 or medium requests are available.
>
> Above the maximum, None Requests are not serviced
> when HARDLIM is 1
>
> So if we keep the minimum and the maximum controls values always the same then
> all traffic will be given "high" preference until the target bandwidth is
> reached. For some MPAM systems it is recommended to set the minimum value as 5%
> less than the maximum value to get a reliable target bandwidth. As 5% seems
> implementation specific and some systems don't have min controls it seemed
> better to just match the MB control with a maximum bandwidth control and let the
> user have freedom to choose the minimum bandwidth control when MB_MIN support is
> added.
>
> If a default for the minimum of the maximum possible bandwidth is used (100%)
> then any change of the maximum won't have any effect as it's always less than
> minimum (if that's unchanged) and so all traffic is high preference. I now see
> from your reply below that you are planning on not allowing this kind of
> configuration.
>
> If the minimum always tracks the maximum then we lose the distinction between
> medium and high preference traffic and so to reserve some high preference
> bandwidth for one control group we'd have to change the configuration in the
> other controls groups so that they're bandwidth preference is medium (minimum
> value at 0).
I do not think we are talking about the same thing here. I am *not* saying
that minimum and maximum controls should always be the same.
The discussion is about a proposed change to resctrl_get_default_ctrlval(). resctrl
uses this function in two places:
- When creating a new resource group:
The intention here is that when user space creates a new resource group it should
be created with maximum allocations possible. For MBA this means "unthrottled".
After creating the resource group user space can adjust allocations to match
workload requirements.
- When unmounting the resctrl fs.
The intention here is that all controls are set to unthrottled to stop any possible
impact to system when user space stops using resctrl.
resctrl_get_default_ctrlval() is thus intended to support an unthrottled baseline from
where user space can make configuration changes as supported by hardware and required
by workloads.
I see that the MPAM driver internally uses resctrl_get_default_ctrlval() in a couple
of places and I am not considering this usage here. If internally MPAM has other
usages for this function where it does not mean "unthrottled" then perhaps
it would be better to create a new function that matches the usage?
>>>> - No support for "read-modify-write" usage of schemata file. This is where we
>>>> discussed (without agreement) on possibly introducing the "#" prefix to schemata
>>>> file entries. This PoC does not support this prefix and the current assumption/expectation
>>>> is that when user space changes a configuration only the new control values are
>>>> written to schemata file. I thus do not have a plan to support this so please
>>>> share opinions in this regard if you have some.
>>>
>>> There is now less motivation from the MPAM side for this than when this was
>>> initially discussed. In pre-upstream versions of the MPAM patches a change in
>>> the MB resource control value would change both the mpam h/w mbw_min and mbw_max
>>> values but now (on non-broken h/w) we just change the mbw_max. (mbw_min kept at 0).
>>
>> Ah, thanks for the correction. The email I linked above indeed refers to changing
>> both min and max.
>>
>>>
>>> However, it would be useful not to be limited by percentages. In my quick
>>
>> Indeed. Not being limited by percentages while still needing to have a backward
>> compatible user interface is how we ended up with "emulated controls".
>>
>>> experimentation with your patches I used a percentage value for MB_MIN but it
>>> would be best to move away from this. For new controls I think we can mandate
>>> that user space has to discover the resolution from the info directly but how
>>> can we retrofit this. For MPAM, MB and MB_MAX, would control the same things.
>>> Could we just add MB_MAX with a h/w friendly scale and then reflect changes in
>>> MB_MAX in MB and vica versa with MB taking precedent if both are set? Old
>>> software can continue setting MB can move to using MB_MAX and take advantage of
>>> the improved control. (I don't think we should expose the MPAM hardware value
>>> directly as it has confusion over whether all 1s is 100% or not and we'd like to
>>> have something generic and friendly to the user.)
>>
>> Sounds to me as though you are describing emulated controls. Exposing two
>> controls in schemata file that essentially controls the same thing is what the
>> emulated controls aim to solve and the resctrl hierarchies presented in slide #6
>> of that presentation (and discussed in the email thread) is how we contemplated how
>> to represent the relationship among these controls to user space. So, considering
>> your example resctrl may display something like:
>>
>> info//
>> └── MB/
>> └── resource_schemata/
>> └── MB/
>> └── MB_MAX/
>>
>> Above hierarchy describes the relationship to user space that if MB is changed it
>> will impact MB_MAX and vice-versa.
>>
>> The one open I am aware of surrounding emulated controls is how to present some
>> semblance of consistency to user space when considering all the possibilities
>> the different architectures (and even within architectures) may have.
>
> What other use cases do we have apart from MB and MB_MAX? I was wondering if
> this could be limited to a default control (L2, L3, MB..) with a single new
> style control (L2_*, L3_*, MB_ ...) under it.
The motivation for these emulated controls is to not break a user space that does
not understand the "info/<resource>/resource_schemata" interface. At this time
user space expects every resource (not control) to have an entry in the schemata file.
So yes, I also see this as limited to the default control.
Whether it implies that only a single (finer grained/hardware) control would be under
it is not obvious to me since we already had one scenario where a legacy control is
emulated by two hardware controls when considering the example on MPAM where the "MB"
legacy control can be emulated with MPAM's "min" and "max" controls. An additional
complication is that some of these architecture specs describe several controls but have
their implementation as "optional" which presents a challenge when trying to create a
sane and consistent hierarchy.
>>>> - Controls are independent for now. This means that, for example, if a resource
>>>> supports a "MIN" and "MAX" control then this implementation would allow user to
>>>> set the "maximum" control values to be less than the "minimum" control values.
>>>
>>> I think this is ok as long as adding support for new controls in resctrl doesn't
>>> change the existing behaviour. In MPAM we dodged this by introducing MB as only
>>> affecting the h/w mbw_max and not mbw_min (as mentioned above).
>>
>> I understand this to be a requirement for Intel where the spec contains "The Maximum Cap
>> should be programmed to be greater than or equal to the Minimum and Optimal caps.
>> Undesirable and undefined performance effects may result if cap programming guidelines
>> are not followed."
>>
>> I am currently thinking that resctrl should not try to be too smart here and if user
>> space wants to make dramatic changes to min and max values then it should just ensure
>> the ordering is appropriate. For example, attempting to set a new min to be larger than
>> the old max would fail and user space should first increase the old max and then set
>> a new min.
>
> Ok with me.
Thank you for considering this.
...
>>>> Primary resctrl fs data structure changes
>>>> =========================================
>>>>
>>>> Introduces a control represented by struct resctrl_ctrl that looks as below. To make
>>>> the changes easier to follow I kept some of the original names to help communicate
>>>> where familiar data structures land.
>>>>
>>>> What to notice about a control is that it has some common properties required
>>>> from all controls (scope, type, etc.) and then depending on the type of control
>>>> (RESCTRL_CTRL_BITMAP or RESCTRL_CTRL_SCALAR) there are type specific properties.
>>>>
>>>> /**
>>>> * struct resctrl_ctrl - A resource control
>>>> * @entry: List entry of rdt_resource::controls
>>>> * @scope: Scope of the resource that this control allocates
>>>> * @domains: RCU list of all control domains
>>>> * @type: The control type that determines the properties of the control,
>>>> * format string for displaying control values to user space, and
>>>> * parser of control values provided by user space.
>>>> * @name: Name of the control. Appended to final resource name
>>>> * (rdt_resource_final::name) to create final schema entry.
>>>> * Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
>>>> * For example, with resource name "MB" and control name "MAX" the
>>>> * schema entry will be "MB_MAX".
>>>> * @cache: Cache allocation control properties.
>>>> * @membw: Bandwidth control properties.
>>>> */
>>>> struct resctrl_ctrl {
>>>> struct list_head entry;
>>>> enum resctrl_scope scope;
>>>> struct list_head domains;
>>>> enum resctrl_ctrl_type type;
>>>> enum resctrl_ctrl_name name;
>>>> union {
>>>> struct resctrl_cache cache;
>>>> struct resctrl_membw membw;
>>>> };
>>>> };
>>>>
>>>> Two members summarize how this new structure fits into the rest of resctrl:
>>>> a) resctrl_ctrl::entry
>>>> Since a resource can support multiple controls there is a new list
>>>> in struct rdt_resource named "controls" that contains the list of all
>>>> controls supported by the resource.
>>>> b) resctrl_ctrl::domains
>>>> Instead of the list of control domains belonging to a resource they
>>>> now belong to the control self. By doing so resctrl can support resource
>>>> controls at different scope for the same resource. This is intended to
>>>> support some upcoming MPAM and RISC-V usages.
>>>
>>> Please can you expand a bit on part b).
>>>
>>> In an MPAM system we consider 3 resctrl resources, RDT_RESOURCE_L3,
>>> RDT_RESOURCE_L2 and RDT_RESOURCE_MBA which correspond to the L3 caches, L2
>>> caches and memory bandwidth on egress from the L3 caches. The domain for each of
>>> these corresponds to the instance of the resource. That is, for RDT_RESOURCE_L2
>>> there is a resource for each L2 instance, similarly for L3, and for
>>
>> (I'm assuming above is typo and it is "there is a domain for each L2 instance"?)
>
> yes, a mistake
>
>>
>>> RDT_RESOURCE_MBA there is a domain for each L3 cache. If we were to add suport
>>> for controls on a new cache level, say the L4, then I'd expect to add a new
>>> resource. For memory bandwidth, we'd like to be able to control b/w on the L2
>>> egress (e.g. in a DSU). Wouldn't this too be a separate resource or would this
>>> be a new set of controls on the same resource?
>>>
>>> New controls on the same resource
>>> MB_MIN2
>>> MB_MAX2
>>> MB_PROP2
>>> ...
>>>
>>> or
>>> MB2_MIN
>>> MB2_MAX
>>> MB2_PROP
>>
>>
>> The way I currently see it is that controlling bandwidth at a different scope would
>> be a new set of controls associated with the MB resource. There are more scenarios
>> coming this way with AMD's "Global MBA" that is memory bandwidth allocation at
>> NUMA node scope. If I understand correctly the "CPU-less Memory Node" that Nvidia
>> shared at plumbers would need this also and control memory bandwidth allocation
>> at the NUMA node scope.
>
> Yes, in general for MSC at the memory controlers it would be good to scope these
> by NUMA node whether or not they are CPU-less or not.
>
>> A related technology is Intel's region-aware MBA, which is
>> still at L3 scope.
>>
>> I fully agree that we need to figure out how to represent all of this to user space
>> without turning the interface into something unintelligible. In the end this is
>> required for user space to know what a domain ID represents.
>>
>> Would it help to make the scope part of the control name? The ship has sailed for
>> MB being associated with L3 scope but this could mean the "default" scope of MB
>> resource is L3 (which user space can still confirm by looking at the control's
>> "scope" file) and the others include scope in the name? Consider for example:
>> https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@intel.com/
>
> This certainly helps with the naming.
>
> The scope does have an effect on what causes a domain to be present or not. For
> existing scopes, such as L3 scope, that whether a domain is online or not is
> dependent on whether or not a set cpu is online and the cpu_read_lock is taken.
> However, for NUMA scope in MPAM (maybe not GMBA?) then whether or not the domain
> is online would need to depend on whether the memory is online or not and the
> memory hotplug lock will be needed to be taken. I am wondering if this sort of
> configuration means it's better to have the NUMA scoped memory bandwidth on a
> different resource or we just say ok and always take the memory hotplug lock ,
> get_online_mems(), where we take the cpu_read_lock.
oh, thank you for bringing this up. I have not considered how the memory hotplug lock
needs to be integrated. Taking cpus_read_lock() has permeated the entire subsystem.
My initial thought is that having unique per-resource locking sounds complicated while
always taking memory hotplug lock sounds much simpler. I do not see many users of
get_online_mems() though.
>>> AFAIK, the DSU h/w just supports proportional bandwidth controls at the moment
>>> but we should consider what to do about the potential naming.
>>
>> ack.
>>
>>>
>>> In the MPAM driver, we collect MSC into components (based on instances) and
>>> those into classes (components of the same type). Currently, a resource is
>>> mapped to a single class. (Two resources may map to the same class.)
>>>
>>> I expect it is useful in the memory region and sub numa cases but I'd still
>>> expect the common case to be that the domains are the same within a control. Or
>>> am I missing something?
>>
>> Domains of a control should all be at the same scope. Since the schemata file
>> exposes the control with the different IDs representing the instances of the
>> resource needing to be controlled it has to be clear to user space what the
>> domain ID represents.
>
> Agreed. (I meant to say the domains within a resource are likely to be the same
> for each control within the same resource.)
This seems accurate for the resources that have implicit scope (the caches) but
memory bandwidth as a resource is looking more like it needs to support allocation
at different scopes.
Reinette
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-05 15:39 ` Reinette Chatre
@ 2026-06-05 16:37 ` Ben Horgan
2026-06-08 16:16 ` Reinette Chatre
0 siblings, 1 reply; 66+ messages in thread
From: Ben Horgan @ 2026-06-05 16:37 UTC (permalink / raw)
To: Reinette Chatre, Tony Luck, James Morse, Dave Martin, Babu Moger,
Drew Fustini, Fenghua Yu, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi Reinette,
On 6/5/26 16:39, Reinette Chatre wrote:
> Hi Ben,
>
> On 6/5/26 7:53 AM, Ben Horgan wrote:
>> On 6/4/26 18:43, Reinette Chatre wrote:
>>> On 6/3/26 8:15 AM, Ben Horgan wrote:
>>>> On 5/29/26 19:06, Reinette Chatre wrote:
>
> ...
>
>>>
>>>> I plumbed in support for the MB_MIN resource schema which also works under light
>>>> testing. The only fs resctrl code change I needed was:
>>>>
>>>> --- a/include/linux/resctrl.h
>>>> +++ b/include/linux/resctrl.h
>>>> @@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct
>>>> resctrl_ctrl *ctrl)
>>>> case RESCTRL_CTRL_BITMAP:
>>>> return BIT_MASK(ctrl->cache.cbm_len) - 1;
>>>> case RESCTRL_CTRL_SCALAR:
>>>> + if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
>>>> + return ctrl->membw.min_bw;
>>>> +
>>>> return ctrl->membw.max_bw;
>>>> }
>>>>
>>>>
>>>> At least on MPAM systems, we use a default of 0 for minimum bandwidth controls
>>>> as the maximum bandwidth controls only take effect if their value is higher than
>>>> the minimum bandwidth value. I have specialised this on the ctrl->name which
>>>> breaks your ctrl->type based classification but that's fixable by just adding a
>>>> default field to membw.
>>>
>>> This I am not sure about. In my understanding a typical "default" value means
>>> "no throttling" and, at least on Intel, this default hardware state has been
>>> summarized as "min" == "max" == "optimal".
>>
>> Ok, this sounds odd to me but that is probably because I don't know what Intel
>> systems do. On MPAM systems a MIN control is a boost rather than a throttling
>> control. Although, you can always think of that as throttling the traffic with
>> the other PARTIDs.
>>
>>>
>>> Are you saying that on MPAM systems if "min" == "max" then max bandwidth controls
>>> do not take effect? Could you please elaborate what happens if "min" == "max"?
>>
>> Table 5-4 from section 5.2.8 of the IHI0099B.b shows the interaction between the
>> min and maximum controls.
>>
>> If used bandwidth is The preference is Description
>> Below the minimum High Only high requests compete with this
>> request.
>> Above the minimum:
>> Below the maximum Medium High requests are serviced first then
>>
>> this request competes with other
>> medium requests.
>>
>> Above the maximum, Low Requests are not serviced if any high
>> when HARDLIM is 0 or medium requests are available.
>>
>> Above the maximum, None Requests are not serviced
>> when HARDLIM is 1
>>
>> So if we keep the minimum and the maximum controls values always the same then
>> all traffic will be given "high" preference until the target bandwidth is
>> reached. For some MPAM systems it is recommended to set the minimum value as 5%
>> less than the maximum value to get a reliable target bandwidth. As 5% seems
>> implementation specific and some systems don't have min controls it seemed
>> better to just match the MB control with a maximum bandwidth control and let the
>> user have freedom to choose the minimum bandwidth control when MB_MIN support is
>> added.
>>
>> If a default for the minimum of the maximum possible bandwidth is used (100%)
>> then any change of the maximum won't have any effect as it's always less than
>> minimum (if that's unchanged) and so all traffic is high preference. I now see
>> from your reply below that you are planning on not allowing this kind of
>> configuration.
>>
>> If the minimum always tracks the maximum then we lose the distinction between
>> medium and high preference traffic and so to reserve some high preference
>> bandwidth for one control group we'd have to change the configuration in the
>> other controls groups so that they're bandwidth preference is medium (minimum
>> value at 0).
>
> I do not think we are talking about the same thing here. I am *not* saying
> that minimum and maximum controls should always be the same.
>
> The discussion is about a proposed change to resctrl_get_default_ctrlval(). resctrl
> uses this function in two places:
> - When creating a new resource group:
> The intention here is that when user space creates a new resource group it should
> be created with maximum allocations possible. For MBA this means "unthrottled".
I would contend that for minimum controls that a policy of 'maximum allocation
possible' isn't a useful default. I try and explain a bit more below.
> After creating the resource group user space can adjust allocations to match
> workload requirements.
> - When unmounting the resctrl fs.
> The intention here is that all controls are set to unthrottled to stop any possible
> impact to system when user space stops using resctrl.
>
> resctrl_get_default_ctrlval() is thus intended to support an unthrottled baseline from
> where user space can make configuration changes as supported by hardware and required
> by workloads.
The baseline that I see makes most sense for a minimum control is to have the
default as 0. This just means that there is no "guaranteed"/high preference
bandwidth reserved for the control group. I would say this still unthrottled but
just not giving a boost. With this default the user can use MB (backed by max
bandwidth) without having to know about MB_MIN (keeping it constant). If the
default is 100% for min bandwidth then the user needs to know to set MB_MIN to
be able to use MB. Having a default of 100% for max bandwidth, correspondingly
means a user can change MB_MIN and see guaranteed bandwidth effects without
having to know about MB/MB_MAX.
Does this make sense?
>
> I see that the MPAM driver internally uses resctrl_get_default_ctrlval() in a couple
> of places and I am not considering this usage here. If internally MPAM has other
> usages for this function where it does not mean "unthrottled" then perhaps
> it would be better to create a new function that matches the usage?
I don't think the internal usage makes a difference here.
One process thing I was wondering about so that I know how to structure my
patches. In the series you have a few patches which touch all architectures;
these have the prefix mpam,x86,fs/resctrl. Is this how you would like cross
architectures patches to look like or is it just for convenience in the rfc and
a patch per-architecture is preferable?
Thanks,
Ben
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-05 16:37 ` Ben Horgan
@ 2026-06-08 16:16 ` Reinette Chatre
2026-06-09 10:10 ` Ben Horgan
0 siblings, 1 reply; 66+ messages in thread
From: Reinette Chatre @ 2026-06-08 16:16 UTC (permalink / raw)
To: Ben Horgan, Tony Luck, James Morse, Dave Martin, Babu Moger,
Drew Fustini, Fenghua Yu, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi Ben,
On 6/5/26 9:37 AM, Ben Horgan wrote:
> Hi Reinette,
>
> On 6/5/26 16:39, Reinette Chatre wrote:
>> Hi Ben,
>>
>> On 6/5/26 7:53 AM, Ben Horgan wrote:
>>> On 6/4/26 18:43, Reinette Chatre wrote:
>>>> On 6/3/26 8:15 AM, Ben Horgan wrote:
>>>>> On 5/29/26 19:06, Reinette Chatre wrote:
>>
>> ...
>>
>>>>
>>>>> I plumbed in support for the MB_MIN resource schema which also works under light
>>>>> testing. The only fs resctrl code change I needed was:
>>>>>
>>>>> --- a/include/linux/resctrl.h
>>>>> +++ b/include/linux/resctrl.h
>>>>> @@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct
>>>>> resctrl_ctrl *ctrl)
>>>>> case RESCTRL_CTRL_BITMAP:
>>>>> return BIT_MASK(ctrl->cache.cbm_len) - 1;
>>>>> case RESCTRL_CTRL_SCALAR:
>>>>> + if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
>>>>> + return ctrl->membw.min_bw;
>>>>> +
>>>>> return ctrl->membw.max_bw;
>>>>> }
>>>>>
>>>>>
>>>>> At least on MPAM systems, we use a default of 0 for minimum bandwidth controls
>>>>> as the maximum bandwidth controls only take effect if their value is higher than
>>>>> the minimum bandwidth value. I have specialised this on the ctrl->name which
>>>>> breaks your ctrl->type based classification but that's fixable by just adding a
>>>>> default field to membw.
>>>>
>>>> This I am not sure about. In my understanding a typical "default" value means
>>>> "no throttling" and, at least on Intel, this default hardware state has been
>>>> summarized as "min" == "max" == "optimal".
>>>
>>> Ok, this sounds odd to me but that is probably because I don't know what Intel
>>> systems do. On MPAM systems a MIN control is a boost rather than a throttling
>>> control. Although, you can always think of that as throttling the traffic with
>>> the other PARTIDs.
>>>
>>>>
>>>> Are you saying that on MPAM systems if "min" == "max" then max bandwidth controls
>>>> do not take effect? Could you please elaborate what happens if "min" == "max"?
>>>
>>> Table 5-4 from section 5.2.8 of the IHI0099B.b shows the interaction between the
>>> min and maximum controls.
>>>
>>> If used bandwidth is The preference is Description
>>> Below the minimum High Only high requests compete with this
>>> request.
>>> Above the minimum:
>>> Below the maximum Medium High requests are serviced first then
>>>
>>> this request competes with other
>>> medium requests.
>>>
>>> Above the maximum, Low Requests are not serviced if any high
>>> when HARDLIM is 0 or medium requests are available.
>>>
>>> Above the maximum, None Requests are not serviced
>>> when HARDLIM is 1
>>>
>>> So if we keep the minimum and the maximum controls values always the same then
>>> all traffic will be given "high" preference until the target bandwidth is
>>> reached. For some MPAM systems it is recommended to set the minimum value as 5%
>>> less than the maximum value to get a reliable target bandwidth. As 5% seems
>>> implementation specific and some systems don't have min controls it seemed
>>> better to just match the MB control with a maximum bandwidth control and let the
>>> user have freedom to choose the minimum bandwidth control when MB_MIN support is
>>> added.
>>>
>>> If a default for the minimum of the maximum possible bandwidth is used (100%)
>>> then any change of the maximum won't have any effect as it's always less than
>>> minimum (if that's unchanged) and so all traffic is high preference. I now see
>>> from your reply below that you are planning on not allowing this kind of
>>> configuration.
>>>
>>> If the minimum always tracks the maximum then we lose the distinction between
>>> medium and high preference traffic and so to reserve some high preference
>>> bandwidth for one control group we'd have to change the configuration in the
>>> other controls groups so that they're bandwidth preference is medium (minimum
>>> value at 0).
>>
>> I do not think we are talking about the same thing here. I am *not* saying
>> that minimum and maximum controls should always be the same.
>>
>> The discussion is about a proposed change to resctrl_get_default_ctrlval(). resctrl
>> uses this function in two places:
>> - When creating a new resource group:
>> The intention here is that when user space creates a new resource group it should
>> be created with maximum allocations possible. For MBA this means "unthrottled".
>
> I would contend that for minimum controls that a policy of 'maximum allocation
> possible' isn't a useful default. I try and explain a bit more below.
>
>> After creating the resource group user space can adjust allocations to match
>> workload requirements.
>> - When unmounting the resctrl fs.
>> The intention here is that all controls are set to unthrottled to stop any possible
>> impact to system when user space stops using resctrl.
>>
>> resctrl_get_default_ctrlval() is thus intended to support an unthrottled baseline from
>> where user space can make configuration changes as supported by hardware and required
>> by workloads.
>
> The baseline that I see makes most sense for a minimum control is to have the
> default as 0. This just means that there is no "guaranteed"/high preference
> bandwidth reserved for the control group. I would say this still unthrottled but
> just not giving a boost. With this default the user can use MB (backed by max
> bandwidth) without having to know about MB_MIN (keeping it constant). If the
> default is 100% for min bandwidth then the user needs to know to set MB_MIN to
> be able to use MB. Having a default of 100% for max bandwidth, correspondingly
> means a user can change MB_MIN and see guaranteed bandwidth effects without
> having to know about MB/MB_MAX.
>
> Does this make sense?
I see. I've only considered the original scenario where MPAM's MB is emulated with
both MIN and MAX controls based on examples in
https://lore.kernel.org/lkml/aPJP52jXJvRYAjjV@e133380.arm.com/
There seems to be two issues here:
a) Since some systems require MIN to be 5% less than MAX MPAM driver may not know what
the system MIN and MAX difference should be to get optimal bandwidth.
b) Some MPAM systems have MAX control but not the MIN control.
The proposal to only emulate MB with MAX is clear but it is not obvious why, on a
system that support both MIN and MAX, MB cannot be emulated with both as in the
original proposal. Is this motivated by (a) where MPAM driver just does not know
what the "max" of a MIN control value should be? Just emulating MB with MAX control
does not seem to eliminate this problem since between the fs and arch resctrl still needs
to ensure that when user space writes a control value to the MIN control that it is valid
for the underlying MPAM system.
It almost sounds as though there is an attempt to eliminate resctrl's usage of a "max"
value for the MIN control since that is effectively unknown to MPAM but that does
not look possible to me?
>> I see that the MPAM driver internally uses resctrl_get_default_ctrlval() in a couple
>> of places and I am not considering this usage here. If internally MPAM has other
>> usages for this function where it does not mean "unthrottled" then perhaps
>> it would be better to create a new function that matches the usage?
>
> I don't think the internal usage makes a difference here.
Thanks for checking.
>
> One process thing I was wondering about so that I know how to structure my
> patches. In the series you have a few patches which touch all architectures;
> these have the prefix mpam,x86,fs/resctrl. Is this how you would like cross
> architectures patches to look like or is it just for convenience in the rfc and
> a patch per-architecture is preferable?
A per-architecture patch is preferable. I did try to keep changes separate as much
as possible but some places all parts needed to be changed together to ensure that
bisect continue to work. I expect I may have missed some places in the rush to get
the PoC done and those need to be fixed.
Reinette
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-08 16:16 ` Reinette Chatre
@ 2026-06-09 10:10 ` Ben Horgan
2026-06-09 15:28 ` Reinette Chatre
0 siblings, 1 reply; 66+ messages in thread
From: Ben Horgan @ 2026-06-09 10:10 UTC (permalink / raw)
To: Reinette Chatre, Tony Luck, James Morse, Dave Martin, Babu Moger,
Drew Fustini, Fenghua Yu, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi Reinette,
On 6/8/26 17:16, Reinette Chatre wrote:
> Hi Ben,
>
> On 6/5/26 9:37 AM, Ben Horgan wrote:
>> Hi Reinette,
>>
>> On 6/5/26 16:39, Reinette Chatre wrote:
>>> Hi Ben,
>>>
>>> On 6/5/26 7:53 AM, Ben Horgan wrote:
>>>> On 6/4/26 18:43, Reinette Chatre wrote:
>>>>> On 6/3/26 8:15 AM, Ben Horgan wrote:
>>>>>> On 5/29/26 19:06, Reinette Chatre wrote:
>>>
>>> ...
>>>
>>>>>
>>>>>> I plumbed in support for the MB_MIN resource schema which also works under light
>>>>>> testing. The only fs resctrl code change I needed was:
>>>>>>
>>>>>> --- a/include/linux/resctrl.h
>>>>>> +++ b/include/linux/resctrl.h
>>>>>> @@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct
>>>>>> resctrl_ctrl *ctrl)
>>>>>> case RESCTRL_CTRL_BITMAP:
>>>>>> return BIT_MASK(ctrl->cache.cbm_len) - 1;
>>>>>> case RESCTRL_CTRL_SCALAR:
>>>>>> + if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
>>>>>> + return ctrl->membw.min_bw;
>>>>>> +
>>>>>> return ctrl->membw.max_bw;
>>>>>> }
>>>>>>
>>>>>>
>>>>>> At least on MPAM systems, we use a default of 0 for minimum bandwidth controls
>>>>>> as the maximum bandwidth controls only take effect if their value is higher than
>>>>>> the minimum bandwidth value. I have specialised this on the ctrl->name which
>>>>>> breaks your ctrl->type based classification but that's fixable by just adding a
>>>>>> default field to membw.
>>>>>
>>>>> This I am not sure about. In my understanding a typical "default" value means
>>>>> "no throttling" and, at least on Intel, this default hardware state has been
>>>>> summarized as "min" == "max" == "optimal".
>>>>
>>>> Ok, this sounds odd to me but that is probably because I don't know what Intel
>>>> systems do. On MPAM systems a MIN control is a boost rather than a throttling
>>>> control. Although, you can always think of that as throttling the traffic with
>>>> the other PARTIDs.
>>>>
>>>>>
>>>>> Are you saying that on MPAM systems if "min" == "max" then max bandwidth controls
>>>>> do not take effect? Could you please elaborate what happens if "min" == "max"?
>>>>
>>>> Table 5-4 from section 5.2.8 of the IHI0099B.b shows the interaction between the
>>>> min and maximum controls.
>>>>
>>>> If used bandwidth is The preference is Description
>>>> Below the minimum High Only high requests compete with this
>>>> request.
>>>> Above the minimum:
>>>> Below the maximum Medium High requests are serviced first then
>>>>
>>>> this request competes with other
>>>> medium requests.
>>>>
>>>> Above the maximum, Low Requests are not serviced if any high
>>>> when HARDLIM is 0 or medium requests are available.
>>>>
>>>> Above the maximum, None Requests are not serviced
>>>> when HARDLIM is 1
>>>>
>>>> So if we keep the minimum and the maximum controls values always the same then
>>>> all traffic will be given "high" preference until the target bandwidth is
>>>> reached. For some MPAM systems it is recommended to set the minimum value as 5%
>>>> less than the maximum value to get a reliable target bandwidth. As 5% seems
>>>> implementation specific and some systems don't have min controls it seemed
>>>> better to just match the MB control with a maximum bandwidth control and let the
>>>> user have freedom to choose the minimum bandwidth control when MB_MIN support is
>>>> added.
>>>>
>>>> If a default for the minimum of the maximum possible bandwidth is used (100%)
>>>> then any change of the maximum won't have any effect as it's always less than
>>>> minimum (if that's unchanged) and so all traffic is high preference. I now see
>>>> from your reply below that you are planning on not allowing this kind of
>>>> configuration.
>>>>
>>>> If the minimum always tracks the maximum then we lose the distinction between
>>>> medium and high preference traffic and so to reserve some high preference
>>>> bandwidth for one control group we'd have to change the configuration in the
>>>> other controls groups so that they're bandwidth preference is medium (minimum
>>>> value at 0).
>>>
>>> I do not think we are talking about the same thing here. I am *not* saying
>>> that minimum and maximum controls should always be the same.
>>>
>>> The discussion is about a proposed change to resctrl_get_default_ctrlval(). resctrl
>>> uses this function in two places:
>>> - When creating a new resource group:
>>> The intention here is that when user space creates a new resource group it should
>>> be created with maximum allocations possible. For MBA this means "unthrottled".
>>
>> I would contend that for minimum controls that a policy of 'maximum allocation
>> possible' isn't a useful default. I try and explain a bit more below.
>>
>>> After creating the resource group user space can adjust allocations to match
>>> workload requirements.
>>> - When unmounting the resctrl fs.
>>> The intention here is that all controls are set to unthrottled to stop any possible
>>> impact to system when user space stops using resctrl.
>>>
>>> resctrl_get_default_ctrlval() is thus intended to support an unthrottled baseline from
>>> where user space can make configuration changes as supported by hardware and required
>>> by workloads.
>>
>> The baseline that I see makes most sense for a minimum control is to have the
>> default as 0. This just means that there is no "guaranteed"/high preference
>> bandwidth reserved for the control group. I would say this still unthrottled but
>> just not giving a boost. With this default the user can use MB (backed by max
>> bandwidth) without having to know about MB_MIN (keeping it constant). If the
>> default is 100% for min bandwidth then the user needs to know to set MB_MIN to
>> be able to use MB. Having a default of 100% for max bandwidth, correspondingly
>> means a user can change MB_MIN and see guaranteed bandwidth effects without
>> having to know about MB/MB_MAX.
>>
>> Does this make sense?
>
> I see. I've only considered the original scenario where MPAM's MB is emulated with
> both MIN and MAX controls based on examples in
> https://lore.kernel.org/lkml/aPJP52jXJvRYAjjV@e133380.arm.com/
Ok, but I'm not sure why that's relevant. I feel we are talking at cross
purposely here and so I'll try and answer your questions but I'm likely missing
the point.
>
> There seems to be two issues here:
> a) Since some systems require MIN to be 5% less than MAX MPAM driver may not know what
> the system MIN and MAX difference should be to get optimal bandwidth.
> b) Some MPAM systems have MAX control but not the MIN control.
>
> The proposal to only emulate MB with MAX is clear but it is not obvious why, on a
> system that support both MIN and MAX, MB cannot be emulated with both as in the
> original proposal.
I don't see the advantage of emulating MB with both MIN and MAX. Just going by
the MPAM specification, a system keeping MIN at 0 and just setting MAX from MB,
(MIN=0, MAX=MB) should behave the same as one always setting both, (MIN=MB,
MAX=MB). In the MIN=0 case there is never any high preference traffic and in the
MIN=MAX_MB case there is never any medium preference traffic. It seemed best to
not rely on any platform specific heuristics to try and guess what's better and
just wait til the time we could support MB_MIN in resctrl (and leave the
decision up to the user). My expectation was that this would be the simplest
course of action.
Is this motivated by (a) where MPAM driver just does not know
> what the "max" of a MIN control value should be?
What's the "max" of a MIN control?
The maximum value we can set it to?
We can work that. For MPAM it's writing all 1s to the register which for the
minumum case represents ((2**mbw_min_wd)/2**mbw_min_wd)) * 100 %
Just emulating MB with MAX control
> does not seem to eliminate this problem since between the fs and arch resctrl still needs
> to ensure that when user space writes a control value to the MIN control that it is valid
> for the underlying MPAM system.
>
> It almost sounds as though there is an attempt to eliminate resctrl's usage of a "max"
> value for the MIN control since that is effectively unknown to MPAM but that does
> not look possible to me?
Sorry but I haven't understood what your saying. What does "resctrl's usage of a
"max" value for MIN control" mean?
>
>>> I see that the MPAM driver internally uses resctrl_get_default_ctrlval() in a couple
>>> of places and I am not considering this usage here. If internally MPAM has other
>>> usages for this function where it does not mean "unthrottled" then perhaps
>>> it would be better to create a new function that matches the usage?
>>
>> I don't think the internal usage makes a difference here.
>
> Thanks for checking.
>
>>
>> One process thing I was wondering about so that I know how to structure my
>> patches. In the series you have a few patches which touch all architectures;
>> these have the prefix mpam,x86,fs/resctrl. Is this how you would like cross
>> architectures patches to look like or is it just for convenience in the rfc and
>> a patch per-architecture is preferable?
>
> A per-architecture patch is preferable. I did try to keep changes separate as much
> as possible but some places all parts needed to be changed together to ensure that
> bisect continue to work. I expect I may have missed some places in the rush to get
> the PoC done and those need to be fixed.
I haven't checked if what you have could be better split but getting it out
without worrying too much about such things seems the right call for the PoC.
Thanks,
Ben
>
> Reinette
>
>
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-09 10:10 ` Ben Horgan
@ 2026-06-09 15:28 ` Reinette Chatre
2026-06-09 16:37 ` Ben Horgan
2026-06-10 4:31 ` Drew Fustini
0 siblings, 2 replies; 66+ messages in thread
From: Reinette Chatre @ 2026-06-09 15:28 UTC (permalink / raw)
To: Ben Horgan, Tony Luck, James Morse, Dave Martin, Babu Moger,
Drew Fustini, Fenghua Yu, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi Ben,
On 6/9/26 3:10 AM, Ben Horgan wrote:
> On 6/8/26 17:16, Reinette Chatre wrote:
>> On 6/5/26 9:37 AM, Ben Horgan wrote:
>>> On 6/5/26 16:39, Reinette Chatre wrote:
>>>> Hi Ben,
>>>>
>>>> On 6/5/26 7:53 AM, Ben Horgan wrote:
>>>>> On 6/4/26 18:43, Reinette Chatre wrote:
>>>>>> On 6/3/26 8:15 AM, Ben Horgan wrote:
>>>>>>> On 5/29/26 19:06, Reinette Chatre wrote:
>>>>
>>>> ...
>>>>
>>>>>>
>>>>>>> I plumbed in support for the MB_MIN resource schema which also works under light
>>>>>>> testing. The only fs resctrl code change I needed was:
>>>>>>>
>>>>>>> --- a/include/linux/resctrl.h
>>>>>>> +++ b/include/linux/resctrl.h
>>>>>>> @@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct
>>>>>>> resctrl_ctrl *ctrl)
>>>>>>> case RESCTRL_CTRL_BITMAP:
>>>>>>> return BIT_MASK(ctrl->cache.cbm_len) - 1;
>>>>>>> case RESCTRL_CTRL_SCALAR:
>>>>>>> + if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
>>>>>>> + return ctrl->membw.min_bw;
>>>>>>> +
>>>>>>> return ctrl->membw.max_bw;
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> At least on MPAM systems, we use a default of 0 for minimum bandwidth controls
>>>>>>> as the maximum bandwidth controls only take effect if their value is higher than
>>>>>>> the minimum bandwidth value. I have specialised this on the ctrl->name which
>>>>>>> breaks your ctrl->type based classification but that's fixable by just adding a
>>>>>>> default field to membw.
>>>>>>
>>>>>> This I am not sure about. In my understanding a typical "default" value means
>>>>>> "no throttling" and, at least on Intel, this default hardware state has been
>>>>>> summarized as "min" == "max" == "optimal".
>>>>>
>>>>> Ok, this sounds odd to me but that is probably because I don't know what Intel
>>>>> systems do. On MPAM systems a MIN control is a boost rather than a throttling
>>>>> control. Although, you can always think of that as throttling the traffic with
>>>>> the other PARTIDs.
>>>>>
>>>>>>
>>>>>> Are you saying that on MPAM systems if "min" == "max" then max bandwidth controls
>>>>>> do not take effect? Could you please elaborate what happens if "min" == "max"?
>>>>>
>>>>> Table 5-4 from section 5.2.8 of the IHI0099B.b shows the interaction between the
>>>>> min and maximum controls.
>>>>>
>>>>> If used bandwidth is The preference is Description
>>>>> Below the minimum High Only high requests compete with this
>>>>> request.
>>>>> Above the minimum:
>>>>> Below the maximum Medium High requests are serviced first then
>>>>>
>>>>> this request competes with other
>>>>> medium requests.
>>>>>
>>>>> Above the maximum, Low Requests are not serviced if any high
>>>>> when HARDLIM is 0 or medium requests are available.
>>>>>
>>>>> Above the maximum, None Requests are not serviced
>>>>> when HARDLIM is 1
>>>>>
>>>>> So if we keep the minimum and the maximum controls values always the same then
>>>>> all traffic will be given "high" preference until the target bandwidth is
>>>>> reached. For some MPAM systems it is recommended to set the minimum value as 5%
>>>>> less than the maximum value to get a reliable target bandwidth. As 5% seems
>>>>> implementation specific and some systems don't have min controls it seemed
>>>>> better to just match the MB control with a maximum bandwidth control and let the
>>>>> user have freedom to choose the minimum bandwidth control when MB_MIN support is
>>>>> added.
>>>>>
>>>>> If a default for the minimum of the maximum possible bandwidth is used (100%)
>>>>> then any change of the maximum won't have any effect as it's always less than
>>>>> minimum (if that's unchanged) and so all traffic is high preference. I now see
>>>>> from your reply below that you are planning on not allowing this kind of
>>>>> configuration.
>>>>>
>>>>> If the minimum always tracks the maximum then we lose the distinction between
>>>>> medium and high preference traffic and so to reserve some high preference
>>>>> bandwidth for one control group we'd have to change the configuration in the
>>>>> other controls groups so that they're bandwidth preference is medium (minimum
>>>>> value at 0).
>>>>
>>>> I do not think we are talking about the same thing here. I am *not* saying
>>>> that minimum and maximum controls should always be the same.
>>>>
>>>> The discussion is about a proposed change to resctrl_get_default_ctrlval(). resctrl
>>>> uses this function in two places:
>>>> - When creating a new resource group:
>>>> The intention here is that when user space creates a new resource group it should
>>>> be created with maximum allocations possible. For MBA this means "unthrottled".
>>>
>>> I would contend that for minimum controls that a policy of 'maximum allocation
>>> possible' isn't a useful default. I try and explain a bit more below.
>>>
>>>> After creating the resource group user space can adjust allocations to match
>>>> workload requirements.
>>>> - When unmounting the resctrl fs.
>>>> The intention here is that all controls are set to unthrottled to stop any possible
>>>> impact to system when user space stops using resctrl.
>>>>
>>>> resctrl_get_default_ctrlval() is thus intended to support an unthrottled baseline from
>>>> where user space can make configuration changes as supported by hardware and required
>>>> by workloads.
>>>
>>> The baseline that I see makes most sense for a minimum control is to have the
>>> default as 0. This just means that there is no "guaranteed"/high preference
>>> bandwidth reserved for the control group. I would say this still unthrottled but
>>> just not giving a boost. With this default the user can use MB (backed by max
>>> bandwidth) without having to know about MB_MIN (keeping it constant). If the
>>> default is 100% for min bandwidth then the user needs to know to set MB_MIN to
>>> be able to use MB. Having a default of 100% for max bandwidth, correspondingly
>>> means a user can change MB_MIN and see guaranteed bandwidth effects without
>>> having to know about MB/MB_MAX.
>>>
>>> Does this make sense?
>>
>> I see. I've only considered the original scenario where MPAM's MB is emulated with
>> both MIN and MAX controls based on examples in
>> https://lore.kernel.org/lkml/aPJP52jXJvRYAjjV@e133380.arm.com/
>
> Ok, but I'm not sure why that's relevant. I feel we are talking at cross
> purposely here and so I'll try and answer your questions but I'm likely missing
> the point.
>
>>
>> There seems to be two issues here:
>> a) Since some systems require MIN to be 5% less than MAX MPAM driver may not know what
>> the system MIN and MAX difference should be to get optimal bandwidth.
>> b) Some MPAM systems have MAX control but not the MIN control.
>>
>> The proposal to only emulate MB with MAX is clear but it is not obvious why, on a
>> system that support both MIN and MAX, MB cannot be emulated with both as in the
>> original proposal.
>
> I don't see the advantage of emulating MB with both MIN and MAX. Just going by
> the MPAM specification, a system keeping MIN at 0 and just setting MAX from MB,
> (MIN=0, MAX=MB) should behave the same as one always setting both, (MIN=MB,
> MAX=MB). In the MIN=0 case there is never any high preference traffic and in the
> MIN=MAX_MB case there is never any medium preference traffic. It seemed best to
> not rely on any platform specific heuristics to try and guess what's better and
> just wait til the time we could support MB_MIN in resctrl (and leave the
> decision up to the user). My expectation was that this would be the simplest
> course of action.
This sounds fair. Two observations:
- The hierarchy exposed by resctrl may be different on systems that have the "same"
controls.
For example, on an MPAM system (if I understand correctly) the user may see:
info/
└── MB/
└── resource_schemata/
├── MB/
│ └── MB_MAX/
└── MB_MIN/
Compared with a possible implementation on Intel that looks like:
info/
└── MB/
└── resource_schemata/
├── MB/
│ └── MB_OPT/
├── MB_MAX/
└── MB_MIN/
On Intel these controls are optional so could even look more like MPAM:
info/
└── MB/
└── resource_schemata/
└── MB/
└── MB_MAX/
Dave Martin had some musings about how to present controls to user space with
some semblance of consistency but we were not able to finalize that.
At this time it seems that there may be agreement on needing hierarchies as
above but I am concerned about the complicated interface that results from
inconsistencies between systems. While complicated it does represent the relationships
accurately and thus could also be seen as working as intended ... so perhaps we
just need to take great care in documenting this hierarchy mechanism as the
source for control information as opposed to the schemata file.
- I see that the default of zero for the MIN control can be appropriate. For the
usage of resctrl_get_default_ctrlval() when creating a new control group this
could be ok as a general setting since the user is expected to, after creating
the resource group, adjust control values as required by the workloads it will
contain. I need to make sure about this from all architectures and if 0 is not
ok then we could use Drew's suggestion of architecture providing a specific reset
value.
For the other usage of resctrl_get_default_ctrlval() that resets controls on
unmount there is not an opportunity for user space to adjust and here more
care needs to be taken to match architecture requirements. The architecture
provided "reset" value does sound more appealing.
>
> Is this motivated by (a) where MPAM driver just does not know
>> what the "max" of a MIN control value should be?
>
> What's the "max" of a MIN control?
>
> The maximum value we can set it to?
The maximum value a user can set it to. I understood from an earlier comment that there
are some MPAM systems where the MIN needs to be set to an implementation specific 5% less
than MAX. I interpreted this to mean that it is difficult for the MPAM driver to always know
what the maximum value is that a user can write to the MIN control.
>
> We can work that. For MPAM it's writing all 1s to the register which for the
> minumum case represents ((2**mbw_min_wd)/2**mbw_min_wd)) * 100 %
>
> Just emulating MB with MAX control
>> does not seem to eliminate this problem since between the fs and arch resctrl still needs
>> to ensure that when user space writes a control value to the MIN control that it is valid
>> for the underlying MPAM system.
>>
>> It almost sounds as though there is an attempt to eliminate resctrl's usage of a "max"
>> value for the MIN control since that is effectively unknown to MPAM but that does
>> not look possible to me?
>
> Sorry but I haven't understood what your saying. What does "resctrl's usage of a
> "max" value for MIN control" mean?
Basically it is resctrl fs's validation of user input. Specifically, in bw_validate() where
the fs does this range check:
if (bw < r->membw.min_bw || bw > r->membw.max_bw)
resctrl fs thus uses the "max" value of a control for user input checking and it seemed to
me that it may be difficult for MPAM to lean that "max" from all systems but it sounds as
though the plan is instead to use the max that the architecture supports?
Reinette
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-09 15:28 ` Reinette Chatre
@ 2026-06-09 16:37 ` Ben Horgan
2026-06-09 17:41 ` Reinette Chatre
2026-06-10 4:31 ` Drew Fustini
1 sibling, 1 reply; 66+ messages in thread
From: Ben Horgan @ 2026-06-09 16:37 UTC (permalink / raw)
To: Reinette Chatre, Tony Luck, James Morse, Dave Martin, Babu Moger,
Drew Fustini, Fenghua Yu, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi Reinette,
On 6/9/26 16:28, Reinette Chatre wrote:
> Hi Ben,
>
> On 6/9/26 3:10 AM, Ben Horgan wrote:
>> On 6/8/26 17:16, Reinette Chatre wrote:
>>> On 6/5/26 9:37 AM, Ben Horgan wrote:
>>>> On 6/5/26 16:39, Reinette Chatre wrote:
>>>>> Hi Ben,
>>>>>
>>>>> On 6/5/26 7:53 AM, Ben Horgan wrote:
>>>>>> On 6/4/26 18:43, Reinette Chatre wrote:
>>>>>>> On 6/3/26 8:15 AM, Ben Horgan wrote:
>>>>>>>> On 5/29/26 19:06, Reinette Chatre wrote:
>>>>>
>>>>> ...
>>>>>
>>>>>>>
>>>>>>>> I plumbed in support for the MB_MIN resource schema which also works under light
>>>>>>>> testing. The only fs resctrl code change I needed was:
>>>>>>>>
>>>>>>>> --- a/include/linux/resctrl.h
>>>>>>>> +++ b/include/linux/resctrl.h
>>>>>>>> @@ -483,6 +483,9 @@ static inline u32 resctrl_get_default_ctrlval(struct
>>>>>>>> resctrl_ctrl *ctrl)
>>>>>>>> case RESCTRL_CTRL_BITMAP:
>>>>>>>> return BIT_MASK(ctrl->cache.cbm_len) - 1;
>>>>>>>> case RESCTRL_CTRL_SCALAR:
>>>>>>>> + if (ctrl->name == RESCTRL_CTRL_NAME_MIN)
>>>>>>>> + return ctrl->membw.min_bw;
>>>>>>>> +
>>>>>>>> return ctrl->membw.max_bw;
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> At least on MPAM systems, we use a default of 0 for minimum bandwidth controls
>>>>>>>> as the maximum bandwidth controls only take effect if their value is higher than
>>>>>>>> the minimum bandwidth value. I have specialised this on the ctrl->name which
>>>>>>>> breaks your ctrl->type based classification but that's fixable by just adding a
>>>>>>>> default field to membw.
>>>>>>>
>>>>>>> This I am not sure about. In my understanding a typical "default" value means
>>>>>>> "no throttling" and, at least on Intel, this default hardware state has been
>>>>>>> summarized as "min" == "max" == "optimal".
>>>>>>
>>>>>> Ok, this sounds odd to me but that is probably because I don't know what Intel
>>>>>> systems do. On MPAM systems a MIN control is a boost rather than a throttling
>>>>>> control. Although, you can always think of that as throttling the traffic with
>>>>>> the other PARTIDs.
>>>>>>
>>>>>>>
>>>>>>> Are you saying that on MPAM systems if "min" == "max" then max bandwidth controls
>>>>>>> do not take effect? Could you please elaborate what happens if "min" == "max"?
>>>>>>
>>>>>> Table 5-4 from section 5.2.8 of the IHI0099B.b shows the interaction between the
>>>>>> min and maximum controls.
>>>>>>
>>>>>> If used bandwidth is The preference is Description
>>>>>> Below the minimum High Only high requests compete with this
>>>>>> request.
>>>>>> Above the minimum:
>>>>>> Below the maximum Medium High requests are serviced first then
>>>>>>
>>>>>> this request competes with other
>>>>>> medium requests.
>>>>>>
>>>>>> Above the maximum, Low Requests are not serviced if any high
>>>>>> when HARDLIM is 0 or medium requests are available.
>>>>>>
>>>>>> Above the maximum, None Requests are not serviced
>>>>>> when HARDLIM is 1
>>>>>>
>>>>>> So if we keep the minimum and the maximum controls values always the same then
>>>>>> all traffic will be given "high" preference until the target bandwidth is
>>>>>> reached. For some MPAM systems it is recommended to set the minimum value as 5%
>>>>>> less than the maximum value to get a reliable target bandwidth. As 5% seems
>>>>>> implementation specific and some systems don't have min controls it seemed
>>>>>> better to just match the MB control with a maximum bandwidth control and let the
>>>>>> user have freedom to choose the minimum bandwidth control when MB_MIN support is
>>>>>> added.
>>>>>>
>>>>>> If a default for the minimum of the maximum possible bandwidth is used (100%)
>>>>>> then any change of the maximum won't have any effect as it's always less than
>>>>>> minimum (if that's unchanged) and so all traffic is high preference. I now see
>>>>>> from your reply below that you are planning on not allowing this kind of
>>>>>> configuration.
>>>>>>
>>>>>> If the minimum always tracks the maximum then we lose the distinction between
>>>>>> medium and high preference traffic and so to reserve some high preference
>>>>>> bandwidth for one control group we'd have to change the configuration in the
>>>>>> other controls groups so that they're bandwidth preference is medium (minimum
>>>>>> value at 0).
>>>>>
>>>>> I do not think we are talking about the same thing here. I am *not* saying
>>>>> that minimum and maximum controls should always be the same.
>>>>>
>>>>> The discussion is about a proposed change to resctrl_get_default_ctrlval(). resctrl
>>>>> uses this function in two places:
>>>>> - When creating a new resource group:
>>>>> The intention here is that when user space creates a new resource group it should
>>>>> be created with maximum allocations possible. For MBA this means "unthrottled".
>>>>
>>>> I would contend that for minimum controls that a policy of 'maximum allocation
>>>> possible' isn't a useful default. I try and explain a bit more below.
>>>>
>>>>> After creating the resource group user space can adjust allocations to match
>>>>> workload requirements.
>>>>> - When unmounting the resctrl fs.
>>>>> The intention here is that all controls are set to unthrottled to stop any possible
>>>>> impact to system when user space stops using resctrl.
>>>>>
>>>>> resctrl_get_default_ctrlval() is thus intended to support an unthrottled baseline from
>>>>> where user space can make configuration changes as supported by hardware and required
>>>>> by workloads.
>>>>
>>>> The baseline that I see makes most sense for a minimum control is to have the
>>>> default as 0. This just means that there is no "guaranteed"/high preference
>>>> bandwidth reserved for the control group. I would say this still unthrottled but
>>>> just not giving a boost. With this default the user can use MB (backed by max
>>>> bandwidth) without having to know about MB_MIN (keeping it constant). If the
>>>> default is 100% for min bandwidth then the user needs to know to set MB_MIN to
>>>> be able to use MB. Having a default of 100% for max bandwidth, correspondingly
>>>> means a user can change MB_MIN and see guaranteed bandwidth effects without
>>>> having to know about MB/MB_MAX.
>>>>
>>>> Does this make sense?
>>>
>>> I see. I've only considered the original scenario where MPAM's MB is emulated with
>>> both MIN and MAX controls based on examples in
>>> https://lore.kernel.org/lkml/aPJP52jXJvRYAjjV@e133380.arm.com/
>>
>> Ok, but I'm not sure why that's relevant. I feel we are talking at cross
>> purposely here and so I'll try and answer your questions but I'm likely missing
>> the point.
>>
>>>
>>> There seems to be two issues here:
>>> a) Since some systems require MIN to be 5% less than MAX MPAM driver may not know what
>>> the system MIN and MAX difference should be to get optimal bandwidth.
>>> b) Some MPAM systems have MAX control but not the MIN control.
>>>
>>> The proposal to only emulate MB with MAX is clear but it is not obvious why, on a
>>> system that support both MIN and MAX, MB cannot be emulated with both as in the
>>> original proposal.
>>
>> I don't see the advantage of emulating MB with both MIN and MAX. Just going by
>> the MPAM specification, a system keeping MIN at 0 and just setting MAX from MB,
>> (MIN=0, MAX=MB) should behave the same as one always setting both, (MIN=MB,
>> MAX=MB). In the MIN=0 case there is never any high preference traffic and in the
>> MIN=MAX_MB case there is never any medium preference traffic. It seemed best to
>> not rely on any platform specific heuristics to try and guess what's better and
>> just wait til the time we could support MB_MIN in resctrl (and leave the
>> decision up to the user). My expectation was that this would be the simplest
>> course of action.
>
> This sounds fair. Two observations:
> - The hierarchy exposed by resctrl may be different on systems that have the "same"
> controls.
> For example, on an MPAM system (if I understand correctly) the user may see:
> info/
> └── MB/
> └── resource_schemata/
> ├── MB/
> │ └── MB_MAX/
> └── MB_MIN/
Yes, this matches my understanding.
>
> Compared with a possible implementation on Intel that looks like:
> info/
> └── MB/
> └── resource_schemata/
> ├── MB/
> │ └── MB_OPT/
> ├── MB_MAX/
> └── MB_MIN/
Not sure if my understanding is correct here...
In the kernel today is it rdt max that backs MB? (Ignoring the sw controller)
If so wouldn't the meaning of MB change within the same platform on a kernel
upgrade once the rdt optimal support is added?
> > On Intel these controls are optional so could even look more like MPAM:
> info/
> └── MB/
> └── resource_schemata/
> └── MB/
> └── MB_MAX/
>
> Dave Martin had some musings about how to present controls to user space with
> some semblance of consistency but we were not able to finalize that.
> At this time it seems that there may be agreement on needing hierarchies as
> above but I am concerned about the complicated interface that results from
> inconsistencies between systems. While complicated it does represent the relationships
> accurately and thus could also be seen as working as intended ... so perhaps we
> just need to take great care in documenting this hierarchy mechanism as the
> source for control information as opposed to the schemata file.
>
> - I see that the default of zero for the MIN control can be appropriate. For the
> usage of resctrl_get_default_ctrlval() when creating a new control group this
> could be ok as a general setting since the user is expected to, after creating
> the resource group, adjust control values as required by the workloads it will
> contain. I need to make sure about this from all architectures and if 0 is not
> ok then we could use Drew's suggestion of architecture providing a specific reset
> value.
> For the other usage of resctrl_get_default_ctrlval() that resets controls on
> unmount there is not an opportunity for user space to adjust and here more
> care needs to be taken to match architecture requirements. The architecture
> provided "reset" value does sound more appealing.
Yes, if 0 isn't generally applicable we'll need an architecture provided value.
>
>>
>> Is this motivated by (a) where MPAM driver just does not know
>>> what the "max" of a MIN control value should be?
>>
>> What's the "max" of a MIN control?
>>
>> The maximum value we can set it to?
>
> The maximum value a user can set it to. I understood from an earlier comment that there
> are some MPAM systems where the MIN needs to be set to an implementation specific 5% less
> than MAX. I interpreted this to mean that it is difficult for the MPAM driver to always know
> what the maximum value is that a user can write to the MIN control.
Sorry for the confusion.
>
>>
>> We can work that. For MPAM it's writing all 1s to the register which for the
>> minumum case represents ((2**mbw_min_wd)/2**mbw_min_wd)) * 100 %
>>
>> Just emulating MB with MAX control
>>> does not seem to eliminate this problem since between the fs and arch resctrl still needs
>>> to ensure that when user space writes a control value to the MIN control that it is valid
>>> for the underlying MPAM system.
>>>
>>> It almost sounds as though there is an attempt to eliminate resctrl's usage of a "max"
>>> value for the MIN control since that is effectively unknown to MPAM but that does
>>> not look possible to me?
>>
>> Sorry but I haven't understood what your saying. What does "resctrl's usage of a
>> "max" value for MIN control" mean?
>
> Basically it is resctrl fs's validation of user input. Specifically, in bw_validate() where
> the fs does this range check:
> if (bw < r->membw.min_bw || bw > r->membw.max_bw)
>
> resctrl fs thus uses the "max" value of a control for user input checking and it seemed to
> me that it may be difficult for MPAM to lean that "max" from all systems but it sounds as
> though the plan is instead to use the max that the architecture supports?
Yes, can just size the max based on the number of configuration bits in the h/w.
Thanks,
Ben
>
>
> Reinette
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-09 16:37 ` Ben Horgan
@ 2026-06-09 17:41 ` Reinette Chatre
2026-06-10 7:09 ` Chen, Yu C
0 siblings, 1 reply; 66+ messages in thread
From: Reinette Chatre @ 2026-06-09 17:41 UTC (permalink / raw)
To: Ben Horgan, Tony Luck, James Morse, Dave Martin, Babu Moger,
Drew Fustini, Fenghua Yu, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi Ben,
On 6/9/26 9:37 AM, Ben Horgan wrote:
> On 6/9/26 16:28, Reinette Chatre wrote:
>> On 6/9/26 3:10 AM, Ben Horgan wrote:
>>> On 6/8/26 17:16, Reinette Chatre wrote:
>>> I don't see the advantage of emulating MB with both MIN and MAX. Just going by
>>> the MPAM specification, a system keeping MIN at 0 and just setting MAX from MB,
>>> (MIN=0, MAX=MB) should behave the same as one always setting both, (MIN=MB,
>>> MAX=MB). In the MIN=0 case there is never any high preference traffic and in the
>>> MIN=MAX_MB case there is never any medium preference traffic. It seemed best to
>>> not rely on any platform specific heuristics to try and guess what's better and
>>> just wait til the time we could support MB_MIN in resctrl (and leave the
>>> decision up to the user). My expectation was that this would be the simplest
>>> course of action.
>>
>> This sounds fair. Two observations:
>> - The hierarchy exposed by resctrl may be different on systems that have the "same"
>> controls.
>> For example, on an MPAM system (if I understand correctly) the user may see:
>> info/
>> └── MB/
>> └── resource_schemata/
>> ├── MB/
>> │ └── MB_MAX/
>> └── MB_MIN/
>
> Yes, this matches my understanding.
>
>>
>> Compared with a possible implementation on Intel that looks like:
>> info/
>> └── MB/
>> └── resource_schemata/
>> ├── MB/
>> │ └── MB_OPT/
>> ├── MB_MAX/
>> └── MB_MIN/
>
> Not sure if my understanding is correct here...
> In the kernel today is it rdt max that backs MB? (Ignoring the sw controller)
resctrl does not have support for the RDT "MAX" controller yet. Since resctrl was
created as part of enabling RDT the resctrl MB control maps exactly to RDT's
original percentage based memory delay value that is an approximate. Newer hardware
support three controls: optimal, minimum, and maximum. These controls have finer
granularity than what the default percentage based control supports so emulation
is needed.
So far I assumed that on these systems the default MB control would be emulated
by the new "optimal" control but after these exchanges I can see there being an
argument for it to be emulated by the new "maximum" control also. Apart from it
implying a cap there is also the idea that the "maximum" control is more likely to
be available on all platforms.
> If so wouldn't the meaning of MB change within the same platform on a kernel
> upgrade once the rdt optimal support is added?
Good point. Thank you for bringing this up. While the existing RDT MB control is
approximate the RDT spec does contain the statement "... should be viewed as a
maximum bandwidth “cap” per-CLOS." which is quite clear. I'll follow up with
RDT folks on this.
...
>>> We can work that. For MPAM it's writing all 1s to the register which for the
>>> minumum case represents ((2**mbw_min_wd)/2**mbw_min_wd)) * 100 %
>>>
>>> Just emulating MB with MAX control
>>>> does not seem to eliminate this problem since between the fs and arch resctrl still needs
>>>> to ensure that when user space writes a control value to the MIN control that it is valid
>>>> for the underlying MPAM system.
>>>>
>>>> It almost sounds as though there is an attempt to eliminate resctrl's usage of a "max"
>>>> value for the MIN control since that is effectively unknown to MPAM but that does
>>>> not look possible to me?
>>>
>>> Sorry but I haven't understood what your saying. What does "resctrl's usage of a
>>> "max" value for MIN control" mean?
>>
>> Basically it is resctrl fs's validation of user input. Specifically, in bw_validate() where
>> the fs does this range check:
>> if (bw < r->membw.min_bw || bw > r->membw.max_bw)
>>
>> resctrl fs thus uses the "max" value of a control for user input checking and it seemed to
>> me that it may be difficult for MPAM to lean that "max" from all systems but it sounds as
>> though the plan is instead to use the max that the architecture supports?
>
> Yes, can just size the max based on the number of configuration bits in the h/w.
Ack. Thank you for the clarification.
Reinette
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-09 17:41 ` Reinette Chatre
@ 2026-06-10 7:09 ` Chen, Yu C
2026-06-10 14:27 ` Chen, Yu C
2026-06-10 15:59 ` Reinette Chatre
0 siblings, 2 replies; 66+ messages in thread
From: Chen, Yu C @ 2026-06-10 7:09 UTC (permalink / raw)
To: Reinette Chatre
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org, Ben Horgan,
Tony Luck, Dave Martin, James Morse, Drew Fustini, Babu Moger,
Fenghua Yu, Chen Yu
Hi Reinette,
On 6/10/2026 1:41 AM, Reinette Chatre wrote:
> Hi Ben,
>
> On 6/9/26 9:37 AM, Ben Horgan wrote:
>> On 6/9/26 16:28, Reinette Chatre wrote:
>>> On 6/9/26 3:10 AM, Ben Horgan wrote:
>>>> On 6/8/26 17:16, Reinette Chatre wrote:
>
>
>>>> I don't see the advantage of emulating MB with both MIN and MAX. Just going by
>>>> the MPAM specification, a system keeping MIN at 0 and just setting MAX from MB,
>>>> (MIN=0, MAX=MB) should behave the same as one always setting both, (MIN=MB,
>>>> MAX=MB). In the MIN=0 case there is never any high preference traffic and in the
>>>> MIN=MAX_MB case there is never any medium preference traffic. It seemed best to
>>>> not rely on any platform specific heuristics to try and guess what's better and
>>>> just wait til the time we could support MB_MIN in resctrl (and leave the
>>>> decision up to the user). My expectation was that this would be the simplest
>>>> course of action.
>>>
>>> This sounds fair. Two observations:
>>> - The hierarchy exposed by resctrl may be different on systems that have the "same"
>>> controls.
>>> For example, on an MPAM system (if I understand correctly) the user may see:
>>> info/
>>> └── MB/
>>> └── resource_schemata/
>>> ├── MB/
>>> │ └── MB_MAX/
>>> └── MB_MIN/
>>
>> Yes, this matches my understanding.
>>
>>>
>>> Compared with a possible implementation on Intel that looks like:
>>> info/
>>> └── MB/
>>> └── resource_schemata/
>>> ├── MB/
>>> │ └── MB_OPT/
>>> ├── MB_MAX/
>>> └── MB_MIN/
>>
>> Not sure if my understanding is correct here...
>> In the kernel today is it rdt max that backs MB? (Ignoring the sw controller)
>
> resctrl does not have support for the RDT "MAX" controller yet. Since resctrl was
> created as part of enabling RDT the resctrl MB control maps exactly to RDT's
> original percentage based memory delay value that is an approximate. Newer hardware
> support three controls: optimal, minimum, and maximum. These controls have finer
> granularity than what the default percentage based control supports so emulation
> is needed.
> So far I assumed that on these systems the default MB control would be emulated
> by the new "optimal" control but after these exchanges I can see there being an
> argument for it to be emulated by the new "maximum" control also. Apart from it
> implying a cap there is also the idea that the "maximum" control is more likely to
> be available on all platforms.
>
Regarding the region-aware RDT case, I wonder if we actually need to
emulate the
legacy MB control using MB_MAX. First, when we refer to the "legacy" for
region-aware
RDT, I suppose it corresponds to "MSR access" plus "percentage-based
control".
case 1:
If the platform does not support region-aware RDT (no ERDT table is
detected),
the MB is naturally the "legacy" MB, and the info directory would look like:
info
└── MB
└── resource_schemata
└── MB
case 2:If the platform supports region-aware RDT (i.e., ERDT parsing
succeeds),
then the structure looks like below:
info
└── MB
└── resource_schema
└── MB <=== legacy
└── MB_REGION0_OPT
└── MB_REGION1_OPT
└── MB_REGION0_MIN
└── MB_REGION1_MIX
└── MB_REGION0_MAX
└── MB_REGION1_MAX
The OS supports runtime switching between the legacy MB and
region-aware MB, via a MB scope control like:
/sys/fs/resctrl/info/MB/resource_schemata/mode
[legacy] native
If the user wants to use the legacy interface, the system
will fall back to the default [MSR-based, percentage]; if the
user wants to use the new interface, the region-aware interfaces
will be used. Since the two modes are mutually exclusive, there
seems to be no need to use MB_REGION0_MAX to emulate legacy MB.
thanks,
Chenyu
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-10 7:09 ` Chen, Yu C
@ 2026-06-10 14:27 ` Chen, Yu C
2026-06-10 16:13 ` Reinette Chatre
2026-06-10 15:59 ` Reinette Chatre
1 sibling, 1 reply; 66+ messages in thread
From: Chen, Yu C @ 2026-06-10 14:27 UTC (permalink / raw)
To: Reinette Chatre
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org, Ben Horgan,
Tony Luck, Dave Martin, James Morse, Drew Fustini, Babu Moger,
Fenghua Yu
Hi Reinette,
On 6/10/2026 3:09 PM, Chen, Yu C wrote:
> Hi Reinette,
>
> On 6/10/2026 1:41 AM, Reinette Chatre wrote:
>> Hi Ben,
>>
>> On 6/9/26 9:37 AM, Ben Horgan wrote:
>>> On 6/9/26 16:28, Reinette Chatre wrote:
>>>> On 6/9/26 3:10 AM, Ben Horgan wrote:
>>>>> On 6/8/26 17:16, Reinette Chatre wrote:
>>
>>
>>>>> I don't see the advantage of emulating MB with both MIN and MAX.
>>>>> Just going by
>>>>> the MPAM specification, a system keeping MIN at 0 and just setting
>>>>> MAX from MB,
>>>>> (MIN=0, MAX=MB) should behave the same as one always setting both,
>>>>> (MIN=MB,
>>>>> MAX=MB). In the MIN=0 case there is never any high preference
>>>>> traffic and in the
>>>>> MIN=MAX_MB case there is never any medium preference traffic. It
>>>>> seemed best to
>>>>> not rely on any platform specific heuristics to try and guess
>>>>> what's better and
>>>>> just wait til the time we could support MB_MIN in resctrl (and
>>>>> leave the
>>>>> decision up to the user). My expectation was that this would be the
>>>>> simplest
>>>>> course of action.
>>>>
>>>> This sounds fair. Two observations:
>>>> - The hierarchy exposed by resctrl may be different on systems that
>>>> have the "same"
>>>> controls.
>>>> For example, on an MPAM system (if I understand correctly) the
>>>> user may see:
>>>> info/
>>>> └── MB/
>>>> └── resource_schemata/
>>>> ├── MB/
>>>> │ └── MB_MAX/
>>>> └── MB_MIN/
>>>
>>> Yes, this matches my understanding.
>>>
>>>>
>>>> Compared with a possible implementation on Intel that looks like:
>>>> info/
>>>> └── MB/
>>>> └── resource_schemata/
>>>> ├── MB/
>>>> │ └── MB_OPT/
>>>> ├── MB_MAX/
>>>> └── MB_MIN/
>>>
>>> Not sure if my understanding is correct here...
>>> In the kernel today is it rdt max that backs MB? (Ignoring the sw
>>> controller)
>>
>> resctrl does not have support for the RDT "MAX" controller yet. Since
>> resctrl was
>> created as part of enabling RDT the resctrl MB control maps exactly to
>> RDT's
>> original percentage based memory delay value that is an approximate.
>> Newer hardware
>> support three controls: optimal, minimum, and maximum. These controls
>> have finer
>> granularity than what the default percentage based control supports so
>> emulation
>> is needed.
>> So far I assumed that on these systems the default MB control would be
>> emulated
>> by the new "optimal" control but after these exchanges I can see there
>> being an
>> argument for it to be emulated by the new "maximum" control also.
>> Apart from it
>> implying a cap there is also the idea that the "maximum" control is
>> more likely to
>> be available on all platforms.
>>
>
> Regarding the region-aware RDT case, I wonder if we actually need to
> emulate the
> legacy MB control using MB_MAX. First, when we refer to the "legacy" for
> region-aware
> RDT, I suppose it corresponds to "MSR access" plus "percentage-based
> control".
>
> case 1:
> If the platform does not support region-aware RDT (no ERDT table is
> detected),
> the MB is naturally the "legacy" MB, and the info directory would look
> like:
>
> info
> └── MB
> └── resource_schemata
> └── MB
>
> case 2:If the platform supports region-aware RDT (i.e., ERDT parsing
> succeeds),
> then the structure looks like below:
>
> info
> └── MB
> └── resource_schema
> └── MB <=== legacy
> └── MB_REGION0_OPT
> └── MB_REGION1_OPT
> └── MB_REGION0_MIN
> └── MB_REGION1_MIX
> └── MB_REGION0_MAX
> └── MB_REGION1_MAX
>
This may be slightly off-topic from MAX emulation, but I have another
thought regarding multi-controllers for rdt_resource:
As we know, with N regions, an MB resource will have a total of N × 3
controllers. Given that the current PoC iterates through every controller
within the resource in resctrl_resource_ctrl_get(), could this increase
lookup latency?
I studied the cgroup code and found that each controller for a cgroup
resource uses a dedicated cftype. For example:
static struct cftype memory_files[] = {
{ .name = "min", .write = memory_min_write, .seq_show =
memory_min_show },
{ .name = "max", .write = memory_max_write, .seq_show =
memory_max_show },
...
};
The min/max memory controllers can be accessed in O(1) time using:
of_cft(of) -> kn->priv, and cft->write(of, buf, ...)
rftype is resctrl's equivalent of cftype, and schemata is currently
implemented
as a single rftype. Would it make sense to define a separate rftype for
each
resctrl controller(or maybe in the future consider that this is not in a
critical path)
thanks,
Chenyu
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-10 14:27 ` Chen, Yu C
@ 2026-06-10 16:13 ` Reinette Chatre
2026-06-10 17:57 ` Chen, Yu C
0 siblings, 1 reply; 66+ messages in thread
From: Reinette Chatre @ 2026-06-10 16:13 UTC (permalink / raw)
To: Chen, Yu C
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org, Ben Horgan,
Tony Luck, Dave Martin, James Morse, Drew Fustini, Babu Moger,
Fenghua Yu
Hi Chenyu,
On 6/10/26 7:27 AM, Chen, Yu C wrote:
> Hi Reinette,
>
> On 6/10/2026 3:09 PM, Chen, Yu C wrote:
>> Hi Reinette,
>>
>> On 6/10/2026 1:41 AM, Reinette Chatre wrote:
>>> Hi Ben,
>>>
>>> On 6/9/26 9:37 AM, Ben Horgan wrote:
>>>> On 6/9/26 16:28, Reinette Chatre wrote:
>>>>> On 6/9/26 3:10 AM, Ben Horgan wrote:
>>>>>> On 6/8/26 17:16, Reinette Chatre wrote:
>>>
>>>
>>>>>> I don't see the advantage of emulating MB with both MIN and MAX. Just going by
>>>>>> the MPAM specification, a system keeping MIN at 0 and just setting MAX from MB,
>>>>>> (MIN=0, MAX=MB) should behave the same as one always setting both, (MIN=MB,
>>>>>> MAX=MB). In the MIN=0 case there is never any high preference traffic and in the
>>>>>> MIN=MAX_MB case there is never any medium preference traffic. It seemed best to
>>>>>> not rely on any platform specific heuristics to try and guess what's better and
>>>>>> just wait til the time we could support MB_MIN in resctrl (and leave the
>>>>>> decision up to the user). My expectation was that this would be the simplest
>>>>>> course of action.
>>>>>
>>>>> This sounds fair. Two observations:
>>>>> - The hierarchy exposed by resctrl may be different on systems that have the "same"
>>>>> controls.
>>>>> For example, on an MPAM system (if I understand correctly) the user may see:
>>>>> info/
>>>>> └── MB/
>>>>> └── resource_schemata/
>>>>> ├── MB/
>>>>> │ └── MB_MAX/
>>>>> └── MB_MIN/
>>>>
>>>> Yes, this matches my understanding.
>>>>
>>>>>
>>>>> Compared with a possible implementation on Intel that looks like:
>>>>> info/
>>>>> └── MB/
>>>>> └── resource_schemata/
>>>>> ├── MB/
>>>>> │ └── MB_OPT/
>>>>> ├── MB_MAX/
>>>>> └── MB_MIN/
>>>>
>>>> Not sure if my understanding is correct here...
>>>> In the kernel today is it rdt max that backs MB? (Ignoring the sw controller)
>>>
>>> resctrl does not have support for the RDT "MAX" controller yet. Since resctrl was
>>> created as part of enabling RDT the resctrl MB control maps exactly to RDT's
>>> original percentage based memory delay value that is an approximate. Newer hardware
>>> support three controls: optimal, minimum, and maximum. These controls have finer
>>> granularity than what the default percentage based control supports so emulation
>>> is needed.
>>> So far I assumed that on these systems the default MB control would be emulated
>>> by the new "optimal" control but after these exchanges I can see there being an
>>> argument for it to be emulated by the new "maximum" control also. Apart from it
>>> implying a cap there is also the idea that the "maximum" control is more likely to
>>> be available on all platforms.
>>>
>>
>> Regarding the region-aware RDT case, I wonder if we actually need to emulate the
>> legacy MB control using MB_MAX. First, when we refer to the "legacy" for region-aware
>> RDT, I suppose it corresponds to "MSR access" plus "percentage-based control".
>>
>> case 1:
>> If the platform does not support region-aware RDT (no ERDT table is detected),
>> the MB is naturally the "legacy" MB, and the info directory would look like:
>>
>> info
>> └── MB
>> └── resource_schemata
>> └── MB
>>
>> case 2:If the platform supports region-aware RDT (i.e., ERDT parsing succeeds),
>> then the structure looks like below:
>>
>> info
>> └── MB
>> └── resource_schema
>> └── MB <=== legacy
>> └── MB_REGION0_OPT
>> └── MB_REGION1_OPT
>> └── MB_REGION0_MIN
>> └── MB_REGION1_MIX
>> └── MB_REGION0_MAX
>> └── MB_REGION1_MAX
>>
>
> This may be slightly off-topic from MAX emulation, but I have another
> thought regarding multi-controllers for rdt_resource:
> As we know, with N regions, an MB resource will have a total of N × 3
> controllers. Given that the current PoC iterates through every controller
> within the resource in resctrl_resource_ctrl_get(), could this increase
> lookup latency?
resctrl_resource_ctrl_get() is only called when the user writes to the
schemata file and since user can modify any of the enabled controls with
a write to the schemata file it is necessary to iterate through all controls
to find a match.
>
> I studied the cgroup code and found that each controller for a cgroup
> resource uses a dedicated cftype. For example:
> static struct cftype memory_files[] = {
> { .name = "min", .write = memory_min_write, .seq_show = memory_min_show },
> { .name = "max", .write = memory_max_write, .seq_show = memory_max_show },
> ...
> };
The PoC currently has "static struct rftype ctrl_files[] " that associates an
rftype with every property of a control which enables all user space interactions
with the individual controls to be direct.
>
> The min/max memory controllers can be accessed in O(1) time using:
> of_cft(of) -> kn->priv, and cft->write(of, buf, ...)
>
> rftype is resctrl's equivalent of cftype, and schemata is currently implemented
> as a single rftype. Would it make sense to define a separate rftype for each
> resctrl controller(or maybe in the future consider that this is not in a critical path)
Your suggestion is not clear to me. The schemata file is associated with a control group,
a resource group that has multiple allocations each backed by a different controller.
I do not think I fully understand your suggestion so would appreciate if you could
provide more detail.
Reinette
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-10 16:13 ` Reinette Chatre
@ 2026-06-10 17:57 ` Chen, Yu C
2026-06-10 18:10 ` Reinette Chatre
0 siblings, 1 reply; 66+ messages in thread
From: Chen, Yu C @ 2026-06-10 17:57 UTC (permalink / raw)
To: Reinette Chatre
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org, Ben Horgan,
Tony Luck, Dave Martin, James Morse, Drew Fustini, Babu Moger,
Fenghua Yu
Hi Reinette,
On 6/11/2026 12:13 AM, Reinette Chatre wrote:
> Hi Chenyu,
>
[ ... ]
>>> Regarding the region-aware RDT case, I wonder if we actually need to emulate the
>>> legacy MB control using MB_MAX. First, when we refer to the "legacy" for region-aware
>>> RDT, I suppose it corresponds to "MSR access" plus "percentage-based control".
>>>
>>> case 1:
>>> If the platform does not support region-aware RDT (no ERDT table is detected),
>>> the MB is naturally the "legacy" MB, and the info directory would look like:
>>>
>>> info
>>> └── MB
>>> └── resource_schemata
>>> └── MB
>>>
>>> case 2:If the platform supports region-aware RDT (i.e., ERDT parsing succeeds),
>>> then the structure looks like below:
>>>
>>> info
>>> └── MB
>>> └── resource_schema
>>> └── MB <=== legacy
>>> └── MB_REGION0_OPT
>>> └── MB_REGION1_OPT
>>> └── MB_REGION0_MIN
>>> └── MB_REGION1_MIX
>>> └── MB_REGION0_MAX
>>> └── MB_REGION1_MAX
>>>
>>
>> This may be slightly off-topic from MAX emulation, but I have another
>> thought regarding multi-controllers for rdt_resource:
>> As we know, with N regions, an MB resource will have a total of N × 3
>> controllers. Given that the current PoC iterates through every controller
>> within the resource in resctrl_resource_ctrl_get(), could this increase
>> lookup latency?
>
> resctrl_resource_ctrl_get() is only called when the user writes to the
> schemata file and since user can modify any of the enabled controls with
> a write to the schemata file it is necessary to iterate through all controls
> to find a match.
>
Yes, I was thinking if we can speed up this write by not iterating the
control
list.
>>
>> I studied the cgroup code and found that each controller for a cgroup
>> resource uses a dedicated cftype. For example:
>> static struct cftype memory_files[] = {
>> { .name = "min", .write = memory_min_write, .seq_show = memory_min_show },
>> { .name = "max", .write = memory_max_write, .seq_show = memory_max_show },
>> ...
>> };
>
> The PoC currently has "static struct rftype ctrl_files[] " that associates an
> rftype with every property of a control which enables all user space interactions
> with the individual controls to be direct.
>
Yes we have dedicated rftype for the read-only info/properties.
>>
>> The min/max memory controllers can be accessed in O(1) time using:
>> of_cft(of) -> kn->priv, and cft->write(of, buf, ...)
>>
>> rftype is resctrl's equivalent of cftype, and schemata is currently implemented
>> as a single rftype. Would it make sense to define a separate rftype for each
>> resctrl controller(or maybe in the future consider that this is not in a critical path)
>
> Your suggestion is not clear to me. The schemata file is associated with a control group,
> a resource group that has multiple allocations each backed by a different controller.
>
> I do not think I fully understand your suggestion so would appreciate if you could
> provide more detail.
>
Sorry, I did not describe this precisely. My original thought is that
when writing
schemata, we need to locate the control by iterating through the control
list:
rdtgroup_schemata_write
for_each_resource_ctrl(ctrl, r)
if (!strncmp())
return ctrl;
parse_line("0=80;1=60", ctrl)
I just wonder if we can skip iterating the control list, by
giving each control a dedicated rftype(and rftype needs to introduce
a pointer struct resctrl_ctrl *ctrl)
rdtgroup_ctrl_schemata_write()
rft = of->kn->priv
parse_line("0=80;1=60", rft->ctrl)
thanks,
Chenyu
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-10 17:57 ` Chen, Yu C
@ 2026-06-10 18:10 ` Reinette Chatre
0 siblings, 0 replies; 66+ messages in thread
From: Reinette Chatre @ 2026-06-10 18:10 UTC (permalink / raw)
To: Chen, Yu C
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org, Ben Horgan,
Tony Luck, Dave Martin, James Morse, Drew Fustini, Babu Moger,
Fenghua Yu
Hi Chenyu,
On 6/10/26 10:57 AM, Chen, Yu C wrote:
> On 6/11/2026 12:13 AM, Reinette Chatre wrote:
> Sorry, I did not describe this precisely. My original thought is that when writing
> schemata, we need to locate the control by iterating through the control list:
> rdtgroup_schemata_write
> for_each_resource_ctrl(ctrl, r)
> if (!strncmp())
> return ctrl;
> parse_line("0=80;1=60", ctrl)
>
> I just wonder if we can skip iterating the control list, by
> giving each control a dedicated rftype(and rftype needs to introduce
> a pointer struct resctrl_ctrl *ctrl)
> rdtgroup_ctrl_schemata_write()
> rft = of->kn->priv
I interpret this to mean that this new "rftype" would be connected to the private
data associated with the kernfs node associated with a resctrl file. Which resctrl
file do you have in mind here? The schemata resctrl file has a kn but its private
data points to the resource group to which the schemata file belongs. The
only way to have private data of a resctrl file point to a control is if that file
is clearly associated with just that one control. This is true for the files in
the new resource_schemata directory but is not true for the files associated
with a resource group.
When viewing the schemata file associated with a resource group the user expects
to access all controls, not just one.
> parse_line("0=80;1=60", rft->ctrl)
>
Reinette
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-10 7:09 ` Chen, Yu C
2026-06-10 14:27 ` Chen, Yu C
@ 2026-06-10 15:59 ` Reinette Chatre
2026-06-10 18:05 ` Chen, Yu C
2026-06-11 3:26 ` Chen, Yu C
1 sibling, 2 replies; 66+ messages in thread
From: Reinette Chatre @ 2026-06-10 15:59 UTC (permalink / raw)
To: Chen, Yu C
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org, Ben Horgan,
Tony Luck, Dave Martin, James Morse, Drew Fustini, Babu Moger,
Fenghua Yu
Hi Chenyu,
On 6/10/26 12:09 AM, Chen, Yu C wrote:
> Hi Reinette,
>
> On 6/10/2026 1:41 AM, Reinette Chatre wrote:
>> Hi Ben,
>>
>> On 6/9/26 9:37 AM, Ben Horgan wrote:
>>> On 6/9/26 16:28, Reinette Chatre wrote:
>>>> On 6/9/26 3:10 AM, Ben Horgan wrote:
>>>>> On 6/8/26 17:16, Reinette Chatre wrote:
>>
>>
>>>>> I don't see the advantage of emulating MB with both MIN and MAX. Just going by
>>>>> the MPAM specification, a system keeping MIN at 0 and just setting MAX from MB,
>>>>> (MIN=0, MAX=MB) should behave the same as one always setting both, (MIN=MB,
>>>>> MAX=MB). In the MIN=0 case there is never any high preference traffic and in the
>>>>> MIN=MAX_MB case there is never any medium preference traffic. It seemed best to
>>>>> not rely on any platform specific heuristics to try and guess what's better and
>>>>> just wait til the time we could support MB_MIN in resctrl (and leave the
>>>>> decision up to the user). My expectation was that this would be the simplest
>>>>> course of action.
>>>>
>>>> This sounds fair. Two observations:
>>>> - The hierarchy exposed by resctrl may be different on systems that have the "same"
>>>> controls.
>>>> For example, on an MPAM system (if I understand correctly) the user may see:
>>>> info/
>>>> └── MB/
>>>> └── resource_schemata/
>>>> ├── MB/
>>>> │ └── MB_MAX/
>>>> └── MB_MIN/
>>>
>>> Yes, this matches my understanding.
>>>
>>>>
>>>> Compared with a possible implementation on Intel that looks like:
>>>> info/
>>>> └── MB/
>>>> └── resource_schemata/
>>>> ├── MB/
>>>> │ └── MB_OPT/
>>>> ├── MB_MAX/
>>>> └── MB_MIN/
>>>
>>> Not sure if my understanding is correct here...
>>> In the kernel today is it rdt max that backs MB? (Ignoring the sw controller)
>>
>> resctrl does not have support for the RDT "MAX" controller yet. Since resctrl was
>> created as part of enabling RDT the resctrl MB control maps exactly to RDT's
>> original percentage based memory delay value that is an approximate. Newer hardware
>> support three controls: optimal, minimum, and maximum. These controls have finer
>> granularity than what the default percentage based control supports so emulation
>> is needed.
>> So far I assumed that on these systems the default MB control would be emulated
>> by the new "optimal" control but after these exchanges I can see there being an
>> argument for it to be emulated by the new "maximum" control also. Apart from it
>> implying a cap there is also the idea that the "maximum" control is more likely to
>> be available on all platforms.
>>
>
> Regarding the region-aware RDT case, I wonder if we actually need to emulate the
> legacy MB control using MB_MAX. First, when we refer to the "legacy" for region-aware
> RDT, I suppose it corresponds to "MSR access" plus "percentage-based control".
>
> case 1:
> If the platform does not support region-aware RDT (no ERDT table is detected),
> the MB is naturally the "legacy" MB, and the info directory would look like:
>
> info
> └── MB
> └── resource_schemata
> └── MB
>
> case 2:If the platform supports region-aware RDT (i.e., ERDT parsing succeeds),
> then the structure looks like below:
>
> info
> └── MB
> └── resource_schema
> └── MB <=== legacy
> └── MB_REGION0_OPT
> └── MB_REGION1_OPT
> └── MB_REGION0_MIN
> └── MB_REGION1_MIX
> └── MB_REGION0_MAX
> └── MB_REGION1_MAX
>
> The OS supports runtime switching between the legacy MB and
> region-aware MB, via a MB scope control like:
> /sys/fs/resctrl/info/MB/resource_schemata/mode
> [legacy] native
> If the user wants to use the legacy interface, the system
> will fall back to the default [MSR-based, percentage]; if the
> user wants to use the new interface, the region-aware interfaces
> will be used. Since the two modes are mutually exclusive, there
> seems to be no need to use MB_REGION0_MAX to emulate legacy MB.
- This implies that all current and future ERDT capable systems will keep
supporting MSR access with the legacy "percentage based control". Is this
accurate? What will behavior be on an ERDT system that does not support
MSR access with the percentage-based control?
- It is not obvious to me what you plan to have as default interface -
I believe this should be "legacy" interface to ensure this will not
break user space. resctrl should only switch to "native" after user
demonstrates (by switching) familiarity with the new interface.
Reinette
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-10 15:59 ` Reinette Chatre
@ 2026-06-10 18:05 ` Chen, Yu C
2026-06-11 3:26 ` Chen, Yu C
1 sibling, 0 replies; 66+ messages in thread
From: Chen, Yu C @ 2026-06-10 18:05 UTC (permalink / raw)
To: Reinette Chatre
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org, Ben Horgan,
Tony Luck, Dave Martin, James Morse, Drew Fustini, Babu Moger,
Fenghua Yu
Hi Reinette,
On 6/10/2026 11:59 PM, Reinette Chatre wrote:
> Hi Chenyu,
>
[ ... ]
>> Regarding the region-aware RDT case, I wonder if we actually need to emulate the
>> legacy MB control using MB_MAX. First, when we refer to the "legacy" for region-aware
>> RDT, I suppose it corresponds to "MSR access" plus "percentage-based control".
>>
>> case 1:
>> If the platform does not support region-aware RDT (no ERDT table is detected),
>> the MB is naturally the "legacy" MB, and the info directory would look like:
>>
>> info
>> └── MB
>> └── resource_schemata
>> └── MB
>>
>> case 2:If the platform supports region-aware RDT (i.e., ERDT parsing succeeds),
>> then the structure looks like below:
>>
>> info
>> └── MB
>> └── resource_schema
>> └── MB <=== legacy
>> └── MB_REGION0_OPT
>> └── MB_REGION1_OPT
>> └── MB_REGION0_MIN
>> └── MB_REGION1_MIX
>> └── MB_REGION0_MAX
>> └── MB_REGION1_MAX
>>
>> The OS supports runtime switching between the legacy MB and
>> region-aware MB, via a MB scope control like:
>> /sys/fs/resctrl/info/MB/resource_schemata/mode
>> [legacy] native
>> If the user wants to use the legacy interface, the system
>> will fall back to the default [MSR-based, percentage]; if the
>> user wants to use the new interface, the region-aware interfaces
>> will be used. Since the two modes are mutually exclusive, there
>> seems to be no need to use MB_REGION0_MAX to emulate legacy MB.
>
> - This implies that all current and future ERDT capable systems will keep
> supporting MSR access with the legacy "percentage based control". Is this
> accurate?
I don't know the answer for now; it may depend on how the hardware
maintains
backward compatibility. Let me have a check.
> What will behavior be on an ERDT system that does not support
> MSR access with the percentage-based control?
Then in this case, we may need to emulate the legacy MBA if users do
not wish to switch to the new interface.
> - It is not obvious to me what you plan to have as default interface -
> I believe this should be "legacy" interface to ensure this will not
> break user space. resctrl should only switch to "native" after user
> demonstrates (by switching) familiarity with the new interface.
>
Yes, setting "legacy" as the default will accommodate users who rely on
the old interfaces.
> Reinette
>
>
>
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-10 15:59 ` Reinette Chatre
2026-06-10 18:05 ` Chen, Yu C
@ 2026-06-11 3:26 ` Chen, Yu C
2026-06-11 15:45 ` Reinette Chatre
1 sibling, 1 reply; 66+ messages in thread
From: Chen, Yu C @ 2026-06-11 3:26 UTC (permalink / raw)
To: Reinette Chatre
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org, Ben Horgan,
Tony Luck, Dave Martin, James Morse, Drew Fustini, Babu Moger,
Fenghua Yu
Hi Reinette,
On 6/10/2026 11:59 PM, Reinette Chatre wrote:
> Hi Chenyu,
>
[ ... ]
>
> - This implies that all current and future ERDT capable systems will keep
> supporting MSR access with the legacy "percentage based control". Is this
> accurate? What will behavior be on an ERDT system that does not support
> MSR access with the percentage-based control?
Would it make sense to encourage users to switch to MMIO-based access if
the
platform replaces MSR interfaces with MMIO-based ones? One issue with
emulating
MSR-based access is that there may not be a strict 1:1 mapping between
the MSR
range and the MMIO range. Additionally, I am unsure whether both are
strictly
linearly scalable, so it could be challenging to create a proper mapping for
msr_value_emulated = map(mmio_value) IIUC.
thanks,
Chenyu
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-11 3:26 ` Chen, Yu C
@ 2026-06-11 15:45 ` Reinette Chatre
2026-06-26 15:46 ` Chen, Yu C
0 siblings, 1 reply; 66+ messages in thread
From: Reinette Chatre @ 2026-06-11 15:45 UTC (permalink / raw)
To: Chen, Yu C
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org, Ben Horgan,
Tony Luck, Dave Martin, James Morse, Drew Fustini, Babu Moger,
Fenghua Yu
Hi Chenyu,
On 6/10/26 8:26 PM, Chen, Yu C wrote:
> On 6/10/2026 11:59 PM, Reinette Chatre wrote:
> [ ... ]
>
>>
>> - This implies that all current and future ERDT capable systems will keep
>> supporting MSR access with the legacy "percentage based control". Is this
>> accurate? What will behavior be on an ERDT system that does not support
>> MSR access with the percentage-based control?
>
> Would it make sense to encourage users to switch to MMIO-based access if the
> platform replaces MSR interfaces with MMIO-based ones? One issue with emulating
> MSR-based access is that there may not be a strict 1:1 mapping between the MSR
> range and the MMIO range. Additionally, I am unsure whether both are strictly
> linearly scalable, so it could be challenging to create a proper mapping for
> msr_value_emulated = map(mmio_value) IIUC.
resctrl needs to continue to offer the MB percentage based control to not break any
existing tools that may take a while to transition to the new control interface.
I believe all architectures will encourage their users to use the finer grained
controls while I also expect users and their tools would gradually switch by themselves
to benefit from the improved capabilities.
Reinette
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-11 15:45 ` Reinette Chatre
@ 2026-06-26 15:46 ` Chen, Yu C
2026-07-02 14:27 ` Ben Horgan
0 siblings, 1 reply; 66+ messages in thread
From: Chen, Yu C @ 2026-06-26 15:46 UTC (permalink / raw)
To: Reinette Chatre
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org, Ben Horgan,
Tony Luck, Dave Martin, James Morse, Drew Fustini, Babu Moger,
Fenghua Yu, chen.yu
Hi Reinette,
On 6/11/2026 11:45 PM, Reinette Chatre wrote:
> Hi Chenyu,
>
> On 6/10/26 8:26 PM, Chen, Yu C wrote:
>> On 6/10/2026 11:59 PM, Reinette Chatre wrote:
>> [ ... ]
>>
>>>
>>> - This implies that all current and future ERDT capable systems will keep
>>> supporting MSR access with the legacy "percentage based control". Is this
>>> accurate? What will behavior be on an ERDT system that does not support
>>> MSR access with the percentage-based control?
>>
>> Would it make sense to encourage users to switch to MMIO-based access if the
>> platform replaces MSR interfaces with MMIO-based ones? One issue with emulating
>> MSR-based access is that there may not be a strict 1:1 mapping between the MSR
>> range and the MMIO range. Additionally, I am unsure whether both are strictly
>> linearly scalable, so it could be challenging to create a proper mapping for
>> msr_value_emulated = map(mmio_value) IIUC.
>
> resctrl needs to continue to offer the MB percentage based control to not break any
> existing tools that may take a while to transition to the new control interface.
>
> I believe all architectures will encourage their users to use the finer grained
> controls while I also expect users and their tools would gradually switch by themselves
> to benefit from the improved capabilities.
>
The following are current thoughts on how Control B can emulate Control A.
May I know if this direction is acceptable?
The main idea is that, Legacy MBA controller delegates to region-aware MBA
controller via a pointer, with its hw_update function converting
percentages
to the target's value range.
struct resctrl_hw_ctrl {
struct resctrl_ctrl r_ctrl;
unsigned int msr_base;
void (*hw_update)(struct hw_param *m);
/* points to the backing controller */
struct resctrl_ctrl *r_ctrl_emul; <-- newly added
};
On boot, region-aware controllers shall be created first, followed by the
legacy MBA controller. When ERDT is enabled for legacy MBA, hw_update is
assigned to mba_emul_update() instead of mba_wrmsr_intel(). Meanwhile,
r_ctrl_emul points to the MAX controller of region0.
In mba_emul_update(), the legacy MSR value is scaled to the range of
region0
MAX controller by referencing r_ctrl_emul->membw.max_bw and
r_ctrl_emul->membw.min_bw.
This maps the original input range [1, 100] to a new range [1, L], where
L <= 255.
This emulation logic can also be extended to other controllers.
thanks,
Chenyu
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-26 15:46 ` Chen, Yu C
@ 2026-07-02 14:27 ` Ben Horgan
2026-07-03 9:01 ` Chen, Yu C
0 siblings, 1 reply; 66+ messages in thread
From: Ben Horgan @ 2026-07-02 14:27 UTC (permalink / raw)
To: Chen, Yu C, Reinette Chatre
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org, Tony Luck,
Dave Martin, James Morse, Drew Fustini, Babu Moger, Fenghua Yu,
chen.yu
Hi Chenyu,
On 6/26/26 16:46, Chen, Yu C wrote:
> Hi Reinette,
>
> On 6/11/2026 11:45 PM, Reinette Chatre wrote:
>> Hi Chenyu,
>>
>> On 6/10/26 8:26 PM, Chen, Yu C wrote:
>>> On 6/10/2026 11:59 PM, Reinette Chatre wrote:
>>> [ ... ]
>>>
>>>>
>>>> - This implies that all current and future ERDT capable systems
>>>> will keep
>>>> supporting MSR access with the legacy "percentage based
>>>> control". Is this
>>>> accurate? What will behavior be on an ERDT system that does not
>>>> support
>>>> MSR access with the percentage-based control?
>>>
>>> Would it make sense to encourage users to switch to MMIO-based access
>>> if the
>>> platform replaces MSR interfaces with MMIO-based ones? One issue with
>>> emulating
>>> MSR-based access is that there may not be a strict 1:1 mapping
>>> between the MSR
>>> range and the MMIO range. Additionally, I am unsure whether both are
>>> strictly
>>> linearly scalable, so it could be challenging to create a proper
>>> mapping for
>>> msr_value_emulated = map(mmio_value) IIUC.
>>
>> resctrl needs to continue to offer the MB percentage based control to
>> not break any
>> existing tools that may take a while to transition to the new control
>> interface.
>>
>> I believe all architectures will encourage their users to use the
>> finer grained
>> controls while I also expect users and their tools would gradually
>> switch by themselves
>> to benefit from the improved capabilities.
>>
>
> The following are current thoughts on how Control B can emulate Control A.
> May I know if this direction is acceptable?
>
> The main idea is that, Legacy MBA controller delegates to region-aware MBA
> controller via a pointer, with its hw_update function converting
> percentages
> to the target's value range.
>
> struct resctrl_hw_ctrl {
> struct resctrl_ctrl r_ctrl;
> unsigned int msr_base;
> void (*hw_update)(struct hw_param *m);
> /* points to the backing controller */
> struct resctrl_ctrl *r_ctrl_emul; <-- newly added
> };
What's the reason for this being in the architecture specific code? I
would expect that the resctrl core code would need to know about any
emulation even if the exact emulation is architecture specific?
Thanks,
Ben
>
> On boot, region-aware controllers shall be created first, followed by the
> legacy MBA controller. When ERDT is enabled for legacy MBA, hw_update is
> assigned to mba_emul_update() instead of mba_wrmsr_intel(). Meanwhile,
> r_ctrl_emul points to the MAX controller of region0.
>
> In mba_emul_update(), the legacy MSR value is scaled to the range of
> region0
> MAX controller by referencing r_ctrl_emul->membw.max_bw and r_ctr_emul-
>>membw.min_bw.
> This maps the original input range [1, 100] to a new range [1, L], where
> L <= 255.
>
> This emulation logic can also be extended to other controllers.
>
> thanks,
> Chenyu
>
>
>
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-07-02 14:27 ` Ben Horgan
@ 2026-07-03 9:01 ` Chen, Yu C
0 siblings, 0 replies; 66+ messages in thread
From: Chen, Yu C @ 2026-07-03 9:01 UTC (permalink / raw)
To: Ben Horgan
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org, Tony Luck,
Dave Martin, James Morse, Drew Fustini, Babu Moger, Fenghua Yu,
chen.yu, Reinette Chatre
Hi Ben,
On 7/2/2026 10:27 PM, Ben Horgan wrote:
> Hi Chenyu,
[ ... ]
>> The following are current thoughts on how Control B can emulate Control A.
>> May I know if this direction is acceptable?
>>
>> The main idea is that, Legacy MBA controller delegates to region-aware MBA
>> controller via a pointer, with its hw_update function converting
>> percentages
>> to the target's value range.
>>
>> struct resctrl_hw_ctrl {
>> struct resctrl_ctrl r_ctrl;
>> unsigned int msr_base;
>> void (*hw_update)(struct hw_param *m);
>> /* points to the backing controller */
>> struct resctrl_ctrl *r_ctrl_emul; <-- newly added
>> };
>
> What's the reason for this being in the architecture specific code? I
> would expect that the resctrl core code would need to know about any
> emulation even if the exact emulation is architecture specific?
>
I was thinking that the low-level emulation was implemented by the
hardware/architecture, so I put it within resctrl_hw_ctrl. But you
are right - the generic resctrl might need to be aware of the emulation,
because the "info" directory needs to display the emulation hierarchy.
Let me have a try at moving it into the generic resctrl, like into
struct resctrl_ctrl.
thanks,
Chenyu
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-09 15:28 ` Reinette Chatre
2026-06-09 16:37 ` Ben Horgan
@ 2026-06-10 4:31 ` Drew Fustini
2026-06-10 15:14 ` Reinette Chatre
1 sibling, 1 reply; 66+ messages in thread
From: Drew Fustini @ 2026-06-10 4:31 UTC (permalink / raw)
To: Reinette Chatre
Cc: Ben Horgan, Tony Luck, James Morse, Dave Martin, Babu Moger,
Fenghua Yu, Chen Yu, Borislav Petkov, Thomas Gleixner,
Dave Hansen, Peter Newman, x86@kernel.org,
linux-kernel@vger.kernel.org
On Tue, Jun 09, 2026 at 08:28:09AM -0700, Reinette Chatre wrote:
> - I see that the default of zero for the MIN control can be appropriate. For the
> usage of resctrl_get_default_ctrlval() when creating a new control group this
> could be ok as a general setting since the user is expected to, after creating
> the resource group, adjust control values as required by the workloads it will
> contain. I need to make sure about this from all architectures and if 0 is not
> ok then we could use Drew's suggestion of architecture providing a specific reset
> value.
Unfortunately, 0 will not work for the default value for Reserved
bandwidth Blocks (Rbwb) in RISC-V CBQRI as it is required by the spec to
be at least 1.
Thanks,
Drew
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-10 4:31 ` Drew Fustini
@ 2026-06-10 15:14 ` Reinette Chatre
0 siblings, 0 replies; 66+ messages in thread
From: Reinette Chatre @ 2026-06-10 15:14 UTC (permalink / raw)
To: Drew Fustini
Cc: Ben Horgan, Tony Luck, James Morse, Dave Martin, Babu Moger,
Fenghua Yu, Chen Yu, Borislav Petkov, Thomas Gleixner,
Dave Hansen, Peter Newman, x86@kernel.org,
linux-kernel@vger.kernel.org
Hi Drew,
On 6/9/26 9:31 PM, Drew Fustini wrote:
> On Tue, Jun 09, 2026 at 08:28:09AM -0700, Reinette Chatre wrote:
>> - I see that the default of zero for the MIN control can be appropriate. For the
>> usage of resctrl_get_default_ctrlval() when creating a new control group this
>> could be ok as a general setting since the user is expected to, after creating
>> the resource group, adjust control values as required by the workloads it will
>> contain. I need to make sure about this from all architectures and if 0 is not
>> ok then we could use Drew's suggestion of architecture providing a specific reset
>> value.
>
> Unfortunately, 0 will not work for the default value for Reserved
> bandwidth Blocks (Rbwb) in RISC-V CBQRI as it is required by the spec to
> be at least 1.
My mistake, I should not have written a hardcoded 0 but instead mirrored Ben's original
suggestion of using ctrl->membw.min_bw.
Reinette
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-05-29 18:06 [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept Reinette Chatre
` (2 preceding siblings ...)
2026-06-03 15:15 ` Ben Horgan
@ 2026-06-03 18:46 ` Luck, Tony
2026-06-04 10:02 ` Ben Horgan
2026-06-04 21:42 ` Reinette Chatre
2026-06-03 22:14 ` Drew Fustini
` (2 subsequent siblings)
6 siblings, 2 replies; 66+ messages in thread
From: Luck, Tony @ 2026-06-03 18:46 UTC (permalink / raw)
To: Reinette Chatre
Cc: Ben Horgan, James Morse, Dave Martin, Babu Moger, Drew Fustini,
Fenghua Yu, Chen Yu, Borislav Petkov, Thomas Gleixner,
Dave Hansen, Peter Newman, x86@kernel.org,
linux-kernel@vger.kernel.org
Reinette,
Tiny bug in "mpam,x86,fs/resctrl: Transition resource control to a list"
> /*
> * Return length needed to display longest control suffix.
> * Add 1 for the "_" character when control name exists.
> */
> size_t resctrl_resource_ctrl_max_len(struct rdt_resource *r)
> {
> struct resctrl_ctrl *ctrl;
> size_t total = 0;
> size_t len;
>
> for_each_resource_ctrl(ctrl,r) {
> len = strlen(resctrl_ctrl_name_str(ctrl->name));
> if (len)
> total += 1 + len;
Should be:
total = max(total, 1 + len);
> }
>
> return total;
> }
Your sample code just converts "MB" over to using a list of controls.
Would it also work for code & data priority? E.g. by having controls
for "_CODE" and "_DATA" that are on the list when CDP is enabled, and
just a no name control for the default?
-Tony
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-03 18:46 ` Luck, Tony
@ 2026-06-04 10:02 ` Ben Horgan
2026-06-04 21:42 ` Reinette Chatre
1 sibling, 0 replies; 66+ messages in thread
From: Ben Horgan @ 2026-06-04 10:02 UTC (permalink / raw)
To: Luck, Tony, Reinette Chatre
Cc: James Morse, Dave Martin, Babu Moger, Drew Fustini, Fenghua Yu,
Chen Yu, Borislav Petkov, Thomas Gleixner, Dave Hansen,
Peter Newman, x86@kernel.org, linux-kernel@vger.kernel.org
Hi Tony,
On 6/3/26 19:46, Luck, Tony wrote:
> Reinette,
>
> Tiny bug in "mpam,x86,fs/resctrl: Transition resource control to a list"
>
>> /*
>> * Return length needed to display longest control suffix.
>> * Add 1 for the "_" character when control name exists.
>> */
>> size_t resctrl_resource_ctrl_max_len(struct rdt_resource *r)
>> {
>> struct resctrl_ctrl *ctrl;
>> size_t total = 0;
>> size_t len;
>>
>> for_each_resource_ctrl(ctrl,r) {
>> len = strlen(resctrl_ctrl_name_str(ctrl->name));
>> if (len)
>> total += 1 + len;
>
> Should be:
>
> total = max(total, 1 + len);
>
>> }
>>
>> return total;
>> }
>
> Your sample code just converts "MB" over to using a list of controls.
> Would it also work for code & data priority? E.g. by having controls
> for "_CODE" and "_DATA" that are on the list when CDP is enabled, and
> just a no name control for the default?
Is there a plan to expand to support CDP for things other than cache allocation?
This would help for MPAM as we currently don't support the MB schema when CDP is
enabled for L2 or L3 as we have no way of reliably supporting it for the caches
but not the memory allocation.
Thanks,
Ben
>
> -Tony
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-03 18:46 ` Luck, Tony
2026-06-04 10:02 ` Ben Horgan
@ 2026-06-04 21:42 ` Reinette Chatre
2026-07-08 12:56 ` Chen, Yu C
1 sibling, 1 reply; 66+ messages in thread
From: Reinette Chatre @ 2026-06-04 21:42 UTC (permalink / raw)
To: Luck, Tony
Cc: Ben Horgan, James Morse, Dave Martin, Babu Moger, Drew Fustini,
Fenghua Yu, Chen Yu, Borislav Petkov, Thomas Gleixner,
Dave Hansen, Peter Newman, x86@kernel.org,
linux-kernel@vger.kernel.org
Hi Tony,
On 6/3/26 11:46 AM, Luck, Tony wrote:
> Reinette,
>
> Tiny bug in "mpam,x86,fs/resctrl: Transition resource control to a list"
>
>> /*
>> * Return length needed to display longest control suffix.
>> * Add 1 for the "_" character when control name exists.
>> */
>> size_t resctrl_resource_ctrl_max_len(struct rdt_resource *r)
>> {
>> struct resctrl_ctrl *ctrl;
>> size_t total = 0;
>> size_t len;
>>
>> for_each_resource_ctrl(ctrl,r) {
>> len = strlen(resctrl_ctrl_name_str(ctrl->name));
>> if (len)
>> total += 1 + len;
>
> Should be:
>
> total = max(total, 1 + len);
Thank you very much. I squashed this locally.
>
>> }
>>
>> return total;
>> }
>
> Your sample code just converts "MB" over to using a list of controls.
> Would it also work for code & data priority? E.g. by having controls
> for "_CODE" and "_DATA" that are on the list when CDP is enabled, and
> just a no name control for the default?
This PoC also converts existing cache allocation to use a (one entry) list
of controls. The current idea is that whether CDP is enabled or not there is
just one list of controls associated with the resource but when CDP is enabled
every domain associated with every control can accommodate both a "CODE" and
"DATA" control value.
For cache controls this PoC indeed still has many assumptions about there
being only one cache control. Even with the single control there are many
CDP special cases so there may be some unexplored corners when adding multiple
cache controls.
Reinette
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-04 21:42 ` Reinette Chatre
@ 2026-07-08 12:56 ` Chen, Yu C
0 siblings, 0 replies; 66+ messages in thread
From: Chen, Yu C @ 2026-07-08 12:56 UTC (permalink / raw)
To: Reinette Chatre
Cc: Ben Horgan, James Morse, Dave Martin, Babu Moger, Drew Fustini,
Fenghua Yu, Borislav Petkov, Thomas Gleixner, Dave Hansen,
Peter Newman, x86@kernel.org, linux-kernel@vger.kernel.org,
Luck, Tony, chen.yu
Hi Reinette,
On 6/5/2026 5:42 AM, Reinette Chatre wrote:
> Hi Tony,
>
> On 6/3/26 11:46 AM, Luck, Tony wrote:
>> Reinette,
>>
>> Tiny bug in "mpam,x86,fs/resctrl: Transition resource control to a list"
>>
>>> /*
>>> * Return length needed to display longest control suffix.
>>> * Add 1 for the "_" character when control name exists.
>>> */
>>> size_t resctrl_resource_ctrl_max_len(struct rdt_resource *r)
>>> {
>>> struct resctrl_ctrl *ctrl;
>>> size_t total = 0;
>>> size_t len;
>>>
>>> for_each_resource_ctrl(ctrl,r) {
>>> len = strlen(resctrl_ctrl_name_str(ctrl->name));
>>> if (len)
>>> total += 1 + len;
>>
>> Should be:
>>
>> total = max(total, 1 + len);
>
> Thank you very much. I squashed this locally.
>
A minor alignment schemata issue is found when testing region-aware RDT:
[resctrl]# cat schemata
MB:0=100
MB_REGION0_OPT:0=255
MB_REGION0_MIN:0=255
MB_REGION0_MAX:0=255
MB_REGION1_OPT:0=255
MB_REGION1_MIN:0=255
MB_REGION1_MAX:0=255
L2:0=ffff;2=ffff;4=ffff
L3:0=3ff
The schemata file right-justified only the resource name in a
max_name_width field, then printed the control suffix afterwards
without padding. Since max_name_width reserves space for name plus
suffix, this left a run of spaces before every line. A small hacky
fix:
diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c
index 9b51464b66b7..7d517f3457bf 100644
--- a/fs/resctrl/ctrlmondata.c
+++ b/fs/resctrl/ctrlmondata.c
@@ -521,6 +521,7 @@ ssize_t rdtgroup_schemata_write(struct
kernfs_open_file *of,
static void show_doms(struct seq_file *s, struct rdt_resource_final *f,
bool print_ctrl, int closid, struct resctrl_ctrl
*ctrl)
{
+ char name[20];
struct rdt_resource *r = f->res;
struct rdt_ctrl_domain *dom;
bool sep = false;
@@ -529,11 +530,13 @@ static void show_doms(struct seq_file *s, struct
rdt_resource_final *f,
/* Walking r->domains, ensure it can't race with cpuhp */
lockdep_assert_cpus_held();
- if (print_ctrl)
- seq_printf(s, "%*s%s%s:", max_name_width, f->name,
- resctrl_ctrl_is_default(ctrl) ? "" : "_",
- resctrl_ctrl_is_default(ctrl) ?
- "" : resctrl_ctrl_name_str(ctrl->name));
+ if (print_ctrl) {
+ snprintf(name, sizeof(name), "%s%s%s", f->name,
+ resctrl_ctrl_is_default(ctrl) ? "" : "_",
+ resctrl_ctrl_is_default(ctrl) ?
+ "" : resctrl_ctrl_name_str(ctrl->name));
+ seq_printf(s, "%*s:", max_name_width, name);
+ }
list_for_each_entry(dom, &ctrl->domains, hdr.list) {
if (sep)
seq_puts(s, ";");
diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
index c1aac0dcd332..916bf6a8eab4 100644
--- a/fs/resctrl/rdtgroup.c
+++ b/fs/resctrl/rdtgroup.c
@@ -1654,6 +1654,7 @@ static int rdtgroup_size_show(struct
kernfs_open_file *of,
struct rdtgroup *rdtgrp;
struct rdt_resource *r;
unsigned int size;
+ char name[20];
u32 ctrl_val;
int ret = 0;
u32 closid;
@@ -1693,11 +1694,11 @@ static int rdtgroup_size_show(struct
kernfs_open_file *of,
type = f->conf_type;
for_each_resource_ctrl(ctrl, r) {
sep = false;
- seq_printf(s, "%*s", max_name_width, f->name);
- if (!resctrl_ctrl_is_default(ctrl))
- seq_printf(s, "_%s:",
resctrl_ctrl_name_str(ctrl->name));
- else
- seq_putc(s, ':');
+ snprintf(name, sizeof(name), "%s%s%s", f->name,
+ resctrl_ctrl_is_default(ctrl) ? "" : "_",
+ resctrl_ctrl_is_default(ctrl) ?
+ "" : resctrl_ctrl_name_str(ctrl->name));
+ seq_printf(s, "%*s:", max_name_width, name);
list_for_each_entry(d, &ctrl->domains, hdr.list) {
if (sep)
seq_putc(s, ';');
And it becomes:
[ resctrl]# cat schemata
MB:0=100
MB_REGION0_OPT:0=255
MB_REGION0_MIN:0=255
MB_REGION0_MAX:0=255
MB_REGION1_OPT:0=255
MB_REGION1_MIN:0=255
MB_REGION1_MAX:0=255
L2:0=ffff;2=ffff;4=ffff
L3:0=3ff
thanks,
Chenyu
^ permalink raw reply related [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-05-29 18:06 [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept Reinette Chatre
` (3 preceding siblings ...)
2026-06-03 18:46 ` Luck, Tony
@ 2026-06-03 22:14 ` Drew Fustini
2026-06-04 21:47 ` Reinette Chatre
2026-06-15 21:05 ` Moger, Babu
2026-06-24 19:08 ` Fenghua Yu
6 siblings, 1 reply; 66+ messages in thread
From: Drew Fustini @ 2026-06-03 22:14 UTC (permalink / raw)
To: Reinette Chatre
Cc: Tony Luck, Ben Horgan, James Morse, Dave Martin, Babu Moger,
Fenghua Yu, Chen Yu, Borislav Petkov, Thomas Gleixner,
Dave Hansen, Peter Newman, x86@kernel.org,
linux-kernel@vger.kernel.org
On Fri, May 29, 2026 at 11:06:07AM -0700, Reinette Chatre wrote:
> /**
> * struct resctrl_ctrl - A resource control
> * @entry: List entry of rdt_resource::controls
> * @scope: Scope of the resource that this control allocates
> * @domains: RCU list of all control domains
> * @type: The control type that determines the properties of the control,
> * format string for displaying control values to user space, and
> * parser of control values provided by user space.
> * @name: Name of the control. Appended to final resource name
> * (rdt_resource_final::name) to create final schema entry.
> * Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
> * For example, with resource name "MB" and control name "MAX" the
> * schema entry will be "MB_MAX".
> * @cache: Cache allocation control properties.
> * @membw: Bandwidth control properties.
> */
> struct resctrl_ctrl {
> struct list_head entry;
> enum resctrl_scope scope;
> struct list_head domains;
> enum resctrl_ctrl_type type;
> enum resctrl_ctrl_name name;
> union {
> struct resctrl_cache cache;
> struct resctrl_membw membw;
> };
> };
>
> Two members summarize how this new structure fits into the rest of resctrl:
> a) resctrl_ctrl::entry
> Since a resource can support multiple controls there is a new list
> in struct rdt_resource named "controls" that contains the list of all
> controls supported by the resource.
> b) resctrl_ctrl::domains
> Instead of the list of control domains belonging to a resource they
> now belong to the control self. By doing so resctrl can support resource
> controls at different scope for the same resource. This is intended to
> support some upcoming MPAM and RISC-V usages.
The ability to change scope is much needed for RISC-V. There are
compromises in my RFC [1] as a result of trying to map everything to
either L2 or L3 scope.
I would also like to see a non-cpu cache scope for monitoring too, but
would that be better discussed outside the context of this proof of
concept?
Thanks,
Drew
[1] https://lore.kernel.org/all/20260601-ssqosid-cbqri-rqsc-v7-0-v6-0-baf00f50028a@kernel.org/
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-03 22:14 ` Drew Fustini
@ 2026-06-04 21:47 ` Reinette Chatre
2026-06-05 19:48 ` Drew Fustini
0 siblings, 1 reply; 66+ messages in thread
From: Reinette Chatre @ 2026-06-04 21:47 UTC (permalink / raw)
To: Drew Fustini
Cc: Tony Luck, Ben Horgan, James Morse, Dave Martin, Babu Moger,
Fenghua Yu, Chen Yu, Borislav Petkov, Thomas Gleixner,
Dave Hansen, Peter Newman, x86@kernel.org,
linux-kernel@vger.kernel.org
Hi Drew,
On 6/3/26 3:14 PM, Drew Fustini wrote:
> On Fri, May 29, 2026 at 11:06:07AM -0700, Reinette Chatre wrote:
>> /**
>> * struct resctrl_ctrl - A resource control
>> * @entry: List entry of rdt_resource::controls
>> * @scope: Scope of the resource that this control allocates
>> * @domains: RCU list of all control domains
>> * @type: The control type that determines the properties of the control,
>> * format string for displaying control values to user space, and
>> * parser of control values provided by user space.
>> * @name: Name of the control. Appended to final resource name
>> * (rdt_resource_final::name) to create final schema entry.
>> * Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
>> * For example, with resource name "MB" and control name "MAX" the
>> * schema entry will be "MB_MAX".
>> * @cache: Cache allocation control properties.
>> * @membw: Bandwidth control properties.
>> */
>> struct resctrl_ctrl {
>> struct list_head entry;
>> enum resctrl_scope scope;
>> struct list_head domains;
>> enum resctrl_ctrl_type type;
>> enum resctrl_ctrl_name name;
>> union {
>> struct resctrl_cache cache;
>> struct resctrl_membw membw;
>> };
>> };
>>
>> Two members summarize how this new structure fits into the rest of resctrl:
>> a) resctrl_ctrl::entry
>> Since a resource can support multiple controls there is a new list
>> in struct rdt_resource named "controls" that contains the list of all
>> controls supported by the resource.
>> b) resctrl_ctrl::domains
>> Instead of the list of control domains belonging to a resource they
>> now belong to the control self. By doing so resctrl can support resource
>> controls at different scope for the same resource. This is intended to
>> support some upcoming MPAM and RISC-V usages.
>
> The ability to change scope is much needed for RISC-V. There are
> compromises in my RFC [1] as a result of trying to map everything to
> either L2 or L3 scope.
>
> I would also like to see a non-cpu cache scope for monitoring too, but
> would that be better discussed outside the context of this proof of
> concept?
I also think it would be good for it to be clear that monitoring is based on
scope, not a resource. With the MB controls supporting different scope I do think
that this would be a good next step. A previous musing from me on this topic can
be found (at the end of ) https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@intel.com/
I have not yet considered how this can be built on top of this PoC though.
Reinette
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-04 21:47 ` Reinette Chatre
@ 2026-06-05 19:48 ` Drew Fustini
0 siblings, 0 replies; 66+ messages in thread
From: Drew Fustini @ 2026-06-05 19:48 UTC (permalink / raw)
To: Reinette Chatre
Cc: Tony Luck, Ben Horgan, James Morse, Dave Martin, Babu Moger,
Fenghua Yu, Chen Yu, Borislav Petkov, Thomas Gleixner,
Dave Hansen, Peter Newman, x86@kernel.org,
linux-kernel@vger.kernel.org
On Thu, Jun 04, 2026 at 02:47:39PM -0700, Reinette Chatre wrote:
> > The ability to change scope is much needed for RISC-V. There are
> > compromises in my RFC [1] as a result of trying to map everything to
> > either L2 or L3 scope.
> >
> > I would also like to see a non-cpu cache scope for monitoring too, but
> > would that be better discussed outside the context of this proof of
> > concept?
>
> I also think it would be good for it to be clear that monitoring is based on
> scope, not a resource. With the MB controls supporting different scope I do think
> that this would be a good next step. A previous musing from me on this topic can
> be found (at the end of ) https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@intel.com/
>
> I have not yet considered how this can be built on top of this PoC though.
Thanks for explaining. I like how how you show an example of
mon_data/mon_NODE_00/mbm_total_bytes in that thread. I believe that sort
of scheme would work well for RISc-V as a bandwidth controller
implementing the CBQRI spec can be located anywhere within the system.
Drew
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-05-29 18:06 [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept Reinette Chatre
` (4 preceding siblings ...)
2026-06-03 22:14 ` Drew Fustini
@ 2026-06-15 21:05 ` Moger, Babu
2026-06-17 17:18 ` Reinette Chatre
2026-06-24 19:08 ` Fenghua Yu
6 siblings, 1 reply; 66+ messages in thread
From: Moger, Babu @ 2026-06-15 21:05 UTC (permalink / raw)
To: Reinette Chatre, Tony Luck, Ben Horgan, James Morse, Dave Martin,
Moger, Babu, Drew Fustini, Fenghua Yu, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi Reinette,
On 5/29/2026 1:06 PM, Reinette Chatre wrote:
> Hi Everybody,
>
> It has been a while since we discussed the resctrl changes required to support
> hardware that has controls with fine granularity or hardware that has multiple
> controls per resource. For reference, the most recent email discussion can
> be found at [1] with a summary of discussions in last year's plumbers slides [2].
>
> I created a PoC that I believe supports what folks have agreed to so far. I
> hope this can help us to restart the discussion with the goal that resctrl gains
> support for upcoming hardware that require these features.
>
> Request regarding this PoC
> ==========================
>
> Please consider this PoC as a "direction check" on the schema description and multiple
> control discussions held thus far.
>
> Could folks working on enabling new hardware requiring this capability please consider
> if this is something you can build on and how it should be improved to support these
> upcoming capabilities?
>
> Opens
> =====
>
> While the PoC aims to support what folks agreed on some opens remain:
> - I attempted to make some MPAM supporting changes but these are all just compile
> tested. While MPAM should benefit from the new control properties I did not
> initialize them on MPAM and did not attempt refactor to separate out
> the architecture specific control properties (more on what this means later).
> I did attempt some MPAM refactoring that duplicates the MPAM domain to the
> control domain and monitoring domain lists in support of there being multiple
> controls each with its own list of control domains but it is definitely not good
> design.
> - No support for emulated controls (yet). The PoC is quite large already
> but I think it can be used as a base for emulated controls for which the software
> controller could be a potential first customer. In this PoC mounting with
> software controller will still display the original controller's properties.
> - One open that needs to be addressed as part of support for emulated controls is
> how best to display emulation relationship via resctrl hierarchy.
> - No support for "read-modify-write" usage of schemata file. This is where we
> discussed (without agreement) on possibly introducing the "#" prefix to schemata
> file entries. This PoC does not support this prefix and the current assumption/expectation
> is that when user space changes a configuration only the new control values are
> written to schemata file. I thus do not have a plan to support this so please
> share opinions in this regard if you have some.
> - Controls are independent for now. This means that, for example, if a resource
> supports a "MIN" and "MAX" control then this implementation would allow user to
> set the "maximum" control values to be less than the "minimum" control values.
> - PoC supports the "bitmap" control but does not (yet) expose properties of a bitmap
> control to the new info/<resource>/resource_schemata directory.
>
> Accessing PoC
> =============
>
> Please consider the PoC as a rough draft. It has only been compile tested for Arm
> and known to be incomplete in Arm support. To help with experimenting I only
> fully adapted the Intel MBA resource to demo two dummy additional MBA controls.
> All architectures should immediately benefit from the new schema descriptions
> and new info/MB/resource_schemata hierarchy.
>
> I considered the patches self too many for email. Instead, the PoC can be found at:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git branch resctrl/controls_rfc_v1
>
> The work is based on v7.1-rc2 that also includes the following series (two of which has
> since been queued) included:
>
> "selftests/resctrl: Fixes and improvements focused on Intel platforms"
> https://lore.kernel.org/lkml/cover.1775266384.git.reinette.chatre@intel.com/
>
> "x86,fs/resctrl: Improve resctrl quality and consistency"
> https://lore.kernel.org/lkml/cover.1777419024.git.reinette.chatre@intel.com/
>
> "x86,fs/resctrl: Pave the way for MPAM counter assignment"
> https://lore.kernel.org/lkml/20260506082855.3694761-1-ben.horgan@arm.com/
>
>
> Primary resctrl fs data structure changes
> =========================================
>
> Introduces a control represented by struct resctrl_ctrl that looks as below. To make
> the changes easier to follow I kept some of the original names to help communicate
> where familiar data structures land.
>
> What to notice about a control is that it has some common properties required
> from all controls (scope, type, etc.) and then depending on the type of control
> (RESCTRL_CTRL_BITMAP or RESCTRL_CTRL_SCALAR) there are type specific properties.
>
> /**
> * struct resctrl_ctrl - A resource control
> * @entry: List entry of rdt_resource::controls
> * @scope: Scope of the resource that this control allocates
> * @domains: RCU list of all control domains
> * @type: The control type that determines the properties of the control,
> * format string for displaying control values to user space, and
> * parser of control values provided by user space.
> * @name: Name of the control. Appended to final resource name
> * (rdt_resource_final::name) to create final schema entry.
> * Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
> * For example, with resource name "MB" and control name "MAX" the
> * schema entry will be "MB_MAX".
> * @cache: Cache allocation control properties.
> * @membw: Bandwidth control properties.
> */
> struct resctrl_ctrl {
> struct list_head entry;
> enum resctrl_scope scope;
> struct list_head domains;
> enum resctrl_ctrl_type type;
> enum resctrl_ctrl_name name;
> union {
> struct resctrl_cache cache;
> struct resctrl_membw membw;
> };
> };
>
> Two members summarize how this new structure fits into the rest of resctrl:
> a) resctrl_ctrl::entry
> Since a resource can support multiple controls there is a new list
> in struct rdt_resource named "controls" that contains the list of all
> controls supported by the resource.
> b) resctrl_ctrl::domains
> Instead of the list of control domains belonging to a resource they
> now belong to the control self. By doing so resctrl can support resource
> controls at different scope for the same resource. This is intended to
> support some upcoming MPAM and RISC-V usages.
>
> Example architectural data structure changes
> ============================================
>
> An architecture can use the new control by following a similar pattern to
> resource and domain use by architectures. Consider the following for x86
> where a new architecture specific struct resctrl_hw_ctrl includes
> struct resctrl_ctrl and any architecture private data needed to support
> the control:
>
> /*
> * struct resctrl_hw_ctrl - Arch private properties of a resource control
> * @r_ctrl: Control properties exposed to resctrl file system
> * @msr_base: Base MSR address where control values should be programmed
> * @msr_update: Function pointer to update control values
> */
> struct resctrl_hw_ctrl {
> struct resctrl_ctrl r_ctrl;
> unsigned int msr_base;
> void (*msr_update)(struct msr_param *m);
> };
>
> Structure of patch series
> =========================
>
> As a PoC the series is not perfectly structured but to help navigate this work
> on a high level the changes can be categorized as follows:
>
> Patch 1 to 11:
> With a vision of what a "control" is, remove unused/unnecessary
> members, make clear what is a *resource* property vs a *control*
> property, do some renaming to help with the PoC.
>
> Patch 12:
> Introduce struct resctrl_ctrl and re-arrange existing struct rdt_resource
> members to form part of new rdt_resource::ctrl
>
> Patch 13 to 44:
> A lot of wrangling to introduce struct resctrl_ctrl to all code that needs
> to work with a control and/or domain without assuming that the control is
> the one and only control embedded in the resource it belongs to. Essentially,
> a lot of changes passing the control around in addition to the resource/domain.
>
> Patch 45:
> Switch the single struct resctrl_ctrl member of struct rdt_resource to be
> a list of struct resctrl_ctrl.
>
> Patch 47 to 49:
> Introduce new info/<resource>/resource_schemata hierarchy to first only
> consist of properties already known to resctrl fs.
>
> Patch 50 to 52:
> Introduce the new control properties per [1], initialize them for x86,
> and expose them via info/<resource>/resource_schemata
>
> Patch 53:
> Let the new struct resctrl_hw_ctrl contain architecture's control properties.
>
> Patch 54:
> Teach resctrl fs about "MIN" and "MAX" controls.
>
> Patch 55:
> Sample of "MIN" and "MAX" memory bandwidth controls for x86.
>
> Example interactions
> ====================
>
> This series can be used on an x86 system where it will show two new dummy controls
> where it is possible to interact with the new controls.
> For example:
>
> # cat schemata
> MB_MAX:0=100;1=100
> MB_MIN:0=100;1=100
> MB:0=100;1=100
> L3:0=fff;1=fff
> # echo 'MB_MIN:0=50' > schemata
> # cat schemata
> MB_MAX:0=100;1=100
> MB_MIN:0=50;1=100
> MB:0=100;1=100
> L3:0=fff;1=fff
>
> Writing to the dummy control will call a dummy callback that just prints to the
> kernel log:
> "resctrl: Updata temporary MIN control on domain 0 with user value 50"
>
>
> Example output of info/MB/:
> /sys/fs/resctrl/info/MB/thread_throttle_mode:max
> /sys/fs/resctrl/info/MB/num_closids:15
> /sys/fs/resctrl/info/MB/delay_linear:1
> /sys/fs/resctrl/info/MB/min_bandwidth:10
> /sys/fs/resctrl/info/MB/resource_schemata/MB/resolution:100
> /sys/fs/resctrl/info/MB/resource_schemata/MB/tolerance:5
> /sys/fs/resctrl/info/MB/resource_schemata/MB/type:scalar
> /sys/fs/resctrl/info/MB/resource_schemata/MB/min:10
> /sys/fs/resctrl/info/MB/resource_schemata/MB/scale:1
> /sys/fs/resctrl/info/MB/resource_schemata/MB/scope:L3
> /sys/fs/resctrl/info/MB/resource_schemata/MB/unit:all
> /sys/fs/resctrl/info/MB/resource_schemata/MB/max:100
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/resolution:100
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/tolerance:5
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/type:scalar
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/min:10
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scale:1
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scope:L3
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/unit:all
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/max:100
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/resolution:100
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/tolerance:5
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/type:scalar
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/min:10
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scale:1
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scope:L3
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/unit:all
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/max:100
> /sys/fs/resctrl/info/MB/bandwidth_gran:10
>
> Any feedback is appreciated.
>
> Reinette
>
> [1] https://lore.kernel.org/lkml/aPtfMFfLV1l%2FRB0L@e133380.arm.com/
> [2] https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf
>
Re-based GMBA patches[3] on top of this series.
[3] https://lore.kernel.org/lkml/cover.1776980182.git.babu.moger@amd.com/
#sudo mount -t resctrl resctrl /sys/fs/resctrl
#cat /sys/fs/resctrl/schemata
SMBA_GBL:0=4096;1=4096
SMBA:0=8192;1=8192;2=8192;3=8192
MB_GBL:0=4096;1=4096
MB:0=8192;1=8192;2=8192;3=8192
L3:0=ffff;1=ffff;2=ffff;3=ffff
Needed this change to get the schemata alignment correctly.
Note: Renamed GMB to MB_GBL and GSMBA to SMBA_GBL.
===========================================================
diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c
index 979176b054e8..6123689e1250 100644
--- a/fs/resctrl/ctrlmondata.c
+++ b/fs/resctrl/ctrlmondata.c
@@ -512,17 +512,22 @@ static void show_doms(struct seq_file *s, struct
rdt_resource_final *f,
{
struct rdt_resource *r = f->res;
struct rdt_ctrl_domain *dom;
+ char res_name[16];
bool sep = false;
u32 ctrl_val;
/* Walking r->domains, ensure it can't race with cpuhp */
lockdep_assert_cpus_held();
- if (print_ctrl)
- seq_printf(s, "%*s%s%s:", max_name_width, f->name,
- resctrl_ctrl_is_default(ctrl) ? "" : "_",
- resctrl_ctrl_is_default(ctrl) ?
- "" : resctrl_ctrl_name_str(ctrl->name));
+ if (print_ctrl) {
+ if (resctrl_ctrl_is_default(ctrl))
+ snprintf(res_name, sizeof(res_name), "%s", f->name);
+ else
+ snprintf(res_name, sizeof(res_name), "%s_%s",
+ f->name,
resctrl_ctrl_name_str(ctrl->name));
+ seq_printf(s, "%*s:", max_name_width, res_name);
+ }
+
list_for_each_entry(dom, &ctrl->domains, hdr.list) {
if (sep)
seq_puts(s, ";");
====================================================================
$ tree /sys/fs/resctrl/info/MB/
/sys/fs/resctrl/info/MB/
├── bandwidth_gran
├── delay_linear
├── min_bandwidth
├── num_closids
└── resource_schemata
├── MB
│ ├── max
│ ├── min
│ ├── resolution
│ ├── scale
│ ├── scope
│ ├── tolerance
│ ├── type
│ └── unit
└── MB_GBL
├── max
├── min
├── resolution
├── scale
├── scope
├── tolerance
├── type
└── unit
4 directories, 20 files
$ tree /sys/fs/resctrl/info/SMBA/
/sys/fs/resctrl/info/SMBA/
├── bandwidth_gran
├── delay_linear
├── min_bandwidth
├── num_closids
└── resource_schemata
├── SMBA
│ ├── max
│ ├── min
│ ├── resolution
│ ├── scale
│ ├── scope
│ ├── tolerance
│ ├── type
│ └── unit
└── SMBA_GBL
├── max
├── min
├── resolution
├── scale
├── scope
├── tolerance
├── type
└── unit
4 directories, 20 files
Does the info directory hierarchy look right?
After going through the comments, it seems like it should cover most of
the upcoming features. At least the Global MBA/SMBA stuff looks like it
fits in well with the RFC.
What is your expectation for the RFC going forward?
Thanks
Babu
^ permalink raw reply related [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-15 21:05 ` Moger, Babu
@ 2026-06-17 17:18 ` Reinette Chatre
2026-06-17 20:29 ` Babu Moger
0 siblings, 1 reply; 66+ messages in thread
From: Reinette Chatre @ 2026-06-17 17:18 UTC (permalink / raw)
To: Moger, Babu, Tony Luck, Ben Horgan, James Morse, Dave Martin,
Moger, Babu, Drew Fustini, Fenghua Yu, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi Babu,
On 6/15/26 2:05 PM, Moger, Babu wrote:
>
> Re-based GMBA patches[3] on top of this series.
Thank you very much for trying it out.
>
> [3] https://lore.kernel.org/lkml/cover.1776980182.git.babu.moger@amd.com/
>
> #sudo mount -t resctrl resctrl /sys/fs/resctrl
>
> #cat /sys/fs/resctrl/schemata
> SMBA_GBL:0=4096;1=4096
> SMBA:0=8192;1=8192;2=8192;3=8192
> MB_GBL:0=4096;1=4096
> MB:0=8192;1=8192;2=8192;3=8192
> L3:0=ffff;1=ffff;2=ffff;3=ffff
>
>
> Needed this change to get the schemata alignment correctly.
>
> Note: Renamed GMB to MB_GBL and GSMBA to SMBA_GBL.
>
> ===========================================================
> diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c
> index 979176b054e8..6123689e1250 100644
> --- a/fs/resctrl/ctrlmondata.c
> +++ b/fs/resctrl/ctrlmondata.c
> @@ -512,17 +512,22 @@ static void show_doms(struct seq_file *s, struct rdt_resource_final *f,
> {
> struct rdt_resource *r = f->res;
> struct rdt_ctrl_domain *dom;
> + char res_name[16];
> bool sep = false;
> u32 ctrl_val;
>
> /* Walking r->domains, ensure it can't race with cpuhp */
> lockdep_assert_cpus_held();
>
> - if (print_ctrl)
> - seq_printf(s, "%*s%s%s:", max_name_width, f->name,
> - resctrl_ctrl_is_default(ctrl) ? "" : "_",
> - resctrl_ctrl_is_default(ctrl) ?
> - "" : resctrl_ctrl_name_str(ctrl->name));
> + if (print_ctrl) {
> + if (resctrl_ctrl_is_default(ctrl))
> + snprintf(res_name, sizeof(res_name), "%s", f->name);
> + else
> + snprintf(res_name, sizeof(res_name), "%s_%s",
> + f->name, resctrl_ctrl_name_str(ctrl->name));
> + seq_printf(s, "%*s:", max_name_width, res_name);
> + }
> +
> list_for_each_entry(dom, &ctrl->domains, hdr.list) {
> if (sep)
> seq_puts(s, ";");
>
This looks like a workaround for a bug. Are you perhaps running with Tony's
fixup included?
https://lore.kernel.org/lkml/aiB2oJFPF-sqUVQx@agluck-desk3/
> ====================================================================
>
> $ tree /sys/fs/resctrl/info/MB/
> /sys/fs/resctrl/info/MB/
> ├── bandwidth_gran
> ├── delay_linear
> ├── min_bandwidth
> ├── num_closids
> └── resource_schemata
> ├── MB
> │ ├── max
> │ ├── min
> │ ├── resolution
> │ ├── scale
> │ ├── scope
> │ ├── tolerance
> │ ├── type
> │ └── unit
> └── MB_GBL
> ├── max
> ├── min
> ├── resolution
> ├── scale
> ├── scope
> ├── tolerance
> ├── type
> └── unit
>
> 4 directories, 20 files
>
>
> $ tree /sys/fs/resctrl/info/SMBA/
> /sys/fs/resctrl/info/SMBA/
> ├── bandwidth_gran
> ├── delay_linear
> ├── min_bandwidth
> ├── num_closids
> └── resource_schemata
> ├── SMBA
> │ ├── max
> │ ├── min
> │ ├── resolution
> │ ├── scale
> │ ├── scope
> │ ├── tolerance
> │ ├── type
> │ └── unit
> └── SMBA_GBL
> ├── max
> ├── min
> ├── resolution
> ├── scale
> ├── scope
> ├── tolerance
> ├── type
> └── unit
>
> 4 directories, 20 files
>
> Does the info directory hierarchy look right?
I am not sure if the "GBL" extension is most informative. In this thread there
were some motivations for the control names to include the scope. It could be useful
to create some control naming customs to not have the schemata file become chaotic.
Apart from that the hierarchy above matches what we have so far for the new schema
descriptions.
Could you please also share the contents of the new files resulting from this
implementation?
> After going through the comments, it seems like it should cover most
> of the upcoming features. At least the Global MBA/SMBA stuff looks
> like it fits in well with the RFC.
That is good news.
> What is your expectation for the RFC going forward?
The "opens" of the RFC needs to be finalized as well as the items identified during this
discussion. How to support emulated controls looks to be the biggest one. While the
initial schema description may not need to support emulated controls, how to support
it needs to be clear since the initial schema description needs to be accompanied
with enough user documentation to support such future change.
It should be possible to start upstreaming at least the preparatory patches.
Reinette
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-17 17:18 ` Reinette Chatre
@ 2026-06-17 20:29 ` Babu Moger
0 siblings, 0 replies; 66+ messages in thread
From: Babu Moger @ 2026-06-17 20:29 UTC (permalink / raw)
To: Reinette Chatre, Moger, Babu, Tony Luck, Ben Horgan, James Morse,
Dave Martin, Drew Fustini, Fenghua Yu, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi Reinette,
On 6/17/26 12:18, Reinette Chatre wrote:
> Hi Babu,
>
> On 6/15/26 2:05 PM, Moger, Babu wrote:
>>
>> Re-based GMBA patches[3] on top of this series.
>
> Thank you very much for trying it out.
>
>>
>> [3] https://lore.kernel.org/lkml/cover.1776980182.git.babu.moger@amd.com/
>>
>> #sudo mount -t resctrl resctrl /sys/fs/resctrl
>>
>> #cat /sys/fs/resctrl/schemata
>> SMBA_GBL:0=4096;1=4096
>> SMBA:0=8192;1=8192;2=8192;3=8192
>> MB_GBL:0=4096;1=4096
>> MB:0=8192;1=8192;2=8192;3=8192
>> L3:0=ffff;1=ffff;2=ffff;3=ffff
>>
>>
>> Needed this change to get the schemata alignment correctly.
>>
>> Note: Renamed GMB to MB_GBL and GSMBA to SMBA_GBL.
>>
>> ===========================================================
>> diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c
>> index 979176b054e8..6123689e1250 100644
>> --- a/fs/resctrl/ctrlmondata.c
>> +++ b/fs/resctrl/ctrlmondata.c
>> @@ -512,17 +512,22 @@ static void show_doms(struct seq_file *s, struct rdt_resource_final *f,
>> {
>> struct rdt_resource *r = f->res;
>> struct rdt_ctrl_domain *dom;
>> + char res_name[16];
>> bool sep = false;
>> u32 ctrl_val;
>>
>> /* Walking r->domains, ensure it can't race with cpuhp */
>> lockdep_assert_cpus_held();
>>
>> - if (print_ctrl)
>> - seq_printf(s, "%*s%s%s:", max_name_width, f->name,
>> - resctrl_ctrl_is_default(ctrl) ? "" : "_",
>> - resctrl_ctrl_is_default(ctrl) ?
>> - "" : resctrl_ctrl_name_str(ctrl->name));
>> + if (print_ctrl) {
>> + if (resctrl_ctrl_is_default(ctrl))
>> + snprintf(res_name, sizeof(res_name), "%s", f->name);
>> + else
>> + snprintf(res_name, sizeof(res_name), "%s_%s",
>> + f->name, resctrl_ctrl_name_str(ctrl->name));
>> + seq_printf(s, "%*s:", max_name_width, res_name);
>> + }
>> +
>> list_for_each_entry(dom, &ctrl->domains, hdr.list) {
>> if (sep)
>> seq_puts(s, ";");
>>
>
> This looks like a workaround for a bug. Are you perhaps running with Tony's
> fixup included?
> https://lore.kernel.org/lkml/aiB2oJFPF-sqUVQx@agluck-desk3/
Ah.. I missed that.
>
>> ====================================================================
>>
>> $ tree /sys/fs/resctrl/info/MB/
>> /sys/fs/resctrl/info/MB/
>> ├── bandwidth_gran
>> ├── delay_linear
>> ├── min_bandwidth
>> ├── num_closids
>> └── resource_schemata
>> ├── MB
>> │ ├── max
>> │ ├── min
>> │ ├── resolution
>> │ ├── scale
>> │ ├── scope
>> │ ├── tolerance
>> │ ├── type
>> │ └── unit
>> └── MB_GBL
>> ├── max
>> ├── min
>> ├── resolution
>> ├── scale
>> ├── scope
>> ├── tolerance
>> ├── type
>> └── unit
>>
>> 4 directories, 20 files
>>
>>
>> $ tree /sys/fs/resctrl/info/SMBA/
>> /sys/fs/resctrl/info/SMBA/
>> ├── bandwidth_gran
>> ├── delay_linear
>> ├── min_bandwidth
>> ├── num_closids
>> └── resource_schemata
>> ├── SMBA
>> │ ├── max
>> │ ├── min
>> │ ├── resolution
>> │ ├── scale
>> │ ├── scope
>> │ ├── tolerance
>> │ ├── type
>> │ └── unit
>> └── SMBA_GBL
>> ├── max
>> ├── min
>> ├── resolution
>> ├── scale
>> ├── scope
>> ├── tolerance
>> ├── type
>> └── unit
>>
>> 4 directories, 20 files
>>
>> Does the info directory hierarchy look right?
>
> I am not sure if the "GBL" extension is most informative. In this thread there
> were some motivations for the control names to include the scope. It could be useful
> to create some control naming customs to not have the schemata file become chaotic.
Make sense.
> Apart from that the hierarchy above matches what we have so far for the new schema
> descriptions.
>
> Could you please also share the contents of the new files resulting from this
> implementation?
Used this script to print:
$tree -fi info/MB | while read f; do
if [ -f "$f" ]; then
printf "%s: %s\n" "$f" "$(cat "$f")"
else
echo "$f"
fi
done
info/MB
info/MB/bandwidth_gran: 1
info/MB/delay_linear: 0
info/MB/min_bandwidth: 0
info/MB/num_closids: 16
info/MB/resource_schemata
info/MB/resource_schemata/MB
info/MB/resource_schemata/MB/max: 8192
info/MB/resource_schemata/MB/min: 0
info/MB/resource_schemata/MB/resolution: 8
info/MB/resource_schemata/MB/scale: 1
info/MB/resource_schemata/MB/scope: L3
info/MB/resource_schemata/MB/tolerance: 0
info/MB/resource_schemata/MB/type: scalar
info/MB/resource_schemata/MB/unit: GBps
info/MB/resource_schemata/MB_GBL
info/MB/resource_schemata/MB_GBL/max: 4096
info/MB/resource_schemata/MB_GBL/min: 0
info/MB/resource_schemata/MB_GBL/resolution: 1
info/MB/resource_schemata/MB_GBL/scale: 1
info/MB/resource_schemata/MB_GBL/scope: Unsupported control scope
info/MB/resource_schemata/MB_GBL/tolerance: 0
info/MB/resource_schemata/MB_GBL/type: scalar
info/MB/resource_schemata/MB_GBL/unit: GBps
4 directories, 20 files
info/SMBA
info/SMBA/bandwidth_gran: 1
info/SMBA/delay_linear: 0
info/SMBA/min_bandwidth: 0
info/SMBA/num_closids: 16
info/SMBA/resource_schemata
info/SMBA/resource_schemata/SMBA
info/SMBA/resource_schemata/SMBA/max: 8192
info/SMBA/resource_schemata/SMBA/min: 0
info/SMBA/resource_schemata/SMBA/resolution: 8
info/SMBA/resource_schemata/SMBA/scale: 1
info/SMBA/resource_schemata/SMBA/scope: L3
info/SMBA/resource_schemata/SMBA/tolerance: 0
info/SMBA/resource_schemata/SMBA/type: scalar
info/SMBA/resource_schemata/SMBA/unit: GBps
info/SMBA/resource_schemata/SMBA_GBL
info/SMBA/resource_schemata/SMBA_GBL/max: 4096
info/SMBA/resource_schemata/SMBA_GBL/min: 0
info/SMBA/resource_schemata/SMBA_GBL/resolution: 1
info/SMBA/resource_schemata/SMBA_GBL/scale: 1
info/SMBA/resource_schemata/SMBA_GBL/scope: Unsupported control scope
info/SMBA/resource_schemata/SMBA_GBL/tolerance: 0
info/SMBA/resource_schemata/SMBA_GBL/type: scalar
info/SMBA/resource_schemata/SMBA_GBL/unit: GBps
4 directories, 20 files
>
>> After going through the comments, it seems like it should cover most
>> of the upcoming features. At least the Global MBA/SMBA stuff looks
>> like it fits in well with the RFC.
>
> That is good news.
>
>> What is your expectation for the RFC going forward?
>
> The "opens" of the RFC needs to be finalized as well as the items identified during this
> discussion. How to support emulated controls looks to be the biggest one. While the
> initial schema description may not need to support emulated controls, how to support
> it needs to be clear since the initial schema description needs to be accompanied
> with enough user documentation to support such future change.
>
> It should be possible to start upstreaming at least the preparatory patches.
>
Sounds good. Will keep an eye out for it.
Thanks
Babu
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-05-29 18:06 [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept Reinette Chatre
` (5 preceding siblings ...)
2026-06-15 21:05 ` Moger, Babu
@ 2026-06-24 19:08 ` Fenghua Yu
2026-06-24 22:22 ` Reinette Chatre
6 siblings, 1 reply; 66+ messages in thread
From: Fenghua Yu @ 2026-06-24 19:08 UTC (permalink / raw)
To: Reinette Chatre, Tony Luck, Ben Horgan, James Morse, Dave Martin,
Babu Moger, Drew Fustini, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi, Reinette, Ben, Shaopen, et al,
On 5/29/26 11:06, Reinette Chatre wrote:
As Shaopen and Ben mentioned earlier, we are working on two MPAM
features that may need to change schemata interface. The CPU-less
feature was discussed on LPC (although the interfaces will be slightly
different from the LPC). Hardlimit feature was not discussed yet.
It's good to discuss this further in this RFC thread before the new
features RFCs will be sent out.
Overall, the new features can fit into this RFC well.
> Hi Everybody,
>
> It has been a while since we discussed the resctrl changes required to support
> hardware that has controls with fine granularity or hardware that has multiple
> controls per resource. For reference, the most recent email discussion can
> be found at [1] with a summary of discussions in last year's plumbers slides [2].
>
> I created a PoC that I believe supports what folks have agreed to so far. I
> hope this can help us to restart the discussion with the goal that resctrl gains
> support for upcoming hardware that require these features.
>
> Request regarding this PoC
> ==========================
>
> Please consider this PoC as a "direction check" on the schema description and multiple
> control discussions held thus far.
>
> Could folks working on enabling new hardware requiring this capability please consider
> if this is something you can build on and how it should be improved to support these
> upcoming capabilities?
>
> Opens
> =====
>
> While the PoC aims to support what folks agreed on some opens remain:
> - I attempted to make some MPAM supporting changes but these are all just compile
> tested. While MPAM should benefit from the new control properties I did not
> initialize them on MPAM and did not attempt refactor to separate out
> the architecture specific control properties (more on what this means later).
> I did attempt some MPAM refactoring that duplicates the MPAM domain to the
> control domain and monitoring domain lists in support of there being multiple
> controls each with its own list of control domains but it is definitely not good
> design.
> - No support for emulated controls (yet). The PoC is quite large already
> but I think it can be used as a base for emulated controls for which the software
> controller could be a potential first customer. In this PoC mounting with
> software controller will still display the original controller's properties.
> - One open that needs to be addressed as part of support for emulated controls is
> how best to display emulation relationship via resctrl hierarchy.
> - No support for "read-modify-write" usage of schemata file. This is where we
> discussed (without agreement) on possibly introducing the "#" prefix to schemata
> file entries. This PoC does not support this prefix and the current assumption/expectation
> is that when user space changes a configuration only the new control values are
> written to schemata file. I thus do not have a plan to support this so please
> share opinions in this regard if you have some.
> - Controls are independent for now. This means that, for example, if a resource
> supports a "MIN" and "MAX" control then this implementation would allow user to
> set the "maximum" control values to be less than the "minimum" control values.
> - PoC supports the "bitmap" control but does not (yet) expose properties of a bitmap
> control to the new info/<resource>/resource_schemata directory.
>
> Accessing PoC
> =============
>
> Please consider the PoC as a rough draft. It has only been compile tested for Arm
> and known to be incomplete in Arm support. To help with experimenting I only
> fully adapted the Intel MBA resource to demo two dummy additional MBA controls.
> All architectures should immediately benefit from the new schema descriptions
> and new info/MB/resource_schemata hierarchy.
>
> I considered the patches self too many for email. Instead, the PoC can be found at:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git branch resctrl/controls_rfc_v1
>
> The work is based on v7.1-rc2 that also includes the following series (two of which has
> since been queued) included:
>
> "selftests/resctrl: Fixes and improvements focused on Intel platforms"
> https://lore.kernel.org/lkml/cover.1775266384.git.reinette.chatre@intel.com/
>
> "x86,fs/resctrl: Improve resctrl quality and consistency"
> https://lore.kernel.org/lkml/cover.1777419024.git.reinette.chatre@intel.com/
>
> "x86,fs/resctrl: Pave the way for MPAM counter assignment"
> https://lore.kernel.org/lkml/20260506082855.3694761-1-ben.horgan@arm.com/
>
>
> Primary resctrl fs data structure changes
> =========================================
>
> Introduces a control represented by struct resctrl_ctrl that looks as below. To make
> the changes easier to follow I kept some of the original names to help communicate
> where familiar data structures land.
>
> What to notice about a control is that it has some common properties required
> from all controls (scope, type, etc.) and then depending on the type of control
> (RESCTRL_CTRL_BITMAP or RESCTRL_CTRL_SCALAR) there are type specific properties.
>
> /**
> * struct resctrl_ctrl - A resource control
> * @entry: List entry of rdt_resource::controls
> * @scope: Scope of the resource that this control allocates
> * @domains: RCU list of all control domains
> * @type: The control type that determines the properties of the control,
> * format string for displaying control values to user space, and
> * parser of control values provided by user space.
> * @name: Name of the control. Appended to final resource name
> * (rdt_resource_final::name) to create final schema entry.
> * Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
> * For example, with resource name "MB" and control name "MAX" the
> * schema entry will be "MB_MAX".
> * @cache: Cache allocation control properties.
> * @membw: Bandwidth control properties.
> */
> struct resctrl_ctrl {
> struct list_head entry;
> enum resctrl_scope scope;
> struct list_head domains;
> enum resctrl_ctrl_type type;
> enum resctrl_ctrl_name name;
> union {
> struct resctrl_cache cache;
> struct resctrl_membw membw;
> };
> };
>
> Two members summarize how this new structure fits into the rest of resctrl:
> a) resctrl_ctrl::entry
> Since a resource can support multiple controls there is a new list
> in struct rdt_resource named "controls" that contains the list of all
> controls supported by the resource.
> b) resctrl_ctrl::domains
> Instead of the list of control domains belonging to a resource they
> now belong to the control self. By doing so resctrl can support resource
> controls at different scope for the same resource. This is intended to
> support some upcoming MPAM and RISC-V usages.
>
> Example architectural data structure changes
> ============================================
>
> An architecture can use the new control by following a similar pattern to
> resource and domain use by architectures. Consider the following for x86
> where a new architecture specific struct resctrl_hw_ctrl includes
> struct resctrl_ctrl and any architecture private data needed to support
> the control:
>
> /*
> * struct resctrl_hw_ctrl - Arch private properties of a resource control
> * @r_ctrl: Control properties exposed to resctrl file system
> * @msr_base: Base MSR address where control values should be programmed
> * @msr_update: Function pointer to update control values
> */
> struct resctrl_hw_ctrl {
> struct resctrl_ctrl r_ctrl;
> unsigned int msr_base;
> void (*msr_update)(struct msr_param *m);
> };
>
> Structure of patch series
> =========================
>
> As a PoC the series is not perfectly structured but to help navigate this work
> on a high level the changes can be categorized as follows:
>
> Patch 1 to 11:
> With a vision of what a "control" is, remove unused/unnecessary
> members, make clear what is a *resource* property vs a *control*
> property, do some renaming to help with the PoC.
>
> Patch 12:
> Introduce struct resctrl_ctrl and re-arrange existing struct rdt_resource
> members to form part of new rdt_resource::ctrl
>
> Patch 13 to 44:
> A lot of wrangling to introduce struct resctrl_ctrl to all code that needs
> to work with a control and/or domain without assuming that the control is
> the one and only control embedded in the resource it belongs to. Essentially,
> a lot of changes passing the control around in addition to the resource/domain.
>
> Patch 45:
> Switch the single struct resctrl_ctrl member of struct rdt_resource to be
> a list of struct resctrl_ctrl.
>
> Patch 47 to 49:
> Introduce new info/<resource>/resource_schemata hierarchy to first only
> consist of properties already known to resctrl fs.
>
> Patch 50 to 52:
> Introduce the new control properties per [1], initialize them for x86,
> and expose them via info/<resource>/resource_schemata
>
> Patch 53:
> Let the new struct resctrl_hw_ctrl contain architecture's control properties.
>
> Patch 54:
> Teach resctrl fs about "MIN" and "MAX" controls.
>
> Patch 55:
> Sample of "MIN" and "MAX" memory bandwidth controls for x86.
>
> Example interactions
> ====================
>
> This series can be used on an x86 system where it will show two new dummy controls
> where it is possible to interact with the new controls.
> For example:
>
> # cat schemata
> MB_MAX:0=100;1=100
> MB_MIN:0=100;1=100
> MB:0=100;1=100
Some platforms may support CPU-less node which is represented by numa
node id, examples:
1. CXL type 2 memory node which provides CXL memory without CPU and L3
on the node
2. GPU memory node that can be accessed by all CPUs but doesn't have a
local CPU and L3 bound to.
etc.
MPAM can allocate and monitor mem bandwidth on these memory node.
Since no CPU and L3 on the node, cache id cannot be used in "MB:" line.
Instead, numa ids are used to identify MB allocation and monitoring.
For example, the MB allocation on CPU-less platforms could be:
MB:0=100;1=100;2=100;10=100;18=100;26=100
Where: domain id 0, 1, 2, etc are numa node id shown in
/sys/devices/system/node directory or by numctl.
0: socket 0, node 0, CPUs, memory
1: socket 1, node 1, CPUs, memory
2: GPU 0, node 2, no CPU, memory only
10: GPU 1, node 10, no CPU, memory only
18: GPU 3, node 18, no CPU, memory only
26: GPU 4, node 26, no CPU, memory only
Arch specific driver (e.g. MPAM) detects CPU-less node. If there is any
CPU-less node, use numa id in "MB:". Otherwise, fallback to legacy cache id.
There is another MPAM feature called MBW Max hardlimit which sets "MB:"
allocation as hardlimit (i.e. MBW throttling percentage must be
satisfied) per domain. Adding a new "MB_HLIM:" line in schemata. It's
1:1 mapped to "MB:" to control hardlimit of MB throttling percentage on
each domain. By default hardlimit is off (0) and can be turned on to set
MBW Max hardlimit on a domain.
For exmple:
MB_HLIM: 0=0;1=0;2=1;10=0;18=0;26=0
MB:0=100;1=100;2=80;10=100;18=100;26=100
On GPU memory numa node 2: cannot use more than 80% of total max mbw
even if there is still idle mem bandwidth on this node).
MBW allocations on all other domains are soft limited, meaning MBW can
be used more than specified if mem is idle.
> L3:0=fff;1=fff
> # echo 'MB_MIN:0=50' > schemata
> # cat schemata
> MB_MAX:0=100;1=100
> MB_MIN:0=50;1=100
> MB:0=100;1=100
> L3:0=fff;1=fff
>
> Writing to the dummy control will call a dummy callback that just prints to the
> kernel log:
> "resctrl: Updata temporary MIN control on domain 0 with user value 50"
>
>
> Example output of info/MB/:
> /sys/fs/resctrl/info/MB/thread_throttle_mode:max
> /sys/fs/resctrl/info/MB/num_closids:15
> /sys/fs/resctrl/info/MB/delay_linear:1
> /sys/fs/resctrl/info/MB/min_bandwidth:10
Add two new MB info RO files:
1. /sys/fs/resctrl/info/MB/domain_id
It shows "numa" for using numa id in "MB:" or "cache" for using legacy
cache id.
2. /sys/fs/resctrl/info/MB/max_lim
It shows number 0-3 for MPAM MBW max limit behaviors: 0 for supporting
both softlimit and hardlimit, etc.
> /sys/fs/resctrl/info/MB/resource_schemata/MB/resolution:100
Is it more concise to s/resource_schemata/schemata/? "resource_" seems
redundant in the context "info/MB".
> /sys/fs/resctrl/info/MB/resource_schemata/MB/tolerance:5
> /sys/fs/resctrl/info/MB/resource_schemata/MB/type:scalar
> /sys/fs/resctrl/info/MB/resource_schemata/MB/min:10
> /sys/fs/resctrl/info/MB/resource_schemata/MB/scale:1
> /sys/fs/resctrl/info/MB/resource_schemata/MB/scope:L3
> /sys/fs/resctrl/info/MB/resource_schemata/MB/unit:all
> /sys/fs/resctrl/info/MB/resource_schemata/MB/max:100
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/resolution:100
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/tolerance:5
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/type:scalar
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/min:10
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scale:1
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scope:L3
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/unit:all
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/max:100
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/resolution:100
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/tolerance:5
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/type:scalar
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/min:10
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scale:1
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scope:L3
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/unit:all
> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/max:100
> /sys/fs/resctrl/info/MB/bandwidth_gran:10
For MBW monitoring, extend mon_data/ directory to monitor CPU-less
memory node. For example,
On legacy platforms (i.e. L3 and memory are described in same MPAM ACPI
MSC wich doesn't support CPU-less nodes):
mon_data/mbm_L3_01/llc_occupancy
mon_data/mbm_L3_01/mbm_total_bytes
mon_data/mbm_L3_02/llc_occupancy
mon_data/mbm_L3_02/mbm_total_bytes
On platforms with L3 and memory in separate MPAM ACPI MSCs
but there is no CPU-less node:
mon_data/mbm_L3_01/llc_occupancy <- cache id 1
mon_data/mbm_L3_02/llc_occupancy <- cache id 2
mon_data/mbm_MB_00/mbm_total_bytes <- numa node 0 (socket 0)
mon_data/mbm_MB_01/mbm_total_bytes <- numa node 1 (socket 1)
On platforms with L3 and memory in separate MPAM ACPI MSCs
and there are CPU-less nodes:
mon_data/mbm_L3_01/llc_occupancy <- cache id 1
mon_data/mbm_L3_02/llc_occupancy <- cache id 2
mon_data/mbm_MB_00/mbm_total_bytes <- numa node 0 (socket 0)
mon_data/mbm_MB_01/mbm_total_bytes <- numa node 1 (socket 1)
mon_data/mbm_MB_02/mbm_total_bytes <- numa node 2 (GPU 0 mem)
mon_data/mbm_MB_10/mbm_total_bytes <- numa node 10 (GPU 1 mem)
mon_data/mbm_MB_18/mbm_total_bytes <- numa node 18 (GPU 2 mem)
mon_data/mbm_MB_26/mbm_total_bytes <- numa node 26 (GPU 3 mem)
Thanks.
-Fenghua
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-24 19:08 ` Fenghua Yu
@ 2026-06-24 22:22 ` Reinette Chatre
2026-06-25 1:26 ` Fenghua Yu
0 siblings, 1 reply; 66+ messages in thread
From: Reinette Chatre @ 2026-06-24 22:22 UTC (permalink / raw)
To: Fenghua Yu, Tony Luck, Ben Horgan, James Morse, Dave Martin,
Babu Moger, Drew Fustini, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi Fenghua,
On 6/24/26 12:08 PM, Fenghua Yu wrote:
> Hi, Reinette, Ben, Shaopen, et al,
>
> On 5/29/26 11:06, Reinette Chatre wrote:
>
> As Shaopen and Ben mentioned earlier, we are working on two MPAM
> features that may need to change schemata interface. The CPU-less
> feature was discussed on LPC (although the interfaces will be
> slightly different from the LPC).
I know. Here is where I tried to engage with you on needed interfaces after LPC:
https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@intel.com/
> Hardlimit feature was not discussed yet.
It was considered. See discussion starting at
https://lore.kernel.org/lkml/1c4b6b46-16f9-4887-93f5-e0f5e7f30a6f@intel.com/
> It's good to discuss this further in this RFC thread before the new features RFCs will be sent out.
ack.
>
> Overall, the new features can fit into this RFC well.
>
>> Hi Everybody,
>>
>> It has been a while since we discussed the resctrl changes required to support
>> hardware that has controls with fine granularity or hardware that has multiple
>> controls per resource. For reference, the most recent email discussion can
>> be found at [1] with a summary of discussions in last year's plumbers slides [2].
>>
>> I created a PoC that I believe supports what folks have agreed to so far. I
>> hope this can help us to restart the discussion with the goal that resctrl gains
>> support for upcoming hardware that require these features.
>>
>> Request regarding this PoC
>> ==========================
>>
>> Please consider this PoC as a "direction check" on the schema description and multiple
>> control discussions held thus far.
>>
>> Could folks working on enabling new hardware requiring this capability please consider
>> if this is something you can build on and how it should be improved to support these
>> upcoming capabilities?
>>
>> Opens
>> =====
>>
>> While the PoC aims to support what folks agreed on some opens remain:
>> - I attempted to make some MPAM supporting changes but these are all just compile
>> tested. While MPAM should benefit from the new control properties I did not
>> initialize them on MPAM and did not attempt refactor to separate out
>> the architecture specific control properties (more on what this means later).
>> I did attempt some MPAM refactoring that duplicates the MPAM domain to the
>> control domain and monitoring domain lists in support of there being multiple
>> controls each with its own list of control domains but it is definitely not good
>> design.
>> - No support for emulated controls (yet). The PoC is quite large already
>> but I think it can be used as a base for emulated controls for which the software
>> controller could be a potential first customer. In this PoC mounting with
>> software controller will still display the original controller's properties.
>> - One open that needs to be addressed as part of support for emulated controls is
>> how best to display emulation relationship via resctrl hierarchy.
>> - No support for "read-modify-write" usage of schemata file. This is where we
>> discussed (without agreement) on possibly introducing the "#" prefix to schemata
>> file entries. This PoC does not support this prefix and the current assumption/expectation
>> is that when user space changes a configuration only the new control values are
>> written to schemata file. I thus do not have a plan to support this so please
>> share opinions in this regard if you have some.
>> - Controls are independent for now. This means that, for example, if a resource
>> supports a "MIN" and "MAX" control then this implementation would allow user to
>> set the "maximum" control values to be less than the "minimum" control values.
>> - PoC supports the "bitmap" control but does not (yet) expose properties of a bitmap
>> control to the new info/<resource>/resource_schemata directory.
>>
>> Accessing PoC
>> =============
>>
>> Please consider the PoC as a rough draft. It has only been compile tested for Arm
>> and known to be incomplete in Arm support. To help with experimenting I only
>> fully adapted the Intel MBA resource to demo two dummy additional MBA controls.
>> All architectures should immediately benefit from the new schema descriptions
>> and new info/MB/resource_schemata hierarchy.
>>
>> I considered the patches self too many for email. Instead, the PoC can be found at:
>>
>> git://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git branch resctrl/controls_rfc_v1
>>
>> The work is based on v7.1-rc2 that also includes the following series (two of which has
>> since been queued) included:
>>
>> "selftests/resctrl: Fixes and improvements focused on Intel platforms"
>> https://lore.kernel.org/lkml/cover.1775266384.git.reinette.chatre@intel.com/
>>
>> "x86,fs/resctrl: Improve resctrl quality and consistency"
>> https://lore.kernel.org/lkml/cover.1777419024.git.reinette.chatre@intel.com/
>>
>> "x86,fs/resctrl: Pave the way for MPAM counter assignment"
>> https://lore.kernel.org/lkml/20260506082855.3694761-1-ben.horgan@arm.com/
>>
>>
>> Primary resctrl fs data structure changes
>> =========================================
>>
>> Introduces a control represented by struct resctrl_ctrl that looks as below. To make
>> the changes easier to follow I kept some of the original names to help communicate
>> where familiar data structures land.
>>
>> What to notice about a control is that it has some common properties required
>> from all controls (scope, type, etc.) and then depending on the type of control
>> (RESCTRL_CTRL_BITMAP or RESCTRL_CTRL_SCALAR) there are type specific properties.
>>
>> /**
>> * struct resctrl_ctrl - A resource control
>> * @entry: List entry of rdt_resource::controls
>> * @scope: Scope of the resource that this control allocates
>> * @domains: RCU list of all control domains
>> * @type: The control type that determines the properties of the control,
>> * format string for displaying control values to user space, and
>> * parser of control values provided by user space.
>> * @name: Name of the control. Appended to final resource name
>> * (rdt_resource_final::name) to create final schema entry.
>> * Specifically, "rdt_resource_final::name"_"resctrl_ctrl::name".
>> * For example, with resource name "MB" and control name "MAX" the
>> * schema entry will be "MB_MAX".
>> * @cache: Cache allocation control properties.
>> * @membw: Bandwidth control properties.
>> */
>> struct resctrl_ctrl {
>> struct list_head entry;
>> enum resctrl_scope scope;
>> struct list_head domains;
>> enum resctrl_ctrl_type type;
>> enum resctrl_ctrl_name name;
>> union {
>> struct resctrl_cache cache;
>> struct resctrl_membw membw;
>> };
>> };
>>
>> Two members summarize how this new structure fits into the rest of resctrl:
>> a) resctrl_ctrl::entry
>> Since a resource can support multiple controls there is a new list
>> in struct rdt_resource named "controls" that contains the list of all
>> controls supported by the resource.
>> b) resctrl_ctrl::domains
>> Instead of the list of control domains belonging to a resource they
>> now belong to the control self. By doing so resctrl can support resource
>> controls at different scope for the same resource. This is intended to
>> support some upcoming MPAM and RISC-V usages.
>>
>> Example architectural data structure changes
>> ============================================
>>
>> An architecture can use the new control by following a similar pattern to
>> resource and domain use by architectures. Consider the following for x86
>> where a new architecture specific struct resctrl_hw_ctrl includes
>> struct resctrl_ctrl and any architecture private data needed to support
>> the control:
>>
>> /*
>> * struct resctrl_hw_ctrl - Arch private properties of a resource control
>> * @r_ctrl: Control properties exposed to resctrl file system
>> * @msr_base: Base MSR address where control values should be programmed
>> * @msr_update: Function pointer to update control values
>> */
>> struct resctrl_hw_ctrl {
>> struct resctrl_ctrl r_ctrl;
>> unsigned int msr_base;
>> void (*msr_update)(struct msr_param *m);
>> };
>>
>> Structure of patch series
>> =========================
>>
>> As a PoC the series is not perfectly structured but to help navigate this work
>> on a high level the changes can be categorized as follows:
>>
>> Patch 1 to 11:
>> With a vision of what a "control" is, remove unused/unnecessary
>> members, make clear what is a *resource* property vs a *control*
>> property, do some renaming to help with the PoC.
>>
>> Patch 12:
>> Introduce struct resctrl_ctrl and re-arrange existing struct rdt_resource
>> members to form part of new rdt_resource::ctrl
>>
>> Patch 13 to 44:
>> A lot of wrangling to introduce struct resctrl_ctrl to all code that needs
>> to work with a control and/or domain without assuming that the control is
>> the one and only control embedded in the resource it belongs to. Essentially,
>> a lot of changes passing the control around in addition to the resource/domain.
>>
>> Patch 45:
>> Switch the single struct resctrl_ctrl member of struct rdt_resource to be
>> a list of struct resctrl_ctrl.
>>
>> Patch 47 to 49:
>> Introduce new info/<resource>/resource_schemata hierarchy to first only
>> consist of properties already known to resctrl fs.
>>
>> Patch 50 to 52:
>> Introduce the new control properties per [1], initialize them for x86,
>> and expose them via info/<resource>/resource_schemata
>>
>> Patch 53:
>> Let the new struct resctrl_hw_ctrl contain architecture's control properties.
>>
>> Patch 54:
>> Teach resctrl fs about "MIN" and "MAX" controls.
>>
>> Patch 55:
>> Sample of "MIN" and "MAX" memory bandwidth controls for x86.
>>
>> Example interactions
>> ====================
>>
>> This series can be used on an x86 system where it will show two new dummy controls
>> where it is possible to interact with the new controls.
>> For example:
>>
>> # cat schemata
>> MB_MAX:0=100;1=100
>> MB_MIN:0=100;1=100
>> MB:0=100;1=100
>
> Some platforms may support CPU-less node which is represented by numa node id, examples:
> 1. CXL type 2 memory node which provides CXL memory without CPU and L3 on the node
> 2. GPU memory node that can be accessed by all CPUs but doesn't have a local CPU and L3 bound to.
> etc.
>
> MPAM can allocate and monitor mem bandwidth on these memory node.
> Since no CPU and L3 on the node, cache id cannot be used in "MB:" line. Instead, numa ids are used to identify MB allocation and monitoring.
>
> For example, the MB allocation on CPU-less platforms could be:
> MB:0=100;1=100;2=100;10=100;18=100;26=100
>
> Where: domain id 0, 1, 2, etc are numa node id shown in /sys/devices/system/node directory or by numctl.
> 0: socket 0, node 0, CPUs, memory
> 1: socket 1, node 1, CPUs, memory
> 2: GPU 0, node 2, no CPU, memory only
> 10: GPU 1, node 10, no CPU, memory only
> 18: GPU 3, node 18, no CPU, memory only
> 26: GPU 4, node 26, no CPU, memory only
>
> Arch specific driver (e.g. MPAM) detects CPU-less node. If there is any CPU-less node, use numa id in "MB:". Otherwise, fallback to legacy cache id.
We always have to consider backward compatibility and to do so we cannot just retroactively
change what domain ID represents when user space interacts with the "MB" control.
The legacy "MB" control is already defined and its domain ID represents an L3 cache ID. To
support these new devices resctrl would need to expose a new control.
>
> There is another MPAM feature called MBW Max hardlimit which sets
> "MB:" allocation as hardlimit (i.e. MBW throttling percentage must
> be satisfied) per domain. Adding a new "MB_HLIM:" line in schemata.
> It's 1:1 mapped to "MB:" to control hardlimit of MB throttling
> percentage on each domain. By default hardlimit is off (0) and can
> be turned on to set MBW Max hardlimit on a domain.
ack. This sounds like a new control associated with the MB resource.
This is a boolean control as Dave highlighted in previous discussion so
resctrl would need to know its properties.
See https://lore.kernel.org/lkml/aO0Oazuxt54hQFbx@e133380.arm.com/
> For exmple:
> MB_HLIM: 0=0;1=0;2=1;10=0;18=0;26=0
> MB:0=100;1=100;2=80;10=100;18=100;26=100
>
> On GPU memory numa node 2: cannot use more than 80% of total max mbw even if there is still idle mem bandwidth on this node).
>
> MBW allocations on all other domains are soft limited, meaning MBW can be used more than specified if mem is idle.
>
ack.
>> L3:0=fff;1=fff
>> # echo 'MB_MIN:0=50' > schemata
>> # cat schemata
>> MB_MAX:0=100;1=100
>> MB_MIN:0=50;1=100
>> MB:0=100;1=100
>> L3:0=fff;1=fff
>>
>> Writing to the dummy control will call a dummy callback that just prints to the
>> kernel log:
>> "resctrl: Updata temporary MIN control on domain 0 with user value 50"
>>
>>
>> Example output of info/MB/:
>> /sys/fs/resctrl/info/MB/thread_throttle_mode:max
>> /sys/fs/resctrl/info/MB/num_closids:15
>> /sys/fs/resctrl/info/MB/delay_linear:1
>> /sys/fs/resctrl/info/MB/min_bandwidth:10
>
> Add two new MB info RO files:
> 1. /sys/fs/resctrl/info/MB/domain_id
> It shows "numa" for using numa id in "MB:" or "cache" for using legacy cache id.
This proposal introduces a *global* property to the MB *resource*? It does not seem as though
this takes into account *anything* about how resctrl can support new hardware that has been
discussed before, during, or after LPC. You have not participated in these discussions and
now make an orthogonal proposal that does not take into account *any* of the requirements
that we have been struggling with for months.
Why should this proposal be taken seriously? In your absence folks have been trying to
accommodate how these upcoming products and be supported and the "scope" file associated with
a control is intended to communicate to user space how the domain ID should be interpreted.
Why are you proposing something entirely different here without even acknowledging current
approach and explaining why it does not work for you?
>
> 2. /sys/fs/resctrl/info/MB/max_lim
> It shows number 0-3 for MPAM MBW max limit behaviors: 0 for supporting both softlimit and hardlimit, etc.
Again this adds another *global* property to the MB resource but then above you
describe the new "MB_HLIM" schemata file entry that implies that it is a new control
for the MB resource. Having it be a new control for the MB resource matches earlier
discussions. To support this I thus expect it to be exposed as a new control with
potentially a new type if any of the existing planned types do not suffice.
>> /sys/fs/resctrl/info/MB/resource_schemata/MB/resolution:100
>
> Is it more concise to s/resource_schemata/schemata/? "resource_" seems redundant in the context "info/MB".
We could do this, yes.
>
>> /sys/fs/resctrl/info/MB/resource_schemata/MB/tolerance:5
>> /sys/fs/resctrl/info/MB/resource_schemata/MB/type:scalar
>> /sys/fs/resctrl/info/MB/resource_schemata/MB/min:10
>> /sys/fs/resctrl/info/MB/resource_schemata/MB/scale:1
>> /sys/fs/resctrl/info/MB/resource_schemata/MB/scope:L3
>> /sys/fs/resctrl/info/MB/resource_schemata/MB/unit:all
>> /sys/fs/resctrl/info/MB/resource_schemata/MB/max:100
>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/resolution:100
>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/tolerance:5
>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/type:scalar
>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/min:10
>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scale:1
>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scope:L3
>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/unit:all
>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/max:100
>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/resolution:100
>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/tolerance:5
>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/type:scalar
>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/min:10
>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scale:1
>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scope:L3
>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/unit:all
>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/max:100
>> /sys/fs/resctrl/info/MB/bandwidth_gran:10
>
> For MBW monitoring, extend mon_data/ directory to monitor CPU-less memory node. For example,
Here is where I attempted to discuss with you how to support monitoring on these systems:
https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@intel.com/
Here again you respond with something completely different without acknowledging
the previous discussion or noting why that does not work for you.
>
> On legacy platforms (i.e. L3 and memory are described in same MPAM ACPI MSC wich doesn't support CPU-less nodes):
> mon_data/mbm_L3_01/llc_occupancy
> mon_data/mbm_L3_01/mbm_total_bytes
> mon_data/mbm_L3_02/llc_occupancy
> mon_data/mbm_L3_02/mbm_total_bytes
>
> On platforms with L3 and memory in separate MPAM ACPI MSCs
> but there is no CPU-less node:
> mon_data/mbm_L3_01/llc_occupancy <- cache id 1
> mon_data/mbm_L3_02/llc_occupancy <- cache id 2
> mon_data/mbm_MB_00/mbm_total_bytes <- numa node 0 (socket 0)
> mon_data/mbm_MB_01/mbm_total_bytes <- numa node 1 (socket 1)
Here too I do not find it appropriate for resctrl to retroactively
change its interface. "MB" is a resource and the above switches the
resctrl interface to imply the monitoring data of a resource can be
found in the mon directory that matches the resource name. This is
not what resctrl does today. Doing something like above will result in
resctrl having a confusing interface where "sometimes" memory bandwidth
data can be found in the L3 directory and "sometimes" memory bandwidth
data can be found in the MB directory.
As I described to you in December resctrl already exposes the monitoring
data based on the *scope*. As you also point out above, today the memory
bandwidth monitoring data at L3 scope can be found in the L3 directory.
"L3" should thus not be interpreted as the resource L3 but the scope L3
since it contains MBM data today. When viewing it as such resctrl could
internally be more explicit and separate monitoring scope from monitoring
resource and present the monitoring data based on scope to remain intuitively
backward compatible while obtaining support for these memory nodes.
>
> On platforms with L3 and memory in separate MPAM ACPI MSCs
> and there are CPU-less nodes:
> mon_data/mbm_L3_01/llc_occupancy <- cache id 1
> mon_data/mbm_L3_02/llc_occupancy <- cache id 2
> mon_data/mbm_MB_00/mbm_total_bytes <- numa node 0 (socket 0)
> mon_data/mbm_MB_01/mbm_total_bytes <- numa node 1 (socket 1)
> mon_data/mbm_MB_02/mbm_total_bytes <- numa node 2 (GPU 0 mem)
> mon_data/mbm_MB_10/mbm_total_bytes <- numa node 10 (GPU 1 mem)
> mon_data/mbm_MB_18/mbm_total_bytes <- numa node 18 (GPU 2 mem)
> mon_data/mbm_MB_26/mbm_total_bytes <- numa node 26 (GPU 3 mem)
To be backward compatible I find it more intuitive if instead
this data is exposed as below:
mon_data/mbm_L3_01/llc_occupancy <- cache id 1
mon_data/mbm_L3_02/llc_occupancy <- cache id 2
mon_data/mbm_NODE_00/mbm_total_bytes <- numa node 0 (socket 0)
mon_data/mbm_NODE_01/mbm_total_bytes <- numa node 1 (socket 1)
mon_data/mbm_NODE_02/mbm_total_bytes <- numa node 2 (GPU 0 mem)
mon_data/mbm_NODE_10/mbm_total_bytes <- numa node 10 (GPU 1 mem)
mon_data/mbm_NODE_18/mbm_total_bytes <- numa node 18 (GPU 2 mem)
mon_data/mbm_NODE_26/mbm_total_bytes <- numa node 26 (GPU 3 mem)
When "mbm_total_bytes" move from mon_data/mbm_L3_x to
mon_data/mbm_NODE_x it clearly indicates that it is memory bandwidth
monitoring data moving from "L3" scope to "NODE" scope.
Reinette
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-24 22:22 ` Reinette Chatre
@ 2026-06-25 1:26 ` Fenghua Yu
2026-06-25 15:43 ` Reinette Chatre
2026-07-02 13:37 ` Ben Horgan
0 siblings, 2 replies; 66+ messages in thread
From: Fenghua Yu @ 2026-06-25 1:26 UTC (permalink / raw)
To: Reinette Chatre, Tony Luck, Ben Horgan, James Morse, Dave Martin,
Babu Moger, Drew Fustini, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi, Reinette,
On 6/24/26 15:22, Reinette Chatre wrote:
> Hi Fenghua,
>
> On 6/24/26 12:08 PM, Fenghua Yu wrote:
>> Hi, Reinette, Ben, Shaopen, et al,
>>
>> On 5/29/26 11:06, Reinette Chatre wrote:
>>
>> As Shaopen and Ben mentioned earlier, we are working on two MPAM
>> features that may need to change schemata interface. The CPU-less
>> feature was discussed on LPC (although the interfaces will be
>> slightly different from the LPC).
>
> I know. Here is where I tried to engage with you on needed interfaces after LPC:
> https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@intel.com/
MPAM ACPI defines MSC (Memory System Control) is defined in one of two
ways (not both) on one platform:
1. L3 and memory together on each processor MSC
2. L3 in processor MSC and memory control/monitoring in different memory
MSCs.
On type 1 platform, schemata is legacy:
MB:1=100;2=100 <-- cache id 1 and 2 as domain id
On type 2 platform, I will not reuse "MB:" name. Instead, define new
resource name "MBN:" for numa node and schemata is:
MBN:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10, 18,
26 as domain id
On type 2 platform, there won't be "MB:" line. Numa 0 and 1
are for mbm allocation on socket 0 and 1. 2,10, 18 and 26 are for GPU
memory nodes allocation.
BTW, Slow MBA (SMBA) is different from MBA Numa (MBN). SMBA still relies
on L3 and the domain id in SMBA is still cache id. MBN depends on each
memory controlor with numa id as domain id for both CPU and CPU-less
memory nodes.
On type 1 platform, there is only MB:
info
└── MB
└── resource_schemata
├── MB
│ ├── max
│ ├── min
│ ├── resolution
│ ├── scale
│ ├── scope <== contains "L3"
│ ├── tolerance
│ ├── type
│ └── unit
On type 2 platform, there is only MBN:
info
└── MBN
└── resource_schemata
├── MBN
│ ├── max
│ ├── min
│ ├── resolution
│ ├── scale
│ ├── scope <== contains "NUMA"
│ ├── tolerance
│ ├── type
│ └── unit
This is different from the "scope" hierarchy discussed in the link. "MB"
and "MBN" won't exist on the same platform.
I find it's hard (and not useful) to split "MB" for memory with CPU and
"MBN" for CPU-less memory node. It's easier to have either "MB" for
legacy memory with CPU or "MBN" for CPU-less memory.
Any thoughts? Does this update make sense?
>
>> Hardlimit feature was not discussed yet.
>
> It was considered. See discussion starting at
> https://lore.kernel.org/lkml/1c4b6b46-16f9-4887-93f5-e0f5e7f30a6f@intel.com/
>
>> It's good to discuss this further in this RFC thread before the new features RFCs will be sent out.
>
> ack.
>
>>
>> Overall, the new features can fit into this RFC well.
>>
>>> Hi Everybody,
>>>
[SNIP]
>>>
>>> This series can be used on an x86 system where it will show two new dummy controls
>>> where it is possible to interact with the new controls.
>>> For example:
>>>
>>> # cat schemata
>>> MB_MAX:0=100;1=100
>>> MB_MIN:0=100;1=100
>>> MB:0=100;1=100
>>
>> Some platforms may support CPU-less node which is represented by numa node id, examples:
>> 1. CXL type 2 memory node which provides CXL memory without CPU and L3 on the node
>> 2. GPU memory node that can be accessed by all CPUs but doesn't have a local CPU and L3 bound to.
>> etc.
>>
>> MPAM can allocate and monitor mem bandwidth on these memory node.
>> Since no CPU and L3 on the node, cache id cannot be used in "MB:" line. Instead, numa ids are used to identify MB allocation and monitoring.
>>
>> For example, the MB allocation on CPU-less platforms could be:
>> MB:0=100;1=100;2=100;10=100;18=100;26=100
>>
>> Where: domain id 0, 1, 2, etc are numa node id shown in /sys/devices/system/node directory or by numctl.
>> 0: socket 0, node 0, CPUs, memory
>> 1: socket 1, node 1, CPUs, memory
>> 2: GPU 0, node 2, no CPU, memory only
>> 10: GPU 1, node 10, no CPU, memory only
>> 18: GPU 3, node 18, no CPU, memory only
>> 26: GPU 4, node 26, no CPU, memory only
>>
>> Arch specific driver (e.g. MPAM) detects CPU-less node. If there is any CPU-less node, use numa id in "MB:". Otherwise, fallback to legacy cache id.
>
> We always have to consider backward compatibility and to do so we cannot just retroactively
> change what domain ID represents when user space interacts with the "MB" control.
>
> The legacy "MB" control is already defined and its domain ID represents an L3 cache ID. To
> support these new devices resctrl would need to expose a new control.
>
Agree. Add new control "MBN" for Memory Bandwidth Allocation on numa
node. See above.
>>
>
>> There is another MPAM feature called MBW Max hardlimit which sets
>> "MB:" allocation as hardlimit (i.e. MBW throttling percentage must
>> be satisfied) per domain. Adding a new "MB_HLIM:" line in schemata.
>> It's 1:1 mapped to "MB:" to control hardlimit of MB throttling
>> percentage on each domain. By default hardlimit is off (0) and can
>> be turned on to set MBW Max hardlimit on a domain.
>
> ack. This sounds like a new control associated with the MB resource.
> This is a boolean control as Dave highlighted in previous discussion so
> resctrl would need to know its properties.
> See https://lore.kernel.org/lkml/aO0Oazuxt54hQFbx@e133380.arm.com/
>
Right. ("MB_HLIM" name may be adjusted accordingly when "MB_MAX" is
available.)
>> For exmple:
>> MB_HLIM: 0=0;1=0;2=1;10=0;18=0;26=0
>> MB:0=100;1=100;2=80;10=100;18=100;26=100
>>
>> On GPU memory numa node 2: cannot use more than 80% of total max mbw even if there is still idle mem bandwidth on this node).
>>
>> MBW allocations on all other domains are soft limited, meaning MBW can be used more than specified if mem is idle.
>>
>
> ack.
>
>>> L3:0=fff;1=fff
>>> # echo 'MB_MIN:0=50' > schemata
>>> # cat schemata
>>> MB_MAX:0=100;1=100
>>> MB_MIN:0=50;1=100
>>> MB:0=100;1=100
>>> L3:0=fff;1=fff
>>>
>>> Writing to the dummy control will call a dummy callback that just prints to the
>>> kernel log:
>>> "resctrl: Updata temporary MIN control on domain 0 with user value 50"
>>>
>>>
>>> Example output of info/MB/:
>>> /sys/fs/resctrl/info/MB/thread_throttle_mode:max
>>> /sys/fs/resctrl/info/MB/num_closids:15
>>> /sys/fs/resctrl/info/MB/delay_linear:1
>>> /sys/fs/resctrl/info/MB/min_bandwidth:10
>>
>> Add two new MB info RO files:
>> 1. /sys/fs/resctrl/info/MB/domain_id
>> It shows "numa" for using numa id in "MB:" or "cache" for using legacy cache id.
>
> This proposal introduces a *global* property to the MB *resource*? It does not seem as though
> this takes into account *anything* about how resctrl can support new hardware that has been
> discussed before, during, or after LPC. You have not participated in these discussions and
> now make an orthogonal proposal that does not take into account *any* of the requirements
> that we have been struggling with for months.
>
> Why should this proposal be taken seriously? In your absence folks have been trying to
> accommodate how these upcoming products and be supported and the "scope" file associated with
> a control is intended to communicate to user space how the domain ID should be interpreted.
>
> Why are you proposing something entirely different here without even acknowledging current
> approach and explaining why it does not work for you?
>
So can I change this part to adding the following files in info dirctory?
1. For numa memory bw allocation (MBN):
/sys/fs/resctrl/info/MBN/resource_schemata/MBN/
/sys/fs/resctrl/info/MBN/resource_schemata/MBN/resolution:100
/sys/fs/resctrl/info/MBN/resource_schemata/MBN/tolerance:5
/sys/fs/resctrl/info/MBN/resource_schemata/MBN/type:scalar
/sys/fs/resctrl/info/MBN/resource_schemata/MBN/min:10
/sys/fs/resctrl/info/MBN/resource_schemata/MBN/scale:1
/sys/fs/resctrl/info/MBN/resource_schemata/MBN/scope:NUMA
/sys/fs/resctrl/info/MBN/resource_schemata/MBN/unit:all
/sys/fs/resctrl/info/MBN/resource_schemata/MBN/max:100
>> 2. /sys/fs/resctrl/info/MB/max_lim
>> It shows number 0-3 for MPAM MBW max limit behaviors: 0 for supporting both softlimit and hardlimit, etc.
>
> Again this adds another *global* property to the MB resource but then above you
> describe the new "MB_HLIM" schemata file entry that implies that it is a new control
> for the MB resource. Having it be a new control for the MB resource matches earlier
> discussions. To support this I thus expect it to be exposed as a new control with
> potentially a new type if any of the existing planned types do not suffice.
>
How about adding these MB_HLIM dir and files in info?
/sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/type: boolean
/sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/max_lim: 0
>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/resolution:100
>>
>> Is it more concise to s/resource_schemata/schemata/? "resource_" seems redundant in the context "info/MB".
>
> We could do this, yes.
>
>>
>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/tolerance:5
>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/type:scalar
>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/min:10
>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/scale:1
>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/scope:L3
>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/unit:all
>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/max:100
>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/resolution:100
>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/tolerance:5
>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/type:scalar
>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/min:10
>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scale:1
>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scope:L3
>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/unit:all
>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/max:100
>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/resolution:100
>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/tolerance:5
>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/type:scalar
>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/min:10
>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scale:1
>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scope:L3
>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/unit:all
>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/max:100
>>> /sys/fs/resctrl/info/MB/bandwidth_gran:10
>>
>> For MBW monitoring, extend mon_data/ directory to monitor CPU-less memory node. For example,
>
> Here is where I attempted to discuss with you how to support monitoring on these systems:
> https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@intel.com/
>
> Here again you respond with something completely different without acknowledging
> the previous discussion or noting why that does not work for you.
>
>>
>> On legacy platforms (i.e. L3 and memory are described in same MPAM ACPI MSC wich doesn't support CPU-less nodes):
>> mon_data/mbm_L3_01/llc_occupancy
>> mon_data/mbm_L3_01/mbm_total_bytes
>> mon_data/mbm_L3_02/llc_occupancy
>> mon_data/mbm_L3_02/mbm_total_bytes
>>
>> On platforms with L3 and memory in separate MPAM ACPI MSCs
>> but there is no CPU-less node:
>> mon_data/mbm_L3_01/llc_occupancy <- cache id 1
>> mon_data/mbm_L3_02/llc_occupancy <- cache id 2
>> mon_data/mbm_MB_00/mbm_total_bytes <- numa node 0 (socket 0)
>> mon_data/mbm_MB_01/mbm_total_bytes <- numa node 1 (socket 1)
>
> Here too I do not find it appropriate for resctrl to retroactively
> change its interface. "MB" is a resource and the above switches the
> resctrl interface to imply the monitoring data of a resource can be
> found in the mon directory that matches the resource name. This is
> not what resctrl does today. Doing something like above will result in
> resctrl having a confusing interface where "sometimes" memory bandwidth
> data can be found in the L3 directory and "sometimes" memory bandwidth
> data can be found in the MB directory.
>
> As I described to you in December resctrl already exposes the monitoring
> data based on the *scope*. As you also point out above, today the memory
> bandwidth monitoring data at L3 scope can be found in the L3 directory.
> "L3" should thus not be interpreted as the resource L3 but the scope L3
> since it contains MBM data today. When viewing it as such resctrl could
> internally be more explicit and separate monitoring scope from monitoring
> resource and present the monitoring data based on scope to remain intuitively
> backward compatible while obtaining support for these memory nodes.
>
>>
>> On platforms with L3 and memory in separate MPAM ACPI MSCs
>> and there are CPU-less nodes:
>> mon_data/mbm_L3_01/llc_occupancy <- cache id 1
>> mon_data/mbm_L3_02/llc_occupancy <- cache id 2
>> mon_data/mbm_MB_00/mbm_total_bytes <- numa node 0 (socket 0)
>> mon_data/mbm_MB_01/mbm_total_bytes <- numa node 1 (socket 1)
>> mon_data/mbm_MB_02/mbm_total_bytes <- numa node 2 (GPU 0 mem)
>> mon_data/mbm_MB_10/mbm_total_bytes <- numa node 10 (GPU 1 mem)
>> mon_data/mbm_MB_18/mbm_total_bytes <- numa node 18 (GPU 2 mem)
>> mon_data/mbm_MB_26/mbm_total_bytes <- numa node 26 (GPU 3 mem)
> To be backward compatible I find it more intuitive if instead
> this data is exposed as below:
>
> mon_data/mbm_L3_01/llc_occupancy <- cache id 1
> mon_data/mbm_L3_02/llc_occupancy <- cache id 2
> mon_data/mbm_NODE_00/mbm_total_bytes <- numa node 0 (socket 0)
> mon_data/mbm_NODE_01/mbm_total_bytes <- numa node 1 (socket 1)
> mon_data/mbm_NODE_02/mbm_total_bytes <- numa node 2 (GPU 0 mem)
> mon_data/mbm_NODE_10/mbm_total_bytes <- numa node 10 (GPU 1 mem)
> mon_data/mbm_NODE_18/mbm_total_bytes <- numa node 18 (GPU 2 mem)
> mon_data/mbm_NODE_26/mbm_total_bytes <- numa node 26 (GPU 3 mem)
>
> When "mbm_total_bytes" move from mon_data/mbm_L3_x to
> mon_data/mbm_NODE_x it clearly indicates that it is memory bandwidth
> monitoring data moving from "L3" scope to "NODE" scope.
The scope in mon_data shown in
https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@intel.com/:
mon_data
├── mon_L3_00 <== monitoring data at scope L3
│ ├── llc_occupancy
│ ├── mbm_local_bytes
│ └── mbm_total_bytes
├── mon_L3_01 <== monitoring data at scope L3
│ ├── llc_occupancy
│ ├── mbm_local_bytes
│ └── mbm_total_bytes
├── mon_NODE_00 <== monitoring data at scope NODE
│ └── mbm_total_bytes
└── mon_NODE_01 <== monitoring data at scope NODE
└── mbm_total_bytes
On some ARM platforms, for example, socket 0 (CPUs+L3) and socket 1
(CPUs+L3):
llc_occupancy is monitored through processor/L3.
total_bytes are monitored through memory controlor.
Then the above "scope" is confused:
mon_L3_00 and mon_L3_01 only has llc_occupancy. The scope "L3" is fine here.
But mon_NODE_00 and 01 are confused because numa node 00 and 01 have
both L3 and memory. The name "mon_NODE_00" seems monitor both
llc_occupany and total_bytes but it only monitor total_bytes. And on
CPU-less platforms, a CPU-less node does only have total_bytes which
seems match "scope NODE".
With "scope NODE", can user tell if it has llc_occupancy or total_bytes
or both?
If change "mon_MB_01" to "mon_MM_01", the scope is "MM" now which means
monitoring memory (total_bytes) in numa node 1 with "scope MM" (not L3)?
Thanks.
-Fenghua
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-25 1:26 ` Fenghua Yu
@ 2026-06-25 15:43 ` Reinette Chatre
2026-07-10 20:59 ` Fenghua Yu
2026-07-02 13:37 ` Ben Horgan
1 sibling, 1 reply; 66+ messages in thread
From: Reinette Chatre @ 2026-06-25 15:43 UTC (permalink / raw)
To: Fenghua Yu, Tony Luck, Ben Horgan, James Morse, Dave Martin,
Babu Moger, Drew Fustini, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi Fenghua,
On 6/24/26 6:26 PM, Fenghua Yu wrote:
> Hi, Reinette,
>
> On 6/24/26 15:22, Reinette Chatre wrote:
>> Hi Fenghua,
>>
>> On 6/24/26 12:08 PM, Fenghua Yu wrote:
>>> Hi, Reinette, Ben, Shaopen, et al,
>>>
>>> On 5/29/26 11:06, Reinette Chatre wrote:
>>>
>>> As Shaopen and Ben mentioned earlier, we are working on two MPAM
>>> features that may need to change schemata interface. The CPU-less
>>> feature was discussed on LPC (although the interfaces will be
>>> slightly different from the LPC).
>>
>> I know. Here is where I tried to engage with you on needed interfaces after LPC:
>> https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@intel.com/
>
> MPAM ACPI defines MSC (Memory System Control) is defined in one of two ways (not both) on one platform:
> 1. L3 and memory together on each processor MSC
> 2. L3 in processor MSC and memory control/monitoring in different memory MSCs.
>
> On type 1 platform, schemata is legacy:
> MB:1=100;2=100 <-- cache id 1 and 2 as domain id
>
> On type 2 platform, I will not reuse "MB:" name. Instead, define new resource name "MBN:" for numa node and schemata is:
> MBN:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10, 18,
> 26 as domain id
> On type 2 platform, there won't be "MB:" line. Numa 0 and 1
> are for mbm allocation on socket 0 and 1. 2,10, 18 and 26 are for GPU
> memory nodes allocation.
(to help make things explicit I will refer to what you call "MBN" as "MB_NODE" to make it
explicit that it is memory bandwidth allocation at node scope)
I am trying to consider how this can be accomplished while also considering all the other
new hardware features that resctrl need to support. Consider, for example, AMD's "Global
MBA" (https://lore.kernel.org/lkml/cover.1776980182.git.babu.moger@amd.com/) that throttles
memory bandwidth at L3 scope but the user configures allocations at NODE scope. At this time
the plan is to support this with a second control associated with the MB resource that can
allocate memory bandwidth at node scope. See
https://lore.kernel.org/lkml/430ffb48-29f4-44d9-9164-9f8b743b2739@amd.com/
If resctrl creates a new resource for node scoped memory bandwidth allocations to support these
"type 2" systems then that will result in an inconsistent interface between architectures that
we should avoid.
Have you been listening in on the discussions surrounding emulated controls? Considering that,
would it be possible to support the "MB" control on a type "2" system but have it be backed by
(emulated by) the underlying "MB_NODE" control?
resctrl could expose both controls on these "type 2" systems but make it clear that "MB"
is emulated by "MB_NODE". For example:
info/
└── MB/
└── resource_schemata/
└── MB/
└── MB_NODE/
User will see both controls in schemata file but when changes are made to "MB" control it
will show in the "MB_NODE" control and vice-versa. User could also disable the "MB" control
that will establish familiarity with the interface at which point resctrl can drop the
"MB" control from the schemata file on these "type 2" systems.
Having the MB resource available with an MB control will keep resctrl backward compatible
if there are any tools that expect that. If backward compatibility is not of concern then
resctrl could initialize with the emulated control disabled by default. See discussion at
https://lore.kernel.org/lkml/5e575bc2-e67f-4696-9332-33c54023c057@intel.com/
that describes a new resctrl capability in support of RISC-V and RDT.
With this resctrl could initialize with:
info/
└── MB/
└── resource_schemata/
├── MB/
│ ├── MB_NODE/
│ │ └── status:enabled
│ └── status:disabled
└── mode:legacy [native]
With above a "type 2" system will boot with its schemata file just containing the "MB_NODE"
control while info/MB describes the memory bandwidth resource.
If AMD is ok with naming their "Global MBA" allocations "MB_NODE" then a user working with
AMD and these MPAM systems could find the controls in the same hierarchy and need not
use external knowledge to determine how to interact with resctrl fs.
> BTW, Slow MBA (SMBA) is different from MBA Numa (MBN). SMBA still
> relies on L3 and the domain id in SMBA is still cache id. MBN
> depends on each memory controlor with numa id as domain id for both
> CPU and CPU-less memory nodes.
ack. Similar to AMD's "Global MBA" there is also a new feature of "Global SMBA" that
needs to be supported by resctrl.
> On type 1 platform, there is only MB:
>
> info
> └── MB
> └── resource_schemata
> ├── MB
> │ ├── max
> │ ├── min
> │ ├── resolution
> │ ├── scale
> │ ├── scope <== contains "L3"
> │ ├── tolerance
> │ ├── type
> │ └── unit
>
> On type 2 platform, there is only MBN:
> info
> └── MBN
> └── resource_schemata
> ├── MBN
> │ ├── max
> │ ├── min
> │ ├── resolution
> │ ├── scale
> │ ├── scope <== contains "NUMA"
> │ ├── tolerance
> │ ├── type
> │ └── unit
>
> This is different from the "scope" hierarchy discussed in the link. "MB" and "MBN" won't exist on the same platform.
>
> I find it's hard (and not useful) to split "MB" for memory with CPU
> and "MBN" for CPU-less memory node. It's easier to have either "MB"
> for legacy memory with CPU or "MBN" for CPU-less memory.
Please widen your considerations to include how resctrl can maintain backward compatibility
and how enabling of these platforms can fit well with the "similar but not identical" hardware
features from other architectures that also needs to be supported by resctrl.
> Any thoughts? Does this update make sense?
>
>>
>>> Hardlimit feature was not discussed yet.
>>
>> It was considered. See discussion starting at
>> https://lore.kernel.org/lkml/1c4b6b46-16f9-4887-93f5-e0f5e7f30a6f@intel.com/
>>
>>> It's good to discuss this further in this RFC thread before the new features RFCs will be sent out.
>>
>> ack.
>>
>>>
>>> Overall, the new features can fit into this RFC well.
>>>
>>>> Hi Everybody,
>>>>
>
> [SNIP]
>
>>>>
>>>> This series can be used on an x86 system where it will show two new dummy controls
>>>> where it is possible to interact with the new controls.
>>>> For example:
>>>>
>>>> # cat schemata
>>>> MB_MAX:0=100;1=100
>>>> MB_MIN:0=100;1=100
>>>> MB:0=100;1=100
>>>
>>> Some platforms may support CPU-less node which is represented by numa node id, examples:
>>> 1. CXL type 2 memory node which provides CXL memory without CPU and L3 on the node
>>> 2. GPU memory node that can be accessed by all CPUs but doesn't have a local CPU and L3 bound to.
>>> etc.
>>>
>>> MPAM can allocate and monitor mem bandwidth on these memory node.
>>> Since no CPU and L3 on the node, cache id cannot be used in "MB:" line. Instead, numa ids are used to identify MB allocation and monitoring.
>>>
>>> For example, the MB allocation on CPU-less platforms could be:
>>> MB:0=100;1=100;2=100;10=100;18=100;26=100
>>>
>>> Where: domain id 0, 1, 2, etc are numa node id shown in /sys/devices/system/node directory or by numctl.
>>> 0: socket 0, node 0, CPUs, memory
>>> 1: socket 1, node 1, CPUs, memory
>>> 2: GPU 0, node 2, no CPU, memory only
>>> 10: GPU 1, node 10, no CPU, memory only
>>> 18: GPU 3, node 18, no CPU, memory only
>>> 26: GPU 4, node 26, no CPU, memory only
>>>
>>> Arch specific driver (e.g. MPAM) detects CPU-less node. If there is any CPU-less node, use numa id in "MB:". Otherwise, fallback to legacy cache id.
>>
>> We always have to consider backward compatibility and to do so we cannot just retroactively
>> change what domain ID represents when user space interacts with the "MB" control.
>>
>> The legacy "MB" control is already defined and its domain ID represents an L3 cache ID. To
>> support these new devices resctrl would need to expose a new control.
>>
>
> Agree. Add new control "MBN" for Memory Bandwidth Allocation on numa node. See above.
>
>>>
>>
>>> There is another MPAM feature called MBW Max hardlimit which sets
>>> "MB:" allocation as hardlimit (i.e. MBW throttling percentage must
>>> be satisfied) per domain. Adding a new "MB_HLIM:" line in schemata.
>>> It's 1:1 mapped to "MB:" to control hardlimit of MB throttling
>>> percentage on each domain. By default hardlimit is off (0) and can
>>> be turned on to set MBW Max hardlimit on a domain.
>>
>> ack. This sounds like a new control associated with the MB resource.
>> This is a boolean control as Dave highlighted in previous discussion so
>> resctrl would need to know its properties.
>> See https://lore.kernel.org/lkml/aO0Oazuxt54hQFbx@e133380.arm.com/
>>
>
> Right. ("MB_HLIM" name may be adjusted accordingly when "MB_MAX" is available.)
>
>>> For exmple:
>>> MB_HLIM: 0=0;1=0;2=1;10=0;18=0;26=0
>>> MB:0=100;1=100;2=80;10=100;18=100;26=100
>>>
>>> On GPU memory numa node 2: cannot use more than 80% of total max mbw even if there is still idle mem bandwidth on this node).
>>>
>>> MBW allocations on all other domains are soft limited, meaning MBW can be used more than specified if mem is idle.
>>>
>>
>> ack.
>>
>>>> L3:0=fff;1=fff
>>>> # echo 'MB_MIN:0=50' > schemata
>>>> # cat schemata
>>>> MB_MAX:0=100;1=100
>>>> MB_MIN:0=50;1=100
>>>> MB:0=100;1=100
>>>> L3:0=fff;1=fff
>>>>
>>>> Writing to the dummy control will call a dummy callback that just prints to the
>>>> kernel log:
>>>> "resctrl: Updata temporary MIN control on domain 0 with user value 50"
>>>>
>>>>
>>>> Example output of info/MB/:
>>>> /sys/fs/resctrl/info/MB/thread_throttle_mode:max
>>>> /sys/fs/resctrl/info/MB/num_closids:15
>>>> /sys/fs/resctrl/info/MB/delay_linear:1
>>>> /sys/fs/resctrl/info/MB/min_bandwidth:10
>>>
>>> Add two new MB info RO files:
>>> 1. /sys/fs/resctrl/info/MB/domain_id
>>> It shows "numa" for using numa id in "MB:" or "cache" for using legacy cache id.
>>
>> This proposal introduces a *global* property to the MB *resource*? It does not seem as though
>> this takes into account *anything* about how resctrl can support new hardware that has been
>> discussed before, during, or after LPC. You have not participated in these discussions and
>> now make an orthogonal proposal that does not take into account *any* of the requirements
>> that we have been struggling with for months.
>>
>> Why should this proposal be taken seriously? In your absence folks have been trying to
>> accommodate how these upcoming products and be supported and the "scope" file associated with
>> a control is intended to communicate to user space how the domain ID should be interpreted.
>>
>> Why are you proposing something entirely different here without even acknowledging current
>> approach and explaining why it does not work for you?
>>
>
> So can I change this part to adding the following files in info dirctory?
>
> 1. For numa memory bw allocation (MBN):
> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/
> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/resolution:100
> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/tolerance:5
> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/type:scalar
> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/min:10
> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/scale:1
> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/scope:NUMA
> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/unit:all
> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/max:100
This is not just about adding files to the info directory. The files, directories, their relationships,
and content have meaning. All I see from these proposals is an attempt to slap some new files into
resctrl without any consideration to present consistent interface to users and without consideration of
other architectures that need to be supported by resctrl.
resctrl needs to provide a generic and consistent interface to user space irrespective of the
underlying architecture. Architectures cannot just slap some new files for their convenience.
>
>>> 2. /sys/fs/resctrl/info/MB/max_lim
>>> It shows number 0-3 for MPAM MBW max limit behaviors: 0 for supporting both softlimit and hardlimit, etc.
>>
>> Again this adds another *global* property to the MB resource but then above you
>> describe the new "MB_HLIM" schemata file entry that implies that it is a new control
>> for the MB resource. Having it be a new control for the MB resource matches earlier
>> discussions. To support this I thus expect it to be exposed as a new control with
>> potentially a new type if any of the existing planned types do not suffice.
>>
>
> How about adding these MB_HLIM dir and files in info?
>
> /sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/type: boolean
> /sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/max_lim: 0
This presents "MB_HLIM" as a *resource* to user space. It is not a resource
but a *control* of a resource, no? I thus expect it to instead look something like
below that makes it clear that MB_HARDMAX is a control of the MB resource.
info
└── MB
└── resource_schemata
├── MB
└── MB_HARDMAX
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/resolution:100
>>>
>>> Is it more concise to s/resource_schemata/schemata/? "resource_" seems redundant in the context "info/MB".
>>
>> We could do this, yes.
>>
>>>
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/tolerance:5
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/type:scalar
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/min:10
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/scale:1
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/scope:L3
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/unit:all
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/max:100
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/resolution:100
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/tolerance:5
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/type:scalar
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/min:10
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scale:1
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scope:L3
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/unit:all
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/max:100
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/resolution:100
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/tolerance:5
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/type:scalar
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/min:10
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scale:1
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scope:L3
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/unit:all
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/max:100
>>>> /sys/fs/resctrl/info/MB/bandwidth_gran:10
>>>
>>> For MBW monitoring, extend mon_data/ directory to monitor CPU-less memory node. For example,
>>
>> Here is where I attempted to discuss with you how to support monitoring on these systems:
>> https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@intel.com/
>>
>> Here again you respond with something completely different without acknowledging
>> the previous discussion or noting why that does not work for you.
>>
>>>
>>> On legacy platforms (i.e. L3 and memory are described in same MPAM ACPI MSC wich doesn't support CPU-less nodes):
>>> mon_data/mbm_L3_01/llc_occupancy
>>> mon_data/mbm_L3_01/mbm_total_bytes
>>> mon_data/mbm_L3_02/llc_occupancy
>>> mon_data/mbm_L3_02/mbm_total_bytes
>>>
>>> On platforms with L3 and memory in separate MPAM ACPI MSCs
>>> but there is no CPU-less node:
>>> mon_data/mbm_L3_01/llc_occupancy <- cache id 1
>>> mon_data/mbm_L3_02/llc_occupancy <- cache id 2
>>> mon_data/mbm_MB_00/mbm_total_bytes <- numa node 0 (socket 0)
>>> mon_data/mbm_MB_01/mbm_total_bytes <- numa node 1 (socket 1)
>>
>> Here too I do not find it appropriate for resctrl to retroactively
>> change its interface. "MB" is a resource and the above switches the
>> resctrl interface to imply the monitoring data of a resource can be
>> found in the mon directory that matches the resource name. This is
>> not what resctrl does today. Doing something like above will result in
>> resctrl having a confusing interface where "sometimes" memory bandwidth
>> data can be found in the L3 directory and "sometimes" memory bandwidth
>> data can be found in the MB directory.
>>
>> As I described to you in December resctrl already exposes the monitoring
>> data based on the *scope*. As you also point out above, today the memory
>> bandwidth monitoring data at L3 scope can be found in the L3 directory.
>> "L3" should thus not be interpreted as the resource L3 but the scope L3
>> since it contains MBM data today. When viewing it as such resctrl could
>> internally be more explicit and separate monitoring scope from monitoring
>> resource and present the monitoring data based on scope to remain intuitively
>> backward compatible while obtaining support for these memory nodes.
>>
>>>
>>> On platforms with L3 and memory in separate MPAM ACPI MSCs
>>> and there are CPU-less nodes:
>>> mon_data/mbm_L3_01/llc_occupancy <- cache id 1
>>> mon_data/mbm_L3_02/llc_occupancy <- cache id 2
>>> mon_data/mbm_MB_00/mbm_total_bytes <- numa node 0 (socket 0)
>>> mon_data/mbm_MB_01/mbm_total_bytes <- numa node 1 (socket 1)
>>> mon_data/mbm_MB_02/mbm_total_bytes <- numa node 2 (GPU 0 mem)
>>> mon_data/mbm_MB_10/mbm_total_bytes <- numa node 10 (GPU 1 mem)
>>> mon_data/mbm_MB_18/mbm_total_bytes <- numa node 18 (GPU 2 mem)
>>> mon_data/mbm_MB_26/mbm_total_bytes <- numa node 26 (GPU 3 mem)
>> To be backward compatible I find it more intuitive if instead
>> this data is exposed as below:
>>
>> mon_data/mbm_L3_01/llc_occupancy <- cache id 1
>> mon_data/mbm_L3_02/llc_occupancy <- cache id 2
>> mon_data/mbm_NODE_00/mbm_total_bytes <- numa node 0 (socket 0)
>> mon_data/mbm_NODE_01/mbm_total_bytes <- numa node 1 (socket 1)
>> mon_data/mbm_NODE_02/mbm_total_bytes <- numa node 2 (GPU 0 mem)
>> mon_data/mbm_NODE_10/mbm_total_bytes <- numa node 10 (GPU 1 mem)
>> mon_data/mbm_NODE_18/mbm_total_bytes <- numa node 18 (GPU 2 mem)
>> mon_data/mbm_NODE_26/mbm_total_bytes <- numa node 26 (GPU 3 mem)
>>
>> When "mbm_total_bytes" move from mon_data/mbm_L3_x to
>> mon_data/mbm_NODE_x it clearly indicates that it is memory bandwidth
>> monitoring data moving from "L3" scope to "NODE" scope.
>
> The scope in mon_data shown in https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@intel.com/:
>
> mon_data
> ├── mon_L3_00 <== monitoring data at scope L3
> │ ├── llc_occupancy
> │ ├── mbm_local_bytes
> │ └── mbm_total_bytes
> ├── mon_L3_01 <== monitoring data at scope L3
> │ ├── llc_occupancy
> │ ├── mbm_local_bytes
> │ └── mbm_total_bytes
> ├── mon_NODE_00 <== monitoring data at scope NODE
> │ └── mbm_total_bytes
> └── mon_NODE_01 <== monitoring data at scope NODE
> └── mbm_total_bytes
>
> On some ARM platforms, for example, socket 0 (CPUs+L3) and socket 1
> (CPUs+L3):
> llc_occupancy is monitored through processor/L3.
> total_bytes are monitored through memory controlor.
>
> Then the above "scope" is confused:
> mon_L3_00 and mon_L3_01 only has llc_occupancy. The scope "L3" is fine here.
> But mon_NODE_00 and 01 are confused because numa node 00 and 01 have
> both L3 and memory. The name "mon_NODE_00" seems monitor both
While the node may have L3 and memory the directory only represents what is
monitored at the particular scope.
> llc_occupany and total_bytes but it only monitor total_bytes. And on
> CPU-less platforms, a CPU-less node does only have total_bytes which
> seems match "scope NODE".
> With "scope NODE", can user tell if it has llc_occupancy or total_bytes or both?
resctrl has a "mon_features" file associated with the monitoring scope that
informs user space which events can be expected in a resource group's monitoring
data directories.
>
> If change "mon_MB_01" to "mon_MM_01", the scope is "MM" now which
> means monitoring memory (total_bytes) in numa node 1 with "scope MM"
> (not L3)?
This sounds redundant to me since the event names already have the resource embedded.
"llc_occupancy" implies L3 occupancy
"mbm_local_bytes" ... the "mbm" implies this is memory bandwidth monitoring data.
User space can infer the scope at which the monitoring data is collected from the
directory the event file is in.
Reinette
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-25 15:43 ` Reinette Chatre
@ 2026-07-10 20:59 ` Fenghua Yu
0 siblings, 0 replies; 66+ messages in thread
From: Fenghua Yu @ 2026-07-10 20:59 UTC (permalink / raw)
To: Reinette Chatre, Tony Luck, Ben Horgan, James Morse, Dave Martin,
Babu Moger, Drew Fustini, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi, Reinette,
On 6/25/26 08:43, Reinette Chatre wrote:
> Hi Fenghua,
>
> On 6/24/26 6:26 PM, Fenghua Yu wrote:
>> Hi, Reinette,
>>
>> On 6/24/26 15:22, Reinette Chatre wrote:
>>> Hi Fenghua,
>>>
>>> On 6/24/26 12:08 PM, Fenghua Yu wrote:
>>>> Hi, Reinette, Ben, Shaopen, et al,
>>>>
>>>> On 5/29/26 11:06, Reinette Chatre wrote:
>>>>
>>>> As Shaopen and Ben mentioned earlier, we are working on two MPAM
>>>> features that may need to change schemata interface. The CPU-less
>>>> feature was discussed on LPC (although the interfaces will be
>>>> slightly different from the LPC).
>>>
>>> I know. Here is where I tried to engage with you on needed interfaces after LPC:
>>> https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@intel.com/
>>
>> MPAM ACPI defines MSC (Memory System Control) is defined in one of two ways (not both) on one platform:
>> 1. L3 and memory together on each processor MSC
>> 2. L3 in processor MSC and memory control/monitoring in different memory MSCs.
Ben said there is type 3 platform:
3. L3 cache and memory bandwidth in processor MSCs and memory bandwidth
in different memory MSCs.
>>
>> On type 1 platform, schemata is legacy:
>> MB:1=100;2=100 <-- cache id 1 and 2 as domain id
>>
>> On type 2 platform, I will not reuse "MB:" name. Instead, define new resource name "MBN:" for numa node and schemata is:
>> MBN:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10, 18,
>> 26 as domain id
>> On type 2 platform, there won't be "MB:" line. Numa 0 and 1
>> are for mbm allocation on socket 0 and 1. 2,10, 18 and 26 are for GPU
>> memory nodes allocation.
On type 3 platform, there could be "MB:" line for L3 cache and
"MB_NODE:" for numa node. Example schemata is:
MB:1=100;2=100 <-- cache id 1 and 2 as domain id
MB_NODE:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10,
18, 26 as domain id
>
> (to help make things explicit I will refer to what you call "MBN" as "MB_NODE" to make it
> explicit that it is memory bandwidth allocation at node scope)
>
> I am trying to consider how this can be accomplished while also considering all the other
> new hardware features that resctrl need to support. Consider, for example, AMD's "Global
> MBA" (https://lore.kernel.org/lkml/cover.1776980182.git.babu.moger@amd.com/) that throttles
> memory bandwidth at L3 scope but the user configures allocations at NODE scope. At this time
> the plan is to support this with a second control associated with the MB resource that can
> allocate memory bandwidth at node scope. See
> https://lore.kernel.org/lkml/430ffb48-29f4-44d9-9164-9f8b743b2739@amd.com/
>
> If resctrl creates a new resource for node scoped memory bandwidth allocations to support these
> "type 2" systems then that will result in an inconsistent interface between architectures that
> we should avoid.
>
> Have you been listening in on the discussions surrounding emulated controls? Considering that,
> would it be possible to support the "MB" control on a type "2" system but have it be backed by
> (emulated by) the underlying "MB_NODE" control?
>
> resctrl could expose both controls on these "type 2" systems but make it clear that "MB"
> is emulated by "MB_NODE". For example:
>
> info/
> └── MB/
> └── resource_schemata/
> └── MB/
> └── MB_NODE/
>
> User will see both controls in schemata file but when changes are made to "MB" control it
> will show in the "MB_NODE" control and vice-versa. User could also disable the "MB" control
> that will establish familiarity with the interface at which point resctrl can drop the
> "MB" control from the schemata file on these "type 2" systems.
>
> Having the MB resource available with an MB control will keep resctrl backward compatible
> if there are any tools that expect that. If backward compatibility is not of concern then
> resctrl could initialize with the emulated control disabled by default. See discussion at
> https://lore.kernel.org/lkml/5e575bc2-e67f-4696-9332-33c54023c057@intel.com/
> that describes a new resctrl capability in support of RISC-V and RDT.
> With this resctrl could initialize with:
>
> info/
> └── MB/
> └── resource_schemata/
> ├── MB/
> │ ├── MB_NODE/
> │ │ └── status:enabled
> │ └── status:disabled
> └── mode:legacy [native]
>
> With above a "type 2" system will boot with its schemata file just containing the "MB_NODE"
> control while info/MB describes the memory bandwidth resource.
>
On type 3 machine, schemata has both MB in legacy mode with cache id as
domain id and MB_NODE with numa id as domain id.
Is this directory OK?
info/
└── MB/
└── resource_schemata/
├── MB/
│ ├── MB_NODE/
│ │ └── status:disabled
│ └── status:enabled
├── MB_NODE/
└── mode:node
1. MB and MB_NODE are shown in parallel in inf/MB/resource_schemata/
2. mode is set as "node" meaning "MB" is for L3 and "MB_NODE" is for
numa node
3. Emulation "MB_NODE" is disabled (or should the "MB_NODE" sub-dir be
invisible?)
> If AMD is ok with naming their "Global MBA" allocations "MB_NODE" then a user working with
> AMD and these MPAM systems could find the controls in the same hierarchy and need not
> use external knowledge to determine how to interact with resctrl fs.
MB_NODE is good for me.
>
>
>> BTW, Slow MBA (SMBA) is different from MBA Numa (MBN). SMBA still
>> relies on L3 and the domain id in SMBA is still cache id. MBN
>> depends on each memory controlor with numa id as domain id for both
>> CPU and CPU-less memory nodes.
>
> ack. Similar to AMD's "Global MBA" there is also a new feature of "Global SMBA" that
> needs to be supported by resctrl.
>
>
>> On type 1 platform, there is only MB:
>>
>> info
>> └── MB
>> └── resource_schemata
>> ├── MB
>> │ ├── max
>> │ ├── min
>> │ ├── resolution
>> │ ├── scale
>> │ ├── scope <== contains "L3"
>> │ ├── tolerance
>> │ ├── type
>> │ └── unit
>>
>> On type 2 platform, there is only MBN:
>> info
>> └── MBN
>> └── resource_schemata
>> ├── MBN
>> │ ├── max
>> │ ├── min
>> │ ├── resolution
>> │ ├── scale
>> │ ├── scope <== contains "NUMA"
>> │ ├── tolerance
>> │ ├── type
>> │ └── unit
>>
>> This is different from the "scope" hierarchy discussed in the link. "MB" and "MBN" won't exist on the same platform.
>>
>> I find it's hard (and not useful) to split "MB" for memory with CPU
>> and "MBN" for CPU-less memory node. It's easier to have either "MB"
>> for legacy memory with CPU or "MBN" for CPU-less memory.
>
> Please widen your considerations to include how resctrl can maintain backward compatibility
> and how enabling of these platforms can fit well with the "similar but not identical" hardware
> features from other architectures that also needs to be supported by resctrl.
>
>
>> Any thoughts? Does this update make sense?
>>
>>>
>>>> Hardlimit feature was not discussed yet.
>>>
>>> It was considered. See discussion starting at
>>> https://lore.kernel.org/lkml/1c4b6b46-16f9-4887-93f5-e0f5e7f30a6f@intel.com/
>>>
>>>> It's good to discuss this further in this RFC thread before the new features RFCs will be sent out.
>>>
>>> ack.
>>>
>>>>
>>>> Overall, the new features can fit into this RFC well.
>>>>
>>>>> Hi Everybody,
>>>>>
>>
>> [SNIP]
>>
>>>>>
>>>>> This series can be used on an x86 system where it will show two new dummy controls
>>>>> where it is possible to interact with the new controls.
>>>>> For example:
>>>>>
>>>>> # cat schemata
>>>>> MB_MAX:0=100;1=100
>>>>> MB_MIN:0=100;1=100
>>>>> MB:0=100;1=100
>>>>
>>>> Some platforms may support CPU-less node which is represented by numa node id, examples:
>>>> 1. CXL type 2 memory node which provides CXL memory without CPU and L3 on the node
>>>> 2. GPU memory node that can be accessed by all CPUs but doesn't have a local CPU and L3 bound to.
>>>> etc.
>>>>
>>>> MPAM can allocate and monitor mem bandwidth on these memory node.
>>>> Since no CPU and L3 on the node, cache id cannot be used in "MB:" line. Instead, numa ids are used to identify MB allocation and monitoring.
>>>>
>>>> For example, the MB allocation on CPU-less platforms could be:
>>>> MB:0=100;1=100;2=100;10=100;18=100;26=100
>>>>
>>>> Where: domain id 0, 1, 2, etc are numa node id shown in /sys/devices/system/node directory or by numctl.
>>>> 0: socket 0, node 0, CPUs, memory
>>>> 1: socket 1, node 1, CPUs, memory
>>>> 2: GPU 0, node 2, no CPU, memory only
>>>> 10: GPU 1, node 10, no CPU, memory only
>>>> 18: GPU 3, node 18, no CPU, memory only
>>>> 26: GPU 4, node 26, no CPU, memory only
>>>>
>>>> Arch specific driver (e.g. MPAM) detects CPU-less node. If there is any CPU-less node, use numa id in "MB:". Otherwise, fallback to legacy cache id.
>>>
>>> We always have to consider backward compatibility and to do so we cannot just retroactively
>>> change what domain ID represents when user space interacts with the "MB" control.
>>>
>>> The legacy "MB" control is already defined and its domain ID represents an L3 cache ID. To
>>> support these new devices resctrl would need to expose a new control.
>>>
>>
>> Agree. Add new control "MBN" for Memory Bandwidth Allocation on numa node. See above.
>>
>>>>
>>>
>>>> There is another MPAM feature called MBW Max hardlimit which sets
>>>> "MB:" allocation as hardlimit (i.e. MBW throttling percentage must
>>>> be satisfied) per domain. Adding a new "MB_HLIM:" line in schemata.
>>>> It's 1:1 mapped to "MB:" to control hardlimit of MB throttling
>>>> percentage on each domain. By default hardlimit is off (0) and can
>>>> be turned on to set MBW Max hardlimit on a domain.
>>>
>>> ack. This sounds like a new control associated with the MB resource.
>>> This is a boolean control as Dave highlighted in previous discussion so
>>> resctrl would need to know its properties.
>>> See https://lore.kernel.org/lkml/aO0Oazuxt54hQFbx@e133380.arm.com/
>>>
>>
>> Right. ("MB_HLIM" name may be adjusted accordingly when "MB_MAX" is available.)
>>
>>>> For exmple:
>>>> MB_HLIM: 0=0;1=0;2=1;10=0;18=0;26=0
>>>> MB:0=100;1=100;2=80;10=100;18=100;26=100
>>>>
>>>> On GPU memory numa node 2: cannot use more than 80% of total max mbw even if there is still idle mem bandwidth on this node).
>>>>
>>>> MBW allocations on all other domains are soft limited, meaning MBW can be used more than specified if mem is idle.
>>>>
>>>
>>> ack.
>>>
>>>>> L3:0=fff;1=fff
>>>>> # echo 'MB_MIN:0=50' > schemata
>>>>> # cat schemata
>>>>> MB_MAX:0=100;1=100
>>>>> MB_MIN:0=50;1=100
>>>>> MB:0=100;1=100
>>>>> L3:0=fff;1=fff
>>>>>
>>>>> Writing to the dummy control will call a dummy callback that just prints to the
>>>>> kernel log:
>>>>> "resctrl: Updata temporary MIN control on domain 0 with user value 50"
>>>>>
>>>>>
>>>>> Example output of info/MB/:
>>>>> /sys/fs/resctrl/info/MB/thread_throttle_mode:max
>>>>> /sys/fs/resctrl/info/MB/num_closids:15
>>>>> /sys/fs/resctrl/info/MB/delay_linear:1
>>>>> /sys/fs/resctrl/info/MB/min_bandwidth:10
>>>>
>>>> Add two new MB info RO files:
>>>> 1. /sys/fs/resctrl/info/MB/domain_id
>>>> It shows "numa" for using numa id in "MB:" or "cache" for using legacy cache id.
>>>
>>> This proposal introduces a *global* property to the MB *resource*? It does not seem as though
>>> this takes into account *anything* about how resctrl can support new hardware that has been
>>> discussed before, during, or after LPC. You have not participated in these discussions and
>>> now make an orthogonal proposal that does not take into account *any* of the requirements
>>> that we have been struggling with for months.
>>>
>>> Why should this proposal be taken seriously? In your absence folks have been trying to
>>> accommodate how these upcoming products and be supported and the "scope" file associated with
>>> a control is intended to communicate to user space how the domain ID should be interpreted.
>>>
>>> Why are you proposing something entirely different here without even acknowledging current
>>> approach and explaining why it does not work for you?
>>>
>>
>> So can I change this part to adding the following files in info dirctory?
>>
>> 1. For numa memory bw allocation (MBN):
>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/
>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/resolution:100
>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/tolerance:5
>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/type:scalar
>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/min:10
>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/scale:1
>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/scope:NUMA
>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/unit:all
>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/max:100
>
> This is not just about adding files to the info directory. The files, directories, their relationships,
> and content have meaning. All I see from these proposals is an attempt to slap some new files into
> resctrl without any consideration to present consistent interface to users and without consideration of
> other architectures that need to be supported by resctrl.
>
> resctrl needs to provide a generic and consistent interface to user space irrespective of the
> underlying architecture. Architectures cannot just slap some new files for their convenience.
>
>>
>>>> 2. /sys/fs/resctrl/info/MB/max_lim
>>>> It shows number 0-3 for MPAM MBW max limit behaviors: 0 for supporting both softlimit and hardlimit, etc.
>>>
>>> Again this adds another *global* property to the MB resource but then above you
>>> describe the new "MB_HLIM" schemata file entry that implies that it is a new control
>>> for the MB resource. Having it be a new control for the MB resource matches earlier
>>> discussions. To support this I thus expect it to be exposed as a new control with
>>> potentially a new type if any of the existing planned types do not suffice.
>>>
>>
>> How about adding these MB_HLIM dir and files in info?
>>
>> /sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/type: boolean
>> /sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/max_lim: 0
>
> This presents "MB_HLIM" as a *resource* to user space. It is not a resource
> but a *control* of a resource, no? I thus expect it to instead look something like
> below that makes it clear that MB_HARDMAX is a control of the MB resource.
>
> info
> └── MB
> └── resource_schemata
> ├── MB
> └── MB_HARDMAX
Yes, this makes sense. I have changed to this hierarchy.
>
>
>
>>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/resolution:100
>>>>
>>>> Is it more concise to s/resource_schemata/schemata/? "resource_" seems redundant in the context "info/MB".
>>>
>>> We could do this, yes.
>>>
>>>>
>>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/tolerance:5
>>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/type:scalar
>>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/min:10
>>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/scale:1
>>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/scope:L3
>>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/unit:all
>>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/max:100
>>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/resolution:100
>>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/tolerance:5
>>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/type:scalar
>>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/min:10
>>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scale:1
>>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scope:L3
>>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/unit:all
>>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/max:100
>>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/resolution:100
>>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/tolerance:5
>>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/type:scalar
>>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/min:10
>>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scale:1
>>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scope:L3
>>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/unit:all
>>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/max:100
>>>>> /sys/fs/resctrl/info/MB/bandwidth_gran:10
>>>>
>>>> For MBW monitoring, extend mon_data/ directory to monitor CPU-less memory node. For example,
>>>
>>> Here is where I attempted to discuss with you how to support monitoring on these systems:
>>> https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@intel.com/
>>>
>>> Here again you respond with something completely different without acknowledging
>>> the previous discussion or noting why that does not work for you.
>>>
>>>>
>>>> On legacy platforms (i.e. L3 and memory are described in same MPAM ACPI MSC wich doesn't support CPU-less nodes):
>>>> mon_data/mbm_L3_01/llc_occupancy
>>>> mon_data/mbm_L3_01/mbm_total_bytes
>>>> mon_data/mbm_L3_02/llc_occupancy
>>>> mon_data/mbm_L3_02/mbm_total_bytes
>>>>
>>>> On platforms with L3 and memory in separate MPAM ACPI MSCs
>>>> but there is no CPU-less node:
>>>> mon_data/mbm_L3_01/llc_occupancy <- cache id 1
>>>> mon_data/mbm_L3_02/llc_occupancy <- cache id 2
>>>> mon_data/mbm_MB_00/mbm_total_bytes <- numa node 0 (socket 0)
>>>> mon_data/mbm_MB_01/mbm_total_bytes <- numa node 1 (socket 1)
>>>
>>> Here too I do not find it appropriate for resctrl to retroactively
>>> change its interface. "MB" is a resource and the above switches the
>>> resctrl interface to imply the monitoring data of a resource can be
>>> found in the mon directory that matches the resource name. This is
>>> not what resctrl does today. Doing something like above will result in
>>> resctrl having a confusing interface where "sometimes" memory bandwidth
>>> data can be found in the L3 directory and "sometimes" memory bandwidth
>>> data can be found in the MB directory.
>>>
>>> As I described to you in December resctrl already exposes the monitoring
>>> data based on the *scope*. As you also point out above, today the memory
>>> bandwidth monitoring data at L3 scope can be found in the L3 directory.
>>> "L3" should thus not be interpreted as the resource L3 but the scope L3
>>> since it contains MBM data today. When viewing it as such resctrl could
>>> internally be more explicit and separate monitoring scope from monitoring
>>> resource and present the monitoring data based on scope to remain intuitively
>>> backward compatible while obtaining support for these memory nodes.
>>>
>>>>
>>>> On platforms with L3 and memory in separate MPAM ACPI MSCs
>>>> and there are CPU-less nodes:
>>>> mon_data/mbm_L3_01/llc_occupancy <- cache id 1
>>>> mon_data/mbm_L3_02/llc_occupancy <- cache id 2
>>>> mon_data/mbm_MB_00/mbm_total_bytes <- numa node 0 (socket 0)
>>>> mon_data/mbm_MB_01/mbm_total_bytes <- numa node 1 (socket 1)
>>>> mon_data/mbm_MB_02/mbm_total_bytes <- numa node 2 (GPU 0 mem)
>>>> mon_data/mbm_MB_10/mbm_total_bytes <- numa node 10 (GPU 1 mem)
>>>> mon_data/mbm_MB_18/mbm_total_bytes <- numa node 18 (GPU 2 mem)
>>>> mon_data/mbm_MB_26/mbm_total_bytes <- numa node 26 (GPU 3 mem)
>>> To be backward compatible I find it more intuitive if instead
>>> this data is exposed as below:
>>>
>>> mon_data/mbm_L3_01/llc_occupancy <- cache id 1
>>> mon_data/mbm_L3_02/llc_occupancy <- cache id 2
>>> mon_data/mbm_NODE_00/mbm_total_bytes <- numa node 0 (socket 0)
>>> mon_data/mbm_NODE_01/mbm_total_bytes <- numa node 1 (socket 1)
>>> mon_data/mbm_NODE_02/mbm_total_bytes <- numa node 2 (GPU 0 mem)
>>> mon_data/mbm_NODE_10/mbm_total_bytes <- numa node 10 (GPU 1 mem)
>>> mon_data/mbm_NODE_18/mbm_total_bytes <- numa node 18 (GPU 2 mem)
>>> mon_data/mbm_NODE_26/mbm_total_bytes <- numa node 26 (GPU 3 mem)
>>>
>>> When "mbm_total_bytes" move from mon_data/mbm_L3_x to
>>> mon_data/mbm_NODE_x it clearly indicates that it is memory bandwidth
>>> monitoring data moving from "L3" scope to "NODE" scope.
>>
>> The scope in mon_data shown in https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@intel.com/:
>>
>> mon_data
>> ├── mon_L3_00 <== monitoring data at scope L3
>> │ ├── llc_occupancy
>> │ ├── mbm_local_bytes
>> │ └── mbm_total_bytes
>> ├── mon_L3_01 <== monitoring data at scope L3
>> │ ├── llc_occupancy
>> │ ├── mbm_local_bytes
>> │ └── mbm_total_bytes
>> ├── mon_NODE_00 <== monitoring data at scope NODE
>> │ └── mbm_total_bytes
>> └── mon_NODE_01 <== monitoring data at scope NODE
>> └── mbm_total_bytes
>>
>> On some ARM platforms, for example, socket 0 (CPUs+L3) and socket 1
>> (CPUs+L3):
>> llc_occupancy is monitored through processor/L3.
>> total_bytes are monitored through memory controlor.
>>
>> Then the above "scope" is confused:
>> mon_L3_00 and mon_L3_01 only has llc_occupancy. The scope "L3" is fine here.
>> But mon_NODE_00 and 01 are confused because numa node 00 and 01 have
>> both L3 and memory. The name "mon_NODE_00" seems monitor both
> While the node may have L3 and memory the directory only represents what is
> monitored at the particular scope.
>
>> llc_occupany and total_bytes but it only monitor total_bytes. And on
>> CPU-less platforms, a CPU-less node does only have total_bytes which
>> seems match "scope NODE".
>> With "scope NODE", can user tell if it has llc_occupancy or total_bytes or both?
>
> resctrl has a "mon_features" file associated with the monitoring scope that
> informs user space which events can be expected in a resource group's monitoring
> data directories.
Right.
>
>>
>> If change "mon_MB_01" to "mon_MM_01", the scope is "MM" now which
>> means monitoring memory (total_bytes) in numa node 1 with "scope MM"
>> (not L3)?
> This sounds redundant to me since the event names already have the resource embedded.
> "llc_occupancy" implies L3 occupancy
> "mbm_local_bytes" ... the "mbm" implies this is memory bandwidth monitoring data.
>
> User space can infer the scope at which the monitoring data is collected from the
> directory the event file is in.
>
Right.
Thanks.
-Fenghua
^ permalink raw reply [flat|nested] 66+ messages in thread
* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-06-25 1:26 ` Fenghua Yu
2026-06-25 15:43 ` Reinette Chatre
@ 2026-07-02 13:37 ` Ben Horgan
2026-07-02 15:16 ` Fenghua Yu
1 sibling, 1 reply; 66+ messages in thread
From: Ben Horgan @ 2026-07-02 13:37 UTC (permalink / raw)
To: Fenghua Yu, Reinette Chatre, Tony Luck, James Morse, Dave Martin,
Babu Moger, Drew Fustini, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi Fenghua,
On 6/25/26 02:26, Fenghua Yu wrote:
> Hi, Reinette,
>
> On 6/24/26 15:22, Reinette Chatre wrote:
>> Hi Fenghua,
>>
>> On 6/24/26 12:08 PM, Fenghua Yu wrote:
>>> Hi, Reinette, Ben, Shaopen, et al,
>>>
>>> On 5/29/26 11:06, Reinette Chatre wrote:
>>>
>>> As Shaopen and Ben mentioned earlier, we are working on two MPAM
>>> features that may need to change schemata interface. The CPU-less
>>> feature was discussed on LPC (although the interfaces will be
>>> slightly different from the LPC).
>>
>> I know. Here is where I tried to engage with you on needed interfaces
>> after LPC:
>> https://lore.kernel.org/lkml/fb1e2686-237b-4536-
>> acd6-15159abafcba@intel.com/
>
> MPAM ACPI defines MSC (Memory System Control) is defined in one of two
> ways (not both) on one platform:
> 1. L3 and memory together on each processor MSC
> 2. L3 in processor MSC and memory control/monitoring in different memory
> MSCs.
On one platform, if there are MSC with memory bandwidth monitors or
controls in both your slc and at the memory controllers then the MPAM
ACPI tables would describe those at the memory as being at the memory
and those at the cache.
This could lead to having memory bandwidth controls/monitors at both L3
and memory scope.
>
> On type 1 platform, schemata is legacy:
> MB:1=100;2=100 <-- cache id 1 and 2 as domain id
>
> On type 2 platform, I will not reuse "MB:" name. Instead, define new
> resource name "MBN:" for numa node and schemata is:
> MBN:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10, 18,
> 26 as domain id
> On type 2 platform, there won't be "MB:" line. Numa 0 and 1
> are for mbm allocation on socket 0 and 1. 2,10, 18 and 26 are for GPU
> memory nodes allocation.
>
> BTW, Slow MBA (SMBA) is different from MBA Numa (MBN). SMBA still relies
> on L3 and the domain id in SMBA is still cache id. MBN depends on each
> memory controlor with numa id as domain id for both CPU and CPU-less
> memory nodes.
>
> On type 1 platform, there is only MB:
>
> info
> └── MB
> └── resource_schemata
> ├── MB
> │ ├── max
> │ ├── min
> │ ├── resolution
> │ ├── scale
> │ ├── scope <== contains "L3"
> │ ├── tolerance
> │ ├── type
> │ └── unit
>
> On type 2 platform, there is only MBN:
> info
> └── MBN
> └── resource_schemata
> ├── MBN
> │ ├── max
> │ ├── min
> │ ├── resolution
> │ ├── scale
> │ ├── scope <== contains "NUMA"
> │ ├── tolerance
> │ ├── type
> │ └── unit
>
> This is different from the "scope" hierarchy discussed in the link. "MB"
> and "MBN" won't exist on the same platform.
>
> I find it's hard (and not useful) to split "MB" for memory with CPU and
> "MBN" for CPU-less memory node. It's easier to have either "MB" for
> legacy memory with CPU or "MBN" for CPU-less memory.
yes, I don't think CPU-less memory needs special casing in the interface
once there is support for NUMA scope.
>
> Any thoughts? Does this update make sense?
I think a _NODE postfix for controls with NUMA scope makes sense. I
brought up naming of controls when they are the same but have different
scope earlier in the thread and Reinette pointed me at this earlier
discussion.
https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@intel.com/
Thanks,
Ben
>
>>
>>> Hardlimit feature was not discussed yet.
>>
>> It was considered. See discussion starting at
>> https://lore.kernel.org/lkml/1c4b6b46-16f9-4887-93f5-
>> e0f5e7f30a6f@intel.com/
>>
>>> It's good to discuss this further in this RFC thread before the new
>>> features RFCs will be sent out.
>>
>> ack.
>>
>>>
>>> Overall, the new features can fit into this RFC well.
>>>
>>>> Hi Everybody,
>>>>
>
> [SNIP]
>
>>>>
>>>> This series can be used on an x86 system where it will show two new
>>>> dummy controls
>>>> where it is possible to interact with the new controls.
>>>> For example:
>>>>
>>>> # cat schemata
>>>> MB_MAX:0=100;1=100
>>>> MB_MIN:0=100;1=100
>>>> MB:0=100;1=100
>>>
>>> Some platforms may support CPU-less node which is represented by numa
>>> node id, examples:
>>> 1. CXL type 2 memory node which provides CXL memory without CPU and
>>> L3 on the node
>>> 2. GPU memory node that can be accessed by all CPUs but doesn't have
>>> a local CPU and L3 bound to.
>>> etc.
>>>
>>> MPAM can allocate and monitor mem bandwidth on these memory node.
>>> Since no CPU and L3 on the node, cache id cannot be used in "MB:"
>>> line. Instead, numa ids are used to identify MB allocation and
>>> monitoring.
>>>
>>> For example, the MB allocation on CPU-less platforms could be:
>>> MB:0=100;1=100;2=100;10=100;18=100;26=100
>>>
>>> Where: domain id 0, 1, 2, etc are numa node id shown in /sys/devices/
>>> system/node directory or by numctl.
>>> 0: socket 0, node 0, CPUs, memory
>>> 1: socket 1, node 1, CPUs, memory
>>> 2: GPU 0, node 2, no CPU, memory only
>>> 10: GPU 1, node 10, no CPU, memory only
>>> 18: GPU 3, node 18, no CPU, memory only
>>> 26: GPU 4, node 26, no CPU, memory only
>>>
>>> Arch specific driver (e.g. MPAM) detects CPU-less node. If there is
>>> any CPU-less node, use numa id in "MB:". Otherwise, fallback to
>>> legacy cache id.
>>
>> We always have to consider backward compatibility and to do so we
>> cannot just retroactively
>> change what domain ID represents when user space interacts with the
>> "MB" control.
>>
>> The legacy "MB" control is already defined and its domain ID
>> represents an L3 cache ID. To
>> support these new devices resctrl would need to expose a new control.
>>
>
> Agree. Add new control "MBN" for Memory Bandwidth Allocation on numa
> node. See above.
>
>>>
>>
>>> There is another MPAM feature called MBW Max hardlimit which sets
>>> "MB:" allocation as hardlimit (i.e. MBW throttling percentage must
>>> be satisfied) per domain. Adding a new "MB_HLIM:" line in schemata.
>>> It's 1:1 mapped to "MB:" to control hardlimit of MB throttling
>>> percentage on each domain. By default hardlimit is off (0) and can
>>> be turned on to set MBW Max hardlimit on a domain.
>>
>> ack. This sounds like a new control associated with the MB resource.
>> This is a boolean control as Dave highlighted in previous discussion so
>> resctrl would need to know its properties.
>> See https://lore.kernel.org/lkml/aO0Oazuxt54hQFbx@e133380.arm.com/
>>
>
> Right. ("MB_HLIM" name may be adjusted accordingly when "MB_MAX" is
> available.)
>
>>> For exmple:
>>> MB_HLIM: 0=0;1=0;2=1;10=0;18=0;26=0
>>> MB:0=100;1=100;2=80;10=100;18=100;26=100
>>>
>>> On GPU memory numa node 2: cannot use more than 80% of total max mbw
>>> even if there is still idle mem bandwidth on this node).
>>>
>>> MBW allocations on all other domains are soft limited, meaning MBW
>>> can be used more than specified if mem is idle.
>>>
>>
>> ack.
>>
>>>> L3:0=fff;1=fff
>>>> # echo 'MB_MIN:0=50' > schemata
>>>> # cat schemata
>>>> MB_MAX:0=100;1=100
>>>> MB_MIN:0=50;1=100
>>>> MB:0=100;1=100
>>>> L3:0=fff;1=fff
>>>>
>>>> Writing to the dummy control will call a dummy callback that just
>>>> prints to the
>>>> kernel log:
>>>> "resctrl: Updata temporary MIN control on domain 0 with user value 50"
>>>>
>>>>
>>>> Example output of info/MB/:
>>>> /sys/fs/resctrl/info/MB/thread_throttle_mode:max
>>>> /sys/fs/resctrl/info/MB/num_closids:15
>>>> /sys/fs/resctrl/info/MB/delay_linear:1
>>>> /sys/fs/resctrl/info/MB/min_bandwidth:10
>>>
>>> Add two new MB info RO files:
>>> 1. /sys/fs/resctrl/info/MB/domain_id
>>> It shows "numa" for using numa id in "MB:" or "cache" for using
>>> legacy cache id.
>>
>> This proposal introduces a *global* property to the MB *resource*? It
>> does not seem as though
>> this takes into account *anything* about how resctrl can support new
>> hardware that has been
>> discussed before, during, or after LPC. You have not participated in
>> these discussions and
>> now make an orthogonal proposal that does not take into account *any*
>> of the requirements
>> that we have been struggling with for months.
>>
>> Why should this proposal be taken seriously? In your absence folks
>> have been trying to
>> accommodate how these upcoming products and be supported and the
>> "scope" file associated with
>> a control is intended to communicate to user space how the domain ID
>> should be interpreted.
>>
>> Why are you proposing something entirely different here without even
>> acknowledging current
>> approach and explaining why it does not work for you?
>>
>
> So can I change this part to adding the following files in info dirctory?
>
> 1. For numa memory bw allocation (MBN):
> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/
> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/resolution:100
> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/tolerance:5
> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/type:scalar
> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/min:10
> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/scale:1
> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/scope:NUMA
> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/unit:all
> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/max:100
>
>>> 2. /sys/fs/resctrl/info/MB/max_lim
>>> It shows number 0-3 for MPAM MBW max limit behaviors: 0 for
>>> supporting both softlimit and hardlimit, etc.
>>
>> Again this adds another *global* property to the MB resource but then
>> above you
>> describe the new "MB_HLIM" schemata file entry that implies that it is
>> a new control
>> for the MB resource. Having it be a new control for the MB resource
>> matches earlier
>> discussions. To support this I thus expect it to be exposed as a new
>> control with
>> potentially a new type if any of the existing planned types do not
>> suffice.
>>
>
> How about adding these MB_HLIM dir and files in info?
>
> /sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/type: boolean
> /sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/max_lim: 0
>
>
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/resolution:100
>>>
>>> Is it more concise to s/resource_schemata/schemata/? "resource_"
>>> seems redundant in the context "info/MB".
>>
>> We could do this, yes.
>>
>>>
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/tolerance:5
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/type:scalar
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/min:10
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/scale:1
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/scope:L3
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/unit:all
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/max:100
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/resolution:100
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/tolerance:5
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/type:scalar
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/min:10
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scale:1
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scope:L3
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/unit:all
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/max:100
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/resolution:100
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/tolerance:5
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/type:scalar
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/min:10
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scale:1
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scope:L3
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/unit:all
>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/max:100
>>>> /sys/fs/resctrl/info/MB/bandwidth_gran:10
>>>
>>> For MBW monitoring, extend mon_data/ directory to monitor CPU-less
>>> memory node. For example,
>>
>> Here is where I attempted to discuss with you how to support
>> monitoring on these systems:
>> https://lore.kernel.org/lkml/fb1e2686-237b-4536-
>> acd6-15159abafcba@intel.com/
>>
>> Here again you respond with something completely different without
>> acknowledging
>> the previous discussion or noting why that does not work for you.
>>
>>>
>>> On legacy platforms (i.e. L3 and memory are described in same MPAM
>>> ACPI MSC wich doesn't support CPU-less nodes):
>>> mon_data/mbm_L3_01/llc_occupancy
>>> mon_data/mbm_L3_01/mbm_total_bytes
>>> mon_data/mbm_L3_02/llc_occupancy
>>> mon_data/mbm_L3_02/mbm_total_bytes
>>>
>>> On platforms with L3 and memory in separate MPAM ACPI MSCs
>>> but there is no CPU-less node:
>>> mon_data/mbm_L3_01/llc_occupancy <- cache id 1
>>> mon_data/mbm_L3_02/llc_occupancy <- cache id 2
>>> mon_data/mbm_MB_00/mbm_total_bytes <- numa node 0 (socket 0)
>>> mon_data/mbm_MB_01/mbm_total_bytes <- numa node 1 (socket 1)
>>
>> Here too I do not find it appropriate for resctrl to retroactively
>> change its interface. "MB" is a resource and the above switches the
>> resctrl interface to imply the monitoring data of a resource can be
>> found in the mon directory that matches the resource name. This is
>> not what resctrl does today. Doing something like above will result in
>> resctrl having a confusing interface where "sometimes" memory bandwidth
>> data can be found in the L3 directory and "sometimes" memory bandwidth
>> data can be found in the MB directory.
>>
>> As I described to you in December resctrl already exposes the monitoring
>> data based on the *scope*. As you also point out above, today the memory
>> bandwidth monitoring data at L3 scope can be found in the L3 directory.
>> "L3" should thus not be interpreted as the resource L3 but the scope L3
>> since it contains MBM data today. When viewing it as such resctrl could
>> internally be more explicit and separate monitoring scope from monitoring
>> resource and present the monitoring data based on scope to remain
>> intuitively
>> backward compatible while obtaining support for these memory nodes.
>>
>>>
>>> On platforms with L3 and memory in separate MPAM ACPI MSCs
>>> and there are CPU-less nodes:
>>> mon_data/mbm_L3_01/llc_occupancy <- cache id 1
>>> mon_data/mbm_L3_02/llc_occupancy <- cache id 2
>>> mon_data/mbm_MB_00/mbm_total_bytes <- numa node 0 (socket 0)
>>> mon_data/mbm_MB_01/mbm_total_bytes <- numa node 1 (socket 1)
>>> mon_data/mbm_MB_02/mbm_total_bytes <- numa node 2 (GPU 0 mem)
>>> mon_data/mbm_MB_10/mbm_total_bytes <- numa node 10 (GPU 1 mem)
>>> mon_data/mbm_MB_18/mbm_total_bytes <- numa node 18 (GPU 2 mem)
>>> mon_data/mbm_MB_26/mbm_total_bytes <- numa node 26 (GPU 3 mem)
>> To be backward compatible I find it more intuitive if instead
>> this data is exposed as below:
>>
>> mon_data/mbm_L3_01/llc_occupancy <- cache id 1
>> mon_data/mbm_L3_02/llc_occupancy <- cache id 2
>> mon_data/mbm_NODE_00/mbm_total_bytes <- numa node 0 (socket 0)
>> mon_data/mbm_NODE_01/mbm_total_bytes <- numa node 1 (socket 1)
>> mon_data/mbm_NODE_02/mbm_total_bytes <- numa node 2 (GPU 0 mem)
>> mon_data/mbm_NODE_10/mbm_total_bytes <- numa node 10 (GPU 1 mem)
>> mon_data/mbm_NODE_18/mbm_total_bytes <- numa node 18 (GPU 2 mem)
>> mon_data/mbm_NODE_26/mbm_total_bytes <- numa node 26 (GPU 3 mem)
>>
>> When "mbm_total_bytes" move from mon_data/mbm_L3_x to
>> mon_data/mbm_NODE_x it clearly indicates that it is memory bandwidth
>> monitoring data moving from "L3" scope to "NODE" scope.
>
> The scope in mon_data shown in https://lore.kernel.org/lkml/
> fb1e2686-237b-4536-acd6-15159abafcba@intel.com/:
>
> mon_data
> ├── mon_L3_00 <== monitoring data at scope L3
> │ ├── llc_occupancy
> │ ├── mbm_local_bytes
> │ └── mbm_total_bytes
> ├── mon_L3_01 <== monitoring data at scope L3
> │ ├── llc_occupancy
> │ ├── mbm_local_bytes
> │ └── mbm_total_bytes
> ├── mon_NODE_00 <== monitoring data at scope NODE
> │ └── mbm_total_bytes
> └── mon_NODE_01 <== monitoring data at scope NODE
> └── mbm_total_bytes
>
> On some ARM platforms, for example, socket 0 (CPUs+L3) and socket 1
> (CPUs+L3):
> llc_occupancy is monitored through processor/L3.
> total_bytes are monitored through memory controlor.
>
> Then the above "scope" is confused:
> mon_L3_00 and mon_L3_01 only has llc_occupancy. The scope "L3" is fine
> here.
> But mon_NODE_00 and 01 are confused because numa node 00 and 01 have
> both L3 and memory. The name "mon_NODE_00" seems monitor both
> llc_occupany and total_bytes but it only monitor total_bytes. And on
> CPU-less platforms, a CPU-less node does only have total_bytes which
> seems match "scope NODE".
>
> With "scope NODE", can user tell if it has llc_occupancy or total_bytes
> or both?
>
> If change "mon_MB_01" to "mon_MM_01", the scope is "MM" now which means
> monitoring memory (total_bytes) in numa node 1 with "scope MM" (not L3)?
>
> Thanks.
>
> -Fenghua
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-07-02 13:37 ` Ben Horgan
@ 2026-07-02 15:16 ` Fenghua Yu
2026-07-03 13:42 ` Ben Horgan
0 siblings, 1 reply; 66+ messages in thread
From: Fenghua Yu @ 2026-07-02 15:16 UTC (permalink / raw)
To: Ben Horgan, Reinette Chatre, Tony Luck, James Morse, Dave Martin,
Babu Moger, Drew Fustini, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi, Ben,
On 7/2/26 06:37, Ben Horgan wrote:
> Hi Fenghua,
>
> On 6/25/26 02:26, Fenghua Yu wrote:
>> Hi, Reinette,
>>
>> On 6/24/26 15:22, Reinette Chatre wrote:
>>> Hi Fenghua,
>>>
>>> On 6/24/26 12:08 PM, Fenghua Yu wrote:
>>>> Hi, Reinette, Ben, Shaopen, et al,
>>>>
>>>> On 5/29/26 11:06, Reinette Chatre wrote:
>>>>
>>>> As Shaopen and Ben mentioned earlier, we are working on two MPAM
>>>> features that may need to change schemata interface. The CPU-less
>>>> feature was discussed on LPC (although the interfaces will be
>>>> slightly different from the LPC).
>>>
>>> I know. Here is where I tried to engage with you on needed interfaces
>>> after LPC:
>>> https://lore.kernel.org/lkml/fb1e2686-237b-4536-
>>> acd6-15159abafcba@intel.com/
>>
>> MPAM ACPI defines MSC (Memory System Control) is defined in one of two
>> ways (not both) on one platform:
>> 1. L3 and memory together on each processor MSC
>> 2. L3 in processor MSC and memory control/monitoring in different memory
>> MSCs.
>
> On one platform, if there are MSC with memory bandwidth monitors or
> controls in both your slc and at the memory controllers then the MPAM
> ACPI tables would describe those at the memory as being at the memory
> and those at the cache.
>
> This could lead to having memory bandwidth controls/monitors at both L3
> and memory scope.
>
The locator type in MSC for this L3 is still 1, right?
So the control and monitor example could be:
MSC0: type 1 L3 with cache id 0 on socket 0
MSC1: type 1 L3 with cache id 1 on socket 1
MSC2: type 2 memory with numa node 1 on socket 0
MSC3: type 2 memory with numa node 1 on socket 1
The schemata file could be:
L3: 0=fff;1=fff <-- cache control on cache id
MB: 0=fff;1=fff <-- memory bandwidth control on cache id. legacy.
MB_NODE: 1=100;2=100 <-- memory bandwidth conttrol on node id. CPU-less
Cache and memory bandwidth monitoring:
On cache id 0, both llc_occupancy and total_bytes are monitored:
mon_data/mon_L3_00/mbm_llc_occupancy
mon_data/mon_L3_00/mbm_total_byptes
On cache id 1, both llc_occupancy and total_bytes are monitored::
mon_data/mon_L3_01/mbm_llc_occupancy
mon_data/mon_L3_01/mbm_total_bytes
On NUMA node 1, only total_bytes is monitored:
mon_data/mon_NODE_01/mbm_total_bytes
On NUMA node 2, only total bytes is monitored:
mon_data/mon_NODE_02/mbm_total_bytes
>> On type 1 platform, schemata is legacy:
>> MB:1=100;2=100 <-- cache id 1 and 2 as domain id
>>
>> On type 2 platform, I will not reuse "MB:" name. Instead, define new
>> resource name "MBN:" for numa node and schemata is:
>> MBN:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10, 18,
>> 26 as domain id
>> On type 2 platform, there won't be "MB:" line. Numa 0 and 1
>> are for mbm allocation on socket 0 and 1. 2,10, 18 and 26 are for GPU
>> memory nodes allocation.
>>
>> BTW, Slow MBA (SMBA) is different from MBA Numa (MBN). SMBA still relies
>> on L3 and the domain id in SMBA is still cache id. MBN depends on each
>> memory controlor with numa id as domain id for both CPU and CPU-less
>> memory nodes.
>>
>> On type 1 platform, there is only MB:
>>
>> info
>> └── MB
>> └── resource_schemata
>> ├── MB
>> │ ├── max
>> │ ├── min
>> │ ├── resolution
>> │ ├── scale
>> │ ├── scope <== contains "L3"
>> │ ├── tolerance
>> │ ├── type
>> │ └── unit
>>
>> On type 2 platform, there is only MBN:
>> info
>> └── MBN
>> └── resource_schemata
>> ├── MBN
>> │ ├── max
>> │ ├── min
>> │ ├── resolution
>> │ ├── scale
>> │ ├── scope <== contains "NUMA"
>> │ ├── tolerance
>> │ ├── type
>> │ └── unit
>>
>> This is different from the "scope" hierarchy discussed in the link. "MB"
>> and "MBN" won't exist on the same platform.
>>
>> I find it's hard (and not useful) to split "MB" for memory with CPU and
>> "MBN" for CPU-less memory node. It's easier to have either "MB" for
>> legacy memory with CPU or "MBN" for CPU-less memory.
>
> yes, I don't think CPU-less memory needs special casing in the interface
> once there is support for NUMA scope.
>
>>
>> Any thoughts? Does this update make sense?
>
> I think a _NODE postfix for controls with NUMA scope makes sense. I
> brought up naming of controls when they are the same but have different
> scope earlier in the thread and Reinette pointed me at this earlier
> discussion.
Is _NODE postfix sufficient for future?
e.g. SMMU locator id is IORT table node id. AFAICT, the node id is not a
numa node. If that's the case, _NODE postfix may cause confusion here.
Is explict "_NUMA" postfix clearer?
[SNIP]
Thanks.
-Fenghua
^ permalink raw reply [flat|nested] 66+ messages in thread* Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
2026-07-02 15:16 ` Fenghua Yu
@ 2026-07-03 13:42 ` Ben Horgan
0 siblings, 0 replies; 66+ messages in thread
From: Ben Horgan @ 2026-07-03 13:42 UTC (permalink / raw)
To: Fenghua Yu, Reinette Chatre, Tony Luck, James Morse, Dave Martin,
Babu Moger, Drew Fustini, Chen Yu
Cc: Borislav Petkov, Thomas Gleixner, Dave Hansen, Peter Newman,
x86@kernel.org, linux-kernel@vger.kernel.org
Hi Fenghua,
On 7/2/26 16:16, Fenghua Yu wrote:
> Hi, Ben,
>
> On 7/2/26 06:37, Ben Horgan wrote:
>> Hi Fenghua,
>>
>> On 6/25/26 02:26, Fenghua Yu wrote:
>>> Hi, Reinette,
>>>
>>> On 6/24/26 15:22, Reinette Chatre wrote:
>>>> Hi Fenghua,
>>>>
>>>> On 6/24/26 12:08 PM, Fenghua Yu wrote:
>>>>> Hi, Reinette, Ben, Shaopen, et al,
>>>>>
>>>>> On 5/29/26 11:06, Reinette Chatre wrote:
>>>>>
>>>>> As Shaopen and Ben mentioned earlier, we are working on two MPAM
>>>>> features that may need to change schemata interface. The CPU-less
>>>>> feature was discussed on LPC (although the interfaces will be
>>>>> slightly different from the LPC).
>>>>
>>>> I know. Here is where I tried to engage with you on needed interfaces
>>>> after LPC:
>>>> https://lore.kernel.org/lkml/fb1e2686-237b-4536-
>>>> acd6-15159abafcba@intel.com/
>>>
>>> MPAM ACPI defines MSC (Memory System Control) is defined in one of two
>>> ways (not both) on one platform:
>>> 1. L3 and memory together on each processor MSC
>>> 2. L3 in processor MSC and memory control/monitoring in different memory
>>> MSCs.
>>
>> On one platform, if there are MSC with memory bandwidth monitors or
>> controls in both your slc and at the memory controllers then the MPAM
>> ACPI tables would describe those at the memory as being at the memory
>> and those at the cache.
>>
>> This could lead to having memory bandwidth controls/monitors at both L3
>> and memory scope.
>>
>
> The locator type in MSC for this L3 is still 1, right?
>
> So the control and monitor example could be:
>
> MSC0: type 1 L3 with cache id 0 on socket 0
> MSC1: type 1 L3 with cache id 1 on socket 1
> MSC2: type 2 memory with numa node 1 on socket 0
> MSC3: type 2 memory with numa node 1 on socket 1
>
> The schemata file could be:
> L3: 0=fff;1=fff <-- cache control on cache id
> MB: 0=fff;1=fff <-- memory bandwidth control on cache id. legacy.
> MB_NODE: 1=100;2=100 <-- memory bandwidth conttrol on node id. CPU-less
>
> Cache and memory bandwidth monitoring:
> On cache id 0, both llc_occupancy and total_bytes are monitored:
> mon_data/mon_L3_00/mbm_llc_occupancy
> mon_data/mon_L3_00/mbm_total_byptes
> On cache id 1, both llc_occupancy and total_bytes are monitored::
> mon_data/mon_L3_01/mbm_llc_occupancy
> mon_data/mon_L3_01/mbm_total_bytes
>
> On NUMA node 1, only total_bytes is monitored:
> mon_data/mon_NODE_01/mbm_total_bytes
> On NUMA node 2, only total bytes is monitored:
> mon_data/mon_NODE_02/mbm_total_bytes
Yes, this all looks sensible.
>
>>> On type 1 platform, schemata is legacy:
>>> MB:1=100;2=100 <-- cache id 1 and 2 as domain id
>>>
>>> On type 2 platform, I will not reuse "MB:" name. Instead, define new
>>> resource name "MBN:" for numa node and schemata is:
>>> MBN:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10, 18,
>>> 26 as domain id
>>> On type 2 platform, there won't be "MB:" line. Numa 0 and 1
>>> are for mbm allocation on socket 0 and 1. 2,10, 18 and 26 are for GPU
>>> memory nodes allocation.
>>>
>>> BTW, Slow MBA (SMBA) is different from MBA Numa (MBN). SMBA still relies
>>> on L3 and the domain id in SMBA is still cache id. MBN depends on each
>>> memory controlor with numa id as domain id for both CPU and CPU-less
>>> memory nodes.
>>>
>>> On type 1 platform, there is only MB:
>>>
>>> info
>>> └── MB
>>> └── resource_schemata
>>> ├── MB
>>> │ ├── max
>>> │ ├── min
>>> │ ├── resolution
>>> │ ├── scale
>>> │ ├── scope <== contains "L3"
>>> │ ├── tolerance
>>> │ ├── type
>>> │ └── unit
>>>
>>> On type 2 platform, there is only MBN:
>>> info
>>> └── MBN
>>> └── resource_schemata
>>> ├── MBN
>>> │ ├── max
>>> │ ├── min
>>> │ ├── resolution
>>> │ ├── scale
>>> │ ├── scope <== contains "NUMA"
>>> │ ├── tolerance
>>> │ ├── type
>>> │ └── unit
>>>
>>> This is different from the "scope" hierarchy discussed in the link. "MB"
>>> and "MBN" won't exist on the same platform.
>>>
>>> I find it's hard (and not useful) to split "MB" for memory with CPU and
>>> "MBN" for CPU-less memory node. It's easier to have either "MB" for
>>> legacy memory with CPU or "MBN" for CPU-less memory.
>>
>> yes, I don't think CPU-less memory needs special casing in the interface
>> once there is support for NUMA scope.
>>
>>>
>>> Any thoughts? Does this update make sense?
>>
>> I think a _NODE postfix for controls with NUMA scope makes sense. I
>> brought up naming of controls when they are the same but have different
>> scope earlier in the thread and Reinette pointed me at this earlier
>> discussion.
>
> Is _NODE postfix sufficient for future?
>
> e.g. SMMU locator id is IORT table node id. AFAICT, the node id is not a
> numa node. If that's the case, _NODE postfix may cause confusion here.
>
> Is explict "_NUMA" postfix clearer?
How is the SMMU locator id displayed to the user?
For the NUMA node id we already use the name node in sysfs,
/sys/devices/system/node/node<node_id>
Thanks,
Ben
>
> [SNIP]
>
> Thanks.
>
> -Fenghua
^ permalink raw reply [flat|nested] 66+ messages in thread