Re: [RFC v2] arm,x86,fs/resctrl: Generic schema description Proof of Concept

[Reinette Chatre <reinette.chatre@intel.com>]

Hi Babu and Ben,

On 8/7/26 11:33 AM, Moger, Babu wrote:
> Hi Reinette/Ben,
> 
> On 8/7/2026 10:35 AM, Reinette Chatre wrote:
>> Hi Babu,
>>
>> On 8/7/26 7:14 AM, Moger, Babu wrote:
>>> On 8/6/2026 12:11 PM, Reinette Chatre wrote:
>>>> On 8/6/26 10:04 AM, Luck, Tony wrote:
>>>>> On Wed, Aug 05, 2026 at 04:57:04PM -0700, Reinette Chatre wrote:
>>>>>> On 8/5/26 9:59 AM, Ben Horgan wrote:
>>>>>>> Just given a go at running this on a model with MPAM and I can mount resctrl with this
>>>>>>> small patch to initialise the emulated_by lists so that list_empty() behaves.
>>>>>>>
>>>>>>> diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c
>>>>>>> index 21dcbf5764cc..638095649151 100644
>>>>>>> --- a/drivers/resctrl/mpam_resctrl.c
>>>>>>> +++ b/drivers/resctrl/mpam_resctrl.c
>>>>>>> @@ -1018,6 +1018,7 @@ static int mpam_resctrl_control_init(struct mpam_resctrl_res *res)
>>>>>>>           case RDT_RESOURCE_L3:
>>>>>>>                   mpam_ctrl->r_ctrl.type = RESCTRL_CTRL_BITMAP;
>>>>>>>                   mpam_ctrl->r_ctrl.name = RESCTRL_CTRL_NAME_DEF;
>>>>>>> +               INIT_LIST_HEAD(&mpam_ctrl->r_ctrl.emulated_by);
>>>>>>>                   INIT_LIST_HEAD_RCU(&mpam_ctrl->r_ctrl.domains);
>>>>>>>                   __set_bit(RESCTRL_BITMAP_FLAG_SPARSE, mpam_ctrl->r_ctrl.bitmap.flags);
>>>>>>>                   mpam_ctrl->r_ctrl.bitmap.cbm_len = class->props.cpbm_wd;
>>>>>>> @@ -1048,6 +1049,7 @@ static int mpam_resctrl_control_init(struct mpam_resctrl_res *res)
>>>>>>>                   r->ctrl_scope = RESCTRL_L3_CACHE;
>>>>>>>                   mpam_ctrl->r_ctrl.type = RESCTRL_CTRL_SCALAR;
>>>>>>>                   mpam_ctrl->r_ctrl.name = RESCTRL_CTRL_NAME_DEF;
>>>>>>> +               INIT_LIST_HEAD(&mpam_ctrl->r_ctrl.emulated_by);
>>>>>>>                   INIT_LIST_HEAD_RCU(&mpam_ctrl->r_ctrl.domains);
>>>>>>>
>>>>>>>                   r->bw_throttle_mode = THREAD_THROTTLE_UNDEFINED;
>>>>>>
>>>>>> Thank you for this. Added this and it is now available in branch resctrl/controls_rfc_v2.1
>>>>>
>>>>> Is same needed for x86? I don't see any initialization of the
>>>>> "r_ctrl.emulated_by" lists in similar initialization functions.
>>>> x86 "emulated_by" list initialization should be in both branches.
>>>>
>>>> Branch resctrl/controls_rfc_v2.1 combined all "emulated_by" list initialization
>>>> (for x86 and MPAM) into commit:
>>>> 3407e523c988 ("fs/resctrl: Introduce emulated controls and control mode")
>>>>
>>>> In the original resctrl/controls_rfc_v2 the x86 "emulated_by" list initialization can
>>>> be found in commit fbed64f80515 ("x86/resctrl: SAMPLE: Emulated controls")
>>>>
>>>
>>> Looking at the commit:
>>>
>>> commit 3407e523c988 ("fs/resctrl: Introduce emulated controls and control mode")
>>>
>>> Based on the patch description, emulated controls are intended to be
>>> used only when there is a difference between the native and legacy
>>> controls. If no such difference exists, both modes should operate
>>> identically.
>>
>> The original AMD MBA enabling did not follow the original percentage based MBA
>> control so resctrl essentially has *two* "legacy" MB controls today: one for AMD
>> and one for Intel, MPAM, and RISC-V (planned afaik). That cannot be changed now.
>> AMD would continue to expose the MB control that is not the percentage based "legacy"
>> control but actually AMD's native control. In short, yes, on AMD's "MB" control
>> the "legacy" and "native" control modes should operate identically.
>> At least now users could use the files in info/MB/schemata/MB/* to learn the
>> properties of the control.
>>> For MBA(AMD) and GMBA, there does not appear to be any difference
>>> between the two modes. Is that understanding correct?
>> It is not clear to me how resctrl should support AMD's GLBE. Note I am intentionally
>> not using GMBA since that already makes an assumption on how resctrl will support this.
>> Do you perhaps have an answer for Ben's question in
>> https://lore.kernel.org/lkml/1444b515-752e-4167-87f5-30ace189e05c@arm.com/ ?
> 
> Yes. We are already at RFC v2. Let me respond here. Please see my response below.
> 
>>
>> GLBE claims to enable users to allocate memory bandwidth at node scope but the memory
>> is managed at L3. This results in scenarios where, for example, a NUMA node can be
>> online and used but resctrl cannot expose it for bandwidth allocation when all the CPUs
>> at that NUMA node scope are offline. GLBE is thus not actually allocating memory bandwidth
>> at the NUMA node.
> 
> Yea. That is correct.
> 
> There are a couple of key differences when compared to the pure NUMA scope.
> 
> 1. In some cases, a NUMA node is treated as the entire system.
> 
>  
> https://lore.kernel.org/lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@amd.com/
> 
> 2. When a user updates the settings on a GLBE for a specific node, we need to update the MSRs within that node at the L3 scope:
> 
> https://lore.kernel.org/lkml/a2a06bd290e68f902be9e7cc3ad35f0a2211b950.1776980182.git.babu.moger@amd.com/
> 
> 
> Considering these differences, I think we should probably treat GLBE
> separate from MB_NODE scope. What do you think?

I think so too, but to Ben's point we need to be clear on terminology here. There are two usages
of "scope" to consider:
1) The scope of the *resource* being allocated.
   Here "scope" applies to the resource. For cache there is L2 and L3 that indicates the
   scope of the cache resource. There is also now two different scope to consider for
   memory bandwidth allocation, "L3 scope" for memory bandwidth at egress from L3 and "NUMA/node"
   scope for memory bandwidth at ingress to NUMA node.

2) The scope of the *control* used to allocate the resource.
   So far the scope of the resource has been assumed that to be the same as the scope of the control.
   That is, a resource at particular scope is allocated at that same scope. GLBE taught us this is not
   always the case - memory bandwidth at L3 scope can be allocated at NUMA(with some caveat as you point out)
   scope.

We have gone back and forth on this. Currently upstream resctrl has control scope as a property of
the resource with expectation that the resource and its controls have the same scope.  As a change
to this RFC v1 of this PoC had "scope" as a property of the control to support a control to have
different scope as resource. After deciding to treat MB_NODE as a new resource to support the CPU-less
nodes, RFC v2 changed this back to have scope a property of the resource.

First, it looks to me as though the RFC v1 separating control scope from resource scope needs to
return.

Second, how to name these resources/controls needs to be decided. This is what prompted my question
about future considerations to you because my original proposal in
https://lore.kernel.org/lkml/f5b6cec4-03d8-4a11-884d-d4579dab6b22@intel.com/
was able to convey the resource and allocation scope via info hierarchy but it used the same
name in the schemata file that Ben highlighted could be problematic.

I believe there is agreement that the resource should include the resource scope in its name. This
is currently done for L2 and L3, and planned to be done for MB_NODE (the "MB" resource does not have
"L3" in its name, this cannot be changed now, but "MB" resource is implicitly "L3" scope). The "NODE"
in MB_NODE is thus memory bandwidth allocation at NUMA node scope - "NODE" in MB_NODE is the *resource*
scope.

Considering this I do not think that GLBE should use MB_NODE as you also state above. Since it does
not allocate memory bandwidth *resource* at node scope. 

We also discussed before (https://lore.kernel.org/lkml/c78169bc-e2d6-4583-96ec-09fa6dd6653a@intel.com/ )
of having the control's scope part of the control's name.

We could have a rule of thumb to include both resource scope and control scope in the
control name but only one instance is displayed if they are the same.

For example, what do you think of something like below as an alternate proposal of what I mentioned in 
https://lore.kernel.org/lkml/f5b6cec4-03d8-4a11-884d-d4579dab6b22@intel.com/:

GLBE (booted with NPS < 4):
info/
└── MB/                /* memory bandwidth allocation implicitly/legacy at L3 scope, think of this as "MB_L3" */
    └── schemata/
        ├── MB/          /* control scope = L3 */
        └── MB_L3NODE/   /* control scope = node ("L3" resource scope + "node" control scope) */

GLBE (booted with NPS = 4):
info/
└── MB/                /* memory bandwidth allocation implicitly/legacy at L3 scope, think of this as "MB_L3" */
    └── schemata/
        ├── MB/          /* control scope = L3 */
        └── MB_L3SYSTEM/ /* control scope = system ("L3" resource scope + "system" control scope*/

MPAM MSC:
info/
├── MB/                /* memory bandwidth allocation at L3 scope */
│   └── schemata/
│       └── MB/        /* control scope = L3 */
└── MB_NODE/           /* memory bandwidth allocation at node scope */
    └── schemata/
        └── MB_NODE/   /* control scope = node */

The GLBE controls make it clear that the *same* resource is allocated using two
different controls that have different scope. This creates an implicit dependency
between the two that is not quite captured but having two controls for the same
resource would already give user space some insight that their control values
need to be considered with care.

I'd appreciate your, Ben's, and anybody else's thoughts on this.

>> Should resctrl prepare for some future where an AMD system may support memory bandwidth
>> allocation at the NUMA node while also supporting GLBE (memory bandwidth allocation at L3)?
> 
> No, I don't believe that's the case. At least, I haven't come across any information suggesting that.
Apologies, this was an unreasonable question from my side. We cannot predict the future.
We try to prepare for it with what we know today but new features always seem to come with unique
capabilities.

Reinette