Re: [RFC v2] arm,x86,fs/resctrl: Generic schema description Proof of Concept

[Reinette Chatre <reinette.chatre@intel.com>]

Hi Babu,

On 8/10/26 11:52 AM, Babu Moger wrote:
> On 8/7/26 17:53, Reinette Chatre wrote:

...

>>
>> Second, how to name these resources/controls needs to be decided. This is what prompted my question
>> about future considerations to you because my original proposal in
>> https://lore.kernel.org/lkml/f5b6cec4-03d8-4a11-884d-d4579dab6b22@intel.com/
>> was able to convey the resource and allocation scope via info hierarchy but it used the same
>> name in the schemata file that Ben highlighted could be problematic.
>>
>> I believe there is agreement that the resource should include the resource scope in its name. This
>> is currently done for L2 and L3, and planned to be done for MB_NODE (the "MB" resource does not have
>> "L3" in its name, this cannot be changed now, but "MB" resource is implicitly "L3" scope). The "NODE"
>> in MB_NODE is thus memory bandwidth allocation at NUMA node scope - "NODE" in MB_NODE is the *resource*
>> scope.
>>
>> Considering this I do not think that GLBE should use MB_NODE as you also state above. Since it does
>> not allocate memory bandwidth *resource* at node scope.
>>
>> We also discussed before (https://lore.kernel.org/lkml/c78169bc-e2d6-4583-96ec-09fa6dd6653a@intel.com/ )
>> of having the control's scope part of the control's name.
>>
>> We could have a rule of thumb to include both resource scope and control scope in the
>> control name but only one instance is displayed if they are the same.
>>
>> For example, what do you think of something like below as an alternate proposal of what I mentioned in
>> https://lore.kernel.org/lkml/f5b6cec4-03d8-4a11-884d-d4579dab6b22@intel.com/:
>>
>> GLBE (booted with NPS < 4):
>> info/
>> └── MB/                /* memory bandwidth allocation implicitly/legacy at L3 scope, think of this as "MB_L3" */
>>      └── schemata/
>>          ├── MB/          /* control scope = L3 */
>>          └── MB_L3NODE/   /* control scope = node ("L3" resource scope + "node" control scope) */
>>
>> GLBE (booted with NPS = 4):
>> info/
>> └── MB/                /* memory bandwidth allocation implicitly/legacy at L3 scope, think of this as "MB_L3" */
>>      └── schemata/
>>          ├── MB/          /* control scope = L3 */
>>          └── MB_L3SYSTEM/ /* control scope = system ("L3" resource scope + "system" control scope*/
> 
> 
> Can the names here MB_L3NODE and MB_L3SYSTEM be simplified to
> MB_NODE and MB_SYSTEM? The L3 scope is already implied by the higher-
> level hierarchy.

This would be the alternate proposal I mention above but Ben highlighted in 
https://lore.kernel.org/lkml/1444b515-752e-4167-87f5-30ace189e05c@arm.com/ that such naming would result in
a conflict if GLBE is ever combined with some actual node based MBA that would result in two controllers named
"MB_NODE" that need to be accommodated by schemata file. This is what prompted my question to
you with the conclusion that we cannot predict the future. The above proposal prepares resctrl for some 
hypothetical scenario where a GLBE style system also supports actual node MBA.
>> ii
>> MPAM MSC:
>> info/
>> ├── MB/                /* memory bandwidth allocation at L3 scope */
>> │   └── schemata/
>> │       └── MB/        /* control scope = L3 */
>> └── MB_NODE/           /* memory bandwidth allocation at node scope */
>>      └── schemata/
>>          └── MB_NODE/   /* control scope = node */
>>
>> The GLBE controls make it clear that the *same* resource is allocated using two
>> different controls that have different scope. This creates an implicit dependency
>> between the two that is not quite captured but having two controls for the same
>> resource would already give user space some insight that their control values
>> need to be considered with care.
>>
>> I'd appreciate your, Ben's, and anybody else's thoughts on this.
> 
> Looks good to me.
> 
> 
> There is one more point I'd like to bring up regarding GLBE.
> 
> Consider the following example with 16 CPUs, 2 NUMA domains, and 2 L3 domains per NUMA node:
> 
> NUMA0
>  ├─ L3_0: CPUs 0-3
>  └─ L3_1: CPUs 4-7
> 
> NUMA1
>  ├─ L3_2: CPUs 8-11
>  └─ L3_3: CPUs 12-15
> 
> 
> When updating NUMA0, we need to update one CPU from each L3 domain.
> However, this relationship is not currently represented in our
> domain data structure.  The domain's hdr.cpumask simply contains
> CPUs 0-7 and does not provide any information about the underlying
> L3-domain boundaries.

I expect that the domain associated with a control that has node scope would include all
CPUs associated with that node in is cpumask. In your example I thus expect there to
only be two domains, domain ID 0 would have CPUs 0 - 7 in its cpumask and domain ID 1
would have CPUs 8 to 15 in its cpumask.

> 
> This was the reason we needed additional processing while programming the MSRs:
> 
> https://lore.kernel.org/lkml/a2a06bd290e68f902be9e7cc3ad35f0a2211b950.1776980182.git.babu.moger@amd.com/
> 
> I'm wondering whether this relationship should be represented
> explicitly in the control data structure, or whether that would be
> exposing details that are too specific to a particular hardware
> implementation.

From what I understand this is already accommodated. What am I missing? Reinette