Re: [RFC PATCH 01/19] x86,fs/resctrl: Add support for Global Bandwidth Enforcement (GLBE)

[Reinette Chatre <reinette.chatre@intel.com>]

Hi Babu,

On 2/10/26 5:07 PM, Moger, Babu wrote:
> Hi Reinette,
> 
> 
> On 2/9/2026 12:44 PM, Reinette Chatre wrote:
>> Hi Babu,
>>
>> On 1/21/26 1:12 PM, Babu Moger wrote:
>>> On AMD systems, the existing MBA feature allows the user to set a bandwidth
>>> limit for each QOS domain. However, multiple QOS domains share system
>>> memory bandwidth as a resource. In order to ensure that system memory
>>> bandwidth is not over-utilized, user must statically partition the
>>> available system bandwidth between the active QOS domains. This typically
>>
>> How do you define "active" QoS Domain?
> 
> Some domains may not have any CPUs associated with that CLOSID. Active meant, I'm referring to domains that have CPUs assigned to the CLOSID.

To confirm, is this then specific to assigning CPUs to resource groups via
the cpus/cpus_list files? This refers to how a user needs to partition
available bandwidth so I am still trying to understand the message here since
users still need to do this even when CPUs are not assigned to resource
groups.

> 
>>
>>> results in system memory being under-utilized since not all QOS domains are
>>> using their full bandwidth Allocation.
>>>
>>> AMD PQoS Global Bandwidth Enforcement(GLBE) provides a mechanism
>>> for software to specify bandwidth limits for groups of threads that span
>>> multiple QoS Domains. This collection of QOS domains is referred to as GLBE
>>> control domain. The GLBE ceiling sets a maximum limit on a memory bandwidth
>>> in GLBE control domain. Bandwidth is shared by all threads in a Class of
>>> Service(COS) across every QoS domain managed by the GLBE control domain.
>>
>> How does this bandwidth allocation limit impact existing MBA? For example, if a
>> system has two domains (A and B) that user space separately sets MBA
>> allocations for while also placing both domains within a "GLBE control domain"
>> with a different allocation, does the individual MBA allocations still matter?
> 
> Yes. Both ceilings are enforced at their respective levels.
> The MBA ceiling is applied at the QoS domain level.
> The GLBE ceiling is applied at the GLBE control  domain level.
> If the MBA ceiling exceeds the GLBE ceiling, the effective MBA limit will be capped by the GLBE ceiling.

It sounds as though MBA and GMBA/GLBE operates within the same parameters wrt
the limits but in examples in this series they have different limits. For example,
in the documentation patch [1] there is this:

 # cat schemata
    GMB:0=2048;1=2048;2=2048;3=2048
    MB:0=4096;1=4096;2=4096;3=4096
    L3:0=ffff;1=ffff;2=ffff;3=ffff

followed up with what it will look like in new generation [2]:

   GMB:0=4096;1=4096;2=4096;3=4096
    MB:0=8192;1=8192;2=8192;3=8192
     L3:0=ffff;1=ffff;2=ffff;3=ffff

In both examples the per-domain MB ceiling is higher than the global GMB ceiling. With
above showing defaults and you state "If the MBA ceiling exceeds the GLBE ceiling,
the effective MBA limit will be capped by the GLBE ceiling." - does this mean that
MB ceiling can never be higher than GMB ceiling as shown in the examples? 

Another question, when setting aside possible differences between MB and GMB.

I am trying to understand how user may expect to interact with these interfaces ...

Consider the starting state example as below where the MB and GMB ceilings are the
same:

  # cat schemata
  GMB:0=2048;1=2048;2=2048;3=2048
  MB:0=2048;1=2048;2=2048;3=2048

Would something like below be accurate? Specifically, showing how the GMB limit impacts the
MB limit:
  
  # echo "GMB:0=8;2=8" > schemata
  # cat schemata
  GMB:0=8;1=2048;2=8;3=2048
  MB:0=8;1=2048;2=8;3=2048

... and then when user space resets GMB the MB can reset like ...

  # echo "GMB:0=2048;2=2048" > schemata
  # cat schemata
  GMB:0=2048;1=2048;2=2048;3=2048
  MB:0=2048;1=2048;2=2048;3=2048

if I understand correctly this will only apply if the MB limit was never set so
another scenario may be to keep a previous MB setting after a GMB change:

  # cat schemata
  GMB:0=2048;1=2048;2=2048;3=2048
  MB:0=8;1=2048;2=8;3=2048

  # echo "GMB:0=8;2=8" > schemata
  # cat schemata
  GMB:0=8;1=2048;2=8;3=2048
  MB:0=8;1=2048;2=8;3=2048

  # echo "GMB:0=2048;2=2048" > schemata
  # cat schemata
  GMB:0=2048;1=2048;2=2048;3=2048
  MB:0=8;1=2048;2=8;3=2048

What would be most intuitive way for user to interact with the interfaces?

>>> From the description it sounds as though there is a new "memory bandwidth
>> ceiling/limit" that seems to imply that MBA allocations are limited by
>> GMBA allocations while the proposed user interface present them as independent.
>>
>> If there is indeed some dependency here ... while MBA and GMBA CLOSID are
>> enumerated separately, under which scenario will GMBA and MBA support different
>> CLOSID? As I mentioned in [1] from user space perspective "memory bandwidth"
> 
> I can see the following scenarios where MBA and GMBA can operate independently:
> 1. If the GMBA limit is set to ‘unlimited’, then MBA functions as an independent CLOS.
> 2. If the MBA limit is set to ‘unlimited’, then GMBA functions as an independent CLOS.
> I hope this clarifies your question.

No. When enumerating the features the number of CLOSID supported by each is
enumerated separately. That means GMBA and MBA may support different number of CLOSID.
My question is: "under which scenario will GMBA and MBA support different CLOSID?"

Because of a possible difference in number of CLOSIDs it seems the feature supports possible
scenarios where some resource groups can support global AND per-domain limits while other
resource groups can just support global or just support per-domain limits. Is this correct?

 
>> can be seen as a single "resource" that can be allocated differently based on
>> the various schemata associated with that resource. This currently has a
>> dependency on the various schemata supporting the same number of CLOSID which
>> may be something that we can reconsider?
> 
> After reviewing the new proposal again, I’m still unsure how all the pieces will fit together. MBA and GMBA share the same scope and have inter-dependencies. Without the full implementation details, it’s difficult for me to provide meaningful feedback on new approach.

The new approach is not final so please provide feedback to help improve it so
that the features you are enabling can be supported well.

Reinette

[1] https://lore.kernel.org/lkml/d58f70592a4ce89e744e7378e49d5a36be3fd05e.1769029977.git.babu.moger@amd.com/
[2] https://lore.kernel.org/lkml/e0c79c53-489d-47bf-89b9-f1bb709316c6@amd.com/