Re: [PATCH 1/2] x86/resctrl, Documentation: Keep mbm_assign_mode "default" on boot

[Reinette Chatre <reinette.chatre@intel.com>]

Hi Babu,

On 7/24/26 1:57 PM, Moger, Babu wrote:
> On 7/23/2026 4:46 PM, Reinette Chatre wrote:
>> On 7/23/26 11:03 AM, Babu Moger wrote:
>>> On 7/21/26 15:54, Reinette Chatre wrote:
>>>> On 7/20/26 12:15 PM, Babu Moger wrote:
>>>>> On 7/20/26 13:27, Reinette Chatre wrote:
>>>>>> On 7/20/26 10:00 AM, Babu Moger wrote:
>>>>>>> On 7/17/26 17:56, Reinette Chatre wrote:
>>>>>>>> On 7/17/26 2:13 PM, Babu Moger wrote:
>>>>>>>>> The kernel currently enables the ABMC-based "mbm_event" mode by default on
>>>>>>>>> hardware that supports it. However, this can cause bandwidth monitoring
>>>>>>>>> failures with existing userspace tools such as pqos.
>>>>>>>>>
>>>>>>>>> The pqos tool mounts the resctrl filesystem and creates 16 or more resctrl
>>>>>>>>> groups by default. On systems with 32 or fewer ABMC counters, this default
>>>>>>>>> configuration can consume all available counters, since each group requires
>>>>>>>>> one counter for local MBM and another for total MBM. If additional
>>>>>>>>> monitoring groups are created, counter resources are exhausted and pqos
>>>>>>>>> tool reports memory bandwidth counters as zero for those groups.
>>>>>>>>
>>>>>>>> It is not obvious to me that this is a problem. If I understand correctly
>>>>>>>> there are two scenarios possible with this pqos behavior:
>>>>>>>>
>>>>>>>> - ABMC is not in use ("mbm_assign_mode" is set to "default")
>>>>>>>>       - pqos can create 16 or more monitor groups
>>>>>>>>       - hardware still supports a limited number of counters with consequence that
>>>>>>>>         underlying counters reset at any time as the different monitoring groups
>>>>>>>>         need to be tracked.
>>>>>>>>       - pqos can read monitoring data of all 16 monitor groups, sometimes reading the
>>>>>>>>         events would return "Unavailable", sometimes reading the events return data.
>>>>>>>>       - *None* of the monitoring numbers returned are guaranteed to be accurate.
>>>>>>>>
>>>>>>>> - ABMC is in use ("mbm_assign_mode" is set to "mbm_event"):
>>>>>>>>       - pqos can create 16 or more monitor groups
>>>>>>>>       - only a subset of monitoring groups have counters assigned and these counters
>>>>>>>>         are guaranteed to only track the monitor groups/events they are assigned to
>>>>>>>>       - pqos can read monitoring data of all 16 monitor groups with two possibilities:
>>>>>>>>         - monitor group/event has counter assigned: monitoring numbers are guaranteed to be accurate
>>>>>>>>         - monitor group/event does not have counter assigned: monitoring numbers return 0
>>>>>>>>
>>>>>>>> If my understanding is correct then the preference is to rather have wrong data than
>>>>>>>> see 0? This does not sound right. What am I missing?
>>>>>>>
>>>>>>>
>>>>>>> This hardware can monitor up to 64 RMIDs without any counter resets.
>>>>>>
>>>>>> How many RMIDs does the hardware claim to support via CPUID that ends up being shown to user
>>>>>> space via "num_rmids"?
>>>>>
>>>>> #cat /sys/fs/resctrl/info/L3_MON/num_rmids
>>>>> 4096
>>>>>
>>>>>>
>>>>>> I understood from original ABMC enabling that the underlying hardware counters of "default"
>>>>>> and "mbm_event" mode on AMD are the same. That is, in "default" mode the hardware does a
>>>>>> "best effort" assignment of hardware counters to events while "mbm_event" mode lets the user
>>>>>> control the assignment. It instead sounds like this is not the case and there are actually
>>>>>> two distinct underlying hardware counter mechanisms?
>>>>>
>>>>> That is correct. They are two different counters.
>>>>>
>>>>>>
>>>>>>
>>>>>>> As you know, the pqos tool creates COS1 through COS15 regardless of
>>>>>>> the command-line options used, resulting in a total of 16 groups
>>>>>>> including the default group. With ABMC enabled, this consumes all
>>>>>>> available ABMC counters(32 counters, 2 counters for each group).
>>>>>>
>>>>>>>
>>>>>>> When pqos is invoked with the -m option, it creates additional monitoring groups. For example:
>>>>>>>
>>>>>>> pqos -m all:0     -> creates 1 monitoring group
>>>>>>> pqos -m all:0,1   -> creates 2 monitoring groups
>>>>>>>
>>>>>>>
>>>>>>> Since all ABMC counters have already been allocated to the default
>>>>>>> set of groups, no counters remain for these additional monitoring
>>>>>>> groups. As a result, the monitoring commands report zero values,
>>>>>>> effectively making monitoring unusable.
>>>>>>>
>>>>>>> In contrast, the default monitoring mode can still support up to 48
>>>>>>> additional monitoring groups (64 total RMIDs minus the 16 default
>>>>>>> groups created by pqos).
>>>>>>>
>>>>>>> For this reason, I still believe keeping the default monitoring mode as the default is the better option.
>>>>>>
>>>>>> It is not clear to me where the "64" number comes from. Even if resctrl sets the "default"
>>>>>
>>>>> The count of 64 is known from internal information. It can also be determined by allocating monitoring counters in a loop until the hardware starts returning an "unavailable" response, which occurs after all 64 counters have been assigned. This information is not documented.
>>>>>
>>>>>> mode as default, what will happen to the scenario you describe and 49, instead of 48,
>>>>>> additional groups are created? From what I understand the moment the 65th group is created user will
>>>>>> transition from "accurate per monitor group monitoring data for all 64 monitoring groups" to "inaccurate
>>>>>> per monitor group monitoring data for all 65 monitoring groups" with no indication that this is happening?
>>>>>
>>>>> Yes. That is correct.
>>>>>
>>>>>>
>>>>>> I believe AMD supports more than 64 RMIDs and in this case there seems to be three ranges:
>>>>>> [supported by ABMC, depends on events but lets say RMID 0 to 16] < [RMID 17 to 64] < [RMID 65 to total number of RMIDs supported]
>>>>>>
>>>>>> Current default "mbm_event" mode uses ABMC so as you state this always results in:
>>>>>> - accurate counts for 16 monitor groups
>>>>>> - zero for all other monitor groups up to total number of RMIDs supported
>>>>>
>>>>> That is correct. With ABMC, there are 32 available counters, allowing up to 32 monitoring events to be tracked simultaneously. Since each monitoring group requires two counters—one for local MBM and one for total MBM—the system can support only 16 monitoring groups at a time.
>>>>>
>>>>>>
>>>>>> As I see it switching to the "default" mode would result in:
>>>>>> Scenario 1, 64 or fewer monitor groups are created:
>>>>>> - accurate counts for all monitor groups
>>>>>> Scenario 2, 65 or more monitor groups are created:
>>>>>> - inaccurate counts for all monitor groups
>>>>>
>>>>> That is true.
>>>>>
>>>>>>
>>>>>> I do not believe there is any way for user space to know when or if system switches from "scenario 1" to
>>>>>> "scenario 2" on these systems and this unpredictable behavior that results in wrong data does not
>>>>>> sound ideal to me.
>>>>>    I agree, it's not an ideal situation, but that's how it has always worked. At the moment, I'm not sure about a better alternative.
>>>> I see the current behavior (return accurate data when available and zero when no
>>>> accurate data available) as the "better alternative". Intentionally returning wrong
>>>> data when it can be avoided does not sound right to me.
>>>>
>>>> The changelog states:
>>>>      The kernel currently enables the ABMC-based "mbm_event" mode by default on
>>>>      hardware that supports it. However, this can cause bandwidth monitoring
>>>>      failures with existing userspace tools such as pqos.
>>>>
>>>> Could another view be that mbm_event mode is the *only* reliable accurate bandwidth
>>>> monitoring and any other mode (AMD hardware not supporting assignable counters or
>>>> AMD hardware with assignable counters choosing to use "default" mode) can cause
>>>> bandwidth monitoring failures (with existing userspace tools such as pqos)?
>>>
>>> Yes, that is correct. However, the bandwidth counters become
>>> inaccurate only when users start monitoring more than a certain
>>> number of groups (64 in this case). In typical deployments, users do
>>> not create more than 64 groups, so the counters remain accurate for
>>> the vast majority of use cases. We do not want to introduce
>>
>> oh ... hmmm ... I have a different view of deployments since the message I
>> keep hearing is "we need more RMIDs".
>>
>>
>>> unnecessary complexity or inconvenience for those users.
>>
>> This sounds like a request to (temporarily) change resctrl user interface to
>> accommodate users that create between 16 and 64 monitor groups (where 64 is
>> a magic number not exposed to users) at the expense of users that create between
>> 65 (magic number + 1) and 4096 monitor groups?
>>
>>
>>> Additionally, changing the current behavior could disrupt existing
>>> tools and workflows, which would likely result in a significant
>>> increase in support requests.
>>
>> resctrl should aim to maintain a consistent user interface and changes to
>> that interface should be done with a lot of care and with very good motivation.
>> Above motivation sounds vague to me. You are concerned about support
>> requests from users that create between 16 and 64 monitor groups but not
>> concerned about support requests from users that create between 65 and
>> 4096 monitor groups?
>>
>> One could instead argue that this patch would cause more support requests
>> since users will start seeing unavailable and wrong data the moment they
>> create more than (the magic) 64 monitor groups.
>>  
>>> At the same time, we are not taking away the option for users who
>>> require strictly accurate measurements while monitoring a large
>>> number of groups. That capability remains available. We also plan to
>>> add more counters and enhancements over time, and it is possible
>>> that the mbm_event option could become the default in the future.
>>
>> Similarly, the default capability remains for users that prefer to
>> create between 16 and 64 monitor groups and do not want to assign
>> counters.
>>
>> Without exposing the magic 64 resctrl needs to provide a good default
>> and I continue to find the accurate data the better option.
>>
>>>
>>> For now, however, our goal is for the default mode to remain the
>>> boot-time default. This avoids unexpected behavior changes and
>>> prevents users from being unnecessarily alarmed by issues that do
>>> not affect their typical usage scenarios.
>>
>> At the same time this commit states that, essentially, this commit is
>> planned to be reverted in the future. Users cannot be expected to upgrade
>> their hardware when upgrading the kernel so this behavior is already planned
>> to be reverted on the hardware that it claims to support. Why not just
>> keep existing behavior?
>>
> 
> I wish I could agree with you on this, but I'm receiving significant pushback internally.
> 
> The primary argument is that, according to longstanding Linux
> principles, we should never break existing userspace tools.

Absolutely. Does the existing behavior break existing userspace tools though?
I find it hard to believe that tools would prefer to see inaccurate data over
accurate data just because inaccurate data is all they have seen until now.

> 
> Before ABMC was introduced:
> 
> # sudo pqos -m all:[0-191]
> 
> CORE         IPC      MISSES     LLC[KB]   MBL[MB/s]   MBR[MB/s]
> 0-191        0.01      108184k    385184.0    34100.8       22.1
> 
> 
> After ABMC was introduced and made the default mode:
> 
> # sudo pqos -m all:[0-191]
> 
> CORE         IPC      MISSES     LLC[KB]   MBL[MB/s]   MBR[MB/s]
> 0-191        0.01      109937k    385600.0        0.0        0.0
> 
> As you can see, both MBL (Memory Bandwidth Local) and MBR (Memory
> Bandwidth Remote) are reported as zero when ABMC is enabled by
> default.
> 
> From a user perspective, this represents a regression in
> functionality. A tool that previously provided valid memory
> bandwidth monitoring data now reports zeros unless the user manually

I question the statement "previously provided valid memory bandwidth monitoring
data" and this is where I am blocked.

From what I understand the previously provided memory bandwidth monitoring data
was not actually reliably and consistently valid since counters could be yanked from,
reset, and re-assigned to, workloads at any time while the workload is running and using
memory bandwidth.

I understand that if users only use a fraction of the supported monitoring groups
then the counters will indeed be valid but (a) this is a small number compared to
the supported monitoring groups, and (b) hardware does not expose this number.

> changes the monitoring mode. That effectively breaks existing
> workflows and monitoring applications, which is difficult to justify
> from a Linux userspace compatibility standpoint.

On the other hand I interpret this request as "We returned wrong data before.
We can do better now but we want to keep returning wrong data since that is
what users are used to".

I continue to feel that this is not the right thing to do.

> 
> While ABMC offers advantages and is the direction we want to move
> toward, it is still relatively new and the ecosystem has not fully
> adapted to it. Making it the default before key userspace tools
> support it creates compatibility concerns.

Indeed. Switching from the original unreliable bandwidth monitoring 
to accurate monitoring will require adoption from ecosystem since
assignable counters is a new interface to get used to. This seems to
be the only way in which users can obtain accurate memory bandwidth
data since everything needed to get reliable data from the hardware
cannot be discovered from the hardware (referring to the magic 64 number
here).

resctrl aims to support the adoption by doing automatic assignment.

 
> I'd appreciate your thoughts on how best to address this situation.

We appear to be at an impasse. One way to move forward could be to
request arbitration from the x86 maintainers. I will abide by their
guidance and look forward to learning from them how to navigate
issues impacting user interfaces.

Reinette