Re: [PATCH 1/2] x86/resctrl, Documentation: Keep mbm_assign_mode "default" on boot

[Reinette Chatre <reinette.chatre@intel.com>]

Hi Babu,

On 7/27/26 1:17 PM, Babu Moger wrote:
> Hi Reinette,
> 
> On 7/27/26 13:12, Reinette Chatre wrote:
>> Hi Babu,
>>
>> On 7/27/26 10:24 AM, Babu Moger wrote:
>>> On 7/27/26 10:25, Reinette Chatre wrote:
>>>> On 7/27/26 7:05 AM, Borislav Petkov wrote:
>>>>> On Fri, Jul 24, 2026 at 04:53:26PM -0700, Reinette Chatre wrote:
>>>>>> One clarification here is that as I understand there has not yet been
>>>>>> an actual user complaint. At least not that the pqos utility mentioned in
>>>>>> this thread is aware of. Instead I struggle with the speculation about possible
>>>>>> user complaints with different interpretations on how this change could be
>>>>>> perceived by users.
>>>>>
>>>>> Sure, but don't you think that it is enough that we know about it?
>>>>
>>>> Only if we are sure that we have the complete picture. I do not believe we do, yet.
>>>
>>> We already have an issue open for this, and it is fairly
>>> straightforward to reproduce. Please let me know if there is
>>> anything specific you would like me to try.
>>
>> Just to be clear, when you refer to issue it means that pqos is returning zero for
>> unassigned counters?
> 
> Yes.
> 
>>
>> Is this issue public? I am not seeing any related issues in
>> https://github.com/intel/intel-cmt-cat/issues
> 
> Opened one:
> [1] https://github.com/intel/intel-cmt-cat/issues/311
> 

Oh, now we have a mess, no? You start by stating that you already have an issue for this and when I
ask you about it you quickly go create one with the pqos folks for this behavior we are
currently discussing to change! Why not create an issue for "Unavailable" being treated as zero that
I believe all agree is not correct.

What are you really expecting from pqos folks based on your bug report? Or ... are you now just
using that bug report as leverage and planning to close it when this patch of yours is merged?

To be honest as AMD representative and the one that enabled ABMC in resctrl I would expect that
a bug report from you would contain guidance how pqos could best support assignable counters.

Please take a moment and consider your report from their perspective. I think it may be
reasonable to view the current pqos behavior as correct since that is what the kernel exposes. 
If you find current behavior incorrect then please provide some guidance to pqos folks. For example,
do you expect pqos to always switch to default mode when it encounters mbm_event mode .... or do
you expect the pqos should do counter assignment, if so, how should it go about doing so, or ...?

What are you expecting from the pqos folks in response to your bug report?

>>>> I connected with the friendly pqos folks on this issue. They subsequently did an audit of
>>>> the tool to understand how it may behave under the different scenarios. They found that
>>>> the tool currently parses text return values ("Unavailable", "Unassigned") as zero.
>>>> This could cause incorrect behavior. (more below)
>>>>
>>>>>
>>>>> I mean, the pqos tool shows 0.0 in the MBL/MBR columns now.
>>>>>
>>>>> And is the
>>>>>
>>>>>     pqos -m all:[0-191]
>>>>>
>>>>> invocation not something people would usually run?
>>>>
>>>> ABMC enabled (mode claimed to be incompatible with pqos)
>>>> --------------------------------------------------------
>>>> This example is from Babu's email that highlights that when ABMC is enabled this returns
>>>> zero as bandwidth from pqos perspective. This matches the pqos audit that found with text return
>>>> states, like the "Unassigned" happening underneath the output above, the return value is zero. When
>>>> the counters are not assigned then the events will always return "Unassigned" (until reassigned)
>>>> so having it return zero all the time may not cause issues. Once reassigned the event value
>>>> will always return a valid value (no text return values).
>>>>
>>>> ABMC disabled (mode claimed to be compatible with pqos)
>>>> -------------------------------------------------------
>>>> Without ABMC there are scenarios when events may return "Unavailable". These scenarios vary
>>>> based on how many monitor groups are created and the workloads run. I am not able to test this
>>>> but when I attempt to combine what I know about AMD bandwidth monitoring with the results from
>>>> the pqos audit I am concerned that there may be a problem.
>>>>
>>>> Consider a scenario where an event may return "Unavailable", for example:
>>>>
>>>> <return A>, <return "Unavailable">, <return B (larger than A)>, <return "Unavailable">, ...
>>>>
>>>> pqos will see:
>>>> A, 0, B, 0, ...
>>>>
>>>> The "pqos -m" usage attempts to determine the bandwidth rate and, for example, when going from
>>>> "A" to "0" it will be considered counter wraparound that would appear as a very large and
>>>> wrong bandwidth number.
>>>
>>> Yes. Agree.  This is known issue.
>>
>> Has this been reported? I do not recognize it in https://github.com/intel/intel-cmt-cat/issues
>> and the pqos folks I spoke with was not aware of this behavior.
> 
> I was not able to recreate this issue. I just opened for the one I have seen [1].

So to summarize: it is known that pqos returns zero instead of Unavailable but it is not reproducible? 
>>>> Question to Babu
>>>> ----------------
>>>> Do you perhaps have a test/workloads that, under "default" counter assignment mode, can create
>>>> many monitor groups (more than 64) and cause "Unavailable" to be returned frequently? Would it
>>>> be possible to put pqos through its paces in this environment?
>>>
>>> Yes, I tried to reproduce the "Unavailable" issue with the pqos tool.
>>>
>>> So far, I have not been able to observe the issue. I created 64
>>> monitoring groups, ran MLC, and simultaneously executed pqos.
>>
>> If I understand correctly 64 monitor groups will still have enough underlying hardware counters
>> and always return event counts, never return "Unavailable". It is when the number of
>> monitor groups are 65 or more that the hardware counters will start to be re-assigned, no?
> 
> Yes. I created 64 monitoring groups and then ran pqos, which creates an additional 15 monitoring groups.
> 
> 
>>
>> Even so, I do not know if 65 would easily trigger the issue ... sounds like it is possible to create
>> many more (4096) monitor groups on these systems so there appears to be some room to make it easier to
>> create scenario where "Unavailable" is returned.
>>
>> On a higher level: could you please run the test that can expose pqos to environment where it will
>> encounter "Unavailable" and observe how that is handled?
> 
> I was able to reproduce the condition where the hardware reports "Unavailable" for some of the MBM counters.
> 
> #cat /sys/fs/resctrl/mon_groups/dir*/mon_data/mon_L3_00/mbm_*
> Unavailable
> Unavailable
> Unavailable
> Unavailable
> Unavailable
> Unavailable

Thank you for running this.

Do all of these monitor groups have workloads assigned that generate memory bandwidth so
that they do not just keep returning "Unavailable" but often return data also? I just want
to confirm that this is not just a variant of the "Unassigned" test case.


> However, the pqos tool continues to operate normally and reports valid bandwidth values:
> #sudo pqos -m all:[0-191]
> 
>     CORE         IPC      MISSES     LLC[KB]   MBL[MB/s]   MBR[MB/s]
>    0-191        2.55      55146k     74496.0        20.0        61.4
> 
>>
>>>
>>> The issue has not reproduced in any of my tests so far.
>>
>> I'm confused. It reads like the issue that you mention earlier is a known issue cannot be
>> reproduced.
> 
> I meant that the pqos tool continued to work as expected even after
> I recreated the condition that caused the hardware to report
> "Unavailable".
> 

ack.

> 
>>
>>> This might require specific test scenario to recreate.
>>
>> Right.
>>
>>>> Of course, "AMBC disabled" scenario also includes AMD systems before ABMC was introduced. I do
>>>> not know why the issue with "Unavailable" handling (text returns in general) has not been reported
>>>> until now. I assume the pqos team may do most testing on Intel systems that do not return text.
>>>> ...
>>
>>>>>> The patch notes mention that this change is planned to be reverted at some
>>>>>> point in the future and I have not heard of any changes to this plan.
>>>>>
>>>>> Where does it say that? I don't see that aspect.
>>>>
>>>> For convenience, copy of patch notes from original commit [1] is below:
>>>>
>>>>      There are plans to enable "mbm_event" by default once additional counters
>>>>      are available. For now, keep the default mode to maintain compatibility
>>>>      with existing tools.
>>>
>>>
>>> I added the text below in the comments section (after ---).
>>>
>>> On second thought, that was probably my mistake, and I don't think I
>>> should have added it. I also don't see this happening anytime soon.
>>
>> ok.
>>
>>>
>>> Considering all of this, I believe it would make sense to make the
>>> "default" mode the preferred mode.
>> The original problem statement is that pqos is not able to handle "Unassigned" return values
>> that will be encountered when "assignable counter mode" is enabled.
>>
>> The request is to disable "assignable counter mode" by default to address the problem
>> when handling "Unassigned" counters.
>>
>> Disabling "assignable counter mode" will under certain circumstances cause "Unavailable" to
>> be returned on event read.
>>
>> At this time there seems to be a mismatch in understanding how pqos can handle
>> "Unavailable" return values. For me to sign off on this patch I would like to fully understand
>> the risk doing so. Could you please confirm from your side that pqos can handle all scenarios of
>> the mode that you request to be the default mode?
>>
> 
> I created the maximum number of monitoring groups supported by the
> system and was able to reproduce the condition where the hardware
> reports "Unavailable". Even under those conditions, the pqos tool
> appeared to function correctly and continued to report valid results.

Could you please share the test details? Just having the maximum number of
monitoring groups would not trigger the issue by itself, it needs workloads
that generate memory traffic that intermittently do return data.

> 
> It's worth keeping in mind that ABMC mode is not available on the
> majority of hardware currently accessible to public.
> 
> Furthermore, default mode has been the longstanding behavior in
> deployed systems. Moving back to the default mode is therefore not a
> new change, but a return to the behavior that users are already
> familiar with.
 
ack. "We returned wrong data before. We can do better now but we want to
keep returning wrong data since that is what users are used to". 

Reinette