Re: [PATCH 1/2] x86/resctrl, Documentation: Keep mbm_assign_mode "default" on boot

["Moger, Babu" <bmoger@amd.com>]

Hi Reinette,

I still have to get back to few questions(test details etc..) on this 
thread. Let me get back to on that later tomorrow.
thanks
Babu

On 7/27/2026 5:44 PM, Moger, Babu wrote:
> Hi Reinette,
> 
> On 7/27/2026 4:42 PM, Reinette Chatre wrote:
>> Hi Babu,
>>
>> On 7/27/26 1:17 PM, Babu Moger wrote:
>>> Hi Reinette,
>>>
>>> On 7/27/26 13:12, Reinette Chatre wrote:
>>>> Hi Babu,
>>>>
>>>> On 7/27/26 10:24 AM, Babu Moger wrote:
>>>>> On 7/27/26 10:25, Reinette Chatre wrote:
>>>>>> On 7/27/26 7:05 AM, Borislav Petkov wrote:
>>>>>>> On Fri, Jul 24, 2026 at 04:53:26PM -0700, Reinette Chatre wrote:
>>>>>>>> One clarification here is that as I understand there has not yet 
>>>>>>>> been
>>>>>>>> an actual user complaint. At least not that the pqos utility 
>>>>>>>> mentioned in
>>>>>>>> this thread is aware of. Instead I struggle with the speculation 
>>>>>>>> about possible
>>>>>>>> user complaints with different interpretations on how this 
>>>>>>>> change could be
>>>>>>>> perceived by users.
>>>>>>>
>>>>>>> Sure, but don't you think that it is enough that we know about it?
>>>>>>
>>>>>> Only if we are sure that we have the complete picture. I do not 
>>>>>> believe we do, yet.
>>>>>
>>>>> We already have an issue open for this, and it is fairly
>>>>> straightforward to reproduce. Please let me know if there is
>>>>> anything specific you would like me to try.
>>>>
>>>> Just to be clear, when you refer to issue it means that pqos is 
>>>> returning zero for
>>>> unassigned counters?
>>>
>>> Yes.
>>>
>>>>
>>>> Is this issue public? I am not seeing any related issues in
>>>> https://github.com/intel/intel-cmt-cat/issues
>>>
>>> Opened one:
>>> [1] https://github.com/intel/intel-cmt-cat/issues/311
>>>
>>
>> Oh, now we have a mess, no? You start by stating that you already have 
>> an issue for this and when I
> 
> What mess?
> 
> I mentioned that I have an open issue that was raised against me 
> internally.
> 
> I understood that you wanted me to create a public bug to address the 
> issue, so I went ahead and created one.
> 
> 
>> ask you about it you quickly go create one with the pqos folks for 
>> this behavior we are
>> currently discussing to change! Why not create an issue for 
>> "Unavailable" being treated as zero that
>> I believe all agree is not correct.
>>
>> What are you really expecting from pqos folks based on your bug 
>> report? Or ... are you now just
>> using that bug report as leverage and planning to close it when this 
>> patch of yours is merged?
> 
> I’m not sure what the final fix is at this point. We’ll get to that once 
> we have a confirmed solution.
> 
> 
>>
>> To be honest as AMD representative and the one that enabled ABMC in 
>> resctrl I would expect that
>> a bug report from you would contain guidance how pqos could best 
>> support assignable counters.
> 
> Again, I don’t know what the final resolution will be at this point. 
> Once we agree on the fix, I’ll update the bug with the relevant details.
> 
>>
>> Please take a moment and consider your report from their perspective. 
>> I think it may be
>> reasonable to view the current pqos behavior as correct since that is 
>> what the kernel exposes.
>> If you find current behavior incorrect then please provide some 
>> guidance to pqos folks. For example,
>> do you expect pqos to always switch to default mode when it encounters 
>> mbm_event mode .... or do
>> you expect the pqos should do counter assignment, if so, how should it 
>> go about doing so, or ...?
>>
>> What are you expecting from the pqos folks in response to your bug 
>> report?
> 
> We will discuss that once come to final resolution. They can always say 
> "will not fix" right?
> 
>>
>>>>>> I connected with the friendly pqos folks on this issue. They 
>>>>>> subsequently did an audit of
>>>>>> the tool to understand how it may behave under the different 
>>>>>> scenarios. They found that
>>>>>> the tool currently parses text return values ("Unavailable", 
>>>>>> "Unassigned") as zero.
>>>>>> This could cause incorrect behavior. (more below)
>>>>>>
>>>>>>>
>>>>>>> I mean, the pqos tool shows 0.0 in the MBL/MBR columns now.
>>>>>>>
>>>>>>> And is the
>>>>>>>
>>>>>>>      pqos -m all:[0-191]
>>>>>>>
>>>>>>> invocation not something people would usually run?
>>>>>>
>>>>>> ABMC enabled (mode claimed to be incompatible with pqos)
>>>>>> --------------------------------------------------------
>>>>>> This example is from Babu's email that highlights that when ABMC 
>>>>>> is enabled this returns
>>>>>> zero as bandwidth from pqos perspective. This matches the pqos 
>>>>>> audit that found with text return
>>>>>> states, like the "Unassigned" happening underneath the output 
>>>>>> above, the return value is zero. When
>>>>>> the counters are not assigned then the events will always return 
>>>>>> "Unassigned" (until reassigned)
>>>>>> so having it return zero all the time may not cause issues. Once 
>>>>>> reassigned the event value
>>>>>> will always return a valid value (no text return values).
>>>>>>
>>>>>> ABMC disabled (mode claimed to be compatible with pqos)
>>>>>> -------------------------------------------------------
>>>>>> Without ABMC there are scenarios when events may return 
>>>>>> "Unavailable". These scenarios vary
>>>>>> based on how many monitor groups are created and the workloads 
>>>>>> run. I am not able to test this
>>>>>> but when I attempt to combine what I know about AMD bandwidth 
>>>>>> monitoring with the results from
>>>>>> the pqos audit I am concerned that there may be a problem.
>>>>>>
>>>>>> Consider a scenario where an event may return "Unavailable", for 
>>>>>> example:
>>>>>>
>>>>>> <return A>, <return "Unavailable">, <return B (larger than A)>, 
>>>>>> <return "Unavailable">, ...
>>>>>>
>>>>>> pqos will see:
>>>>>> A, 0, B, 0, ...
>>>>>>
>>>>>> The "pqos -m" usage attempts to determine the bandwidth rate and, 
>>>>>> for example, when going from
>>>>>> "A" to "0" it will be considered counter wraparound that would 
>>>>>> appear as a very large and
>>>>>> wrong bandwidth number.
>>>>>
>>>>> Yes. Agree.  This is known issue.
>>>>
>>>> Has this been reported? I do not recognize it in https://github.com/ 
>>>> intel/intel-cmt-cat/issues
>>>> and the pqos folks I spoke with was not aware of this behavior.
>>>
>>> I was not able to recreate this issue. I just opened for the one I 
>>> have seen [1].
>>
>> So to summarize: it is known that pqos returns zero instead of 
>> Unavailable but it is not reproducible?
>>>>>> Question to Babu
>>>>>> ----------------
>>>>>> Do you perhaps have a test/workloads that, under "default" counter 
>>>>>> assignment mode, can create
>>>>>> many monitor groups (more than 64) and cause "Unavailable" to be 
>>>>>> returned frequently? Would it
>>>>>> be possible to put pqos through its paces in this environment?
>>>>>
>>>>> Yes, I tried to reproduce the "Unavailable" issue with the pqos tool.
>>>>>
>>>>> So far, I have not been able to observe the issue. I created 64
>>>>> monitoring groups, ran MLC, and simultaneously executed pqos.
>>>>
>>>> If I understand correctly 64 monitor groups will still have enough 
>>>> underlying hardware counters
>>>> and always return event counts, never return "Unavailable". It is 
>>>> when the number of
>>>> monitor groups are 65 or more that the hardware counters will start 
>>>> to be re-assigned, no?
>>>
>>> Yes. I created 64 monitoring groups and then ran pqos, which creates 
>>> an additional 15 monitoring groups.
>>>
>>>
>>>>
>>>> Even so, I do not know if 65 would easily trigger the issue ... 
>>>> sounds like it is possible to create
>>>> many more (4096) monitor groups on these systems so there appears to 
>>>> be some room to make it easier to
>>>> create scenario where "Unavailable" is returned.
>>>>
>>>> On a higher level: could you please run the test that can expose 
>>>> pqos to environment where it will
>>>> encounter "Unavailable" and observe how that is handled?
>>>
>>> I was able to reproduce the condition where the hardware reports 
>>> "Unavailable" for some of the MBM counters.
>>>
>>> #cat /sys/fs/resctrl/mon_groups/dir*/mon_data/mon_L3_00/mbm_*
>>> Unavailable
>>> Unavailable
>>> Unavailable
>>> Unavailable
>>> Unavailable
>>> Unavailable
>>
>> Thank you for running this.
>>
>> Do all of these monitor groups have workloads assigned that generate 
>> memory bandwidth so
>> that they do not just keep returning "Unavailable" but often return 
>> data also? I just want
>> to confirm that this is not just a variant of the "Unassigned" test case.
>>
>>
>>> However, the pqos tool continues to operate normally and reports 
>>> valid bandwidth values:
>>> #sudo pqos -m all:[0-191]
>>>
>>>      CORE         IPC      MISSES     LLC[KB]   MBL[MB/s]   MBR[MB/s]
>>>     0-191        2.55      55146k     74496.0        20.0        61.4
>>>
>>>>
>>>>>
>>>>> The issue has not reproduced in any of my tests so far.
>>>>
>>>> I'm confused. It reads like the issue that you mention earlier is a 
>>>> known issue cannot be
>>>> reproduced.
>>>
>>> I meant that the pqos tool continued to work as expected even after
>>> I recreated the condition that caused the hardware to report
>>> "Unavailable".
>>>
>>
>> ack.
>>
>>>
>>>>
>>>>> This might require specific test scenario to recreate.
>>>>
>>>> Right.
>>>>
>>>>>> Of course, "AMBC disabled" scenario also includes AMD systems 
>>>>>> before ABMC was introduced. I do
>>>>>> not know why the issue with "Unavailable" handling (text returns 
>>>>>> in general) has not been reported
>>>>>> until now. I assume the pqos team may do most testing on Intel 
>>>>>> systems that do not return text.
>>>>>> ...
>>>>
>>>>>>>> The patch notes mention that this change is planned to be 
>>>>>>>> reverted at some
>>>>>>>> point in the future and I have not heard of any changes to this 
>>>>>>>> plan.
>>>>>>>
>>>>>>> Where does it say that? I don't see that aspect.
>>>>>>
>>>>>> For convenience, copy of patch notes from original commit [1] is 
>>>>>> below:
>>>>>>
>>>>>>       There are plans to enable "mbm_event" by default once 
>>>>>> additional counters
>>>>>>       are available. For now, keep the default mode to maintain 
>>>>>> compatibility
>>>>>>       with existing tools.
>>>>>
>>>>>
>>>>> I added the text below in the comments section (after ---).
>>>>>
>>>>> On second thought, that was probably my mistake, and I don't think I
>>>>> should have added it. I also don't see this happening anytime soon.
>>>>
>>>> ok.
>>>>
>>>>>
>>>>> Considering all of this, I believe it would make sense to make the
>>>>> "default" mode the preferred mode.
>>>> The original problem statement is that pqos is not able to handle 
>>>> "Unassigned" return values
>>>> that will be encountered when "assignable counter mode" is enabled.
>>>>
>>>> The request is to disable "assignable counter mode" by default to 
>>>> address the problem
>>>> when handling "Unassigned" counters.
>>>>
>>>> Disabling "assignable counter mode" will under certain circumstances 
>>>> cause "Unavailable" to
>>>> be returned on event read.
>>>>
>>>> At this time there seems to be a mismatch in understanding how pqos 
>>>> can handle
>>>> "Unavailable" return values. For me to sign off on this patch I 
>>>> would like to fully understand
>>>> the risk doing so. Could you please confirm from your side that pqos 
>>>> can handle all scenarios of
>>>> the mode that you request to be the default mode?
>>>>
>>>
>>> I created the maximum number of monitoring groups supported by the
>>> system and was able to reproduce the condition where the hardware
>>> reports "Unavailable". Even under those conditions, the pqos tool
>>> appeared to function correctly and continued to report valid results.
>>
>> Could you please share the test details? Just having the maximum 
>> number of
>> monitoring groups would not trigger the issue by itself, it needs 
>> workloads
>> that generate memory traffic that intermittently do return data.
>>
>>>
>>> It's worth keeping in mind that ABMC mode is not available on the
>>> majority of hardware currently accessible to public.
>>>
>>> Furthermore, default mode has been the longstanding behavior in
>>> deployed systems. Moving back to the default mode is therefore not a
>>> new change, but a return to the behavior that users are already
>>> familiar with.
>> ack. "We returned wrong data before. We can do better now but we want to
>> keep returning wrong data since that is what users are used to".
>>
> 
> I think we are getting nowhere here.
> 
> Thanks
> Babu
> 
>