Re: [PATCH 1/2] x86/resctrl, Documentation: Keep mbm_assign_mode "default" on boot

[Babu Moger <babu.moger@amd.com>]

Hi Maintainers,

It's been a while since we last discussed this thread, and this item has 
been outstanding for some time. I'd like to revisit the conversation and 
understand the plan moving forward.

Please advice. More below.

On 8/5/26 17:40, Moger, Babu wrote:
> Hi Reinette/Boris,
> 
> On 8/4/2026 6:35 PM, Reinette Chatre wrote:
>> Hi Boris,
>>
>> On 8/4/26 3:47 PM, Borislav Petkov wrote:
>>> On Tue, Aug 04, 2026 at 03:10:18PM -0700, Reinette Chatre wrote:
>>>>> So, long story short - and I appreciate the explaining - we should 
>>>>> switch
>>>>> AMD's default behavior back to RMID + 64 counters - exactly like it 
>>>>> is on
>>>>> Intel - and the ABMC thing will be explicitly selectable by the user.
>>>>
>>>> nit: s/exactly like it is on Intel//
>>>
>>> What does that mean?
>>>
>>> What's the difference between Intel's RMID mode + 64 counters and AMD's?
>>
>> Expanding on what Tony answered ...
>>
>> The 64 magic number only applies to AMD.
>>
>> In default mode both Intel and AMD enumerates the number of RMIDs 
>> supported
>> by each resource and both Intel and AMD limits the number of monitor 
>> groups
>> a user can create to the minimum RMID ("minRMID" below) supported 
>> across all
>> resources.
>>
>> On Intel the monitoring behavior is consistent no matter how many 
>> monitor groups the
>> user creates. User can create up to minRMID monitor groups and reading 
>> monitoring
>> data from all the monitor groups always return accurate data (never 
>> "Unavailable").
>>
>> On AMD the monitoring behavior (without assignable counters, aka 
>> "default mode") changes
>> depending on how many monitor groups the user creates:
>> User creates 0 to 64 monitor groups:
>>     reading monitoring data from any of these monitor groups will 
>> always return data
>>     and it will be accurate
>>
>> User creates from 65 to minRMID monitor groups:
>>     reading monitoring data from *any* monitor group may return 
>> "Unavailable" and
>>     vary in accuracy
>>
>>
>>>> ok.
>>>>
>>>>> This way there are no surprises when running any tools on either 
>>>>> vendor and if
>>>>> one wants something special, one selects it.
>>>> This patch, once minimized for easier backporting and marked for 
>>>> stable, would
>>>> accomplish this.
>>>>
>>>> Two nitpicks:
>>>> * "no surprises" should be "no surprises (as long as AMD hardware 
>>>> does not return
>>>>    "Unavailable")".
>>>
>>> AFAIK, Babu was unable to reproduce that.
>>
>> If this cannot be reproduced on AMD hardware then AMD did not need to 
>> create ABMC, no?
> 
> I haven't been able to reproduce the issue in my test environment yet, 
> even though this has been the case since the very beginning. I'm working 
> on reproducing it though. It requires a very specific scenario where 
> more than 64 monitoring groups are active within the same L3 domain.
> 
> Prior to ABMC, there was an attempt to address the problem using a 
> "soft-RMID" approach. Saving and restoring the counters in software.
> However, that solution was eventually abandoned because it introduced 
> unacceptable overhead in the context-switch path.
> 
> This is where ABMC comes in - because reading counters can be very 
> expensive on every context switch, with ABMC you get the ability to pin 
> certain RMIDs for longer without the hardware invalidating them as long 
> as it is pinned.
> 
> You can imagine that there are hardware limitations which cannot allow 
> you to pin 2 counters for *each* RMID. So you end up monitoring a subset 
> of groups.
> 
>>
>>>
>>>>     There is the known issue with the default mode on AMD where 
>>>> return of "Unavailable"
>>>>     is treated as wraparound by pqos.
>>>
>>> I guess Babu can address that.
>>>
>>>> * "one wants something special" should be "one wants accurate data".
>>>>     Caveat: User does not know how inaccurate data is in default 
>>>> mode. Users need to learn
>>>>     about existence of accurate data from outside resctrl via 
>>>> external sources, possibly
>>>>     leaving it up to the tools considered here.
>>>
>>> With my simple thinking, I would expect that accurate data means, the 
>>> number
>>> of counters being in use is not hitting the arch limit. The moment that
>>> happens, I guess one could deem that measurement innacurate.
>>
>> Agreed and matches above summary. The arch limit here would be the 64 
>> RMID that can
>> be guaranteed to be counted. resctrl does not limit the number of 
>> monitor groups to
>> this number though but instead uses the number enumerated from 
>> hardware (4096 on this hardware).
>> To guarantee accurate data in "default" mode a solution could be to 
>> add model specific
>> information that teaches resctrl about "64" and it can use that as 
>> RMID limit instead.
>>
>>>> What is the plan with https://github.com/intel/intel-cmt-cat/ 
>>>> issues/311 ?
>>>
>>> I guess that should be closed once we switch back the default.
>>
>> I expect so also. Even so, it does open a new question of if and how 
>> tools are
>> expected to interact with assignable counters. Instead of this bug I 
>> would propose
>> that AMD work with pqos folks on expectations from tools to support 
>> assignable
>> counters. To me this bug implies that AMD considers enabling 
>> assignable mode on
>> a system as a bug.
>>
> 
> We have discussed adding ABMC support to the pqos tool, but it has not 
> been a priority so far. Given the current discussion, we will need to 
> revisit it.
> 
> We would prefer to keep the current mode as the "default" for the 
> following reasons:
> 
> 1. The "Unavailable" issue is not new and has existed for a long time.
> Most users are unlikely to encounter it.
> 
> 2. Users who do encounter the issue can use ABMC with the intended usage 
> model described in [A], where a subset of groups is monitored at a time.
> We will document this properly.
> 
> 3. It solves the current "pqos" tool issue.
> 
> 4. From AMD's perspective, this is the most practical until we find a 
> long term solution.
> 
> Thanks,
> Babu

Wanted to re-start the patch with some more reasoning. Please see if 
this reasoning makes sense.
-----------------------------------------------------------------

x86/resctrl, Documentation: Keep mbm_assign_mode at default on boot

ABMC ("mbm_event" mode) allows explicit assignment of hardware MBM
counters to RMID/event pairs.  It targets deployments that deliberately
manage counter assignment on platforms with more monitoring groups than
dynamically shareable hardware counters.  It is suited to workflows 
where users snapshot bandwidth for a limited set of groups of interest, 
then move on to another set—not as a general replacement for legacy 
default monitoring on every ABMC-capable system.

Commit 0f1576e43adc ("x86/resctrl: Configure mbm_event mode if
supported") enabled ABMC at boot by setting mbm_cntr_assign_enabled
during L3 monitor initialization.  That breaks existing userspace that
assumes legacy default mode, notably the pqos tool from intel-cmt-cat.

When pqos starts, it mounts resctrl and creates 16 or more monitoring
groups by default (two counters per group: mbm_local_bytes and
mbm_total_bytes).  On platforms with 32 ABMC counters per domain, the
default pqos layout can consume the entire counter pool.  Additional
groups then fail to obtain counters and pqos reports zero bandwidth for
those groups.  The workaround today is to switch back to default mode
before running pqos:

   echo default > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode

Stop enabling ABMC implicitly at boot.  Leave mbm_assign_mode in
"default" mode during initialization; users who need ABMC can enable
it explicitly:

   echo mbm_event > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode


Default mode has a long-standing limitation on AMD platforms with more
monitoring groups than hardware counters: bandwidth reads may report
"Unavailable" or misleading values due to counter re-allocation between
reads.  This typically affects deployments with a large number of
groups (e.g. 64 or more), not typical usage with fewer groups.  Users
who need stable readings for many groups should switch to mbm_event
mode and use a snapshot/rotation workflow.

ABMC capability detection and the mbm_assign_mode interface are
unchanged—only the boot-time opt-in is removed.

Update Documentation/filesystems/resctrl.rst to reflect the boot-time
default, document pqos compatibility, and adjust mbm_assign_mode
examples accordingly.

Link: 
https://lore.kernel.org/lkml/1fe5cb09-aa2e-4b3d-9fbd-293d8e9f33d7@amd.com/
Signed-off-by: Babu Moger <babu.moger@amd.com>

---------------------------------------------------------------------
Thanks
Babu