Re: [PATCH v4 3/3] x86,fs/resctrl: Keep mbm_assign_mode in default mode at boot

[Babu Moger <babu.moger@amd.com>]

Hi Reinette,

On 10/12/26 13:47, Reinette Chatre wrote:
> Hi Babu,
> 
> On 10/9/26 4:44 PM, Babu Moger wrote:
>> The mbm_event mode assigns hardware MBM counters to RMID/event pairs.
>> It is intended for deployments that need to manage counter assignment
>> on platforms where the number of monitoring groups exceeds the available
>> hardware counters. Hardware counters are scarce on these platforms, so
>> mbm_event mode suits workflows that monitor a subset of groups at a time
>> and rotate assignments as needed.
>>
>> resctrl enables the mbm_event mode by default on hardware that supports
>> it. This breaks the pqos tool [1], which assumes the historical default
>> mode. It creates 16 or more monitoring groups and uses two counters per
>> group. On a platform with 32 counters per domain, that exhausts the
>> mbm_event counters pool, causing additional groups to read "Unassigned".
>> pqos interprets a non-numeric event read as zero bandwidth and reports 0
>> MB/s for those groups:
>>
>>    $sudo pqos -m all:0-4
>>    CORE         IPC      MISSES     LLC[KB]   MBL[MB/s]   MBR[MB/s]
>>       0        0.69         16k       192.0         0.0         0.0
>>       1        0.40          3k         0.0         0.0         0.0
>>       2        0.44          1k        64.0         0.0         0.0
>>       3        0.39          1k        64.0         0.0         0.0
>>       4        0.43          1k         0.0         0.0         0.0
>>
>> Leave mbm_assign_mode in default mode during initialization. Default mode
>> uses one counter per monitoring group. On existing AMD platforms, the
>> default active counters pool is 64 and may be larger on newer hardware.
>> The active counters pool is the number of counters that the hardware can
>> actively track in default mode.
>>
>> Deployments within the active counter pool receive accurate bandwidth
> 
> nit: "active counter pool" -> "active counters pool"
> (to be consistent with all the other instances)

Sure.

>> measurements in the default mode. Users can create more monitoring groups
>> (4096) than there are active counters in the pool. Beyond that pool,
> 
> Unexpected where the hardcoded 4096 comes from. Could this be "(for example,
> 4096 on current hardware)"?

Sure.

>> readings may be misleading or report as "Unavailable", with no user-visible
>> indication.
>>
>> Default mode is still limited once the number of monitoring groups exceeds
>> that active counters pool. After hardware re-allocates a counter, a read
>> may return "Unavailable". pqos still treats "Unavailable" as zero.
>> Successive reads can be a count, "Unavailable", then another count. pqos
>> treats those jumps as wraparound and can report an inconsistent rate:
>>
>>    $sudo pqos -m all:0-4
>>    CORE     IPC      MISSES     LLC[KB]   MBL[MB/s]   MBR[MB/s]
>>    0        0.76         60k        32.0         0.0         0.0
>>    1        0.45          1k        32.0         0.0         0.0
>>    2        1.57        107k        64.0         0.0 17592186044184.9
>>    3        1.62        276k      4928.0         3.2         0.5
>>
>> Users that need stable measurements beyond the active countes pool should
> 
> "countes" -> "counters"

Sure.

> 
>> use mbm_event mode and rotate assignments as needed. Enable it with:
>>
>>    $echo mbm_event > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
>>
>> Fixes: 0f1576e43adc ("x86/resctrl: Configure mbm_event mode if supported")
>> Closes: https://github.com/intel/intel-cmt-cat/issues/311
>> Signed-off-by: Babu Moger <babu.moger@amd.com>
>> Cc: stable@vger.kernel.org
>> Link: https://github.com/intel/intel-cmt-cat # [1]
>> ---
> 
> ...
> 
>> ---
>>   Documentation/filesystems/resctrl.rst | 43 +++++++++++++++++----------
>>   arch/x86/kernel/cpu/resctrl/monitor.c |  1 -
>>   2 files changed, 28 insertions(+), 16 deletions(-)
>>
>> diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
>> index c8507580474a..a896bcae3319 100644
>> --- a/Documentation/filesystems/resctrl.rst
>> +++ b/Documentation/filesystems/resctrl.rst
>> @@ -354,8 +354,8 @@ with the following files:
>>   	::
>>   
>>   	  # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
>> -	  [mbm_event]
>> -	  default
>> +	  [default]
>> +	  mbm_event
>>   
>>   	"mbm_event":
>>   
>> @@ -376,20 +376,35 @@ with the following files:
>>   	"mbm_L3_assignments" after switching to "mbm_event" mode for counter
>>   	assignment states of all monitoring groups.
>>   
>> -	The mode is beneficial for AMD platforms that support more CTRL_MON
>> -	and MON groups than available hardware counters. By default, this
>> -	feature is enabled on AMD platforms with the ABMC (Assignable Bandwidth
>> -	Monitoring Counters) capability, ensuring counters remain assigned even
>> -	when the corresponding RMID is not actively used by any processor.
>> +	It is intended for deployments that need to manage counter assignment on
>> +	platforms where the number of monitoring groups exceeds the available
>> +	hardware counters. Hardware counters are scarce on these platforms, so
>> +	mbm_event mode suits workflows that monitor a subset of groups at a time and
>> +	rotate assignments as needed. The mbm_event mode ensures counters remain
>> +	assigned even when the corresponding RMID is not actively monitored.
>>   
>>   	"default":
>>   
>>   	In default mode, resctrl assumes there is a hardware counter for each
>> -	event within every CTRL_MON and MON group. On AMD platforms, it is
>> -	recommended to use the mbm_event mode, if supported, to prevent reset of MBM
>> -	events between reads resulting from hardware re-allocating counters. This can
>> -	result in misleading values or display "Unavailable" if no counter is assigned
>> -	to the event.
>> +	event within every CTRL_MON and MON group. This mode is enabled by default.
>> +
>> +	On AMD platforms that support more CTRL_MON and MON groups than hardware
>> +	counters, hardware dynamically shares a smaller pool of counters among
>> +	RMIDs. One counter from this pool is used per monitoring group and counts
>> +	every MBM event of that group's RMID. The size of the pool is not
>> +	enumerated to software (unlike "num_mbm_cntrs"), and "num_rmids" may be
>> +	much larger. For example, 64 counters in this pool monitor 64 groups.
>> +	Newer hardware may provide a larger pool.
>> +
>> +	While the number of monitoring groups does not exceed that pool, a counter
>> +	stays attached to each RMID and readings remain accurate. Creating more
>> +	groups than the pool can cause hardware to re-allocate those counters
>> +	between successive reads of an event. Bandwidth values may then be
>> +	misleading, or a read may return "Unavailable" if no counter is allocated
>> +	to the RMID. There is no user-visible indication when this begins.
>> +	Users who need stable readings beyond that pool should switch to mbm_event
>> +	mode, if supported, and assign counters to the groups of interest
>> +	(rotating assignments as needed).
>>   
>>   	* To enable "mbm_event" counter assignment mode:
>>   	  ::
>> @@ -1787,11 +1802,9 @@ View the llc occupancy snapshot::
>>   Examples on working with mbm_assign_mode
>>   ========================================
>>   
>> -a. Check if MBM counter assignment mode is supported.
>> +a. Check if MBM counter assignment mode is supported and enabled.
>>   ::
>>   
>> -  # mount -t resctrl resctrl /sys/fs/resctrl/
>> -
>>     # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
>>     [mbm_event]
>>     default
> 
> After the above doc changes the mbm_assign_on_mkdir description still reads
> "Enabled by default on boot". This is still technically accurate, but this patch
> makes it unobservable by user space. Fine with me if you intentionally decide to
> keep it but I would just like to confirm that you considered it?

Good catch. I noticed that a while ago, but it slipped my mind.

I also see that the MPAM driver initializes it to true at boot. I would 
expect that to be harmless.

Thanks
Babu