[PATCH 1/2] x86/resctrl, Documentation: Keep mbm_assign_mode "default" on boot

[PATCH 1/2] x86/resctrl, Documentation: Keep mbm_assign_mode "default" on boot

[Babu Moger]

The kernel currently enables the ABMC-based "mbm_event" mode by default on
hardware that supports it. However, this can cause bandwidth monitoring
failures with existing userspace tools such as pqos.

The pqos tool mounts the resctrl filesystem and creates 16 or more resctrl
groups by default. On systems with 32 or fewer ABMC counters, this default
configuration can consume all available counters, since each group requires
one counter for local MBM and another for total MBM. If additional
monitoring groups are created, counter resources are exhausted and pqos
tool reports memory bandwidth counters as zero for those groups.

Avoid this compatibility issue by leaving mbm_assign_mode in the "default"
mode during initialization. Users who want to use ABMC can continue to
enable it explicitly:

 echo mbm_event > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode

Update the resctrl documentation to reflect the new boot-time default and
adjust the mbm_assign_mode examples accordingly.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
There are plans to enable "mbm_event" by default once additional counters
are available. For now, keep the default mode to maintain compatibility
with existing tools.
---
 Documentation/filesystems/resctrl.rst | 71 ++++++++++++++++-----------
 arch/x86/kernel/cpu/resctrl/monitor.c |  1 -
 2 files changed, 42 insertions(+), 30 deletions(-)

diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
index e4b66af55ffb..e38bfd15fc3d 100644
--- a/Documentation/filesystems/resctrl.rst
+++ b/Documentation/filesystems/resctrl.rst
@@ -355,8 +355,8 @@ with the following files:
 	::
 
 	  # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
-	  [mbm_event]
-	  default
+	  [default]
+	  mbm_event
 
 	"mbm_event":
 
@@ -375,19 +375,23 @@ with the following files:
 	to the events. Otherwise, the MBM event counters will return 'Unassigned' when read.
 
 	The mode is beneficial for AMD platforms that support more CTRL_MON
-	and MON groups than available hardware counters. By default, this
-	feature is enabled on AMD platforms with the ABMC (Assignable Bandwidth
-	Monitoring Counters) capability, ensuring counters remain assigned even
-	when the corresponding RMID is not actively used by any processor.
+	and MON groups than available hardware counters. On platforms with the
+	ABMC (Assignable Bandwidth Monitoring Counters) capability, mbm_event
+	mode ensures counters remain assigned even when the corresponding RMID
+	is not actively used by any processor.
 
 	"default":
 
 	In default mode, resctrl assumes there is a hardware counter for each
-	event within every CTRL_MON and MON group. On AMD platforms, it is
-	recommended to use the mbm_event mode, if supported, to prevent reset of MBM
-	events between reads resulting from hardware re-allocating counters. This can
-	result in misleading values or display "Unavailable" if no counter is assigned
-	to the event.
+	event within every CTRL_MON and MON group. This mode is enabled by default.
+
+	On AMD platforms with more CTRL_MON and MON groups than the available
+	hardware counters, hardware may re-allocate counters between reads
+	while in default mode. This can result in misleading memory bandwidth values
+	or display "Unavailable" if no counter is allocated to the event. In such
+	cases, it is recommended to use the mbm_event mode, if supported, to prevent
+	reset of MBM events between reads resulting from hardware re-allocating
+	counters.
 
 	* To enable "mbm_event" counter assignment mode:
 	  ::
@@ -471,8 +475,8 @@ with the following files:
 
 	Determines if a counter will automatically be assigned to an RMID, MBM event
 	pair when its associated monitor group is created via mkdir. Enabled by default
-	on boot, also when switched from "default" mode to "mbm_event" counter assignment
-	mode. Users can disable this capability by writing to the interface.
+	when switched to "mbm_event" counter assignment mode. Users can disable this
+	capability by writing to the interface.
 
 	"0":
 		Auto assignment is disabled.
@@ -1788,32 +1792,41 @@ a. Check if MBM counter assignment mode is supported.
 
   # mount -t resctrl resctrl /sys/fs/resctrl/
 
+  # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
+  [default]
+  mbm_event
+
+The "mbm_event" and "default" modes are supported. The "default" mode
+is enabled by default.
+
+b. Enable "mbm_event" counter assignment mode.
+::
+
+  # echo "mbm_event" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
   # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
   [mbm_event]
   default
 
-The "mbm_event" mode is detected and enabled.
-
-b. Check how many assignable counters are supported.
+c. Check how many assignable counters are supported.
 ::
 
   # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
   0=32;1=32
 
-c. Check how many assignable counters are available for assignment in each domain.
+d. Check how many assignable counters are available for assignment in each domain.
 ::
 
   # cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs
   0=30;1=30
 
-d. To list the default group's assign states.
+e. To list the default group's assign states.
 ::
 
   # cat /sys/fs/resctrl/mbm_L3_assignments
   mbm_total_bytes:0=e;1=e
   mbm_local_bytes:0=e;1=e
 
-e.  To unassign the counter associated with the mbm_total_bytes event on domain 0.
+f.  To unassign the counter associated with the mbm_total_bytes event on domain 0.
 ::
 
   # echo "mbm_total_bytes:0=_" > /sys/fs/resctrl/mbm_L3_assignments
@@ -1821,7 +1834,7 @@ e.  To unassign the counter associated with the mbm_total_bytes event on domain
   mbm_total_bytes:0=_;1=e
   mbm_local_bytes:0=e;1=e
 
-f. To unassign the counter associated with the mbm_total_bytes event on all domains.
+g. To unassign the counter associated with the mbm_total_bytes event on all domains.
 ::
 
   # echo "mbm_total_bytes:*=_" > /sys/fs/resctrl/mbm_L3_assignments
@@ -1829,7 +1842,7 @@ f. To unassign the counter associated with the mbm_total_bytes event on all doma
   mbm_total_bytes:0=_;1=_
   mbm_local_bytes:0=e;1=e
 
-g. To assign a counter associated with the mbm_total_bytes event on all domains in
+h. To assign a counter associated with the mbm_total_bytes event on all domains in
 exclusive mode.
 ::
 
@@ -1838,7 +1851,7 @@ exclusive mode.
   mbm_total_bytes:0=e;1=e
   mbm_local_bytes:0=e;1=e
 
-h. Read the events mbm_total_bytes and mbm_local_bytes of the default group. There is
+i. Read the events mbm_total_bytes and mbm_local_bytes of the default group. There is
 no change in reading the events with the assignment.
 ::
 
@@ -1851,7 +1864,7 @@ no change in reading the events with the assignment.
   # cat /sys/fs/resctrl/mon_data/mon_L3_01/mbm_local_bytes
   121212144
 
-i. Check the event configurations.
+j. Check the event configurations.
 ::
 
   # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter
@@ -1861,7 +1874,7 @@ i. Check the event configurations.
   # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter
   local_reads,local_non_temporal_writes,local_reads_slow_memory
 
-j. Change the event configuration for mbm_local_bytes.
+k. Change the event configuration for mbm_local_bytes.
 ::
 
   # echo "local_reads, local_non_temporal_writes, local_reads_slow_memory, remote_reads" >
@@ -1870,7 +1883,7 @@ j. Change the event configuration for mbm_local_bytes.
   # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter
   local_reads,local_non_temporal_writes,local_reads_slow_memory,remote_reads
 
-k. Now read the local events again. The first read may come back with "Unavailable"
+l. Now read the local events again. The first read may come back with "Unavailable"
 status. The subsequent read of mbm_local_bytes will display the current value.
 ::
 
@@ -1883,9 +1896,9 @@ status. The subsequent read of mbm_local_bytes will display the current value.
   # cat /sys/fs/resctrl/mon_data/mon_L3_01/mbm_local_bytes
   1566565
 
-l. Users have the option to go back to 'default' mbm_assign_mode if required. This can be
-done using the following command. Note that switching the mbm_assign_mode may reset all
-the MBM counters (and thus all MBM events) of all the resctrl groups.
+m. Users have the option to switch back to 'default' mbm_assign_mode if required. This
+can be done using the following command. Note that switching the mbm_assign_mode may
+reset all the MBM counters (and thus all MBM events) of all the resctrl groups.
 ::
 
   # echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
@@ -1893,7 +1906,7 @@ the MBM counters (and thus all MBM events) of all the resctrl groups.
   mbm_event
   [default]
 
-m. Unmount the resctrl filesystem.
+n. Unmount the resctrl filesystem.
 ::
 
   # umount /sys/fs/resctrl/
diff --git a/arch/x86/kernel/cpu/resctrl/monitor.c b/arch/x86/kernel/cpu/resctrl/monitor.c
index 3838e0a13d36..8a0d6086518b 100644
--- a/arch/x86/kernel/cpu/resctrl/monitor.c
+++ b/arch/x86/kernel/cpu/resctrl/monitor.c
@@ -471,7 +471,6 @@ int __init rdt_get_l3_mon_config(struct rdt_resource *r)
 		r->mon.mbm_cntr_configurable = true;
 		cpuid_count(0x80000020, 5, &eax, &ebx, &ecx, &edx);
 		r->mon.num_mbm_cntrs = (ebx & GENMASK(15, 0)) + 1;
-		hw_res->mbm_cntr_assign_enabled = true;
 	}
 
 	r->mon_capable = true;
-- 
2.43.0

[PATCH 2/2] x86/resctrl: Fix ABMC counter programming for extended counter ranges

[Babu Moger]

Memory Bandwidth Monitoring (MBM) can report incorrect values when ABMC is
enabled on systems supporting more than 32 ABMC counters. As the number of
active monitoring groups increases beyond the range supported by the
existing counter ID encoding, programming an ABMC counter may inadvertently
affect a different counter, resulting in unexpected counter resets and
abnormally large MBM readings.

The issue originates from the ABMC counter programming interface in the
L3_QOS_ABMC_CFG MSR. The counter ID field is currently defined as 5 bits,
which limits the addressable counter range to 32 counters. On systems
implementing more than 32 ABMC counters, counter IDs above 31 cannot be
encoded correctly. Consequently, programming a counter ID beyond the
supported range may target an unintended counter and reset bandwidth
statistics associated with another monitoring group.

While updating this logic, it was also observed that the bw_src field,
which encodes the RMID, is currently at its 12-bit limit with support for
4096 RMIDs. This field also needs to be updated for future expansion.

Also found one more pre-existing issue. This union structure can truncate
data on 32-bit x86 systems when unsigned long is used.

Fix the issues with the following changes:

1. Update the cntr_id field handling to support the full hardware ABMC
counter range and ensure that counter programming does not interfere with
unrelated counters.

2. Expand the bw_src field to 15 bits.

3. Change "unsigned long" to u64 to fix truncation on 32-bit x86.

The AMD64 Architecture Programmer's Manual [1] available at [2] will be
updated accordingly in a future revision to document the expanded cntr_id
and bw_src field definitions.

[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming,
    Publication #24593, Revision 3.41, Section 19.3.3.3 "Assignable
    Bandwidth Monitoring (ABMC)"

Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
Fixes: 84ecefb76674 ("x86/resctrl: Add data structures and definitions for ABMC assignment")
Signed-off-by: Babu Moger <babu.moger@amd.com>
---
 arch/x86/kernel/cpu/resctrl/internal.h | 18 ++++++++----------
 1 file changed, 8 insertions(+), 10 deletions(-)

diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
index e3cfa0c10e92..b3d780eda8a7 100644
--- a/arch/x86/kernel/cpu/resctrl/internal.h
+++ b/arch/x86/kernel/cpu/resctrl/internal.h
@@ -192,7 +192,6 @@ union cpuid_0x10_x_edx {
  * @bw_type		: Event configuration that represents the memory
  *			  transactions being tracked by the @cntr_id.
  * @bw_src		: Bandwidth source (RMID or CLOSID).
- * @reserved1		: Reserved.
  * @is_clos		: @bw_src field is a CLOSID (not an RMID).
  * @cntr_id		: Counter identifier.
  * @reserved		: Reserved.
@@ -210,16 +209,15 @@ union cpuid_0x10_x_edx {
  */
 union l3_qos_abmc_cfg {
 	struct {
-		unsigned long bw_type  :32,
-			      bw_src   :12,
-			      reserved1: 3,
-			      is_clos  : 1,
-			      cntr_id  : 5,
-			      reserved : 9,
-			      cntr_en  : 1,
-			      cfg_en   : 1;
+		u64	bw_type  :32,
+			bw_src   :15,
+			is_clos  : 1,
+			cntr_id  :12,
+			reserved : 2,
+			cntr_en  : 1,
+			cfg_en   : 1;
 	} split;
-	unsigned long full;
+	u64	full;
 };
 
 void rdt_ctrl_update(void *arg);
-- 
2.43.0

Re: [PATCH 1/2] x86/resctrl, Documentation: Keep mbm_assign_mode "default" on boot

[Reinette Chatre]

Hi Babu,

On 7/17/26 2:13 PM, Babu Moger wrote:
> The kernel currently enables the ABMC-based "mbm_event" mode by default on
> hardware that supports it. However, this can cause bandwidth monitoring
> failures with existing userspace tools such as pqos.
> 
> The pqos tool mounts the resctrl filesystem and creates 16 or more resctrl
> groups by default. On systems with 32 or fewer ABMC counters, this default
> configuration can consume all available counters, since each group requires
> one counter for local MBM and another for total MBM. If additional
> monitoring groups are created, counter resources are exhausted and pqos
> tool reports memory bandwidth counters as zero for those groups.

It is not obvious to me that this is a problem. If I understand correctly 
there are two scenarios possible with this pqos behavior:

- ABMC is not in use ("mbm_assign_mode" is set to "default")
  - pqos can create 16 or more monitor groups
  - hardware still supports a limited number of counters with consequence that
    underlying counters reset at any time as the different monitoring groups
    need to be tracked.
  - pqos can read monitoring data of all 16 monitor groups, sometimes reading the
    events would return "Unavailable", sometimes reading the events return data.
  - *None* of the monitoring numbers returned are guaranteed to be accurate.

- ABMC is in use ("mbm_assign_mode" is set to "mbm_event"):
  - pqos can create 16 or more monitor groups
  - only a subset of monitoring groups have counters assigned and these counters
    are guaranteed to only track the monitor groups/events they are assigned to
  - pqos can read monitoring data of all 16 monitor groups with two possibilities:
    - monitor group/event has counter assigned: monitoring numbers are guaranteed to be accurate
    - monitor group/event does not have counter assigned: monitoring numbers return 0

If my understanding is correct then the preference is to rather have wrong data than
see 0? This does not sound right. What am I missing?

The changelog starts with "this can cause bandwidth monitoring failures with existing
userspace tools such as pqos". How could returning accurate memory bandwidth data be
considered a failure? How does this issue manifest itself?

> 
> Avoid this compatibility issue by leaving mbm_assign_mode in the "default"
> mode during initialization. Users who want to use ABMC can continue to
> enable it explicitly:
> 
>  echo mbm_event > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
> 
> Update the resctrl documentation to reflect the new boot-time default and
> adjust the mbm_assign_mode examples accordingly.
> 
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---
> There are plans to enable "mbm_event" by default once additional counters
> are available. For now, keep the default mode to maintain compatibility
> with existing tools.

This is not ideal. resctrl should aim to provide a consistent user interface
across kernel versions. 

Reinette

Re: [PATCH 1/2] x86/resctrl, Documentation: Keep mbm_assign_mode "default" on boot

[Babu Moger]

Hi Reinette,

Thanks for quick response.

On 7/17/26 17:56, Reinette Chatre wrote:
> Hi Babu,
> 
> On 7/17/26 2:13 PM, Babu Moger wrote:
>> The kernel currently enables the ABMC-based "mbm_event" mode by default on
>> hardware that supports it. However, this can cause bandwidth monitoring
>> failures with existing userspace tools such as pqos.
>>
>> The pqos tool mounts the resctrl filesystem and creates 16 or more resctrl
>> groups by default. On systems with 32 or fewer ABMC counters, this default
>> configuration can consume all available counters, since each group requires
>> one counter for local MBM and another for total MBM. If additional
>> monitoring groups are created, counter resources are exhausted and pqos
>> tool reports memory bandwidth counters as zero for those groups.
> 
> It is not obvious to me that this is a problem. If I understand correctly
> there are two scenarios possible with this pqos behavior:
> 
> - ABMC is not in use ("mbm_assign_mode" is set to "default")
>    - pqos can create 16 or more monitor groups
>    - hardware still supports a limited number of counters with consequence that
>      underlying counters reset at any time as the different monitoring groups
>      need to be tracked.
>    - pqos can read monitoring data of all 16 monitor groups, sometimes reading the
>      events would return "Unavailable", sometimes reading the events return data.
>    - *None* of the monitoring numbers returned are guaranteed to be accurate.
> 
> - ABMC is in use ("mbm_assign_mode" is set to "mbm_event"):
>    - pqos can create 16 or more monitor groups
>    - only a subset of monitoring groups have counters assigned and these counters
>      are guaranteed to only track the monitor groups/events they are assigned to
>    - pqos can read monitoring data of all 16 monitor groups with two possibilities:
>      - monitor group/event has counter assigned: monitoring numbers are guaranteed to be accurate
>      - monitor group/event does not have counter assigned: monitoring numbers return 0
> 
> If my understanding is correct then the preference is to rather have wrong data than
> see 0? This does not sound right. What am I missing?


This hardware can monitor up to 64 RMIDs without any counter resets.

As you know, the pqos tool creates COS1 through COS15 regardless of the 
command-line options used, resulting in a total of 16 groups including 
the default group. With ABMC enabled, this consumes all available ABMC 
counters(32 counters, 2 counters for each group).

When pqos is invoked with the -m option, it creates additional 
monitoring groups. For example:

pqos -m all:0     -> creates 1 monitoring group
pqos -m all:0,1   -> creates 2 monitoring groups


Since all ABMC counters have already been allocated to the default set 
of groups, no counters remain for these additional monitoring groups. As 
a result, the monitoring commands report zero values, effectively making 
monitoring unusable.

In contrast, the default monitoring mode can still support up to 48 
additional monitoring groups (64 total RMIDs minus the 16 default groups 
created by pqos).

For this reason, I still believe keeping the default monitoring mode as 
the default is the better option.

Thanks
Babu

> 
> The changelog starts with "this can cause bandwidth monitoring failures with existing
> userspace tools such as pqos". How could returning accurate memory bandwidth data be
> considered a failure? How does this issue manifest itself?
> 
>>
>> Avoid this compatibility issue by leaving mbm_assign_mode in the "default"
>> mode during initialization. Users who want to use ABMC can continue to
>> enable it explicitly:
>>
>>   echo mbm_event > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
>>
>> Update the resctrl documentation to reflect the new boot-time default and
>> adjust the mbm_assign_mode examples accordingly.
>>
>> Signed-off-by: Babu Moger <babu.moger@amd.com>
>> ---
>> There are plans to enable "mbm_event" by default once additional counters
>> are available. For now, keep the default mode to maintain compatibility
>> with existing tools.
> 
> This is not ideal. resctrl should aim to provide a consistent user interface
> across kernel versions.
> 
> Reinette
> 
> 

Re: [PATCH 1/2] x86/resctrl, Documentation: Keep mbm_assign_mode "default" on boot

[Reinette Chatre]

Hi Babu,

On 7/20/26 10:00 AM, Babu Moger wrote:
> Hi Reinette,
> 
> Thanks for quick response.
> 
> On 7/17/26 17:56, Reinette Chatre wrote:
>> Hi Babu,
>>
>> On 7/17/26 2:13 PM, Babu Moger wrote:
>>> The kernel currently enables the ABMC-based "mbm_event" mode by default on
>>> hardware that supports it. However, this can cause bandwidth monitoring
>>> failures with existing userspace tools such as pqos.
>>>
>>> The pqos tool mounts the resctrl filesystem and creates 16 or more resctrl
>>> groups by default. On systems with 32 or fewer ABMC counters, this default
>>> configuration can consume all available counters, since each group requires
>>> one counter for local MBM and another for total MBM. If additional
>>> monitoring groups are created, counter resources are exhausted and pqos
>>> tool reports memory bandwidth counters as zero for those groups.
>>
>> It is not obvious to me that this is a problem. If I understand correctly
>> there are two scenarios possible with this pqos behavior:
>>
>> - ABMC is not in use ("mbm_assign_mode" is set to "default")
>>    - pqos can create 16 or more monitor groups
>>    - hardware still supports a limited number of counters with consequence that
>>      underlying counters reset at any time as the different monitoring groups
>>      need to be tracked.
>>    - pqos can read monitoring data of all 16 monitor groups, sometimes reading the
>>      events would return "Unavailable", sometimes reading the events return data.
>>    - *None* of the monitoring numbers returned are guaranteed to be accurate.
>>
>> - ABMC is in use ("mbm_assign_mode" is set to "mbm_event"):
>>    - pqos can create 16 or more monitor groups
>>    - only a subset of monitoring groups have counters assigned and these counters
>>      are guaranteed to only track the monitor groups/events they are assigned to
>>    - pqos can read monitoring data of all 16 monitor groups with two possibilities:
>>      - monitor group/event has counter assigned: monitoring numbers are guaranteed to be accurate
>>      - monitor group/event does not have counter assigned: monitoring numbers return 0
>>
>> If my understanding is correct then the preference is to rather have wrong data than
>> see 0? This does not sound right. What am I missing?
> 
> 
> This hardware can monitor up to 64 RMIDs without any counter resets.

How many RMIDs does the hardware claim to support via CPUID that ends up being shown to user
space via "num_rmids"?

I understood from original ABMC enabling that the underlying hardware counters of "default"
and "mbm_event" mode on AMD are the same. That is, in "default" mode the hardware does a
"best effort" assignment of hardware counters to events while "mbm_event" mode lets the user
control the assignment. It instead sounds like this is not the case and there are actually
two distinct underlying hardware counter mechanisms?


> As you know, the pqos tool creates COS1 through COS15 regardless of
> the command-line options used, resulting in a total of 16 groups
> including the default group. With ABMC enabled, this consumes all
> available ABMC counters(32 counters, 2 counters for each group).

> 
> When pqos is invoked with the -m option, it creates additional monitoring groups. For example:
> 
> pqos -m all:0     -> creates 1 monitoring group
> pqos -m all:0,1   -> creates 2 monitoring groups
> 
> 
> Since all ABMC counters have already been allocated to the default
> set of groups, no counters remain for these additional monitoring
> groups. As a result, the monitoring commands report zero values,
> effectively making monitoring unusable.
> 
> In contrast, the default monitoring mode can still support up to 48
> additional monitoring groups (64 total RMIDs minus the 16 default
> groups created by pqos).
> 
> For this reason, I still believe keeping the default monitoring mode as the default is the better option.

It is not clear to me where the "64" number comes from. Even if resctrl sets the "default"
mode as default, what will happen to the scenario you describe and 49, instead of 48,
additional groups are created? From what I understand the moment the 65th group is created user will
transition from "accurate per monitor group monitoring data for all 64 monitoring groups" to "inaccurate
per monitor group monitoring data for all 65 monitoring groups" with no indication that this is happening?

I believe AMD supports more than 64 RMIDs and in this case there seems to be three ranges:
[supported by ABMC, depends on events but lets say RMID 0 to 16] < [RMID 17 to 64] < [RMID 65 to total number of RMIDs supported]

Current default "mbm_event" mode uses ABMC so as you state this always results in:
- accurate counts for 16 monitor groups
- zero for all other monitor groups up to total number of RMIDs supported

As I see it switching to the "default" mode would result in:
Scenario 1, 64 or fewer monitor groups are created:
- accurate counts for all monitor groups
Scenario 2, 65 or more monitor groups are created:
- inaccurate counts for all monitor groups

I do not believe there is any way for user space to know when or if system switches from "scenario 1" to
"scenario 2" on these systems and this unpredictable behavior that results in wrong data does not
sound ideal to me.

Reinette

Re: [PATCH 1/2] x86/resctrl, Documentation: Keep mbm_assign_mode "default" on boot

[Babu Moger]

Hi Reinette,


On 7/20/26 13:27, Reinette Chatre wrote:
> Hi Babu,
> 
> On 7/20/26 10:00 AM, Babu Moger wrote:
>> Hi Reinette,
>>
>> Thanks for quick response.
>>
>> On 7/17/26 17:56, Reinette Chatre wrote:
>>> Hi Babu,
>>>
>>> On 7/17/26 2:13 PM, Babu Moger wrote:
>>>> The kernel currently enables the ABMC-based "mbm_event" mode by default on
>>>> hardware that supports it. However, this can cause bandwidth monitoring
>>>> failures with existing userspace tools such as pqos.
>>>>
>>>> The pqos tool mounts the resctrl filesystem and creates 16 or more resctrl
>>>> groups by default. On systems with 32 or fewer ABMC counters, this default
>>>> configuration can consume all available counters, since each group requires
>>>> one counter for local MBM and another for total MBM. If additional
>>>> monitoring groups are created, counter resources are exhausted and pqos
>>>> tool reports memory bandwidth counters as zero for those groups.
>>>
>>> It is not obvious to me that this is a problem. If I understand correctly
>>> there are two scenarios possible with this pqos behavior:
>>>
>>> - ABMC is not in use ("mbm_assign_mode" is set to "default")
>>>     - pqos can create 16 or more monitor groups
>>>     - hardware still supports a limited number of counters with consequence that
>>>       underlying counters reset at any time as the different monitoring groups
>>>       need to be tracked.
>>>     - pqos can read monitoring data of all 16 monitor groups, sometimes reading the
>>>       events would return "Unavailable", sometimes reading the events return data.
>>>     - *None* of the monitoring numbers returned are guaranteed to be accurate.
>>>
>>> - ABMC is in use ("mbm_assign_mode" is set to "mbm_event"):
>>>     - pqos can create 16 or more monitor groups
>>>     - only a subset of monitoring groups have counters assigned and these counters
>>>       are guaranteed to only track the monitor groups/events they are assigned to
>>>     - pqos can read monitoring data of all 16 monitor groups with two possibilities:
>>>       - monitor group/event has counter assigned: monitoring numbers are guaranteed to be accurate
>>>       - monitor group/event does not have counter assigned: monitoring numbers return 0
>>>
>>> If my understanding is correct then the preference is to rather have wrong data than
>>> see 0? This does not sound right. What am I missing?
>>
>>
>> This hardware can monitor up to 64 RMIDs without any counter resets.
> 
> How many RMIDs does the hardware claim to support via CPUID that ends up being shown to user
> space via "num_rmids"?

#cat /sys/fs/resctrl/info/L3_MON/num_rmids
4096

> 
> I understood from original ABMC enabling that the underlying hardware counters of "default"
> and "mbm_event" mode on AMD are the same. That is, in "default" mode the hardware does a
> "best effort" assignment of hardware counters to events while "mbm_event" mode lets the user
> control the assignment. It instead sounds like this is not the case and there are actually
> two distinct underlying hardware counter mechanisms?

That is correct. They are two different counters.

> 
> 
>> As you know, the pqos tool creates COS1 through COS15 regardless of
>> the command-line options used, resulting in a total of 16 groups
>> including the default group. With ABMC enabled, this consumes all
>> available ABMC counters(32 counters, 2 counters for each group).
> 
>>
>> When pqos is invoked with the -m option, it creates additional monitoring groups. For example:
>>
>> pqos -m all:0     -> creates 1 monitoring group
>> pqos -m all:0,1   -> creates 2 monitoring groups
>>
>>
>> Since all ABMC counters have already been allocated to the default
>> set of groups, no counters remain for these additional monitoring
>> groups. As a result, the monitoring commands report zero values,
>> effectively making monitoring unusable.
>>
>> In contrast, the default monitoring mode can still support up to 48
>> additional monitoring groups (64 total RMIDs minus the 16 default
>> groups created by pqos).
>>
>> For this reason, I still believe keeping the default monitoring mode as the default is the better option.
> 
> It is not clear to me where the "64" number comes from. Even if resctrl sets the "default"

The count of 64 is known from internal information. It can also be 
determined by allocating monitoring counters in a loop until the 
hardware starts returning an "unavailable" response, which occurs after 
all 64 counters have been assigned. This information is not documented.

> mode as default, what will happen to the scenario you describe and 49, instead of 48,
> additional groups are created? From what I understand the moment the 65th group is created user will
> transition from "accurate per monitor group monitoring data for all 64 monitoring groups" to "inaccurate
> per monitor group monitoring data for all 65 monitoring groups" with no indication that this is happening?

Yes. That is correct.

> 
> I believe AMD supports more than 64 RMIDs and in this case there seems to be three ranges:
> [supported by ABMC, depends on events but lets say RMID 0 to 16] < [RMID 17 to 64] < [RMID 65 to total number of RMIDs supported]
> 
> Current default "mbm_event" mode uses ABMC so as you state this always results in:
> - accurate counts for 16 monitor groups
> - zero for all other monitor groups up to total number of RMIDs supported

That is correct. With ABMC, there are 32 available counters, allowing up 
to 32 monitoring events to be tracked simultaneously. Since each 
monitoring group requires two counters—one for local MBM and one for 
total MBM—the system can support only 16 monitoring groups at a time.

> 
> As I see it switching to the "default" mode would result in:
> Scenario 1, 64 or fewer monitor groups are created:
> - accurate counts for all monitor groups
> Scenario 2, 65 or more monitor groups are created:
> - inaccurate counts for all monitor groups

That is true.

> 
> I do not believe there is any way for user space to know when or if system switches from "scenario 1" to
> "scenario 2" on these systems and this unpredictable behavior that results in wrong data does not
> sound ideal to me.
  I agree, it's not an ideal situation, but that's how it has always 
worked. At the moment, I'm not sure about a better alternative.

Thanks,
Babu

Re: [PATCH 1/2] x86/resctrl, Documentation: Keep mbm_assign_mode "default" on boot

[Babu Moger]

On 7/17/26 16:13, Babu Moger wrote:
> The kernel currently enables the ABMC-based "mbm_event" mode by default on
> hardware that supports it. However, this can cause bandwidth monitoring
> failures with existing userspace tools such as pqos.
> 
> The pqos tool mounts the resctrl filesystem and creates 16 or more resctrl
> groups by default. On systems with 32 or fewer ABMC counters, this default
> configuration can consume all available counters, since each group requires
> one counter for local MBM and another for total MBM. If additional
> monitoring groups are created, counter resources are exhausted and pqos
> tool reports memory bandwidth counters as zero for those groups.
> 
> Avoid this compatibility issue by leaving mbm_assign_mode in the "default"
> mode during initialization. Users who want to use ABMC can continue to
> enable it explicitly:
> 
>   echo mbm_event > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
> 
> Update the resctrl documentation to reflect the new boot-time default and
> adjust the mbm_assign_mode examples accordingly.
> 
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---
> There are plans to enable "mbm_event" by default once additional counters
> are available. For now, keep the default mode to maintain compatibility
> with existing tools.
> ---
>   Documentation/filesystems/resctrl.rst | 71 ++++++++++++++++-----------
>   arch/x86/kernel/cpu/resctrl/monitor.c |  1 -
>   2 files changed, 42 insertions(+), 30 deletions(-)
> 
> diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
> index e4b66af55ffb..e38bfd15fc3d 100644
> --- a/Documentation/filesystems/resctrl.rst
> +++ b/Documentation/filesystems/resctrl.rst
> @@ -355,8 +355,8 @@ with the following files:
>   	::
>   
>   	  # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
> -	  [mbm_event]
> -	  default
> +	  [default]
> +	  mbm_event
>   
>   	"mbm_event":
>   
> @@ -375,19 +375,23 @@ with the following files:
>   	to the events. Otherwise, the MBM event counters will return 'Unassigned' when read.
>   
>   	The mode is beneficial for AMD platforms that support more CTRL_MON
> -	and MON groups than available hardware counters. By default, this
> -	feature is enabled on AMD platforms with the ABMC (Assignable Bandwidth
> -	Monitoring Counters) capability, ensuring counters remain assigned even
> -	when the corresponding RMID is not actively used by any processor.
> +	and MON groups than available hardware counters. On platforms with the
> +	ABMC (Assignable Bandwidth Monitoring Counters) capability, mbm_event
> +	mode ensures counters remain assigned even when the corresponding RMID
> +	is not actively used by any processor.
>   
>   	"default":
>   
>   	In default mode, resctrl assumes there is a hardware counter for each
> -	event within every CTRL_MON and MON group. On AMD platforms, it is
> -	recommended to use the mbm_event mode, if supported, to prevent reset of MBM
> -	events between reads resulting from hardware re-allocating counters. This can
> -	result in misleading values or display "Unavailable" if no counter is assigned
> -	to the event.
> +	event within every CTRL_MON and MON group. This mode is enabled by default.
> +
> +	On AMD platforms with more CTRL_MON and MON groups than the available
> +	hardware counters, hardware may re-allocate counters between reads
> +	while in default mode. This can result in misleading memory bandwidth values
> +	or display "Unavailable" if no counter is allocated to the event. In such
> +	cases, it is recommended to use the mbm_event mode, if supported, to prevent
> +	reset of MBM events between reads resulting from hardware re-allocating
> +	counters.
>   
>   	* To enable "mbm_event" counter assignment mode:
>   	  ::
> @@ -471,8 +475,8 @@ with the following files:
>   
>   	Determines if a counter will automatically be assigned to an RMID, MBM event
>   	pair when its associated monitor group is created via mkdir. Enabled by default
> -	on boot, also when switched from "default" mode to "mbm_event" counter assignment
> -	mode. Users can disable this capability by writing to the interface.
> +	when switched to "mbm_event" counter assignment mode. Users can disable this
> +	capability by writing to the interface.
>   
>   	"0":
>   		Auto assignment is disabled.
> @@ -1788,32 +1792,41 @@ a. Check if MBM counter assignment mode is supported.
>   
>     # mount -t resctrl resctrl /sys/fs/resctrl/
>   
> +  # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
> +  [default]
> +  mbm_event
> +
> +The "mbm_event" and "default" modes are supported. The "default" mode
> +is enabled by default.
> +
> +b. Enable "mbm_event" counter assignment mode.
> +::
> +
> +  # echo "mbm_event" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
>     # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
>     [mbm_event]
>     default
>   
> -The "mbm_event" mode is detected and enabled.
> -
> -b. Check how many assignable counters are supported.
> +c. Check how many assignable counters are supported.
>   ::
>   
>     # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
>     0=32;1=32
>   
> -c. Check how many assignable counters are available for assignment in each domain.
> +d. Check how many assignable counters are available for assignment in each domain.
>   ::
>   
>     # cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs
>     0=30;1=30
>   
> -d. To list the default group's assign states.
> +e. To list the default group's assign states.
>   ::
>   
>     # cat /sys/fs/resctrl/mbm_L3_assignments
>     mbm_total_bytes:0=e;1=e
>     mbm_local_bytes:0=e;1=e

Shasiko reported.

https://sashiko.dev/#/patchset/8cb66e18e32e4087a9712c1e68ee6da614efe244.1784322818.git.babu.moger%40amd.com


--- Patch [1]: [PATCH 1/2] x86/resctrl, Documentation: Keep 
mbm_assign_mode "default" on boot ---
commit 41dfae2edbe7538cef040aa9e0f1f0f5a5873564
Author: Babu Moger <babu.moger@amd.com>

x86/resctrl, Documentation: Keep mbm_assign_mode "default" on boot

This commit changes the default MBM assignment mode to "default" on boot to
maintain compatibility with existing userspace tools like pqos. Users 
who want
ABMC features must now explicitly enable the "mbm_event" mode.

 > diff --git a/Documentation/filesystems/resctrl.rst 
b/Documentation/filesystems/resctrl.rst
 > index e4b66af55ffba..e38bfd15fc3d0 100644
 > --- a/Documentation/filesystems/resctrl.rst
 > +++ b/Documentation/filesystems/resctrl.rst

[ ... ]

 > @@ -1788,32 +1792,41 @@ a. Check if MBM counter assignment mode is 
supported.
 >
 >    # mount -t resctrl resctrl /sys/fs/resctrl/
 >
 > +  # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
 > +  [default]
 > +  mbm_event
 > +
 > +The "mbm_event" and "default" modes are supported. The "default" mode
 > +is enabled by default.
 > +
 > +b. Enable "mbm_event" counter assignment mode.
 > +::
 > +
 > +  # echo "mbm_event" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
 >    # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
 >    [mbm_event]
 >    default

[ ... ]

 > -d. To list the default group's assign states.
 > +e. To list the default group's assign states.
 >  ::
 >
 >    # cat /sys/fs/resctrl/mbm_L3_assignments
 >    mbm_total_bytes:0=e;1=e
 >    mbm_local_bytes:0=e;1=e

Does switching to "mbm_event" mode automatically assign counters to the
default group as shown here?

Because this patch changes the boot default to "default", users must now
transition to "mbm_event" dynamically. Looking at the transition path:

fs/resctrl/monitor.c:resctrl_mbm_assign_mode_write() {
     ...
     /* Enable auto assignment when switching to "mbm_event" mode */
     if (enable)
         r->mon.mbm_assign_on_mkdir = true;
     /*
      * Reset all the non-achitectural RMID state and assignable counters.
      */
     list_for_each_entry_rcu(d, &r->mon_domains, hdr.list, 
lockdep_is_cpus_held()) {
         mbm_cntr_free_all(r, d);
         resctrl_reset_rmid_all(r, d);
     }
     ...
}

This path clears all existing counters and sets mbm_assign_on_mkdir for
new groups, but does not appear to assign counters for the already-existing
default group.

Prior to this patch, booting directly into "mbm_event" mode handled the
default group assignment during rdt_get_tree().

Will the default group actually show _ (unassigned) states here instead
of the documented e (assigned) states, causing bandwidth monitoring to
return "Unassigned" until manually fixed?

===========================================================================

Yes. It does not automatically assign the counters to already created 
groups. User need to assign the counters manually to read the events.
The documentation needs to updated.

Thanks
Babu