[PATCH 1/2] x86/resctrl, Documentation: Keep mbm_assign_mode "default" on boot

[PATCH 1/2] x86/resctrl, Documentation: Keep mbm_assign_mode "default" on boot

[Babu Moger]

The kernel currently enables the ABMC-based "mbm_event" mode by default on
hardware that supports it. However, this can cause bandwidth monitoring
failures with existing userspace tools such as pqos.

The pqos tool mounts the resctrl filesystem and creates 16 or more resctrl
groups by default. On systems with 32 or fewer ABMC counters, this default
configuration can consume all available counters, since each group requires
one counter for local MBM and another for total MBM. If additional
monitoring groups are created, counter resources are exhausted and pqos
tool reports memory bandwidth counters as zero for those groups.

Avoid this compatibility issue by leaving mbm_assign_mode in the "default"
mode during initialization. Users who want to use ABMC can continue to
enable it explicitly:

 echo mbm_event > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode

Update the resctrl documentation to reflect the new boot-time default and
adjust the mbm_assign_mode examples accordingly.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
There are plans to enable "mbm_event" by default once additional counters
are available. For now, keep the default mode to maintain compatibility
with existing tools.
---
 Documentation/filesystems/resctrl.rst | 71 ++++++++++++++++-----------
 arch/x86/kernel/cpu/resctrl/monitor.c |  1 -
 2 files changed, 42 insertions(+), 30 deletions(-)

diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
index e4b66af55ffb..e38bfd15fc3d 100644
--- a/Documentation/filesystems/resctrl.rst
+++ b/Documentation/filesystems/resctrl.rst
@@ -355,8 +355,8 @@ with the following files:
 	::
 
 	  # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
-	  [mbm_event]
-	  default
+	  [default]
+	  mbm_event
 
 	"mbm_event":
 
@@ -375,19 +375,23 @@ with the following files:
 	to the events. Otherwise, the MBM event counters will return 'Unassigned' when read.
 
 	The mode is beneficial for AMD platforms that support more CTRL_MON
-	and MON groups than available hardware counters. By default, this
-	feature is enabled on AMD platforms with the ABMC (Assignable Bandwidth
-	Monitoring Counters) capability, ensuring counters remain assigned even
-	when the corresponding RMID is not actively used by any processor.
+	and MON groups than available hardware counters. On platforms with the
+	ABMC (Assignable Bandwidth Monitoring Counters) capability, mbm_event
+	mode ensures counters remain assigned even when the corresponding RMID
+	is not actively used by any processor.
 
 	"default":
 
 	In default mode, resctrl assumes there is a hardware counter for each
-	event within every CTRL_MON and MON group. On AMD platforms, it is
-	recommended to use the mbm_event mode, if supported, to prevent reset of MBM
-	events between reads resulting from hardware re-allocating counters. This can
-	result in misleading values or display "Unavailable" if no counter is assigned
-	to the event.
+	event within every CTRL_MON and MON group. This mode is enabled by default.
+
+	On AMD platforms with more CTRL_MON and MON groups than the available
+	hardware counters, hardware may re-allocate counters between reads
+	while in default mode. This can result in misleading memory bandwidth values
+	or display "Unavailable" if no counter is allocated to the event. In such
+	cases, it is recommended to use the mbm_event mode, if supported, to prevent
+	reset of MBM events between reads resulting from hardware re-allocating
+	counters.
 
 	* To enable "mbm_event" counter assignment mode:
 	  ::
@@ -471,8 +475,8 @@ with the following files:
 
 	Determines if a counter will automatically be assigned to an RMID, MBM event
 	pair when its associated monitor group is created via mkdir. Enabled by default
-	on boot, also when switched from "default" mode to "mbm_event" counter assignment
-	mode. Users can disable this capability by writing to the interface.
+	when switched to "mbm_event" counter assignment mode. Users can disable this
+	capability by writing to the interface.
 
 	"0":
 		Auto assignment is disabled.
@@ -1788,32 +1792,41 @@ a. Check if MBM counter assignment mode is supported.
 
   # mount -t resctrl resctrl /sys/fs/resctrl/
 
+  # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
+  [default]
+  mbm_event
+
+The "mbm_event" and "default" modes are supported. The "default" mode
+is enabled by default.
+
+b. Enable "mbm_event" counter assignment mode.
+::
+
+  # echo "mbm_event" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
   # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
   [mbm_event]
   default
 
-The "mbm_event" mode is detected and enabled.
-
-b. Check how many assignable counters are supported.
+c. Check how many assignable counters are supported.
 ::
 
   # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
   0=32;1=32
 
-c. Check how many assignable counters are available for assignment in each domain.
+d. Check how many assignable counters are available for assignment in each domain.
 ::
 
   # cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs
   0=30;1=30
 
-d. To list the default group's assign states.
+e. To list the default group's assign states.
 ::
 
   # cat /sys/fs/resctrl/mbm_L3_assignments
   mbm_total_bytes:0=e;1=e
   mbm_local_bytes:0=e;1=e
 
-e.  To unassign the counter associated with the mbm_total_bytes event on domain 0.
+f.  To unassign the counter associated with the mbm_total_bytes event on domain 0.
 ::
 
   # echo "mbm_total_bytes:0=_" > /sys/fs/resctrl/mbm_L3_assignments
@@ -1821,7 +1834,7 @@ e.  To unassign the counter associated with the mbm_total_bytes event on domain
   mbm_total_bytes:0=_;1=e
   mbm_local_bytes:0=e;1=e
 
-f. To unassign the counter associated with the mbm_total_bytes event on all domains.
+g. To unassign the counter associated with the mbm_total_bytes event on all domains.
 ::
 
   # echo "mbm_total_bytes:*=_" > /sys/fs/resctrl/mbm_L3_assignments
@@ -1829,7 +1842,7 @@ f. To unassign the counter associated with the mbm_total_bytes event on all doma
   mbm_total_bytes:0=_;1=_
   mbm_local_bytes:0=e;1=e
 
-g. To assign a counter associated with the mbm_total_bytes event on all domains in
+h. To assign a counter associated with the mbm_total_bytes event on all domains in
 exclusive mode.
 ::
 
@@ -1838,7 +1851,7 @@ exclusive mode.
   mbm_total_bytes:0=e;1=e
   mbm_local_bytes:0=e;1=e
 
-h. Read the events mbm_total_bytes and mbm_local_bytes of the default group. There is
+i. Read the events mbm_total_bytes and mbm_local_bytes of the default group. There is
 no change in reading the events with the assignment.
 ::
 
@@ -1851,7 +1864,7 @@ no change in reading the events with the assignment.
   # cat /sys/fs/resctrl/mon_data/mon_L3_01/mbm_local_bytes
   121212144
 
-i. Check the event configurations.
+j. Check the event configurations.
 ::
 
   # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter
@@ -1861,7 +1874,7 @@ i. Check the event configurations.
   # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter
   local_reads,local_non_temporal_writes,local_reads_slow_memory
 
-j. Change the event configuration for mbm_local_bytes.
+k. Change the event configuration for mbm_local_bytes.
 ::
 
   # echo "local_reads, local_non_temporal_writes, local_reads_slow_memory, remote_reads" >
@@ -1870,7 +1883,7 @@ j. Change the event configuration for mbm_local_bytes.
   # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter
   local_reads,local_non_temporal_writes,local_reads_slow_memory,remote_reads
 
-k. Now read the local events again. The first read may come back with "Unavailable"
+l. Now read the local events again. The first read may come back with "Unavailable"
 status. The subsequent read of mbm_local_bytes will display the current value.
 ::
 
@@ -1883,9 +1896,9 @@ status. The subsequent read of mbm_local_bytes will display the current value.
   # cat /sys/fs/resctrl/mon_data/mon_L3_01/mbm_local_bytes
   1566565
 
-l. Users have the option to go back to 'default' mbm_assign_mode if required. This can be
-done using the following command. Note that switching the mbm_assign_mode may reset all
-the MBM counters (and thus all MBM events) of all the resctrl groups.
+m. Users have the option to switch back to 'default' mbm_assign_mode if required. This
+can be done using the following command. Note that switching the mbm_assign_mode may
+reset all the MBM counters (and thus all MBM events) of all the resctrl groups.
 ::
 
   # echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
@@ -1893,7 +1906,7 @@ the MBM counters (and thus all MBM events) of all the resctrl groups.
   mbm_event
   [default]
 
-m. Unmount the resctrl filesystem.
+n. Unmount the resctrl filesystem.
 ::
 
   # umount /sys/fs/resctrl/
diff --git a/arch/x86/kernel/cpu/resctrl/monitor.c b/arch/x86/kernel/cpu/resctrl/monitor.c
index 3838e0a13d36..8a0d6086518b 100644
--- a/arch/x86/kernel/cpu/resctrl/monitor.c
+++ b/arch/x86/kernel/cpu/resctrl/monitor.c
@@ -471,7 +471,6 @@ int __init rdt_get_l3_mon_config(struct rdt_resource *r)
 		r->mon.mbm_cntr_configurable = true;
 		cpuid_count(0x80000020, 5, &eax, &ebx, &ecx, &edx);
 		r->mon.num_mbm_cntrs = (ebx & GENMASK(15, 0)) + 1;
-		hw_res->mbm_cntr_assign_enabled = true;
 	}
 
 	r->mon_capable = true;
-- 
2.43.0

[PATCH 2/2] x86/resctrl: Fix ABMC counter programming for extended counter ranges

[Babu Moger]

Memory Bandwidth Monitoring (MBM) can report incorrect values when ABMC is
enabled on systems supporting more than 32 ABMC counters. As the number of
active monitoring groups increases beyond the range supported by the
existing counter ID encoding, programming an ABMC counter may inadvertently
affect a different counter, resulting in unexpected counter resets and
abnormally large MBM readings.

The issue originates from the ABMC counter programming interface in the
L3_QOS_ABMC_CFG MSR. The counter ID field is currently defined as 5 bits,
which limits the addressable counter range to 32 counters. On systems
implementing more than 32 ABMC counters, counter IDs above 31 cannot be
encoded correctly. Consequently, programming a counter ID beyond the
supported range may target an unintended counter and reset bandwidth
statistics associated with another monitoring group.

While updating this logic, it was also observed that the bw_src field,
which encodes the RMID, is currently at its 12-bit limit with support for
4096 RMIDs. This field also needs to be updated for future expansion.

Also found one more pre-existing issue. This union structure can truncate
data on 32-bit x86 systems when unsigned long is used.

Fix the issues with the following changes:

1. Update the cntr_id field handling to support the full hardware ABMC
counter range and ensure that counter programming does not interfere with
unrelated counters.

2. Expand the bw_src field to 15 bits.

3. Change "unsigned long" to u64 to fix truncation on 32-bit x86.

The AMD64 Architecture Programmer's Manual [1] available at [2] will be
updated accordingly in a future revision to document the expanded cntr_id
and bw_src field definitions.

[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming,
    Publication #24593, Revision 3.41, Section 19.3.3.3 "Assignable
    Bandwidth Monitoring (ABMC)"

Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
Fixes: 84ecefb76674 ("x86/resctrl: Add data structures and definitions for ABMC assignment")
Signed-off-by: Babu Moger <babu.moger@amd.com>
---
 arch/x86/kernel/cpu/resctrl/internal.h | 18 ++++++++----------
 1 file changed, 8 insertions(+), 10 deletions(-)

diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
index e3cfa0c10e92..b3d780eda8a7 100644
--- a/arch/x86/kernel/cpu/resctrl/internal.h
+++ b/arch/x86/kernel/cpu/resctrl/internal.h
@@ -192,7 +192,6 @@ union cpuid_0x10_x_edx {
  * @bw_type		: Event configuration that represents the memory
  *			  transactions being tracked by the @cntr_id.
  * @bw_src		: Bandwidth source (RMID or CLOSID).
- * @reserved1		: Reserved.
  * @is_clos		: @bw_src field is a CLOSID (not an RMID).
  * @cntr_id		: Counter identifier.
  * @reserved		: Reserved.
@@ -210,16 +209,15 @@ union cpuid_0x10_x_edx {
  */
 union l3_qos_abmc_cfg {
 	struct {
-		unsigned long bw_type  :32,
-			      bw_src   :12,
-			      reserved1: 3,
-			      is_clos  : 1,
-			      cntr_id  : 5,
-			      reserved : 9,
-			      cntr_en  : 1,
-			      cfg_en   : 1;
+		u64	bw_type  :32,
+			bw_src   :15,
+			is_clos  : 1,
+			cntr_id  :12,
+			reserved : 2,
+			cntr_en  : 1,
+			cfg_en   : 1;
 	} split;
-	unsigned long full;
+	u64	full;
 };
 
 void rdt_ctrl_update(void *arg);
-- 
2.43.0

Re: [PATCH 1/2] x86/resctrl, Documentation: Keep mbm_assign_mode "default" on boot

[Reinette Chatre]

Hi Babu,

On 7/17/26 2:13 PM, Babu Moger wrote:
> The kernel currently enables the ABMC-based "mbm_event" mode by default on
> hardware that supports it. However, this can cause bandwidth monitoring
> failures with existing userspace tools such as pqos.
> 
> The pqos tool mounts the resctrl filesystem and creates 16 or more resctrl
> groups by default. On systems with 32 or fewer ABMC counters, this default
> configuration can consume all available counters, since each group requires
> one counter for local MBM and another for total MBM. If additional
> monitoring groups are created, counter resources are exhausted and pqos
> tool reports memory bandwidth counters as zero for those groups.

It is not obvious to me that this is a problem. If I understand correctly 
there are two scenarios possible with this pqos behavior:

- ABMC is not in use ("mbm_assign_mode" is set to "default")
  - pqos can create 16 or more monitor groups
  - hardware still supports a limited number of counters with consequence that
    underlying counters reset at any time as the different monitoring groups
    need to be tracked.
  - pqos can read monitoring data of all 16 monitor groups, sometimes reading the
    events would return "Unavailable", sometimes reading the events return data.
  - *None* of the monitoring numbers returned are guaranteed to be accurate.

- ABMC is in use ("mbm_assign_mode" is set to "mbm_event"):
  - pqos can create 16 or more monitor groups
  - only a subset of monitoring groups have counters assigned and these counters
    are guaranteed to only track the monitor groups/events they are assigned to
  - pqos can read monitoring data of all 16 monitor groups with two possibilities:
    - monitor group/event has counter assigned: monitoring numbers are guaranteed to be accurate
    - monitor group/event does not have counter assigned: monitoring numbers return 0

If my understanding is correct then the preference is to rather have wrong data than
see 0? This does not sound right. What am I missing?

The changelog starts with "this can cause bandwidth monitoring failures with existing
userspace tools such as pqos". How could returning accurate memory bandwidth data be
considered a failure? How does this issue manifest itself?

> 
> Avoid this compatibility issue by leaving mbm_assign_mode in the "default"
> mode during initialization. Users who want to use ABMC can continue to
> enable it explicitly:
> 
>  echo mbm_event > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
> 
> Update the resctrl documentation to reflect the new boot-time default and
> adjust the mbm_assign_mode examples accordingly.
> 
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---
> There are plans to enable "mbm_event" by default once additional counters
> are available. For now, keep the default mode to maintain compatibility
> with existing tools.

This is not ideal. resctrl should aim to provide a consistent user interface
across kernel versions. 

Reinette