[RFC PATCH] x86,fs/resctrl: Support for Global Bandwidth Enforcement and Priviledge Level Zero Association

[RFC PATCH] x86,fs/resctrl: Support for Global Bandwidth Enforcement and Priviledge Level Zero Association

[Babu Moger]

This patch series adds support for Global Bandwidth Enforcement (GLBE),
Global Slow Bandwidth Enforcement (GLSBE) and Priviledge Level Zero
Association (PLZA) in the x86 architecture's fs/resctrl subsystem. The
changes include modifications to the resctrl filesystem to allow users to
configure GLBE settings and associate CPUs and tasks with a separate CLOS
when executiing in CPL0.

The feature documentation is not yet publicly available, but it is expected
to be released in the next few weeks. In the meantime, a brief description
of the features is provided below. Sharing this series as an RFC to gather
initial feedback. Comments are welcome.

Global Bandwidth Enforcement (GLBE) 

AMD Global Bandwidth Enforcement (GLBE) provides a mechanism for software
to specify bandwidth limits for groups of threads that span multiple QOoS
Domains. This collection of QOS Domains is referred to as the GLBE Control
Domain.  The GLBE ceiling is a bandwidth ceiling for L3 External Bandwidth
competitively shared between all threads in a COS (Class of Service) across
all QOS Domains within the GLBE Control Domain.  This complements L3BE L3
External Bandwidth Enforcement (L3BE) which provides L3 eExternal Bandwidth
control on a per QOS Domain granularity.  

Global Slow Bandwidth Enforcement (GLSBE) 

AMD PQoS Global Slow Bandwidth Enforcement (GLSBE) provides a mechanism for
software to specify bandwidth limits for groups of threads that span
multiple QOS Domains. GLSBE operates within the same GLBE Control Domains
defined by GLBE.  The GLSBE ceiling is a bandwidth ceiling for L3 External
Bandwidth to Slow Memory competitively shared between all threads in a COS
in all QOS Domains within the GLBE Control Domain.  This complements L3SMBE
which provides Slow Memory bandwidth control on a per QOS Domain
granularity.  
 
Privilege Level Zero Association (PLZA) 

Privilege Level Zero Association (PLZA) allows the hardware to
automatically associate execution in Privilege Level Zero (CPL=0) with a
specific COS (Class of Service) and/or RMID (Resource Monitoring
Identifier). The QoS feature set already has a mechanism to associate
execution on each logical processor with an RMID or COS. PLZA allows the
system to override this per-thread association for a thread that is
executing with CPL=0. 

The patches are based on top of commit (v6.19-rc5)
Commit 0f61b1860cc3 (tag: v6.19-rc5, tip/tip/urgent) Linux 6.19-rc5
 
Changes include:        
 - Introduction of a new max_bandwidth file for each resctrl resource to
   expose the maximum supported bandwidth.
 - Addition of new schemata GMB and GSMBA interfaces for configuring GLBE
   and GSLBE parameters.
 - Modifications to associate resctrl groups with PLZA.
 - Documentation updates to describe the new functionality.

Interface Changes:
1. A new max_bandwidth file has been added under each resource type
   directory (for example, /sys/fs/resctrl/info/GMB/max_bandwidth) to
   report the maximum bandwidth supported by the resource.

2. New resource types, GMB and GSMBA, have been introduced and are exposed
   through the schemata interface:
   # cat /sys/fs/resctrl/schemata
     GSMBA:0=4096;1=4096
      SMBA:0=8192;1=8192
       GMB:0=4096;1=4096
        MB:0=8192;1=8192
        L3:0=ffff;1=ffff

3. A new plza_capable file has been added under each resource type directory
  (for example, /sys/fs/resctrl/info/GMB/plza_capable) to indicate whether
   the resource supports the PLZA feature.

4. A new plza control file has been added to each resctrl group (for example,
  /sys/fs/resctrl/plza) to enable or disable PLZA association for the group.
  Writing 1 enables PLZA for the group, while writing 0 disables it.


Babu Moger (1):
  x86,fs/resctrl: Add support for Global Bandwidth Enforcement (GLBE)

 Documentation/admin-guide/kernel-parameters.txt | 2 +-
 arch/x86/include/asm/cpufeatures.h              | 2 +-
 arch/x86/kernel/cpu/resctrl/core.c              | 2 ++
 arch/x86/kernel/cpu/scattered.c                 | 1 +
 4 files changed, 5 insertions(+), 2 deletions(-)

-- 
2.34.1

[RFC PATCH] x86,fs/resctrl: Add support for Global Bandwidth Enforcement (GLBE)

[Babu Moger]

On AMD systems, the existing MBA feature allows the user to set a bandwidth
limit for each QOS domain. However, multiple QOS domains share system
memory bandwidth as a resource. In order to ensure that system memory
bandwidth is not over-utilized, user must statically partition the
available system bandwidth between the active QOS domains. This typically
results in system memory being under-utilized since not all QOS domains are
using their full bandwidth Allocation.

AMD PQoS Global Bandwidth Enforcement(GLBE) provides a mechanism
for software to specify bandwidth limits for groups of threads that span
multiple QoS Domains. This collection of QOS domains is referred to as GLBE
control domain. The GLBE ceiling sets a maximum limit on a memory bandwidth
in GLBE control domain. Bandwidth is shared by all threads in a Class of
Service(COS) across every QoS domain managed by the GLBE control domain.

GLBE support is reported through CPUID.8000_0020_EBX_x0[GLBE] (bit 7).
When this bit is set to 1, the platform supports GLBE.

Since the AMD Memory Bandwidth Enforcement feature is represented as MBA,
the Global Bandwidth Enforcement feature will be shown as GMBA to maintain
consistent naming.

Add GMBA support to resctrl and introduce a kernel parameter that allows
enabling or disabling the feature at boot time.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
 Documentation/admin-guide/kernel-parameters.txt | 2 +-
 arch/x86/include/asm/cpufeatures.h              | 2 +-
 arch/x86/kernel/cpu/resctrl/core.c              | 2 ++
 arch/x86/kernel/cpu/scattered.c                 | 1 +
 4 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index abd77f39c783..e3058b3d47e9 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -6325,7 +6325,7 @@ Kernel parameters
 	rdt=		[HW,X86,RDT]
 			Turn on/off individual RDT features. List is:
 			cmt, mbmtotal, mbmlocal, l3cat, l3cdp, l2cat, l2cdp,
-			mba, smba, bmec, abmc, sdciae, energy[:guid],
+			mba, gmba, smba, bmec, abmc, sdciae, energy[:guid],
 			perf[:guid].
 			E.g. to turn on cmt and turn off mba use:
 				rdt=cmt,!mba
diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index c3b53beb1300..86d1339cd1bd 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -505,7 +505,6 @@
 #define X86_FEATURE_ABMC		(21*32+15) /* Assignable Bandwidth Monitoring Counters */
 #define X86_FEATURE_MSR_IMM		(21*32+16) /* MSR immediate form instructions */
 #define X86_FEATURE_SGX_EUPDATESVN	(21*32+17) /* Support for ENCLS[EUPDATESVN] instruction */
-
 #define X86_FEATURE_SDCIAE		(21*32+18) /* L3 Smart Data Cache Injection Allocation Enforcement */
 #define X86_FEATURE_CLEAR_CPU_BUF_VM_MMIO (21*32+19) /*
 						      * Clear CPU buffers before VM-Enter if the vCPU
@@ -513,6 +512,7 @@
 						      * and purposes if CLEAR_CPU_BUF_VM is set).
 						      */
 #define X86_FEATURE_X2AVIC_EXT		(21*32+20) /* AMD SVM x2AVIC support for 4k vCPUs */
+#define X86_FEATURE_GMBA		(21*32+21) /* Global Memory Bandwidth Allocation */
 
 /*
  * BUG word(s)
diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
index 9fcc06e9e72e..8b3457518ff4 100644
--- a/arch/x86/kernel/cpu/resctrl/core.c
+++ b/arch/x86/kernel/cpu/resctrl/core.c
@@ -795,6 +795,7 @@ enum {
 	RDT_FLAG_L2_CAT,
 	RDT_FLAG_L2_CDP,
 	RDT_FLAG_MBA,
+	RDT_FLAG_GMBA,
 	RDT_FLAG_SMBA,
 	RDT_FLAG_BMEC,
 	RDT_FLAG_ABMC,
@@ -822,6 +823,7 @@ static struct rdt_options rdt_options[]  __ro_after_init = {
 	RDT_OPT(RDT_FLAG_L2_CAT,    "l2cat",	X86_FEATURE_CAT_L2),
 	RDT_OPT(RDT_FLAG_L2_CDP,    "l2cdp",	X86_FEATURE_CDP_L2),
 	RDT_OPT(RDT_FLAG_MBA,	    "mba",	X86_FEATURE_MBA),
+	RDT_OPT(RDT_FLAG_GMBA,	    "gmba",	X86_FEATURE_GMBA),
 	RDT_OPT(RDT_FLAG_SMBA,	    "smba",	X86_FEATURE_SMBA),
 	RDT_OPT(RDT_FLAG_BMEC,	    "bmec",	X86_FEATURE_BMEC),
 	RDT_OPT(RDT_FLAG_ABMC,	    "abmc",	X86_FEATURE_ABMC),
diff --git a/arch/x86/kernel/cpu/scattered.c b/arch/x86/kernel/cpu/scattered.c
index 42c7eac0c387..d081d167bac9 100644
--- a/arch/x86/kernel/cpu/scattered.c
+++ b/arch/x86/kernel/cpu/scattered.c
@@ -59,6 +59,7 @@ static const struct cpuid_bit cpuid_bits[] = {
 	{ X86_FEATURE_BMEC,			CPUID_EBX,  3, 0x80000020, 0 },
 	{ X86_FEATURE_ABMC,			CPUID_EBX,  5, 0x80000020, 0 },
 	{ X86_FEATURE_SDCIAE,			CPUID_EBX,  6, 0x80000020, 0 },
+	{ X86_FEATURE_GMBA,			CPUID_EBX,  7, 0x80000020, 0 },
 	{ X86_FEATURE_TSA_SQ_NO,		CPUID_ECX,  1, 0x80000021, 0 },
 	{ X86_FEATURE_TSA_L1_NO,		CPUID_ECX,  2, 0x80000021, 0 },
 	{ X86_FEATURE_AMD_WORKLOAD_CLASS,	CPUID_EAX, 22, 0x80000021, 0 },
-- 
2.34.1