[PATCH v3 0/7] Support L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

[PATCH v3 0/7] Support L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

[Babu Moger]

This series adds the support for L3 Smart Data Cache Injection Allocation
Enforcement (SDCIAE) to resctrl infrastructure. It is refered as io_alloc
in resctrl subsystem.

Upcoming AMD hardware implements Smart Data Cache Injection (SDCI).
Smart Data Cache Injection (SDCI) is a mechanism that enables direct
insertion of data from I/O devices into the L3 cache. By directly caching
data from I/O devices rather than first storing the I/O data in DRAM, SDCI
reduces demands on DRAM bandwidth and reduces latency to the processor
consuming the I/O data.

The SDCIAE (SDCI Allocation Enforcement) PQE feature allows system software
to control the portion of the L3 cache used for SDCI devices.

When enabled, SDCIAE forces all SDCI lines to be placed into the L3 cache
partitions identified by the highest-supported L3_MASK_n register, where n
is the maximum supported CLOSID.

The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache
Injection Allocation Enforcement (SDCIAE)
Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537

The feature requires linux support of TPH (TLP Processing Hints).
The support is available in linux kernel after the commit
48d0fd2b903e3 ("PCI/TPH: Add TPH documentation")

The patches are based on top of commit
258b24fa741684 (tip/master) Merge branch into tip/master: 'x86/mm'

# Linux Implementation

Feature adds following interface files when the resctrl "io_alloc" feature is
supported:

/sys/fs/resctrl/info/L3/io_alloc: Reports the feature status. Feature can be
				  enabled/disabled by writing to the interface.

/sys/fs/resctrl/info/L3/io_alloc_cbm: Capacity Bit Masks (CBMs) available to SDCI
				      supported IO devices. CBM can be configured
				      by writing to the interface in the following
				      format::
# Examples

a. Check if io_alloc feature is available
	#mount -t resctrl resctrl /sys/fs/resctrl/

	# cat /sys/fs/resctrl/info/L3/io_alloc
	0

b. Enable the io_alloc feature. 

	# echo 1 > /sys/fs/resctrl/info/L3/io_alloc 

c. Check the CBM values for the io_alloc feature.

	# cat /sys/fs/resctrl/info/L3/io_alloc_cbm 
	L3:0=ffff;1=ffff

d. Change the CBM value for the domain 1:
	# echo L3:1=FF > /sys/fs/resctrl/info/L3/io_alloc_cbm

	# cat /sys/fs/resctrl/info/L3/io_alloc_cbm 
	L3:0=ffff;1=00ff

d. Disable io_alloc feature and exit.

	# echo 0 > /sys/fs/resctrl/info/L3/io_alloc 
	#umount /sys/fs/resctrl/

---
v3: Rewrote commit log for the last 3 patches. Changed the text to bit
    more generic than the AMD specific feature. Added AMD feature
    specifics in the end.

    Renamed the rdt_get_sdciae_alloc_cfg() to rdt_set_io_alloc_capable().
    Renamed the _resctrl_io_alloc_enable() to _resctrl_sdciae_enable()
    as it is arch specific.

    Changed the return to void in _resctrl_sdciae_enable() instead of int.
 
    The number of CLOSIDs is determined based on the minimum supported
    across all resources (in closid_init). It needs to match the max
    supported on the resource. Added the check to verify if MAX CLOSID
    availability on the system.

    Added CDP check to make sure io_alloc is configured in CDP_CODE.
    Highest CLOSID corresponds to CDP_CODE. 

    Added resctrl_io_alloc_closid_free() to free the io_alloc CLOSID.

    Added errors in few cases when CLOSID allocation fails.
    Fixes splat reported when info/L3/bit_usage is accesed when io_alloc is enabled.
    https://lore.kernel.org/lkml/SJ1PR11MB60837B532254E7B23BC27E84FC052@SJ1PR11MB6083.namprd11.prod.outlook.com/

v2: Added dependancy on X86_FEATURE_CAT_L3
    Removed the "" in CPU feature definition.

    Changed sdciae_capable to io_alloc_capable to make it as generic feature.
    Moved io_alloc_capable field in struct resctrl_cache.

    Changed the name of few arch functions similar to ABMC series.
    resctrl_arch_get_io_alloc_enabled()
    resctrl_arch_io_alloc_enable()

    Renamed the feature to "io_alloc".
    Added generic texts for the feature in commit log and resctrl.rst doc.
    Added resctrl_io_alloc_init_cat() to initialize io_alloc to default values
    when enabled.
    Fixed io_alloc interface to show only on L3 resource.
    Added the locks while processing io_alloc CBMs.

Previous versions:
v2: https://lore.kernel.org/lkml/cover.1734556832.git.babu.moger@amd.com/
v1: https://lore.kernel.org/lkml/cover.1723824984.git.babu.moger@amd.com/


Babu Moger (7):
  x86/cpufeatures: Add support for L3 Smart Data Cache Injection
    Allocation Enforcement
  x86/resctrl: Add SDCIAE feature in the command line options
  x86/resctrl: Detect io_alloc feature
  x86/resctrl: Implement "io_alloc" enable/disable handlers
  x86/resctrl: Add interface to enable/disable io_alloc feature
  x86/resctrl: Introduce interface to display io_alloc CBMs
  x86/resctrl: Introduce interface to modify io_alloc Capacity Bit Masks

 .../admin-guide/kernel-parameters.txt         |   2 +-
 Documentation/arch/x86/resctrl.rst            |  46 +++
 arch/x86/include/asm/cpufeatures.h            |   1 +
 arch/x86/include/asm/msr-index.h              |   1 +
 arch/x86/kernel/cpu/cpuid-deps.c              |   1 +
 arch/x86/kernel/cpu/resctrl/core.c            |  13 +
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c     |   4 +-
 arch/x86/kernel/cpu/resctrl/internal.h        |  12 +
 arch/x86/kernel/cpu/resctrl/rdtgroup.c        | 346 ++++++++++++++++++
 arch/x86/kernel/cpu/scattered.c               |   1 +
 include/linux/resctrl.h                       |  12 +
 11 files changed, 436 insertions(+), 3 deletions(-)

-- 
2.34.1

[PATCH v3 1/7] x86/cpufeatures: Add support for L3 Smart Data Cache Injection Allocation Enforcement

[Babu Moger]

Smart Data Cache Injection (SDCI) is a mechanism that enables direct
insertion of data from I/O devices into the L3 cache. By directly caching
data from I/O devices rather than first storing the I/O data in DRAM,
SDCI reduces demands on DRAM bandwidth and reduces latency to the processor
consuming the I/O data.

The SDCIAE (SDCI Allocation Enforcement) PQE feature allows system software
to control the portion of the L3 cache used for SDCI.

When enabled, SDCIAE forces all SDCI lines to be placed into the L3 cache
partitions identified by the highest-supported L3_MASK_n register, where n
is the maximum supported CLOSID.

Add CPUID feature bit that can be used to configure SDCIAE.

The feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache
Injection Allocation Enforcement (SDCIAE)

Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v3: No changes.

v2: Added dependancy on X86_FEATURE_CAT_L3
    Removed the "" in CPU feature definition.
    Minor text changes.
---
 arch/x86/include/asm/cpufeatures.h | 1 +
 arch/x86/kernel/cpu/cpuid-deps.c   | 1 +
 arch/x86/kernel/cpu/scattered.c    | 1 +
 3 files changed, 3 insertions(+)

diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index 508c0dad116b..a738617b1910 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -483,6 +483,7 @@
 #define X86_FEATURE_AMD_FAST_CPPC	(21*32 + 5) /* Fast CPPC */
 #define X86_FEATURE_AMD_HETEROGENEOUS_CORES (21*32 + 6) /* Heterogeneous Core Topology */
 #define X86_FEATURE_AMD_WORKLOAD_CLASS	(21*32 + 7) /* Workload Classification */
+#define X86_FEATURE_SDCIAE		(21*32 + 8) /* L3 Smart Data Cache Injection Allocation Enforcement */
 
 /*
  * BUG word(s)
diff --git a/arch/x86/kernel/cpu/cpuid-deps.c b/arch/x86/kernel/cpu/cpuid-deps.c
index 8bd84114c2d9..8185521ce854 100644
--- a/arch/x86/kernel/cpu/cpuid-deps.c
+++ b/arch/x86/kernel/cpu/cpuid-deps.c
@@ -70,6 +70,7 @@ static const struct cpuid_dep cpuid_deps[] = {
 	{ X86_FEATURE_CQM_MBM_LOCAL,		X86_FEATURE_CQM_LLC   },
 	{ X86_FEATURE_BMEC,			X86_FEATURE_CQM_MBM_TOTAL   },
 	{ X86_FEATURE_BMEC,			X86_FEATURE_CQM_MBM_LOCAL   },
+	{ X86_FEATURE_SDCIAE,			X86_FEATURE_CAT_L3    },
 	{ X86_FEATURE_AVX512_BF16,		X86_FEATURE_AVX512VL  },
 	{ X86_FEATURE_AVX512_FP16,		X86_FEATURE_AVX512BW  },
 	{ X86_FEATURE_ENQCMD,			X86_FEATURE_XSAVES    },
diff --git a/arch/x86/kernel/cpu/scattered.c b/arch/x86/kernel/cpu/scattered.c
index 16f3ca30626a..d18a7ce16388 100644
--- a/arch/x86/kernel/cpu/scattered.c
+++ b/arch/x86/kernel/cpu/scattered.c
@@ -49,6 +49,7 @@ static const struct cpuid_bit cpuid_bits[] = {
 	{ X86_FEATURE_MBA,			CPUID_EBX,  6, 0x80000008, 0 },
 	{ X86_FEATURE_SMBA,			CPUID_EBX,  2, 0x80000020, 0 },
 	{ X86_FEATURE_BMEC,			CPUID_EBX,  3, 0x80000020, 0 },
+	{ X86_FEATURE_SDCIAE,			CPUID_EBX,  6, 0x80000020, 0 },
 	{ X86_FEATURE_AMD_WORKLOAD_CLASS,	CPUID_EAX, 22, 0x80000021, 0 },
 	{ X86_FEATURE_PERFMON_V2,		CPUID_EAX,  0, 0x80000022, 0 },
 	{ X86_FEATURE_AMD_LBR_V2,		CPUID_EAX,  1, 0x80000022, 0 },
-- 
2.34.1

[PATCH v3 2/7] x86/resctrl: Add SDCIAE feature in the command line options

[Babu Moger]

Add the command line options to enable or disable the new resctrl feature
L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE).

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v3: No changes.

v2: No changes.
---
 Documentation/admin-guide/kernel-parameters.txt | 2 +-
 arch/x86/kernel/cpu/resctrl/core.c              | 2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index d0f6c055dfcc..0077c4340d10 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -5942,7 +5942,7 @@
 	rdt=		[HW,X86,RDT]
 			Turn on/off individual RDT features. List is:
 			cmt, mbmtotal, mbmlocal, l3cat, l3cdp, l2cat, l2cdp,
-			mba, smba, bmec.
+			mba, smba, bmec, sdciae.
 			E.g. to turn on cmt and turn off mba use:
 				rdt=cmt,!mba
 
diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
index 3d1735ed8d1f..c2450cd52511 100644
--- a/arch/x86/kernel/cpu/resctrl/core.c
+++ b/arch/x86/kernel/cpu/resctrl/core.c
@@ -811,6 +811,7 @@ enum {
 	RDT_FLAG_MBA,
 	RDT_FLAG_SMBA,
 	RDT_FLAG_BMEC,
+	RDT_FLAG_SDCIAE,
 };
 
 #define RDT_OPT(idx, n, f)	\
@@ -836,6 +837,7 @@ static struct rdt_options rdt_options[]  __initdata = {
 	RDT_OPT(RDT_FLAG_MBA,	    "mba",	X86_FEATURE_MBA),
 	RDT_OPT(RDT_FLAG_SMBA,	    "smba",	X86_FEATURE_SMBA),
 	RDT_OPT(RDT_FLAG_BMEC,	    "bmec",	X86_FEATURE_BMEC),
+	RDT_OPT(RDT_FLAG_SDCIAE,    "sdciae",	X86_FEATURE_SDCIAE),
 };
 #define NUM_RDT_OPTIONS ARRAY_SIZE(rdt_options)
 
-- 
2.34.1

[PATCH v3 3/7] x86/resctrl: Detect io_alloc feature

[Babu Moger]

"io_alloc" feature is a mechanism that enables direct insertion of data
from I/O devices into the L3 cache. By directly caching data from I/O
devices rather than first storing the I/O data in DRAM, feature reduces
the demands on DRAM bandwidth and reduces latency to the processor
consuming the I/O data.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v3: Rewrote commit log. Changed the text to bit generic than the AMD specific.
    Renamed the rdt_get_sdciae_alloc_cfg() to rdt_set_io_alloc_capable().
    Removed leftover comment from v2.

v2: Changed sdciae_capable to io_alloc_capable to make it generic feature.
    Also moved the io_alloc_capable in struct resctrl_cache.
---
 arch/x86/kernel/cpu/resctrl/core.c | 7 +++++++
 include/linux/resctrl.h            | 3 +++
 2 files changed, 10 insertions(+)

diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
index c2450cd52511..1ebdb2dcc009 100644
--- a/arch/x86/kernel/cpu/resctrl/core.c
+++ b/arch/x86/kernel/cpu/resctrl/core.c
@@ -306,6 +306,11 @@ static void rdt_get_cdp_config(int level)
 	rdt_resources_all[level].r_resctrl.cdp_capable = true;
 }
 
+static void rdt_set_io_alloc_capable(struct rdt_resource *r)
+{
+	r->cache.io_alloc_capable = true;
+}
+
 static void rdt_get_cdp_l3_config(void)
 {
 	rdt_get_cdp_config(RDT_RESOURCE_L3);
@@ -931,6 +936,8 @@ static __init bool get_rdt_alloc_resources(void)
 		rdt_get_cache_alloc_cfg(1, r);
 		if (rdt_cpu_has(X86_FEATURE_CDP_L3))
 			rdt_get_cdp_l3_config();
+		if (rdt_cpu_has(X86_FEATURE_SDCIAE))
+			rdt_set_io_alloc_capable(r);
 		ret = true;
 	}
 	if (rdt_cpu_has(X86_FEATURE_CAT_L2)) {
diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h
index d94abba1c716..dbe6461f3fbc 100644
--- a/include/linux/resctrl.h
+++ b/include/linux/resctrl.h
@@ -129,6 +129,8 @@ struct rdt_mon_domain {
  * @arch_has_sparse_bitmasks:	True if a bitmask like f00f is valid.
  * @arch_has_per_cpu_cfg:	True if QOS_CFG register for this cache
  *				level has CPU scope.
+ * @io_alloc_capable:	True if portion of the L3 cache can be allocated
+ *			for I/O traffic.
  */
 struct resctrl_cache {
 	unsigned int	cbm_len;
@@ -136,6 +138,7 @@ struct resctrl_cache {
 	unsigned int	shareable_bits;
 	bool		arch_has_sparse_bitmasks;
 	bool		arch_has_per_cpu_cfg;
+	bool		io_alloc_capable;
 };
 
 /**
-- 
2.34.1

[PATCH v3 4/7] x86/resctrl: Implement "io_alloc" enable/disable handlers

[Babu Moger]

"io_alloc" feature that enables direct insertion of data from I/O devices
into the L3 cache.

On AMD, 'io_alloc" feature is backed by L3 Smart Data Cache Injection
Allocation Enforcement (SDCIAE). SDCIAE feature can be enabled by setting
bit 1 in MSR L3_QOS_EXT_CFG. Apply the updated SDCIAE value across all
logical processors within the QOS domain when modifying its state.

Introduce architecture-specific handlers to manage the detection and
enabling/disabling of 'io_alloc" feature.

The SDCIAE feature details are available in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache
Injection Allocation Enforcement (SDCIAE)

Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v3: Passed the struct rdt_resource to resctrl_arch_get_io_alloc_enabled() instead of resource id.
    Renamed the _resctrl_io_alloc_enable() to _resctrl_sdciae_enable() as it is arch specific.
    Changed the return to void in _resctrl_sdciae_enable() instead of int.
    Added more context in commit log and fixed few typos.

v2: Renamed the functions to simplify the code.
    Renamed sdciae_capable to io_alloc_capable.

    Changed the name of few arch functions similar to ABMC series.
    resctrl_arch_get_io_alloc_enabled()
    resctrl_arch_io_alloc_enable()
---
 arch/x86/include/asm/msr-index.h       |  1 +
 arch/x86/kernel/cpu/resctrl/internal.h | 10 ++++++++
 arch/x86/kernel/cpu/resctrl/rdtgroup.c | 32 ++++++++++++++++++++++++++
 include/linux/resctrl.h                |  9 ++++++++
 4 files changed, 52 insertions(+)

diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
index 9a71880eec07..fea1f3afe197 100644
--- a/arch/x86/include/asm/msr-index.h
+++ b/arch/x86/include/asm/msr-index.h
@@ -1197,6 +1197,7 @@
 /* - AMD: */
 #define MSR_IA32_MBA_BW_BASE		0xc0000200
 #define MSR_IA32_SMBA_BW_BASE		0xc0000280
+#define MSR_IA32_L3_QOS_EXT_CFG		0xc00003ff
 #define MSR_IA32_EVT_CFG_BASE		0xc0000400
 
 /* AMD-V MSRs */
diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
index 20c898f09b7e..61bc609e932b 100644
--- a/arch/x86/kernel/cpu/resctrl/internal.h
+++ b/arch/x86/kernel/cpu/resctrl/internal.h
@@ -56,6 +56,9 @@
 /* Max event bits supported */
 #define MAX_EVT_CONFIG_BITS		GENMASK(6, 0)
 
+/* Setting bit 1 in L3_QOS_EXT_CFG enables the SDCIAE feature. */
+#define SDCIAE_ENABLE_BIT		1
+
 /**
  * cpumask_any_housekeeping() - Choose any CPU in @mask, preferring those that
  *			        aren't marked nohz_full
@@ -479,6 +482,7 @@ struct rdt_parse_data {
  * @mbm_cfg_mask:	Bandwidth sources that can be tracked when Bandwidth
  *			Monitoring Event Configuration (BMEC) is supported.
  * @cdp_enabled:	CDP state of this resource
+ * @sdciae_enabled:	SDCIAE feature is enabled
  *
  * Members of this structure are either private to the architecture
  * e.g. mbm_width, or accessed via helpers that provide abstraction. e.g.
@@ -493,6 +497,7 @@ struct rdt_hw_resource {
 	unsigned int		mbm_width;
 	unsigned int		mbm_cfg_mask;
 	bool			cdp_enabled;
+	bool			sdciae_enabled;
 };
 
 static inline struct rdt_hw_resource *resctrl_to_arch_res(struct rdt_resource *r)
@@ -539,6 +544,11 @@ int resctrl_arch_set_cdp_enabled(enum resctrl_res_level l, bool enable);
 
 void arch_mon_domain_online(struct rdt_resource *r, struct rdt_mon_domain *d);
 
+static inline bool resctrl_arch_get_io_alloc_enabled(struct rdt_resource *r)
+{
+	return resctrl_to_arch_res(r)->sdciae_enabled;
+}
+
 /*
  * To return the common struct rdt_resource, which is contained in struct
  * rdt_hw_resource, walk the resctrl member of struct rdt_hw_resource.
diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
index 6419e04d8a7b..c5a0a31c3a85 100644
--- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
+++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
@@ -1798,6 +1798,38 @@ static ssize_t mbm_local_bytes_config_write(struct kernfs_open_file *of,
 	return ret ?: nbytes;
 }
 
+static void resctrl_sdciae_set_one_amd(void *arg)
+{
+	bool *enable = arg;
+
+	if (*enable)
+		msr_set_bit(MSR_IA32_L3_QOS_EXT_CFG, SDCIAE_ENABLE_BIT);
+	else
+		msr_clear_bit(MSR_IA32_L3_QOS_EXT_CFG, SDCIAE_ENABLE_BIT);
+}
+
+static void _resctrl_sdciae_enable(struct rdt_resource *r, bool enable)
+{
+	struct rdt_ctrl_domain *d;
+
+	/* Update L3_QOS_EXT_CFG MSR on all the CPUs in all domains*/
+	list_for_each_entry(d, &r->ctrl_domains, hdr.list)
+		on_each_cpu_mask(&d->hdr.cpu_mask, resctrl_sdciae_set_one_amd, &enable, 1);
+}
+
+int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable)
+{
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
+
+	if (hw_res->r_resctrl.cache.io_alloc_capable &&
+	    hw_res->sdciae_enabled != enable) {
+		_resctrl_sdciae_enable(r, enable);
+		hw_res->sdciae_enabled = enable;
+	}
+
+	return 0;
+}
+
 /* rdtgroup information files for one cache resource. */
 static struct rftype res_common_files[] = {
 	{
diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h
index dbe6461f3fbc..e77c3b37bad4 100644
--- a/include/linux/resctrl.h
+++ b/include/linux/resctrl.h
@@ -343,6 +343,15 @@ void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,
  */
 void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain *d);
 
+/**
+ * resctrl_arch_io_alloc_enable() - Enable/disable io_alloc feature.
+ * @r:		The resctrl resource.
+ * @enable:	Enable (1) or disable (0) the feature
+ *
+ * This can be called from any CPU.
+ */
+int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable);
+
 extern unsigned int resctrl_rmid_realloc_threshold;
 extern unsigned int resctrl_rmid_realloc_limit;
 
-- 
2.34.1

[PATCH v3 5/7] x86/resctrl: Add interface to enable/disable io_alloc feature

[Babu Moger]

The io_alloc feature in resctrl is a mechanism that enables direct
insertion of data from I/O devices into the L3 cache.

On AMD systems, io_alloc feature is backed by SDCIAE (L3 Smart Data Cache
Injection Allocation Enforcement). When enabled, SDCIAE forces all SDCI
lines to be placed into the L3 cache partitions identified by the
highest-supported L3_MASK_n register as reported by CPUID
Fn0000_0010_EDX_x1.MAX_COS. For example, if MAX_COS=15, SDCI lines will
be allocated into the L3 cache partitions determined by the bitmask in
the L3_MASK_15 register.

When CDP is enabled, io_alloc routes I/O traffic using the highest CLOSID
allocated for the instruction cache.

Introduce interface to enable/disable "io_alloc" feature on user input.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v3: Rewrote the change to make it generic.
    Rewrote the documentation in resctrl.rst to be generic and added
    AMD feature details in the end.
    Added the check to verify if MAX CLOSID availability on the system.
    Added CDP check to make sure io_alloc is configured in CDP_CODE.
    Added resctrl_io_alloc_closid_free() to free the io_alloc CLOSID.
    Added errors in few cases when CLOSID allocation fails.
    Fixes splat reported when info/L3/bit_usage is accesed when io_alloc
    is enabled.
    https://lore.kernel.org/lkml/SJ1PR11MB60837B532254E7B23BC27E84FC052@SJ1PR11MB6083.namprd11.prod.outlook.com/

v2: Renamed the feature to "io_alloc".
    Added generic texts for the feature in commit log and resctrl.rst doc.
    Added resctrl_io_alloc_init_cat() to initialize io_alloc to default
    values when enabled.
    Fixed io_alloc show functinality to display only on L3 resource.
---
 Documentation/arch/x86/resctrl.rst     |  34 ++++++
 arch/x86/kernel/cpu/resctrl/core.c     |   2 +
 arch/x86/kernel/cpu/resctrl/rdtgroup.c | 144 +++++++++++++++++++++++++
 3 files changed, 180 insertions(+)

diff --git a/Documentation/arch/x86/resctrl.rst b/Documentation/arch/x86/resctrl.rst
index 6768fc1fad16..1b67e31d626c 100644
--- a/Documentation/arch/x86/resctrl.rst
+++ b/Documentation/arch/x86/resctrl.rst
@@ -135,6 +135,40 @@ related to allocation:
 			"1":
 			      Non-contiguous 1s value in CBM is supported.
 
+"io_alloc":
+		The "io_alloc" feature in resctrl enables system software to
+		configure the portion of the L3 cache allocated for I/O traffic.
+		By directly caching data from I/O devices rather than first storing
+		the I/O data in DRAM, reduces the demands on DRAM bandwidth and
+		reduces latency to the processor consuming the I/O data.
+
+		The feature routes the I/O traffic via specific CLOSID reserved
+		for io_alloc feature. By configuring the CBM (Capacity Bit Mask)
+		for the CLOSID users can control the L3 portions available for
+		I/O traffic. When enabled, CLOSID reserved for the io_alloc will
+		not be available to the resctrl group.
+		::
+
+		  # cat /sys/fs/resctrl/info/L3/io_alloc
+		  0
+
+		  "0":
+		      io_alloc feature is not enabled.
+		  "1":
+		      io_alloc feature is enabled, allowing users to manage
+		      the portions of the L3 cache allocated for the I/O device.
+
+		Feature can be enabled/disabled by writing to the interface.
+		Example::
+
+			# echo 1 > /sys/fs/resctrl/info/L3/io_alloc
+
+		On AMD systems, the io_alloc feature is supported by the L3 Smart
+		Data Cache Injection Allocation Enforcement (SDCIAE). The CLOSID for
+		io_alloc is determined by the highest CLOSID supported by the resource.
+		When CDP is enabled, io_alloc routes I/O traffic using the highest
+		CLOSID allocated for the instruction cache.
+
 Memory bandwidth(MB) subdirectory contains the following files
 with respect to allocation:
 
diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
index 1ebdb2dcc009..88bc95c14ea8 100644
--- a/arch/x86/kernel/cpu/resctrl/core.c
+++ b/arch/x86/kernel/cpu/resctrl/core.c
@@ -309,6 +309,8 @@ static void rdt_get_cdp_config(int level)
 static void rdt_set_io_alloc_capable(struct rdt_resource *r)
 {
 	r->cache.io_alloc_capable = true;
+	resctrl_file_fflags_init("io_alloc",
+				 RFTYPE_CTRL_INFO | RFTYPE_RES_CACHE);
 }
 
 static void rdt_get_cdp_l3_config(void)
diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
index c5a0a31c3a85..37295dd14abe 100644
--- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
+++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
@@ -62,6 +62,7 @@ static char last_cmd_status_buf[512];
 
 static int rdtgroup_setup_root(struct rdt_fs_context *ctx);
 static void rdtgroup_destroy_root(void);
+static int rdtgroup_init_cat(struct resctrl_schema *s, u32 closid);
 
 struct dentry *debugfs_resctrl;
 
@@ -180,6 +181,19 @@ void closid_free(int closid)
 	__set_bit(closid, &closid_free_map);
 }
 
+static int resctrl_io_alloc_closid_alloc(u32 io_alloc_closid)
+{
+	if (__test_and_clear_bit(io_alloc_closid, &closid_free_map))
+		return io_alloc_closid;
+	else
+		return -ENOSPC;
+}
+
+static void resctrl_io_alloc_closid_free(u32 io_alloc_closid)
+{
+	closid_free(io_alloc_closid);
+}
+
 /**
  * closid_allocated - test if provided closid is in use
  * @closid: closid to be tested
@@ -995,6 +1009,33 @@ static int rdt_shareable_bits_show(struct kernfs_open_file *of,
 	return 0;
 }
 
+/*
+ * io_alloc feature uses max CLOSID to route the IO traffic.
+ * Get the max CLOSID and verify if the CLOSID is available.
+ */
+static int resctrl_io_alloc_closid_get(struct rdt_resource *r,
+				       struct resctrl_schema *s)
+{
+	int num_closids = resctrl_arch_get_num_closid(r);
+
+	/*
+	 * The number of CLOSIDs is determined based on the minimum
+	 * supported across all resources (in closid_init). It is stored
+	 * in s->num_closids. Also, if CDP is enabled number of CLOSIDs
+	 * are halved. To enable io_alloc feature, the number of CLOSIDs
+	 * must match the maximum CLOSID supported by the resource.
+	 */
+	if (resctrl_arch_get_cdp_enabled(r->rid))
+		num_closids /= 2;
+
+	if (s->num_closid != num_closids) {
+		rdt_last_cmd_puts("Max CLOSID to support io_alloc is not available\n");
+		return -ENOSPC;
+	}
+
+	return num_closids - 1;
+}
+
 /*
  * rdt_bit_usage_show - Display current usage of resources
  *
@@ -1038,6 +1079,14 @@ static int rdt_bit_usage_show(struct kernfs_open_file *of,
 		for (i = 0; i < closids_supported(); i++) {
 			if (!closid_allocated(i))
 				continue;
+			/*
+			 * If io_alloc is enabled, the CLOSID will be
+			 * allocated but will not be associated with any
+			 * groups. Skip in that case.
+			 */
+			if (i == resctrl_io_alloc_closid_get(r, s) &&
+			    resctrl_arch_get_io_alloc_enabled(r))
+				continue;
 			ctrl_val = resctrl_arch_get_config(r, dom, i,
 							   s->conf_type);
 			mode = rdtgroup_mode_by_closid(i);
@@ -1830,6 +1879,94 @@ int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable)
 	return 0;
 }
 
+static int resctrl_io_alloc_show(struct kernfs_open_file *of,
+				 struct seq_file *seq, void *v)
+{
+	struct resctrl_schema *s = of->kn->parent->priv;
+	struct rdt_resource *r = s->res;
+
+	seq_printf(seq, "%x\n", resctrl_arch_get_io_alloc_enabled(r));
+	return 0;
+}
+
+/*
+ * Initialize io_alloc CLOSID cache resource with default CBM values.
+ */
+static int resctrl_io_alloc_init_cat(struct rdt_resource *r,
+				     struct resctrl_schema *s, u32 closid)
+{
+	int ret;
+
+	rdt_staged_configs_clear();
+
+	ret = rdtgroup_init_cat(s, closid);
+	if (ret < 0)
+		goto out_init_cat;
+
+	ret = resctrl_arch_update_domains(r, closid);
+
+out_init_cat:
+	rdt_staged_configs_clear();
+	return ret;
+}
+
+static ssize_t resctrl_io_alloc_write(struct kernfs_open_file *of, char *buf,
+				      size_t nbytes, loff_t off)
+{
+	struct resctrl_schema *s = of->kn->parent->priv;
+	struct rdt_resource *r = s->res;
+	u32 io_alloc_closid;
+	bool enable;
+	int ret;
+
+	if (!r->cache.io_alloc_capable || s->conf_type == CDP_DATA) {
+		rdt_last_cmd_puts("io_alloc feature is not supported on the resource\n");
+		return -EINVAL;
+	}
+
+	ret = kstrtobool(buf, &enable);
+	if (ret)
+		return ret;
+
+	cpus_read_lock();
+	mutex_lock(&rdtgroup_mutex);
+
+	rdt_last_cmd_clear();
+
+	io_alloc_closid = resctrl_io_alloc_closid_get(r, s);
+	if (io_alloc_closid < 0) {
+		ret = -EINVAL;
+		goto out_io_alloc;
+	}
+
+	if (resctrl_arch_get_io_alloc_enabled(r) != enable) {
+		if (enable) {
+			ret = resctrl_io_alloc_closid_alloc(io_alloc_closid);
+			if (ret < 0) {
+				rdt_last_cmd_puts("CLOSID for io_alloc is not available\n");
+				goto out_io_alloc;
+			}
+			ret = resctrl_io_alloc_init_cat(r, s, io_alloc_closid);
+			if (ret) {
+				rdt_last_cmd_puts("Failed to initialize io_alloc allocations\n");
+				resctrl_io_alloc_closid_free(io_alloc_closid);
+				goto out_io_alloc;
+			}
+
+		} else {
+			resctrl_io_alloc_closid_free(io_alloc_closid);
+		}
+
+		ret = resctrl_arch_io_alloc_enable(r, enable);
+	}
+
+out_io_alloc:
+	mutex_unlock(&rdtgroup_mutex);
+	cpus_read_unlock();
+
+	return ret ?: nbytes;
+}
+
 /* rdtgroup information files for one cache resource. */
 static struct rftype res_common_files[] = {
 	{
@@ -1982,6 +2119,13 @@ static struct rftype res_common_files[] = {
 		.seq_show	= rdtgroup_schemata_show,
 		.fflags		= RFTYPE_CTRL_BASE,
 	},
+	{
+		.name		= "io_alloc",
+		.mode		= 0644,
+		.kf_ops		= &rdtgroup_kf_single_ops,
+		.seq_show	= resctrl_io_alloc_show,
+		.write		= resctrl_io_alloc_write,
+	},
 	{
 		.name		= "mba_MBps_event",
 		.mode		= 0644,
-- 
2.34.1

[PATCH v3 6/7] x86/resctrl: Introduce interface to display io_alloc CBMs

[Babu Moger]

The io_alloc feature in resctrl enables system software to configure
the portion of the L3 cache allocated for I/O traffic.

Add the interface to display CBMs (Capacity Bit Mask) of io_alloc
feature.

When CDP is enabled, io_alloc routes traffic using the highest CLOSID
which corresponds to CDP_CODE. Add a check for the CDP resource type.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v3: Minor changes due to changes in resctrl_arch_get_io_alloc_enabled()
    and resctrl_io_alloc_closid_get().
    Added the check to verify CDP resource type.
    Updated the commit log.

v2: Fixed to display only on L3 resources.
    Added the locks while processing.
    Rename the displat to io_alloc_cbm (from sdciae_cmd).
---
 arch/x86/kernel/cpu/resctrl/core.c        |  2 ++
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c |  2 +-
 arch/x86/kernel/cpu/resctrl/internal.h    |  1 +
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 38 +++++++++++++++++++++++
 4 files changed, 42 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
index 88bc95c14ea8..030f738dea8d 100644
--- a/arch/x86/kernel/cpu/resctrl/core.c
+++ b/arch/x86/kernel/cpu/resctrl/core.c
@@ -311,6 +311,8 @@ static void rdt_set_io_alloc_capable(struct rdt_resource *r)
 	r->cache.io_alloc_capable = true;
 	resctrl_file_fflags_init("io_alloc",
 				 RFTYPE_CTRL_INFO | RFTYPE_RES_CACHE);
+	resctrl_file_fflags_init("io_alloc_cbm",
+				 RFTYPE_CTRL_INFO | RFTYPE_RES_CACHE);
 }
 
 static void rdt_get_cdp_l3_config(void)
diff --git a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
index 536351159cc2..d272dea43924 100644
--- a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
+++ b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
@@ -444,7 +444,7 @@ u32 resctrl_arch_get_config(struct rdt_resource *r, struct rdt_ctrl_domain *d,
 	return hw_dom->ctrl_val[idx];
 }
 
-static void show_doms(struct seq_file *s, struct resctrl_schema *schema, int closid)
+void show_doms(struct seq_file *s, struct resctrl_schema *schema, int closid)
 {
 	struct rdt_resource *r = schema->res;
 	struct rdt_ctrl_domain *dom;
diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
index 61bc609e932b..07cf8409174d 100644
--- a/arch/x86/kernel/cpu/resctrl/internal.h
+++ b/arch/x86/kernel/cpu/resctrl/internal.h
@@ -668,4 +668,5 @@ void resctrl_file_fflags_init(const char *config, unsigned long fflags);
 void rdt_staged_configs_clear(void);
 bool closid_allocated(unsigned int closid);
 int resctrl_find_cleanest_closid(void);
+void show_doms(struct seq_file *s, struct resctrl_schema *schema, int closid);
 #endif /* _ASM_X86_RESCTRL_INTERNAL_H */
diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
index 37295dd14abe..81b9d8c5dabf 100644
--- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
+++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
@@ -1967,6 +1967,38 @@ static ssize_t resctrl_io_alloc_write(struct kernfs_open_file *of, char *buf,
 	return ret ?: nbytes;
 }
 
+static int resctrl_io_alloc_cbm_show(struct kernfs_open_file *of,
+				     struct seq_file *seq, void *v)
+{
+	struct resctrl_schema *s = of->kn->parent->priv;
+	struct rdt_resource *r = s->res;
+	u32 io_alloc_closid;
+	int ret = 0;
+
+	if (!r->cache.io_alloc_capable || s->conf_type == CDP_DATA) {
+		rdt_last_cmd_puts("io_alloc feature is not supported on the resource\n");
+		return -EINVAL;
+	}
+
+	cpus_read_lock();
+	mutex_lock(&rdtgroup_mutex);
+
+	if (!resctrl_arch_get_io_alloc_enabled(r)) {
+		rdt_last_cmd_puts("io_alloc feature is not enabled\n");
+		ret = -EINVAL;
+		goto cbm_show_out;
+	}
+
+	io_alloc_closid = resctrl_io_alloc_closid_get(r, s);
+
+	show_doms(seq, s, io_alloc_closid);
+
+cbm_show_out:
+	mutex_unlock(&rdtgroup_mutex);
+	cpus_read_unlock();
+	return ret;
+}
+
 /* rdtgroup information files for one cache resource. */
 static struct rftype res_common_files[] = {
 	{
@@ -2126,6 +2158,12 @@ static struct rftype res_common_files[] = {
 		.seq_show	= resctrl_io_alloc_show,
 		.write		= resctrl_io_alloc_write,
 	},
+	{
+		.name		= "io_alloc_cbm",
+		.mode		= 0444,
+		.kf_ops		= &rdtgroup_kf_single_ops,
+		.seq_show	= resctrl_io_alloc_cbm_show,
+	},
 	{
 		.name		= "mba_MBps_event",
 		.mode		= 0644,
-- 
2.34.1

[PATCH v3 7/7] x86/resctrl: Introduce interface to modify io_alloc Capacity Bit Masks

[Babu Moger]

"io_alloc" feature is a mechanism that enables direct insertion of data
from I/O devices into the L3 cache. By directly caching data from I/O
devices rather than first storing the I/O data in DRAM, it reduces the
demands on DRAM bandwidth and reduces latency to the processor consuming
the I/O data.

io_alloc feature uses the highest CLOSID to route the traffic from I/O
devices. Provide the interface to modify io_alloc CBMs (Capacity Bit Mask)
when feature is enabled.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v3: Minor changes due to changes in resctrl_arch_get_io_alloc_enabled()
    and resctrl_io_alloc_closid_get().
    Taken care of handling the CBM update when CDP is enabled.
    Updated the commit log to make it generic.

v2: Added more generic text in documentation.
---
 Documentation/arch/x86/resctrl.rst        |  12 ++
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c |   2 +-
 arch/x86/kernel/cpu/resctrl/internal.h    |   1 +
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 134 +++++++++++++++++++++-
 4 files changed, 147 insertions(+), 2 deletions(-)

diff --git a/Documentation/arch/x86/resctrl.rst b/Documentation/arch/x86/resctrl.rst
index 1b67e31d626c..29c8851bcc7f 100644
--- a/Documentation/arch/x86/resctrl.rst
+++ b/Documentation/arch/x86/resctrl.rst
@@ -169,6 +169,18 @@ related to allocation:
 		When CDP is enabled, io_alloc routes I/O traffic using the highest
 		CLOSID allocated for the instruction cache.
 
+"io_alloc_cbm":
+		Capacity Bit Masks (CBMs) available to supported IO devices which
+		can directly insert cache lines in L3 which can help to reduce the
+		latency. CBM can be configured by writing to the interface in the
+		following format::
+
+			L3:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
+
+		When CDP is enabled, L3 control is divided into two separate resources:
+		L3CODE and L3DATA. However, the CBM can only be updated on the L3CODE
+		resource.
+
 Memory bandwidth(MB) subdirectory contains the following files
 with respect to allocation:
 
diff --git a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
index d272dea43924..4dfee0436c1c 100644
--- a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
+++ b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
@@ -102,7 +102,7 @@ int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s,
  * requires at least two bits set.
  * AMD allows non-contiguous bitmasks.
  */
-static bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r)
+bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r)
 {
 	unsigned long first_bit, zero_bit, val;
 	unsigned int cbm_len = r->cache.cbm_len;
diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
index 07cf8409174d..702f6926bbdf 100644
--- a/arch/x86/kernel/cpu/resctrl/internal.h
+++ b/arch/x86/kernel/cpu/resctrl/internal.h
@@ -669,4 +669,5 @@ void rdt_staged_configs_clear(void);
 bool closid_allocated(unsigned int closid);
 int resctrl_find_cleanest_closid(void);
 void show_doms(struct seq_file *s, struct resctrl_schema *schema, int closid);
+bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r);
 #endif /* _ASM_X86_RESCTRL_INTERNAL_H */
diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
index 81b9d8c5dabf..9997cbfc1c19 100644
--- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
+++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
@@ -1999,6 +1999,137 @@ static int resctrl_io_alloc_cbm_show(struct kernfs_open_file *of,
 	return ret;
 }
 
+/*
+ * Read the CBM and check the validity. Make sure CBM is not shared
+ * with any other exclusive resctrl groups.
+ */
+static int resctrl_io_alloc_parse_cbm(char *buf, struct resctrl_schema *s,
+				      struct rdt_ctrl_domain *d)
+{
+	struct resctrl_staged_config *cfg;
+	struct rdt_resource *r = s->res;
+	u32 io_alloc_closid;
+	u32 cbm_val;
+
+	cfg = &d->staged_config[s->conf_type];
+	if (cfg->have_new_ctrl) {
+		rdt_last_cmd_printf("Duplicate domain %d\n", d->hdr.id);
+		return -EINVAL;
+	}
+
+	if (!cbm_validate(buf, &cbm_val, r))
+		return -EINVAL;
+
+	/*
+	 * The CBM may not overlap with other exclusive group.
+	 */
+	io_alloc_closid = resctrl_io_alloc_closid_get(r, s);
+	if (rdtgroup_cbm_overlaps(s, d, cbm_val, io_alloc_closid, true)) {
+		rdt_last_cmd_puts("Overlaps with exclusive group\n");
+		return -EINVAL;
+	}
+
+	cfg->new_ctrl = cbm_val;
+	cfg->have_new_ctrl = true;
+
+	return 0;
+}
+
+static int resctrl_io_alloc_parse_line(char *line,  struct rdt_resource *r,
+				       struct resctrl_schema *s)
+{
+	struct rdt_ctrl_domain *d;
+	char *dom = NULL, *id;
+	unsigned long dom_id;
+
+next:
+	if (!line || line[0] == '\0')
+		return 0;
+
+	dom = strsep(&line, ";");
+	id = strsep(&dom, "=");
+	if (!dom || kstrtoul(id, 10, &dom_id)) {
+		rdt_last_cmd_puts("Missing '=' or non-numeric domain\n");
+		return -EINVAL;
+	}
+
+	dom = strim(dom);
+	list_for_each_entry(d, &r->ctrl_domains, hdr.list) {
+		if (d->hdr.id == dom_id) {
+			if (resctrl_io_alloc_parse_cbm(dom, s, d))
+				return -EINVAL;
+			goto next;
+		}
+	}
+	return -EINVAL;
+}
+
+static ssize_t resctrl_io_alloc_cbm_write(struct kernfs_open_file *of,
+					  char *buf, size_t nbytes, loff_t off)
+{
+	struct resctrl_schema *s = of->kn->parent->priv;
+	struct rdt_resource *r = s->res;
+	u32 io_alloc_closid;
+	char *resname;
+	int ret = 0;
+
+	/* Valid input requires a trailing newline */
+	if (nbytes == 0 || buf[nbytes - 1] != '\n')
+		return -EINVAL;
+
+	buf[nbytes - 1] = '\0';
+
+	if (!r->cache.io_alloc_capable || s->conf_type == CDP_DATA) {
+		rdt_last_cmd_puts("io_alloc feature is not supported on the resource\n");
+		return -EINVAL;
+	}
+
+	cpus_read_lock();
+	mutex_lock(&rdtgroup_mutex);
+
+	rdt_last_cmd_clear();
+	rdt_staged_configs_clear();
+
+	if (!resctrl_arch_get_io_alloc_enabled(r)) {
+		rdt_last_cmd_puts("io_alloc feature is not enabled\n");
+		ret = -EINVAL;
+		goto cbm_write_out;
+	}
+
+	resname = strim(strsep(&buf, ":"));
+	if (!buf) {
+		rdt_last_cmd_puts("Missing ':'\n");
+		ret = -EINVAL;
+		goto cbm_write_out;
+	}
+
+	if (strcmp(resname, s->name)) {
+		rdt_last_cmd_printf("Unsupported resource name '%s'\n", resname);
+		ret = -EINVAL;
+		goto cbm_write_out;
+	}
+
+	if (buf[0] == '\0') {
+		rdt_last_cmd_printf("Missing '%s' value\n", resname);
+		ret = -EINVAL;
+		goto cbm_write_out;
+	}
+
+	ret = resctrl_io_alloc_parse_line(buf, r, s);
+	if (ret)
+		goto cbm_write_out;
+
+	io_alloc_closid = resctrl_io_alloc_closid_get(r, s);
+	ret = resctrl_arch_update_domains(r, io_alloc_closid);
+
+cbm_write_out:
+	rdt_staged_configs_clear();
+	mutex_unlock(&rdtgroup_mutex);
+	cpus_read_unlock();
+
+	return ret ?: nbytes;
+}
+
 /* rdtgroup information files for one cache resource. */
 static struct rftype res_common_files[] = {
 	{
@@ -2160,9 +2291,10 @@ static struct rftype res_common_files[] = {
 	},
 	{
 		.name		= "io_alloc_cbm",
-		.mode		= 0444,
+		.mode		= 0644,
 		.kf_ops		= &rdtgroup_kf_single_ops,
 		.seq_show	= resctrl_io_alloc_cbm_show,
+		.write		= resctrl_io_alloc_cbm_write,
 	},
 	{
 		.name		= "mba_MBps_event",
-- 
2.34.1

Re: [PATCH v3 0/7] Support L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

[Reinette Chatre]

Hi Babu,

On 1/30/25 1:20 PM, Babu Moger wrote:
> 
> This series adds the support for L3 Smart Data Cache Injection Allocation
> Enforcement (SDCIAE) to resctrl infrastructure. It is refered as io_alloc

"refered as" -> "referred to as"?

> in resctrl subsystem.
> 

...

> 
> # Linux Implementation
> 
> Feature adds following interface files when the resctrl "io_alloc" feature is
> supported:

Please note that, for example when L3 supports io_alloc but L2 does not, the
implementation also creates the files for L2 (since it sets flags for all
cache resources). "io_alloc" may just be present for resource that does not
actually support "io_alloc". (more below)

> 
> /sys/fs/resctrl/info/L3/io_alloc: Reports the feature status. Feature can be
> 				  enabled/disabled by writing to the interface.
> 
> /sys/fs/resctrl/info/L3/io_alloc_cbm: Capacity Bit Masks (CBMs) available to SDCI
> 				      supported IO devices. CBM can be configured

(please use imperative tone throughout)

> 				      by writing to the interface in the following
> 				      format::

("in the following format" is not followed by a format example as expected)

> # Examples
> 
> a. Check if io_alloc feature is available
> 	#mount -t resctrl resctrl /sys/fs/resctrl/
> 
> 	# cat /sys/fs/resctrl/info/L3/io_alloc
> 	0

Considering that io_alloc will be present for resources that may not support
io_alloc I think it may be helpful if it instead display "enabled/disabled/not supported".
User space can still interact with the file with typical boolean variants recognized
by kstrtobool()

> 
> b. Enable the io_alloc feature. 
> 
> 	# echo 1 > /sys/fs/resctrl/info/L3/io_alloc 
> 
> c. Check the CBM values for the io_alloc feature.
> 
> 	# cat /sys/fs/resctrl/info/L3/io_alloc_cbm 
> 	L3:0=ffff;1=ffff

Is it expected that more resources may support IO alloc?
In above output the resource name is redundant, but if
more resources could support IO allocation then it may
be more intuitive to move io_alloc_cbm one level up in hierarchy
and keep resource name in output in order to potentially
manage multiple resources. I do not see impact on feature
in current design if files are kept within resource directories,
but in that case the name seems redundant.

> 
> d. Change the CBM value for the domain 1:
> 	# echo L3:1=FF > /sys/fs/resctrl/info/L3/io_alloc_cbm
> 
> 	# cat /sys/fs/resctrl/info/L3/io_alloc_cbm 
> 	L3:0=ffff;1=00ff
> 
> d. Disable io_alloc feature and exit.
> 
> 	# echo 0 > /sys/fs/resctrl/info/L3/io_alloc 
> 	#umount /sys/fs/resctrl/
> 

AMD also supports what is exposed to user space as "shareable_bits". According
to APM:
	Depending on the implementation, some portions of the L3 Cache may be
	shared by other system functions or used for some other purpose not
	under the control of the PQOS feature set. The L3 Cache Allocation
	Sharing Mask returned by CPUID Fn0000_0010_EBX_x1[L3ShareAllocMask] is a
	bitmask that represents portions of the L3 that may be shared by those
	functions.

Could you please include what (if any) the relationship is between the CBM
discoverable via Fn0000_0010_EBX_x1[L3ShareAllocMask] and the CBM of 
"highest-supported L3_MASK_n register" when SDCIAE is enabled?

On the resctrl interface side the documentation currently states:

	"shareable_bits":                                                               
		Bitmask of shareable resource with other executing              
		entities (e.g. I/O). User can use this when                     
		setting up exclusive cache partitions. Note that                
		some platforms support devices that have their                  
		own settings for cache use which can over-ride                  
		these bits.                             

Even though this was originally used to expose the content of
Fn0000_0010_EBX_x1[L3ShareAllocMask] the intent of the content does
seem to also apply to the "io_alloc" CBM also.

I also expect "bit_usage" to appropriately reflect the io_alloc mask.
From what I can tell SDCIAE does not enforce isolation between hardware and
software use of these cache allocations so it does seem to me that the
existing legends apply. 

Reinette

Re: [PATCH v3 3/7] x86/resctrl: Detect io_alloc feature

[Reinette Chatre]

Hi Babu,

On 1/30/25 1:20 PM, Babu Moger wrote:
> "io_alloc" feature is a mechanism that enables direct insertion of data
> from I/O devices into the L3 cache. By directly caching data from I/O
> devices rather than first storing the I/O data in DRAM, feature reduces
> the demands on DRAM bandwidth and reduces latency to the processor
> consuming the I/O data.

This provides good context but the changelog does not mention what this patch
does.

An idea to get started (please improve):
	Data from I/O devices can be inserted directly into L3 cache. This
	reduces demands on DRAM bandwidth and reduces latency to the processor                  
	consuming the I/O data.                                                         
                                                                                
	Introduce cache resource property "io_alloc_capable" that an                    
	architecture can set if a portion of the L3 cache can be allocated              
	for I/O traffic. Set this property on x86 systems that support SDCIAE.  

> 
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---
> v3: Rewrote commit log. Changed the text to bit generic than the AMD specific.
>     Renamed the rdt_get_sdciae_alloc_cfg() to rdt_set_io_alloc_capable().
>     Removed leftover comment from v2.
> 
> v2: Changed sdciae_capable to io_alloc_capable to make it generic feature.
>     Also moved the io_alloc_capable in struct resctrl_cache.
> ---
>  arch/x86/kernel/cpu/resctrl/core.c | 7 +++++++
>  include/linux/resctrl.h            | 3 +++
>  2 files changed, 10 insertions(+)
> 
> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
> index c2450cd52511..1ebdb2dcc009 100644
> --- a/arch/x86/kernel/cpu/resctrl/core.c
> +++ b/arch/x86/kernel/cpu/resctrl/core.c
> @@ -306,6 +306,11 @@ static void rdt_get_cdp_config(int level)
>  	rdt_resources_all[level].r_resctrl.cdp_capable = true;
>  }
>  
> +static void rdt_set_io_alloc_capable(struct rdt_resource *r)
> +{
> +	r->cache.io_alloc_capable = true;
> +}
> +
>  static void rdt_get_cdp_l3_config(void)
>  {
>  	rdt_get_cdp_config(RDT_RESOURCE_L3);
> @@ -931,6 +936,8 @@ static __init bool get_rdt_alloc_resources(void)
>  		rdt_get_cache_alloc_cfg(1, r);
>  		if (rdt_cpu_has(X86_FEATURE_CDP_L3))
>  			rdt_get_cdp_l3_config();
> +		if (rdt_cpu_has(X86_FEATURE_SDCIAE))
> +			rdt_set_io_alloc_capable(r);
>  		ret = true;
>  	}
>  	if (rdt_cpu_has(X86_FEATURE_CAT_L2)) {
> diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h
> index d94abba1c716..dbe6461f3fbc 100644
> --- a/include/linux/resctrl.h
> +++ b/include/linux/resctrl.h
> @@ -129,6 +129,8 @@ struct rdt_mon_domain {
>   * @arch_has_sparse_bitmasks:	True if a bitmask like f00f is valid.
>   * @arch_has_per_cpu_cfg:	True if QOS_CFG register for this cache
>   *				level has CPU scope.
> + * @io_alloc_capable:	True if portion of the L3 cache can be allocated
> + *			for I/O traffic.

Enforcing that this should be for L3 cache is confusing. On a system with
L3 and L2 cache resources each resource will be described by the properties in
struct resctrl_cache for particular resource. We do not want to set "io_alloc_capable"
to true in the L2's struct if the L3 cache supports this feature.

This can just be: "True if portion of the cache can be allocated for I/O traffic"


>   */
>  struct resctrl_cache {
>  	unsigned int	cbm_len;
> @@ -136,6 +138,7 @@ struct resctrl_cache {
>  	unsigned int	shareable_bits;
>  	bool		arch_has_sparse_bitmasks;
>  	bool		arch_has_per_cpu_cfg;
> +	bool		io_alloc_capable;
>  };
>  
>  /**

Reinette

Re: [PATCH v3 4/7] x86/resctrl: Implement "io_alloc" enable/disable handlers

[Reinette Chatre]

Hi Babu,

On 1/30/25 1:20 PM, Babu Moger wrote:
> "io_alloc" feature that enables direct insertion of data from I/O devices
> into the L3 cache.

Above is not clear to me. Should this maybe be ""io_alloc" enables direct insertion ..."


> 
> On AMD, 'io_alloc" feature is backed by L3 Smart Data Cache Injection

(mismatch quotes around io_alloc ... also below)

> Allocation Enforcement (SDCIAE). SDCIAE feature can be enabled by setting

"can be enabled" -> "is enabled"?

> bit 1 in MSR L3_QOS_EXT_CFG. Apply the updated SDCIAE value across all
> logical processors within the QOS domain when modifying its state.

... it could also just be "Change SDCIAE state by setting (to enable) or
clearing (to disable) bit 1 of MSR L3_QOS_EXT_CFG on all logical processors
within the cache domain."

> 
> Introduce architecture-specific handlers to manage the detection and

This patch only seem to do the enable/disable with detection done in
earlier patch.

> enabling/disabling of 'io_alloc" feature.
> 
> The SDCIAE feature details are available in APM listed below [1].
> [1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
> Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache
> Injection Allocation Enforcement (SDCIAE)
> 
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---

...

> diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h
> index dbe6461f3fbc..e77c3b37bad4 100644
> --- a/include/linux/resctrl.h
> +++ b/include/linux/resctrl.h
> @@ -343,6 +343,15 @@ void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,
>   */
>  void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain *d);
>  
> +/**
> + * resctrl_arch_io_alloc_enable() - Enable/disable io_alloc feature.
> + * @r:		The resctrl resource.
> + * @enable:	Enable (1) or disable (0) the feature

Please be consistent in using periods at end of sentence. 
This could be made more specific with:

	"Enable (true) or disable (false) io_alloc on resource @r."

> + *
> + * This can be called from any CPU.
> + */
> +int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable);
> +
>  extern unsigned int resctrl_rmid_realloc_threshold;
>  extern unsigned int resctrl_rmid_realloc_limit;
>  

Reinette

Re: [PATCH v3 5/7] x86/resctrl: Add interface to enable/disable io_alloc feature

[Reinette Chatre]

Hi Babu,

On 1/30/25 1:20 PM, Babu Moger wrote:
> The io_alloc feature in resctrl is a mechanism that enables direct
> insertion of data from I/O devices into the L3 cache.
> 
> On AMD systems, io_alloc feature is backed by SDCIAE (L3 Smart Data Cache
> Injection Allocation Enforcement). When enabled, SDCIAE forces all SDCI
> lines to be placed into the L3 cache partitions identified by the
> highest-supported L3_MASK_n register as reported by CPUID
> Fn0000_0010_EDX_x1.MAX_COS. For example, if MAX_COS=15, SDCI lines will
> be allocated into the L3 cache partitions determined by the bitmask in
> the L3_MASK_15 register.
> 
> When CDP is enabled, io_alloc routes I/O traffic using the highest CLOSID
> allocated for the instruction cache.

You can append a "L3CODE" to the above to help provide context on what resource
is referred to as "instruction cache".

> 
> Introduce interface to enable/disable "io_alloc" feature on user input.
> 
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---
> v3: Rewrote the change to make it generic.
>     Rewrote the documentation in resctrl.rst to be generic and added
>     AMD feature details in the end.
>     Added the check to verify if MAX CLOSID availability on the system.
>     Added CDP check to make sure io_alloc is configured in CDP_CODE.
>     Added resctrl_io_alloc_closid_free() to free the io_alloc CLOSID.
>     Added errors in few cases when CLOSID allocation fails.
>     Fixes splat reported when info/L3/bit_usage is accesed when io_alloc
>     is enabled.
>     https://lore.kernel.org/lkml/SJ1PR11MB60837B532254E7B23BC27E84FC052@SJ1PR11MB6083.namprd11.prod.outlook.com/
> 
> v2: Renamed the feature to "io_alloc".
>     Added generic texts for the feature in commit log and resctrl.rst doc.
>     Added resctrl_io_alloc_init_cat() to initialize io_alloc to default
>     values when enabled.
>     Fixed io_alloc show functinality to display only on L3 resource.
> ---
>  Documentation/arch/x86/resctrl.rst     |  34 ++++++
>  arch/x86/kernel/cpu/resctrl/core.c     |   2 +
>  arch/x86/kernel/cpu/resctrl/rdtgroup.c | 144 +++++++++++++++++++++++++
>  3 files changed, 180 insertions(+)
> 
> diff --git a/Documentation/arch/x86/resctrl.rst b/Documentation/arch/x86/resctrl.rst
> index 6768fc1fad16..1b67e31d626c 100644
> --- a/Documentation/arch/x86/resctrl.rst
> +++ b/Documentation/arch/x86/resctrl.rst
> @@ -135,6 +135,40 @@ related to allocation:
>  			"1":
>  			      Non-contiguous 1s value in CBM is supported.
>  
> +"io_alloc":
> +		The "io_alloc" feature in resctrl enables system software to

Since this is already resctrl documentation "feature in resctrl" could be dropped to
be just:
		"io_alloc" enables system software ...
	

> +		configure the portion of the L3 cache allocated for I/O traffic.
> +		By directly caching data from I/O devices rather than first storing
> +		the I/O data in DRAM, reduces the demands on DRAM bandwidth and
> +		reduces latency to the processor consuming the I/O data.

hmmm ... looks like "SDCIAE" was deleted from earlier used (marketing?) text and
resulting text left as-is without re-checking if resulting text is still coherent.
I do not think it is needed to motivate/market the feature here, perhaps last
sentence can just be dropped?

> +
> +		The feature routes the I/O traffic via specific CLOSID reserved
> +		for io_alloc feature. By configuring the CBM (Capacity Bit Mask)
> +		for the CLOSID users can control the L3 portions available for
> +		I/O traffic. When enabled, CLOSID reserved for the io_alloc will
> +		not be available to the resctrl group.

Although the above reflects how SDCIAE is implemented it may not be true for how
another architecture may support this. hmmm ... this sounds familiar and looking back it
is the same thing I mentioned in V2 feedback, actually, in V2 I pointed to V1 feedback
that said this also.
If you insist on this text then please change the tone that indicates the
behavior is optional. For example, "An architecture may support io_alloc by reserving
a CLOSID to configure the ..."

> +		::
> +
> +		  # cat /sys/fs/resctrl/info/L3/io_alloc
> +		  0

Please refer to cover-letter about proposal to use enabled/disabled/not supported instead.

> +
> +		  "0":
> +		      io_alloc feature is not enabled.
> +		  "1":
> +		      io_alloc feature is enabled, allowing users to manage
> +		      the portions of the L3 cache allocated for the I/O device.
> +
> +		Feature can be enabled/disabled by writing to the interface.
> +		Example::
> +
> +			# echo 1 > /sys/fs/resctrl/info/L3/io_alloc
> +
> +		On AMD systems, the io_alloc feature is supported by the L3 Smart
> +		Data Cache Injection Allocation Enforcement (SDCIAE). The CLOSID for
> +		io_alloc is determined by the highest CLOSID supported by the resource.
> +		When CDP is enabled, io_alloc routes I/O traffic using the highest
> +		CLOSID allocated for the instruction cache.
> +
>  Memory bandwidth(MB) subdirectory contains the following files
>  with respect to allocation:
>  
> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
> index 1ebdb2dcc009..88bc95c14ea8 100644
> --- a/arch/x86/kernel/cpu/resctrl/core.c
> +++ b/arch/x86/kernel/cpu/resctrl/core.c
> @@ -309,6 +309,8 @@ static void rdt_get_cdp_config(int level)
>  static void rdt_set_io_alloc_capable(struct rdt_resource *r)
>  {
>  	r->cache.io_alloc_capable = true;
> +	resctrl_file_fflags_init("io_alloc",
> +				 RFTYPE_CTRL_INFO | RFTYPE_RES_CACHE);
>  }

Some MPAM changes landed since you created this work. After the fs/arch split the
architecture code should have no insight into the resctrl file flags. Please refer to
the MPAM changes on how this can be managed. You can refer to thread_throttle_mode_init()
and similar to that resctrl can use the io_alloc_capable flag to make the files visible.

>  
>  static void rdt_get_cdp_l3_config(void)
> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> index c5a0a31c3a85..37295dd14abe 100644
> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> @@ -62,6 +62,7 @@ static char last_cmd_status_buf[512];
>  
>  static int rdtgroup_setup_root(struct rdt_fs_context *ctx);
>  static void rdtgroup_destroy_root(void);
> +static int rdtgroup_init_cat(struct resctrl_schema *s, u32 closid);
>  
>  struct dentry *debugfs_resctrl;
>  
> @@ -180,6 +181,19 @@ void closid_free(int closid)
>  	__set_bit(closid, &closid_free_map);
>  }
>  
> +static int resctrl_io_alloc_closid_alloc(u32 io_alloc_closid)
> +{
> +	if (__test_and_clear_bit(io_alloc_closid, &closid_free_map))
> +		return io_alloc_closid;
> +	else
> +		return -ENOSPC;
> +}
> +
> +static void resctrl_io_alloc_closid_free(u32 io_alloc_closid)
> +{
> +	closid_free(io_alloc_closid);
> +}
> +
>  /**
>   * closid_allocated - test if provided closid is in use
>   * @closid: closid to be tested
> @@ -995,6 +1009,33 @@ static int rdt_shareable_bits_show(struct kernfs_open_file *of,
>  	return 0;
>  }
>  
> +/*
> + * io_alloc feature uses max CLOSID to route the IO traffic.
> + * Get the max CLOSID and verify if the CLOSID is available.
> + */
> +static int resctrl_io_alloc_closid_get(struct rdt_resource *r,
> +				       struct resctrl_schema *s)
> +{
> +	int num_closids = resctrl_arch_get_num_closid(r);
> +
> +	/*
> +	 * The number of CLOSIDs is determined based on the minimum
> +	 * supported across all resources (in closid_init). It is stored

closid_init -> closid_init()

> +	 * in s->num_closids. Also, if CDP is enabled number of CLOSIDs
> +	 * are halved. To enable io_alloc feature, the number of CLOSIDs
> +	 * must match the maximum CLOSID supported by the resource.
> +	 */
> +	if (resctrl_arch_get_cdp_enabled(r->rid))
> +		num_closids /= 2;
> +
> +	if (s->num_closid != num_closids) {

Considering from schemata_list_add():
	s->num_closid = resctrl_arch_get_num_closid(r);

... the above "if (s->num_closid != num_closids)" just compares the value to itself, no?

This function does not actually take all resources into account with the above
comparison. I think what you may need here is a comparison with closid_free_map_len?

As I understand it is still possible to use io_alloc when the resource's max CLOSID
is not within closid_free_map, this is just not done simplify implementation.

> +		rdt_last_cmd_puts("Max CLOSID to support io_alloc is not available\n");
> +		return -ENOSPC;
> +	}
> +
> +	return num_closids - 1;
> +}
> +
>  /*
>   * rdt_bit_usage_show - Display current usage of resources
>   *
> @@ -1038,6 +1079,14 @@ static int rdt_bit_usage_show(struct kernfs_open_file *of,
>  		for (i = 0; i < closids_supported(); i++) {
>  			if (!closid_allocated(i))
>  				continue;
> +			/*
> +			 * If io_alloc is enabled, the CLOSID will be
> +			 * allocated but will not be associated with any
> +			 * groups. Skip in that case.

This defeats the purpose of "bit_usage" that gives insight to user space
on how the cache is allocated. Instead of ignoring portions of cache
used for I/O this should display to the user that these portions are
used by/shared with hardware.

> +			 */
> +			if (i == resctrl_io_alloc_closid_get(r, s) &&
> +			    resctrl_arch_get_io_alloc_enabled(r))
> +				continue;
>  			ctrl_val = resctrl_arch_get_config(r, dom, i,
>  							   s->conf_type);
>  			mode = rdtgroup_mode_by_closid(i);
> @@ -1830,6 +1879,94 @@ int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable)
>  	return 0;
>  }
>  
> +static int resctrl_io_alloc_show(struct kernfs_open_file *of,
> +				 struct seq_file *seq, void *v)
> +{
> +	struct resctrl_schema *s = of->kn->parent->priv;
> +	struct rdt_resource *r = s->res;
> +
> +	seq_printf(seq, "%x\n", resctrl_arch_get_io_alloc_enabled(r));
> +	return 0;
> +}
> +
> +/*
> + * Initialize io_alloc CLOSID cache resource with default CBM values.
> + */
> +static int resctrl_io_alloc_init_cat(struct rdt_resource *r,
> +				     struct resctrl_schema *s, u32 closid)
> +{
> +	int ret;
> +
> +	rdt_staged_configs_clear();
> +
> +	ret = rdtgroup_init_cat(s, closid);
> +	if (ret < 0)
> +		goto out_init_cat;
> +
> +	ret = resctrl_arch_update_domains(r, closid);
> +
> +out_init_cat:
> +	rdt_staged_configs_clear();
> +	return ret;
> +}
> +
> +static ssize_t resctrl_io_alloc_write(struct kernfs_open_file *of, char *buf,
> +				      size_t nbytes, loff_t off)
> +{
> +	struct resctrl_schema *s = of->kn->parent->priv;
> +	struct rdt_resource *r = s->res;
> +	u32 io_alloc_closid;
> +	bool enable;
> +	int ret;
> +
> +	if (!r->cache.io_alloc_capable || s->conf_type == CDP_DATA) {
> +		rdt_last_cmd_puts("io_alloc feature is not supported on the resource\n");

rdt_last_cmd_puts() starts with lockdep_assert_held(&rdtgroup_mutex), also expect
rdt_last_cmd_clear() before first use.


> +		return -EINVAL;

Could ENODEV be used instead?

> +	}
> +
> +	ret = kstrtobool(buf, &enable);
> +	if (ret)
> +		return ret;
> +
> +	cpus_read_lock();
> +	mutex_lock(&rdtgroup_mutex);
> +
> +	rdt_last_cmd_clear();
> +
> +	io_alloc_closid = resctrl_io_alloc_closid_get(r, s);
> +	if (io_alloc_closid < 0) {

Could you please add an informative message in last_cmd_status? It may be
possible for user to remedy this and retry.

> +		ret = -EINVAL;
> +		goto out_io_alloc;
> +	}
> +
> +	if (resctrl_arch_get_io_alloc_enabled(r) != enable) {
> +		if (enable) {
> +			ret = resctrl_io_alloc_closid_alloc(io_alloc_closid);
> +			if (ret < 0) {
> +				rdt_last_cmd_puts("CLOSID for io_alloc is not available\n");

If the CLOSID is not available then it may be possible for the user to remedy this by
removing a resource group and retry this operation. Since CLOSID is not useful to user space
(and x86 architecture specific) this could  be improved to give guidance to user
space about which resource group (by name, not CLOSID) is preventing this from succeeding.

(this sounded familiar, looks like I provided the same feedback to V2, to which you
responded "Yes. We can do that.")

> +				goto out_io_alloc;
> +			}
> +			ret = resctrl_io_alloc_init_cat(r, s, io_alloc_closid);
> +			if (ret) {
> +				rdt_last_cmd_puts("Failed to initialize io_alloc allocations\n");
> +				resctrl_io_alloc_closid_free(io_alloc_closid);
> +				goto out_io_alloc;
> +			}
> +
> +		} else {
> +			resctrl_io_alloc_closid_free(io_alloc_closid);
> +		}
> +
> +		ret = resctrl_arch_io_alloc_enable(r, enable);
> +	}
> +
> +out_io_alloc:
> +	mutex_unlock(&rdtgroup_mutex);
> +	cpus_read_unlock();
> +
> +	return ret ?: nbytes;
> +}
> +
>  /* rdtgroup information files for one cache resource. */
>  static struct rftype res_common_files[] = {
>  	{
> @@ -1982,6 +2119,13 @@ static struct rftype res_common_files[] = {
>  		.seq_show	= rdtgroup_schemata_show,
>  		.fflags		= RFTYPE_CTRL_BASE,
>  	},
> +	{
> +		.name		= "io_alloc",
> +		.mode		= 0644,
> +		.kf_ops		= &rdtgroup_kf_single_ops,
> +		.seq_show	= resctrl_io_alloc_show,
> +		.write		= resctrl_io_alloc_write,
> +	},
>  	{
>  		.name		= "mba_MBps_event",
>  		.mode		= 0644,

Reinette

Re: [PATCH v3 6/7] x86/resctrl: Introduce interface to display io_alloc CBMs

[Reinette Chatre]

Hi Babu,

On 1/30/25 1:20 PM, Babu Moger wrote:
> The io_alloc feature in resctrl enables system software to configure
> the portion of the L3 cache allocated for I/O traffic.
> 
> Add the interface to display CBMs (Capacity Bit Mask) of io_alloc
> feature.
> 
> When CDP is enabled, io_alloc routes traffic using the highest CLOSID
> which corresponds to CDP_CODE. Add a check for the CDP resource type.

It is not obvious to me what is meant with "highest CLOSID
which corresponds to CDP_CODE" ... how about "highest CLOSID used by
a L3CODE resource"?

> 
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---
> v3: Minor changes due to changes in resctrl_arch_get_io_alloc_enabled()
>     and resctrl_io_alloc_closid_get().
>     Added the check to verify CDP resource type.
>     Updated the commit log.
> 
> v2: Fixed to display only on L3 resources.
>     Added the locks while processing.
>     Rename the displat to io_alloc_cbm (from sdciae_cmd).
> ---
>  arch/x86/kernel/cpu/resctrl/core.c        |  2 ++
>  arch/x86/kernel/cpu/resctrl/ctrlmondata.c |  2 +-
>  arch/x86/kernel/cpu/resctrl/internal.h    |  1 +
>  arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 38 +++++++++++++++++++++++
>  4 files changed, 42 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
> index 88bc95c14ea8..030f738dea8d 100644
> --- a/arch/x86/kernel/cpu/resctrl/core.c
> +++ b/arch/x86/kernel/cpu/resctrl/core.c
> @@ -311,6 +311,8 @@ static void rdt_set_io_alloc_capable(struct rdt_resource *r)
>  	r->cache.io_alloc_capable = true;
>  	resctrl_file_fflags_init("io_alloc",
>  				 RFTYPE_CTRL_INFO | RFTYPE_RES_CACHE);
> +	resctrl_file_fflags_init("io_alloc_cbm",
> +				 RFTYPE_CTRL_INFO | RFTYPE_RES_CACHE);
>  }
>  
>  static void rdt_get_cdp_l3_config(void)
> diff --git a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
> index 536351159cc2..d272dea43924 100644
> --- a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
> +++ b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
> @@ -444,7 +444,7 @@ u32 resctrl_arch_get_config(struct rdt_resource *r, struct rdt_ctrl_domain *d,
>  	return hw_dom->ctrl_val[idx];
>  }
>  
> -static void show_doms(struct seq_file *s, struct resctrl_schema *schema, int closid)
> +void show_doms(struct seq_file *s, struct resctrl_schema *schema, int closid)
>  {
>  	struct rdt_resource *r = schema->res;
>  	struct rdt_ctrl_domain *dom;
> diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
> index 61bc609e932b..07cf8409174d 100644
> --- a/arch/x86/kernel/cpu/resctrl/internal.h
> +++ b/arch/x86/kernel/cpu/resctrl/internal.h
> @@ -668,4 +668,5 @@ void resctrl_file_fflags_init(const char *config, unsigned long fflags);
>  void rdt_staged_configs_clear(void);
>  bool closid_allocated(unsigned int closid);
>  int resctrl_find_cleanest_closid(void);
> +void show_doms(struct seq_file *s, struct resctrl_schema *schema, int closid);
>  #endif /* _ASM_X86_RESCTRL_INTERNAL_H */
> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> index 37295dd14abe..81b9d8c5dabf 100644
> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> @@ -1967,6 +1967,38 @@ static ssize_t resctrl_io_alloc_write(struct kernfs_open_file *of, char *buf,
>  	return ret ?: nbytes;
>  }
>  
> +static int resctrl_io_alloc_cbm_show(struct kernfs_open_file *of,
> +				     struct seq_file *seq, void *v)
> +{
> +	struct resctrl_schema *s = of->kn->parent->priv;
> +	struct rdt_resource *r = s->res;
> +	u32 io_alloc_closid;
> +	int ret = 0;
> +
> +	if (!r->cache.io_alloc_capable || s->conf_type == CDP_DATA) {
> +		rdt_last_cmd_puts("io_alloc feature is not supported on the resource\n");

rdt_last_cmd_puts() has to be called with rdtgroup_mutex held, also clear it before use.

> +		return -EINVAL;

How about ENODEV?

> +	}
> +
> +	cpus_read_lock();
> +	mutex_lock(&rdtgroup_mutex);
> +
> +	if (!resctrl_arch_get_io_alloc_enabled(r)) {
> +		rdt_last_cmd_puts("io_alloc feature is not enabled\n");
> +		ret = -EINVAL;
> +		goto cbm_show_out;
> +	}
> +
> +	io_alloc_closid = resctrl_io_alloc_closid_get(r, s);
> +
> +	show_doms(seq, s, io_alloc_closid);
> +
> +cbm_show_out:
> +	mutex_unlock(&rdtgroup_mutex);
> +	cpus_read_unlock();
> +	return ret;
> +}
> +
>  /* rdtgroup information files for one cache resource. */
>  static struct rftype res_common_files[] = {
>  	{
> @@ -2126,6 +2158,12 @@ static struct rftype res_common_files[] = {
>  		.seq_show	= resctrl_io_alloc_show,
>  		.write		= resctrl_io_alloc_write,
>  	},
> +	{
> +		.name		= "io_alloc_cbm",
> +		.mode		= 0444,
> +		.kf_ops		= &rdtgroup_kf_single_ops,
> +		.seq_show	= resctrl_io_alloc_cbm_show,
> +	},
>  	{
>  		.name		= "mba_MBps_event",
>  		.mode		= 0644,

Reinette

Re: [PATCH v3 7/7] x86/resctrl: Introduce interface to modify io_alloc Capacity Bit Masks

[Reinette Chatre]

Hi Babu,

On 1/30/25 1:20 PM, Babu Moger wrote:
> "io_alloc" feature is a mechanism that enables direct insertion of data
> from I/O devices into the L3 cache. By directly caching data from I/O
> devices rather than first storing the I/O data in DRAM, it reduces the
> demands on DRAM bandwidth and reduces latency to the processor consuming
> the I/O data.
> 
> io_alloc feature uses the highest CLOSID to route the traffic from I/O
> devices. Provide the interface to modify io_alloc CBMs (Capacity Bit Mask)
> when feature is enabled.
> 
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---
> v3: Minor changes due to changes in resctrl_arch_get_io_alloc_enabled()
>     and resctrl_io_alloc_closid_get().
>     Taken care of handling the CBM update when CDP is enabled.
>     Updated the commit log to make it generic.
> 
> v2: Added more generic text in documentation.
> ---
>  Documentation/arch/x86/resctrl.rst        |  12 ++
>  arch/x86/kernel/cpu/resctrl/ctrlmondata.c |   2 +-
>  arch/x86/kernel/cpu/resctrl/internal.h    |   1 +
>  arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 134 +++++++++++++++++++++-
>  4 files changed, 147 insertions(+), 2 deletions(-)
> 
> diff --git a/Documentation/arch/x86/resctrl.rst b/Documentation/arch/x86/resctrl.rst
> index 1b67e31d626c..29c8851bcc7f 100644
> --- a/Documentation/arch/x86/resctrl.rst
> +++ b/Documentation/arch/x86/resctrl.rst
> @@ -169,6 +169,18 @@ related to allocation:
>  		When CDP is enabled, io_alloc routes I/O traffic using the highest
>  		CLOSID allocated for the instruction cache.
>  
> +"io_alloc_cbm":
> +		Capacity Bit Masks (CBMs) available to supported IO devices which
> +		can directly insert cache lines in L3 which can help to reduce the
> +		latency. CBM can be configured by writing to the interface in the
> +		following format::
> +
> +			L3:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...

This format is dependent on the resource name (not always L3).

> +
> +		When CDP is enabled, L3 control is divided into two separate resources:
> +		L3CODE and L3DATA. However, the CBM can only be updated on the L3CODE
> +		resource.
> +
>  Memory bandwidth(MB) subdirectory contains the following files
>  with respect to allocation:
>  
> diff --git a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
> index d272dea43924..4dfee0436c1c 100644
> --- a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
> +++ b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
> @@ -102,7 +102,7 @@ int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s,
>   * requires at least two bits set.
>   * AMD allows non-contiguous bitmasks.
>   */
> -static bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r)
> +bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r)
>  {
>  	unsigned long first_bit, zero_bit, val;
>  	unsigned int cbm_len = r->cache.cbm_len;
> diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
> index 07cf8409174d..702f6926bbdf 100644
> --- a/arch/x86/kernel/cpu/resctrl/internal.h
> +++ b/arch/x86/kernel/cpu/resctrl/internal.h
> @@ -669,4 +669,5 @@ void rdt_staged_configs_clear(void);
>  bool closid_allocated(unsigned int closid);
>  int resctrl_find_cleanest_closid(void);
>  void show_doms(struct seq_file *s, struct resctrl_schema *schema, int closid);
> +bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r);
>  #endif /* _ASM_X86_RESCTRL_INTERNAL_H */
> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> index 81b9d8c5dabf..9997cbfc1c19 100644
> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> @@ -1999,6 +1999,137 @@ static int resctrl_io_alloc_cbm_show(struct kernfs_open_file *of,
>  	return ret;
>  }
>  
> +/*
> + * Read the CBM and check the validity. Make sure CBM is not shared
> + * with any other exclusive resctrl groups.
> + */
> +static int resctrl_io_alloc_parse_cbm(char *buf, struct resctrl_schema *s,
> +				      struct rdt_ctrl_domain *d)
> +{
> +	struct resctrl_staged_config *cfg;
> +	struct rdt_resource *r = s->res;
> +	u32 io_alloc_closid;
> +	u32 cbm_val;
> +
> +	cfg = &d->staged_config[s->conf_type];
> +	if (cfg->have_new_ctrl) {
> +		rdt_last_cmd_printf("Duplicate domain %d\n", d->hdr.id);
> +		return -EINVAL;
> +	}
> +
> +	if (!cbm_validate(buf, &cbm_val, r))
> +		return -EINVAL;
> +
> +	/*
> +	 * The CBM may not overlap with other exclusive group.
> +	 */
> +	io_alloc_closid = resctrl_io_alloc_closid_get(r, s);
> +	if (rdtgroup_cbm_overlaps(s, d, cbm_val, io_alloc_closid, true)) {
> +		rdt_last_cmd_puts("Overlaps with exclusive group\n");
> +		return -EINVAL;
> +	}
> +
> +	cfg->new_ctrl = cbm_val;
> +	cfg->have_new_ctrl = true;
> +
> +	return 0;
> +}

Could you please reduce amount of duplication with parse_cbm()?

(for rest of patch, please check that related comments from previous patches
are addressed here also)

Reinette

Re: [PATCH v3 0/7] Support L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

[Moger, Babu]

Hi Reinette,

Thanks for the feedback. Sorry for the delay.

On 3/21/25 17:50, Reinette Chatre wrote:
> Hi Babu,
> 
> On 1/30/25 1:20 PM, Babu Moger wrote:
>>
>> This series adds the support for L3 Smart Data Cache Injection Allocation
>> Enforcement (SDCIAE) to resctrl infrastructure. It is refered as io_alloc
> 
> "refered as" -> "referred to as"?

sure.

> 
>> in resctrl subsystem.
>>
> 
> ...
> 
>>
>> # Linux Implementation
>>
>> Feature adds following interface files when the resctrl "io_alloc" feature is
>> supported:
> 
> Please note that, for example when L3 supports io_alloc but L2 does not, the
> implementation also creates the files for L2 (since it sets flags for all
> cache resources). "io_alloc" may just be present for resource that does not
> actually support "io_alloc". (more below)
> 
>>
>> /sys/fs/resctrl/info/L3/io_alloc: Reports the feature status. Feature can be
>> 				  enabled/disabled by writing to the interface.
>>
>> /sys/fs/resctrl/info/L3/io_alloc_cbm: Capacity Bit Masks (CBMs) available to SDCI
>> 				      supported IO devices. CBM can be configured
> 
> (please use imperative tone throughout)

ok.

> 
>> 				      by writing to the interface in the following
>> 				      format::
> 
> ("in the following format" is not followed by a format example as expected)

Sure.
> 
>> # Examples
>>
>> a. Check if io_alloc feature is available
>> 	#mount -t resctrl resctrl /sys/fs/resctrl/
>>
>> 	# cat /sys/fs/resctrl/info/L3/io_alloc
>> 	0
> 
> Considering that io_alloc will be present for resources that may not support
> io_alloc I think it may be helpful if it instead display "enabled/disabled/not supported".
> User space can still interact with the file with typical boolean variants recognized
> by kstrtobool()

Sounds good.

> 
>>
>> b. Enable the io_alloc feature. 
>>
>> 	# echo 1 > /sys/fs/resctrl/info/L3/io_alloc 
>>
>> c. Check the CBM values for the io_alloc feature.
>>
>> 	# cat /sys/fs/resctrl/info/L3/io_alloc_cbm 
>> 	L3:0=ffff;1=ffff
> 
> Is it expected that more resources may support IO alloc?

No. There is no plan for that.

> In above output the resource name is redundant, but if
> more resources could support IO allocation then it may
> be more intuitive to move io_alloc_cbm one level up in hierarchy
> and keep resource name in output in order to potentially
> manage multiple resources. I do not see impact on feature
> in current design if files are kept within resource directories,
> but in that case the name seems redundant.

Yea. Will remove "L3".  Just keep 0=ffff;1=ffff

> 
>>
>> d. Change the CBM value for the domain 1:
>> 	# echo L3:1=FF > /sys/fs/resctrl/info/L3/io_alloc_cbm
>>
>> 	# cat /sys/fs/resctrl/info/L3/io_alloc_cbm 
>> 	L3:0=ffff;1=00ff
>>
>> d. Disable io_alloc feature and exit.
>>
>> 	# echo 0 > /sys/fs/resctrl/info/L3/io_alloc 
>> 	#umount /sys/fs/resctrl/
>>
> 
> AMD also supports what is exposed to user space as "shareable_bits". According
> to APM:
> 	Depending on the implementation, some portions of the L3 Cache may be
> 	shared by other system functions or used for some other purpose not
> 	under the control of the PQOS feature set. The L3 Cache Allocation
> 	Sharing Mask returned by CPUID Fn0000_0010_EBX_x1[L3ShareAllocMask] is a
> 	bitmask that represents portions of the L3 that may be shared by those
> 	functions.

Here is the complete text.

The L3 Cache allocation sharing mask (L3ShareAllocMask) returned in EBX by
CPUID Fn0000_0010 with ECX=1 is a bit vector which represents portions of
the cache which may be shared with other system entities or used for some
other purpose not under the control of the QOS feature set. When software
sets a bit in one of the L3_MASK_n registers at the same bit positions a
bit in the L3ShareAllocMask, processors executing with the corresponding
COS will competitively share that portion of the cache with the other
function. If this mask is all 0’s, then there is no other entity in the
system competing with the processors for use of the L3 cache.

The "L3ShareAllocMask" is always reported as 0 on AMD systems.

> Could you please include what (if any) the relationship is between the CBM
> discoverable via Fn0000_0010_EBX_x1[L3ShareAllocMask] and the CBM of 
> "highest-supported L3_MASK_n register" when SDCIAE is enabled?

No. There is no relationship in here.

> 
> On the resctrl interface side the documentation currently states:
> 
> 	"shareable_bits":                                                               
> 		Bitmask of shareable resource with other executing              
> 		entities (e.g. I/O). User can use this when                     
> 		setting up exclusive cache partitions. Note that                
> 		some platforms support devices that have their                  
> 		own settings for cache use which can over-ride                  
> 		these bits.                             
> 
> Even though this was originally used to expose the content of
> Fn0000_0010_EBX_x1[L3ShareAllocMask] the intent of the content does
> seem to also apply to the "io_alloc" CBM also.

It says "shared by other system functions or used for some other purpose
not under the control of the PQOS feature set".

"io_alloc" is PQOS feature set. I feel it should not affect "shareable_bits".

> 
> I also expect "bit_usage" to appropriately reflect the io_alloc mask.
>>From what I can tell SDCIAE does not enforce isolation between hardware and
> software use of these cache allocations so it does seem to me that the
> existing legends apply. 
> 
> Reinette
> 
> 

-- 
Thanks
Babu Moger

Re: [PATCH v3 3/7] x86/resctrl: Detect io_alloc feature

[Moger, Babu]

Hi Reinette,

On 3/21/25 17:51, Reinette Chatre wrote:
> Hi Babu,
> 
> On 1/30/25 1:20 PM, Babu Moger wrote:
>> "io_alloc" feature is a mechanism that enables direct insertion of data
>> from I/O devices into the L3 cache. By directly caching data from I/O
>> devices rather than first storing the I/O data in DRAM, feature reduces
>> the demands on DRAM bandwidth and reduces latency to the processor
>> consuming the I/O data.
> 
> This provides good context but the changelog does not mention what this patch
> does.
> 
> An idea to get started (please improve):
> 	Data from I/O devices can be inserted directly into L3 cache. This
> 	reduces demands on DRAM bandwidth and reduces latency to the processor                  
> 	consuming the I/O data.                                                         
>                                                                                 
> 	Introduce cache resource property "io_alloc_capable" that an                    
> 	architecture can set if a portion of the L3 cache can be allocated              
> 	for I/O traffic. Set this property on x86 systems that support SDCIAE.  

Looks good. Thanks

> 
>>
>> Signed-off-by: Babu Moger <babu.moger@amd.com>
>> ---
>> v3: Rewrote commit log. Changed the text to bit generic than the AMD specific.
>>     Renamed the rdt_get_sdciae_alloc_cfg() to rdt_set_io_alloc_capable().
>>     Removed leftover comment from v2.
>>
>> v2: Changed sdciae_capable to io_alloc_capable to make it generic feature.
>>     Also moved the io_alloc_capable in struct resctrl_cache.
>> ---
>>  arch/x86/kernel/cpu/resctrl/core.c | 7 +++++++
>>  include/linux/resctrl.h            | 3 +++
>>  2 files changed, 10 insertions(+)
>>
>> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
>> index c2450cd52511..1ebdb2dcc009 100644
>> --- a/arch/x86/kernel/cpu/resctrl/core.c
>> +++ b/arch/x86/kernel/cpu/resctrl/core.c
>> @@ -306,6 +306,11 @@ static void rdt_get_cdp_config(int level)
>>  	rdt_resources_all[level].r_resctrl.cdp_capable = true;
>>  }
>>  
>> +static void rdt_set_io_alloc_capable(struct rdt_resource *r)
>> +{
>> +	r->cache.io_alloc_capable = true;
>> +}
>> +
>>  static void rdt_get_cdp_l3_config(void)
>>  {
>>  	rdt_get_cdp_config(RDT_RESOURCE_L3);
>> @@ -931,6 +936,8 @@ static __init bool get_rdt_alloc_resources(void)
>>  		rdt_get_cache_alloc_cfg(1, r);
>>  		if (rdt_cpu_has(X86_FEATURE_CDP_L3))
>>  			rdt_get_cdp_l3_config();
>> +		if (rdt_cpu_has(X86_FEATURE_SDCIAE))
>> +			rdt_set_io_alloc_capable(r);
>>  		ret = true;
>>  	}
>>  	if (rdt_cpu_has(X86_FEATURE_CAT_L2)) {
>> diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h
>> index d94abba1c716..dbe6461f3fbc 100644
>> --- a/include/linux/resctrl.h
>> +++ b/include/linux/resctrl.h
>> @@ -129,6 +129,8 @@ struct rdt_mon_domain {
>>   * @arch_has_sparse_bitmasks:	True if a bitmask like f00f is valid.
>>   * @arch_has_per_cpu_cfg:	True if QOS_CFG register for this cache
>>   *				level has CPU scope.
>> + * @io_alloc_capable:	True if portion of the L3 cache can be allocated
>> + *			for I/O traffic.
> 
> Enforcing that this should be for L3 cache is confusing. On a system with
> L3 and L2 cache resources each resource will be described by the properties in
> struct resctrl_cache for particular resource. We do not want to set "io_alloc_capable"
> to true in the L2's struct if the L3 cache supports this feature.
> 
> This can just be: "True if portion of the cache can be allocated for I/O traffic"

Sure.

> 
> 
>>   */
>>  struct resctrl_cache {
>>  	unsigned int	cbm_len;
>> @@ -136,6 +138,7 @@ struct resctrl_cache {
>>  	unsigned int	shareable_bits;
>>  	bool		arch_has_sparse_bitmasks;
>>  	bool		arch_has_per_cpu_cfg;
>> +	bool		io_alloc_capable;
>>  };
>>  
>>  /**
> 
> Reinette
> 

-- 
Thanks
Babu Moger

Re: [PATCH v3 4/7] x86/resctrl: Implement "io_alloc" enable/disable handlers

[Moger, Babu]

Hi Reinette,

On 3/21/25 17:53, Reinette Chatre wrote:
> Hi Babu,
> 
> On 1/30/25 1:20 PM, Babu Moger wrote:
>> "io_alloc" feature that enables direct insertion of data from I/O devices
>> into the L3 cache.
> 
> Above is not clear to me. Should this maybe be ""io_alloc" enables direct insertion ..."

Sure.

> 
> 
>>
>> On AMD, 'io_alloc" feature is backed by L3 Smart Data Cache Injection
> 
> (mismatch quotes around io_alloc ... also below)

Sure. Will correct both.

> 
>> Allocation Enforcement (SDCIAE). SDCIAE feature can be enabled by setting
> 
> "can be enabled" -> "is enabled"?

Sure.

> 
>> bit 1 in MSR L3_QOS_EXT_CFG. Apply the updated SDCIAE value across all
>> logical processors within the QOS domain when modifying its state.
> 
> ... it could also just be "Change SDCIAE state by setting (to enable) or
> clearing (to disable) bit 1 of MSR L3_QOS_EXT_CFG on all logical processors
> within the cache domain."

Sure.

> 
>>
>> Introduce architecture-specific handlers to manage the detection and
> 
> This patch only seem to do the enable/disable with detection done in
> earlier patch.

Sure. Will fix it.

> 
>> enabling/disabling of 'io_alloc" feature.

Yes. Mismatched quotes.

>>
>> The SDCIAE feature details are available in APM listed below [1].
>> [1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
>> Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache
>> Injection Allocation Enforcement (SDCIAE)
>>
>> Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
>> Signed-off-by: Babu Moger <babu.moger@amd.com>
>> ---
> 
> ...
> 
>> diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h
>> index dbe6461f3fbc..e77c3b37bad4 100644
>> --- a/include/linux/resctrl.h
>> +++ b/include/linux/resctrl.h
>> @@ -343,6 +343,15 @@ void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,
>>   */
>>  void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain *d);
>>  
>> +/**
>> + * resctrl_arch_io_alloc_enable() - Enable/disable io_alloc feature.
>> + * @r:		The resctrl resource.
>> + * @enable:	Enable (1) or disable (0) the feature
> 
> Please be consistent in using periods at end of sentence. 
> This could be made more specific with:
> 
> 	"Enable (true) or disable (false) io_alloc on resource @r."

Sure.

> 
>> + *
>> + * This can be called from any CPU.
>> + */
>> +int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable);
>> +
>>  extern unsigned int resctrl_rmid_realloc_threshold;
>>  extern unsigned int resctrl_rmid_realloc_limit;
>>  
> 
> Reinette
> 

-- 
Thanks
Babu Moger

Re: [PATCH v3 5/7] x86/resctrl: Add interface to enable/disable io_alloc feature

[Moger, Babu]

Hi Reinette,

On 3/21/25 17:58, Reinette Chatre wrote:
> Hi Babu,
> 
> On 1/30/25 1:20 PM, Babu Moger wrote:
>> The io_alloc feature in resctrl is a mechanism that enables direct
>> insertion of data from I/O devices into the L3 cache.
>>
>> On AMD systems, io_alloc feature is backed by SDCIAE (L3 Smart Data Cache
>> Injection Allocation Enforcement). When enabled, SDCIAE forces all SDCI
>> lines to be placed into the L3 cache partitions identified by the
>> highest-supported L3_MASK_n register as reported by CPUID
>> Fn0000_0010_EDX_x1.MAX_COS. For example, if MAX_COS=15, SDCI lines will
>> be allocated into the L3 cache partitions determined by the bitmask in
>> the L3_MASK_15 register.
>>
>> When CDP is enabled, io_alloc routes I/O traffic using the highest CLOSID
>> allocated for the instruction cache.
> 
> You can append a "L3CODE" to the above to help provide context on what resource
> is referred to as "instruction cache".

Sure.

> 
>>
>> Introduce interface to enable/disable "io_alloc" feature on user input.
>>
>> Signed-off-by: Babu Moger <babu.moger@amd.com>
>> ---
>> v3: Rewrote the change to make it generic.
>>     Rewrote the documentation in resctrl.rst to be generic and added
>>     AMD feature details in the end.
>>     Added the check to verify if MAX CLOSID availability on the system.
>>     Added CDP check to make sure io_alloc is configured in CDP_CODE.
>>     Added resctrl_io_alloc_closid_free() to free the io_alloc CLOSID.
>>     Added errors in few cases when CLOSID allocation fails.
>>     Fixes splat reported when info/L3/bit_usage is accesed when io_alloc
>>     is enabled.
>>     https://lore.kernel.org/lkml/SJ1PR11MB60837B532254E7B23BC27E84FC052@SJ1PR11MB6083.namprd11.prod.outlook.com/
>>
>> v2: Renamed the feature to "io_alloc".
>>     Added generic texts for the feature in commit log and resctrl.rst doc.
>>     Added resctrl_io_alloc_init_cat() to initialize io_alloc to default
>>     values when enabled.
>>     Fixed io_alloc show functinality to display only on L3 resource.
>> ---
>>  Documentation/arch/x86/resctrl.rst     |  34 ++++++
>>  arch/x86/kernel/cpu/resctrl/core.c     |   2 +
>>  arch/x86/kernel/cpu/resctrl/rdtgroup.c | 144 +++++++++++++++++++++++++
>>  3 files changed, 180 insertions(+)
>>
>> diff --git a/Documentation/arch/x86/resctrl.rst b/Documentation/arch/x86/resctrl.rst
>> index 6768fc1fad16..1b67e31d626c 100644
>> --- a/Documentation/arch/x86/resctrl.rst
>> +++ b/Documentation/arch/x86/resctrl.rst
>> @@ -135,6 +135,40 @@ related to allocation:
>>  			"1":
>>  			      Non-contiguous 1s value in CBM is supported.
>>  
>> +"io_alloc":
>> +		The "io_alloc" feature in resctrl enables system software to
> 
> Since this is already resctrl documentation "feature in resctrl" could be dropped to
> be just:
> 		"io_alloc" enables system software ...

Sure.

> 	
> 
>> +		configure the portion of the L3 cache allocated for I/O traffic.
>> +		By directly caching data from I/O devices rather than first storing
>> +		the I/O data in DRAM, reduces the demands on DRAM bandwidth and
>> +		reduces latency to the processor consuming the I/O data.
> 
> hmmm ... looks like "SDCIAE" was deleted from earlier used (marketing?) text and
> resulting text left as-is without re-checking if resulting text is still coherent.
> I do not think it is needed to motivate/market the feature here, perhaps last
> sentence can just be dropped?

Yes. I will drop the last sentence.

> 
>> +
>> +		The feature routes the I/O traffic via specific CLOSID reserved
>> +		for io_alloc feature. By configuring the CBM (Capacity Bit Mask)
>> +		for the CLOSID users can control the L3 portions available for
>> +		I/O traffic. When enabled, CLOSID reserved for the io_alloc will
>> +		not be available to the resctrl group.
> 
> Although the above reflects how SDCIAE is implemented it may not be true for how
> another architecture may support this. hmmm ... this sounds familiar and looking back it
> is the same thing I mentioned in V2 feedback, actually, in V2 I pointed to V1 feedback
> that said this also.
> If you insist on this text then please change the tone that indicates the
> behavior is optional. For example, "An architecture may support io_alloc by reserving
> a CLOSID to configure the ..."

Yes. Sure.

> 
>> +		::
>> +
>> +		  # cat /sys/fs/resctrl/info/L3/io_alloc
>> +		  0
> 
> Please refer to cover-letter about proposal to use enabled/disabled/not supported instead.

Yes. Got it.

> 
>> +
>> +		  "0":
>> +		      io_alloc feature is not enabled.
>> +		  "1":
>> +		      io_alloc feature is enabled, allowing users to manage
>> +		      the portions of the L3 cache allocated for the I/O device.
>> +
>> +		Feature can be enabled/disabled by writing to the interface.
>> +		Example::
>> +
>> +			# echo 1 > /sys/fs/resctrl/info/L3/io_alloc
>> +
>> +		On AMD systems, the io_alloc feature is supported by the L3 Smart
>> +		Data Cache Injection Allocation Enforcement (SDCIAE). The CLOSID for
>> +		io_alloc is determined by the highest CLOSID supported by the resource.
>> +		When CDP is enabled, io_alloc routes I/O traffic using the highest
>> +		CLOSID allocated for the instruction cache.
>> +
>>  Memory bandwidth(MB) subdirectory contains the following files
>>  with respect to allocation:
>>  
>> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
>> index 1ebdb2dcc009..88bc95c14ea8 100644
>> --- a/arch/x86/kernel/cpu/resctrl/core.c
>> +++ b/arch/x86/kernel/cpu/resctrl/core.c
>> @@ -309,6 +309,8 @@ static void rdt_get_cdp_config(int level)
>>  static void rdt_set_io_alloc_capable(struct rdt_resource *r)
>>  {
>>  	r->cache.io_alloc_capable = true;
>> +	resctrl_file_fflags_init("io_alloc",
>> +				 RFTYPE_CTRL_INFO | RFTYPE_RES_CACHE);
>>  }
> 
> Some MPAM changes landed since you created this work. After the fs/arch split the
> architecture code should have no insight into the resctrl file flags. Please refer to
> the MPAM changes on how this can be managed. You can refer to thread_throttle_mode_init()
> and similar to that resctrl can use the io_alloc_capable flag to make the files visible.

Yes. This needs change.

> 
>>  
>>  static void rdt_get_cdp_l3_config(void)
>> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>> index c5a0a31c3a85..37295dd14abe 100644
>> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>> @@ -62,6 +62,7 @@ static char last_cmd_status_buf[512];
>>  
>>  static int rdtgroup_setup_root(struct rdt_fs_context *ctx);
>>  static void rdtgroup_destroy_root(void);
>> +static int rdtgroup_init_cat(struct resctrl_schema *s, u32 closid);
>>  
>>  struct dentry *debugfs_resctrl;
>>  
>> @@ -180,6 +181,19 @@ void closid_free(int closid)
>>  	__set_bit(closid, &closid_free_map);
>>  }
>>  
>> +static int resctrl_io_alloc_closid_alloc(u32 io_alloc_closid)
>> +{
>> +	if (__test_and_clear_bit(io_alloc_closid, &closid_free_map))
>> +		return io_alloc_closid;
>> +	else
>> +		return -ENOSPC;
>> +}
>> +
>> +static void resctrl_io_alloc_closid_free(u32 io_alloc_closid)
>> +{
>> +	closid_free(io_alloc_closid);
>> +}
>> +
>>  /**
>>   * closid_allocated - test if provided closid is in use
>>   * @closid: closid to be tested
>> @@ -995,6 +1009,33 @@ static int rdt_shareable_bits_show(struct kernfs_open_file *of,
>>  	return 0;
>>  }
>>  
>> +/*
>> + * io_alloc feature uses max CLOSID to route the IO traffic.
>> + * Get the max CLOSID and verify if the CLOSID is available.
>> + */
>> +static int resctrl_io_alloc_closid_get(struct rdt_resource *r,
>> +				       struct resctrl_schema *s)
>> +{
>> +	int num_closids = resctrl_arch_get_num_closid(r);
>> +
>> +	/*
>> +	 * The number of CLOSIDs is determined based on the minimum
>> +	 * supported across all resources (in closid_init). It is stored
> 
> closid_init -> closid_init()

Sure.

> 
>> +	 * in s->num_closids. Also, if CDP is enabled number of CLOSIDs
>> +	 * are halved. To enable io_alloc feature, the number of CLOSIDs
>> +	 * must match the maximum CLOSID supported by the resource.
>> +	 */
>> +	if (resctrl_arch_get_cdp_enabled(r->rid))
>> +		num_closids /= 2;
>> +
>> +	if (s->num_closid != num_closids) {
> 
> Considering from schemata_list_add():
> 	s->num_closid = resctrl_arch_get_num_closid(r);
> 
> ... the above "if (s->num_closid != num_closids)" just compares the value to itself, no?
> 
> This function does not actually take all resources into account with the above
> comparison. I think what you may need here is a comparison with closid_free_map_len?

Yea. I need to change the logic here.

The max supported CLOSID on the resource and closid_free_map_len should
match. Will fix it.


> 
> As I understand it is still possible to use io_alloc when the resource's max CLOSID
> is not within closid_free_map, this is just not done simplify implementation.

That is correct.

> 
>> +		rdt_last_cmd_puts("Max CLOSID to support io_alloc is not available\n");
>> +		return -ENOSPC;
>> +	}
>> +
>> +	return num_closids - 1;
>> +}
>> +
>>  /*
>>   * rdt_bit_usage_show - Display current usage of resources
>>   *
>> @@ -1038,6 +1079,14 @@ static int rdt_bit_usage_show(struct kernfs_open_file *of,
>>  		for (i = 0; i < closids_supported(); i++) {
>>  			if (!closid_allocated(i))
>>  				continue;
>> +			/*
>> +			 * If io_alloc is enabled, the CLOSID will be
>> +			 * allocated but will not be associated with any
>> +			 * groups. Skip in that case.
> 
> This defeats the purpose of "bit_usage" that gives insight to user space
> on how the cache is allocated. Instead of ignoring portions of cache
> used for I/O this should display to the user that these portions are
> used by/shared with hardware.

Yes. Will change it.

> 
>> +			 */
>> +			if (i == resctrl_io_alloc_closid_get(r, s) &&
>> +			    resctrl_arch_get_io_alloc_enabled(r))
>> +				continue;
>>  			ctrl_val = resctrl_arch_get_config(r, dom, i,
>>  							   s->conf_type);
>>  			mode = rdtgroup_mode_by_closid(i);
>> @@ -1830,6 +1879,94 @@ int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable)
>>  	return 0;
>>  }
>>  
>> +static int resctrl_io_alloc_show(struct kernfs_open_file *of,
>> +				 struct seq_file *seq, void *v)
>> +{
>> +	struct resctrl_schema *s = of->kn->parent->priv;
>> +	struct rdt_resource *r = s->res;
>> +
>> +	seq_printf(seq, "%x\n", resctrl_arch_get_io_alloc_enabled(r));
>> +	return 0;
>> +}
>> +
>> +/*
>> + * Initialize io_alloc CLOSID cache resource with default CBM values.
>> + */
>> +static int resctrl_io_alloc_init_cat(struct rdt_resource *r,
>> +				     struct resctrl_schema *s, u32 closid)
>> +{
>> +	int ret;
>> +
>> +	rdt_staged_configs_clear();
>> +
>> +	ret = rdtgroup_init_cat(s, closid);
>> +	if (ret < 0)
>> +		goto out_init_cat;
>> +
>> +	ret = resctrl_arch_update_domains(r, closid);
>> +
>> +out_init_cat:
>> +	rdt_staged_configs_clear();
>> +	return ret;
>> +}
>> +
>> +static ssize_t resctrl_io_alloc_write(struct kernfs_open_file *of, char *buf,
>> +				      size_t nbytes, loff_t off)
>> +{
>> +	struct resctrl_schema *s = of->kn->parent->priv;
>> +	struct rdt_resource *r = s->res;
>> +	u32 io_alloc_closid;
>> +	bool enable;
>> +	int ret;
>> +
>> +	if (!r->cache.io_alloc_capable || s->conf_type == CDP_DATA) {
>> +		rdt_last_cmd_puts("io_alloc feature is not supported on the resource\n");
> 
> rdt_last_cmd_puts() starts with lockdep_assert_held(&rdtgroup_mutex), also expect
> rdt_last_cmd_clear() before first use.

Yes.

> 
> 
>> +		return -EINVAL;
> 
> Could ENODEV be used instead?

Sure.

> 
>> +	}
>> +
>> +	ret = kstrtobool(buf, &enable);
>> +	if (ret)
>> +		return ret;
>> +
>> +	cpus_read_lock();
>> +	mutex_lock(&rdtgroup_mutex);
>> +
>> +	rdt_last_cmd_clear();
>> +
>> +	io_alloc_closid = resctrl_io_alloc_closid_get(r, s);
>> +	if (io_alloc_closid < 0) {
> 
> Could you please add an informative message in last_cmd_status? It may be
> possible for user to remedy this and retry.

Yes.

> 
>> +		ret = -EINVAL;
>> +		goto out_io_alloc;
>> +	}
>> +
>> +	if (resctrl_arch_get_io_alloc_enabled(r) != enable) {
>> +		if (enable) {
>> +			ret = resctrl_io_alloc_closid_alloc(io_alloc_closid);
>> +			if (ret < 0) {
>> +				rdt_last_cmd_puts("CLOSID for io_alloc is not available\n");
> 
> If the CLOSID is not available then it may be possible for the user to remedy this by
> removing a resource group and retry this operation. Since CLOSID is not useful to user space
> (and x86 architecture specific) this could  be improved to give guidance to user
> space about which resource group (by name, not CLOSID) is preventing this from succeeding.

Sure.

> 
> (this sounded familiar, looks like I provided the same feedback to V2, to which you
> responded "Yes. We can do that.")

Yes. Clearly, I missed that.

> 
>> +				goto out_io_alloc;
>> +			}
>> +			ret = resctrl_io_alloc_init_cat(r, s, io_alloc_closid);
>> +			if (ret) {
>> +				rdt_last_cmd_puts("Failed to initialize io_alloc allocations\n");
>> +				resctrl_io_alloc_closid_free(io_alloc_closid);
>> +				goto out_io_alloc;
>> +			}
>> +
>> +		} else {
>> +			resctrl_io_alloc_closid_free(io_alloc_closid);
>> +		}
>> +
>> +		ret = resctrl_arch_io_alloc_enable(r, enable);
>> +	}
>> +
>> +out_io_alloc:
>> +	mutex_unlock(&rdtgroup_mutex);
>> +	cpus_read_unlock();
>> +
>> +	return ret ?: nbytes;
>> +}
>> +
>>  /* rdtgroup information files for one cache resource. */
>>  static struct rftype res_common_files[] = {
>>  	{
>> @@ -1982,6 +2119,13 @@ static struct rftype res_common_files[] = {
>>  		.seq_show	= rdtgroup_schemata_show,
>>  		.fflags		= RFTYPE_CTRL_BASE,
>>  	},
>> +	{
>> +		.name		= "io_alloc",
>> +		.mode		= 0644,
>> +		.kf_ops		= &rdtgroup_kf_single_ops,
>> +		.seq_show	= resctrl_io_alloc_show,
>> +		.write		= resctrl_io_alloc_write,
>> +	},
>>  	{
>>  		.name		= "mba_MBps_event",
>>  		.mode		= 0644,
> 
> Reinette
> 

-- 
Thanks
Babu Moger

Re: [PATCH v3 6/7] x86/resctrl: Introduce interface to display io_alloc CBMs

[Moger, Babu]

Hi Reinette,

On 3/21/25 17:58, Reinette Chatre wrote:
> Hi Babu,
> 
> On 1/30/25 1:20 PM, Babu Moger wrote:
>> The io_alloc feature in resctrl enables system software to configure
>> the portion of the L3 cache allocated for I/O traffic.
>>
>> Add the interface to display CBMs (Capacity Bit Mask) of io_alloc
>> feature.
>>
>> When CDP is enabled, io_alloc routes traffic using the highest CLOSID
>> which corresponds to CDP_CODE. Add a check for the CDP resource type.
> 
> It is not obvious to me what is meant with "highest CLOSID
> which corresponds to CDP_CODE" ... how about "highest CLOSID used by
> a L3CODE resource"?

Yes. That is correct.


>>
>> Signed-off-by: Babu Moger <babu.moger@amd.com>
>> ---
>> v3: Minor changes due to changes in resctrl_arch_get_io_alloc_enabled()
>>     and resctrl_io_alloc_closid_get().
>>     Added the check to verify CDP resource type.
>>     Updated the commit log.
>>
>> v2: Fixed to display only on L3 resources.
>>     Added the locks while processing.
>>     Rename the displat to io_alloc_cbm (from sdciae_cmd).
>> ---
>>  arch/x86/kernel/cpu/resctrl/core.c        |  2 ++
>>  arch/x86/kernel/cpu/resctrl/ctrlmondata.c |  2 +-
>>  arch/x86/kernel/cpu/resctrl/internal.h    |  1 +
>>  arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 38 +++++++++++++++++++++++
>>  4 files changed, 42 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
>> index 88bc95c14ea8..030f738dea8d 100644
>> --- a/arch/x86/kernel/cpu/resctrl/core.c
>> +++ b/arch/x86/kernel/cpu/resctrl/core.c
>> @@ -311,6 +311,8 @@ static void rdt_set_io_alloc_capable(struct rdt_resource *r)
>>  	r->cache.io_alloc_capable = true;
>>  	resctrl_file_fflags_init("io_alloc",
>>  				 RFTYPE_CTRL_INFO | RFTYPE_RES_CACHE);
>> +	resctrl_file_fflags_init("io_alloc_cbm",
>> +				 RFTYPE_CTRL_INFO | RFTYPE_RES_CACHE);
>>  }
>>  
>>  static void rdt_get_cdp_l3_config(void)
>> diff --git a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
>> index 536351159cc2..d272dea43924 100644
>> --- a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
>> +++ b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
>> @@ -444,7 +444,7 @@ u32 resctrl_arch_get_config(struct rdt_resource *r, struct rdt_ctrl_domain *d,
>>  	return hw_dom->ctrl_val[idx];
>>  }
>>  
>> -static void show_doms(struct seq_file *s, struct resctrl_schema *schema, int closid)
>> +void show_doms(struct seq_file *s, struct resctrl_schema *schema, int closid)
>>  {
>>  	struct rdt_resource *r = schema->res;
>>  	struct rdt_ctrl_domain *dom;
>> diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
>> index 61bc609e932b..07cf8409174d 100644
>> --- a/arch/x86/kernel/cpu/resctrl/internal.h
>> +++ b/arch/x86/kernel/cpu/resctrl/internal.h
>> @@ -668,4 +668,5 @@ void resctrl_file_fflags_init(const char *config, unsigned long fflags);
>>  void rdt_staged_configs_clear(void);
>>  bool closid_allocated(unsigned int closid);
>>  int resctrl_find_cleanest_closid(void);
>> +void show_doms(struct seq_file *s, struct resctrl_schema *schema, int closid);
>>  #endif /* _ASM_X86_RESCTRL_INTERNAL_H */
>> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>> index 37295dd14abe..81b9d8c5dabf 100644
>> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>> @@ -1967,6 +1967,38 @@ static ssize_t resctrl_io_alloc_write(struct kernfs_open_file *of, char *buf,
>>  	return ret ?: nbytes;
>>  }
>>  
>> +static int resctrl_io_alloc_cbm_show(struct kernfs_open_file *of,
>> +				     struct seq_file *seq, void *v)
>> +{
>> +	struct resctrl_schema *s = of->kn->parent->priv;
>> +	struct rdt_resource *r = s->res;
>> +	u32 io_alloc_closid;
>> +	int ret = 0;
>> +
>> +	if (!r->cache.io_alloc_capable || s->conf_type == CDP_DATA) {
>> +		rdt_last_cmd_puts("io_alloc feature is not supported on the resource\n");
> 
> rdt_last_cmd_puts() has to be called with rdtgroup_mutex held, also clear it before use.

Sure.

> 
>> +		return -EINVAL;
> 
> How about ENODEV?

Sure.

> 
>> +	}
>> +
>> +	cpus_read_lock();
>> +	mutex_lock(&rdtgroup_mutex);
>> +
>> +	if (!resctrl_arch_get_io_alloc_enabled(r)) {
>> +		rdt_last_cmd_puts("io_alloc feature is not enabled\n");
>> +		ret = -EINVAL;
>> +		goto cbm_show_out;
>> +	}
>> +
>> +	io_alloc_closid = resctrl_io_alloc_closid_get(r, s);
>> +
>> +	show_doms(seq, s, io_alloc_closid);
>> +
>> +cbm_show_out:
>> +	mutex_unlock(&rdtgroup_mutex);
>> +	cpus_read_unlock();
>> +	return ret;
>> +}
>> +
>>  /* rdtgroup information files for one cache resource. */
>>  static struct rftype res_common_files[] = {
>>  	{
>> @@ -2126,6 +2158,12 @@ static struct rftype res_common_files[] = {
>>  		.seq_show	= resctrl_io_alloc_show,
>>  		.write		= resctrl_io_alloc_write,
>>  	},
>> +	{
>> +		.name		= "io_alloc_cbm",
>> +		.mode		= 0444,
>> +		.kf_ops		= &rdtgroup_kf_single_ops,
>> +		.seq_show	= resctrl_io_alloc_cbm_show,
>> +	},
>>  	{
>>  		.name		= "mba_MBps_event",
>>  		.mode		= 0644,
> 
> Reinette
> 

-- 
Thanks
Babu Moger

Re: [PATCH v3 7/7] x86/resctrl: Introduce interface to modify io_alloc Capacity Bit Masks

[Moger, Babu]

Hi Reinette,

On 3/21/25 18:00, Reinette Chatre wrote:
> Hi Babu,
> 
> On 1/30/25 1:20 PM, Babu Moger wrote:
>> "io_alloc" feature is a mechanism that enables direct insertion of data
>> from I/O devices into the L3 cache. By directly caching data from I/O
>> devices rather than first storing the I/O data in DRAM, it reduces the
>> demands on DRAM bandwidth and reduces latency to the processor consuming
>> the I/O data.
>>
>> io_alloc feature uses the highest CLOSID to route the traffic from I/O
>> devices. Provide the interface to modify io_alloc CBMs (Capacity Bit Mask)
>> when feature is enabled.
>>
>> Signed-off-by: Babu Moger <babu.moger@amd.com>
>> ---
>> v3: Minor changes due to changes in resctrl_arch_get_io_alloc_enabled()
>>     and resctrl_io_alloc_closid_get().
>>     Taken care of handling the CBM update when CDP is enabled.
>>     Updated the commit log to make it generic.
>>
>> v2: Added more generic text in documentation.
>> ---
>>  Documentation/arch/x86/resctrl.rst        |  12 ++
>>  arch/x86/kernel/cpu/resctrl/ctrlmondata.c |   2 +-
>>  arch/x86/kernel/cpu/resctrl/internal.h    |   1 +
>>  arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 134 +++++++++++++++++++++-
>>  4 files changed, 147 insertions(+), 2 deletions(-)
>>
>> diff --git a/Documentation/arch/x86/resctrl.rst b/Documentation/arch/x86/resctrl.rst
>> index 1b67e31d626c..29c8851bcc7f 100644
>> --- a/Documentation/arch/x86/resctrl.rst
>> +++ b/Documentation/arch/x86/resctrl.rst
>> @@ -169,6 +169,18 @@ related to allocation:
>>  		When CDP is enabled, io_alloc routes I/O traffic using the highest
>>  		CLOSID allocated for the instruction cache.
>>  
>> +"io_alloc_cbm":
>> +		Capacity Bit Masks (CBMs) available to supported IO devices which
>> +		can directly insert cache lines in L3 which can help to reduce the
>> +		latency. CBM can be configured by writing to the interface in the
>> +		following format::
>> +
>> +			L3:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
> 
> This format is dependent on the resource name (not always L3).

Yes. Will remove "L3:"

> 
>> +
>> +		When CDP is enabled, L3 control is divided into two separate resources:
>> +		L3CODE and L3DATA. However, the CBM can only be updated on the L3CODE
>> +		resource.
>> +
>>  Memory bandwidth(MB) subdirectory contains the following files
>>  with respect to allocation:
>>  
>> diff --git a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
>> index d272dea43924..4dfee0436c1c 100644
>> --- a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
>> +++ b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
>> @@ -102,7 +102,7 @@ int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s,
>>   * requires at least two bits set.
>>   * AMD allows non-contiguous bitmasks.
>>   */
>> -static bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r)
>> +bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r)
>>  {
>>  	unsigned long first_bit, zero_bit, val;
>>  	unsigned int cbm_len = r->cache.cbm_len;
>> diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
>> index 07cf8409174d..702f6926bbdf 100644
>> --- a/arch/x86/kernel/cpu/resctrl/internal.h
>> +++ b/arch/x86/kernel/cpu/resctrl/internal.h
>> @@ -669,4 +669,5 @@ void rdt_staged_configs_clear(void);
>>  bool closid_allocated(unsigned int closid);
>>  int resctrl_find_cleanest_closid(void);
>>  void show_doms(struct seq_file *s, struct resctrl_schema *schema, int closid);
>> +bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r);
>>  #endif /* _ASM_X86_RESCTRL_INTERNAL_H */
>> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>> index 81b9d8c5dabf..9997cbfc1c19 100644
>> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>> @@ -1999,6 +1999,137 @@ static int resctrl_io_alloc_cbm_show(struct kernfs_open_file *of,
>>  	return ret;
>>  }
>>  
>> +/*
>> + * Read the CBM and check the validity. Make sure CBM is not shared
>> + * with any other exclusive resctrl groups.
>> + */
>> +static int resctrl_io_alloc_parse_cbm(char *buf, struct resctrl_schema *s,
>> +				      struct rdt_ctrl_domain *d)
>> +{
>> +	struct resctrl_staged_config *cfg;
>> +	struct rdt_resource *r = s->res;
>> +	u32 io_alloc_closid;
>> +	u32 cbm_val;
>> +
>> +	cfg = &d->staged_config[s->conf_type];
>> +	if (cfg->have_new_ctrl) {
>> +		rdt_last_cmd_printf("Duplicate domain %d\n", d->hdr.id);
>> +		return -EINVAL;
>> +	}
>> +
>> +	if (!cbm_validate(buf, &cbm_val, r))
>> +		return -EINVAL;
>> +
>> +	/*
>> +	 * The CBM may not overlap with other exclusive group.
>> +	 */
>> +	io_alloc_closid = resctrl_io_alloc_closid_get(r, s);
>> +	if (rdtgroup_cbm_overlaps(s, d, cbm_val, io_alloc_closid, true)) {
>> +		rdt_last_cmd_puts("Overlaps with exclusive group\n");
>> +		return -EINVAL;
>> +	}
>> +
>> +	cfg->new_ctrl = cbm_val;
>> +	cfg->have_new_ctrl = true;
>> +
>> +	return 0;
>> +}
> 
> Could you please reduce amount of duplication with parse_cbm()?

parse_cbm() needs rdtgrp to read 'mode' and 'closid' which is passed in
rdt_parse_data.


We can call parse_cbm directly if we add 'mode' and closid in
rdt_parse_data.  Will add those changes in next revision.


> 
> (for rest of patch, please check that related comments from previous patches
> are addressed here also)

Sure. Will do.

> 
> Reinette
> 

-- 
Thanks
Babu Moger

Re: [PATCH v3 0/7] Support L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

[Reinette Chatre]

Hi Babu,

On 4/7/25 1:12 PM, Moger, Babu wrote:
> On 3/21/25 17:50, Reinette Chatre wrote:
>> On 1/30/25 1:20 PM, Babu Moger wrote:


>>>
>>
>> AMD also supports what is exposed to user space as "shareable_bits". According
>> to APM:
>> 	Depending on the implementation, some portions of the L3 Cache may be
>> 	shared by other system functions or used for some other purpose not
>> 	under the control of the PQOS feature set. The L3 Cache Allocation
>> 	Sharing Mask returned by CPUID Fn0000_0010_EBX_x1[L3ShareAllocMask] is a
>> 	bitmask that represents portions of the L3 that may be shared by those
>> 	functions.
> 
> Here is the complete text.
> 
> The L3 Cache allocation sharing mask (L3ShareAllocMask) returned in EBX by
> CPUID Fn0000_0010 with ECX=1 is a bit vector which represents portions of
> the cache which may be shared with other system entities or used for some
> other purpose not under the control of the QOS feature set. When software
> sets a bit in one of the L3_MASK_n registers at the same bit positions a
> bit in the L3ShareAllocMask, processors executing with the corresponding
> COS will competitively share that portion of the cache with the other
> function. If this mask is all 0’s, then there is no other entity in the
> system competing with the processors for use of the L3 cache.
> 
> The "L3ShareAllocMask" is always reported as 0 on AMD systems.
> 
>> Could you please include what (if any) the relationship is between the CBM
>> discoverable via Fn0000_0010_EBX_x1[L3ShareAllocMask] and the CBM of 
>> "highest-supported L3_MASK_n register" when SDCIAE is enabled?
> 
> No. There is no relationship in here.
> 
>>
>> On the resctrl interface side the documentation currently states:
>>
>> 	"shareable_bits":                                                               
>> 		Bitmask of shareable resource with other executing              
>> 		entities (e.g. I/O). User can use this when                     
>> 		setting up exclusive cache partitions. Note that                
>> 		some platforms support devices that have their                  
>> 		own settings for cache use which can over-ride                  
>> 		these bits.                             
>>
>> Even though this was originally used to expose the content of
>> Fn0000_0010_EBX_x1[L3ShareAllocMask] the intent of the content does
>> seem to also apply to the "io_alloc" CBM also.
> 
> It says "shared by other system functions or used for some other purpose
> not under the control of the PQOS feature set".

This is a quote from the AMD spec, not the resctrl user interface documentation.

Please consider this from resctrl user interface perspective.

> 
> "io_alloc" is PQOS feature set. I feel it should not affect "shareable_bits".

When I read the resctrl user interface documentation for "shareable_bits" it
sounds relevant to io_alloc. The "shareable_bits" contains a bitmask "of
shareable resource with other executing entities (e.g. I/O)" ... is this
not exactly io_alloc? 


Reinette

Re: [PATCH v3 0/7] Support L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

[Moger, Babu]

Hi Reinette,

On 4/8/2025 4:44 PM, Reinette Chatre wrote:
> Hi Babu,
> 
> On 4/7/25 1:12 PM, Moger, Babu wrote:
>> On 3/21/25 17:50, Reinette Chatre wrote:
>>> On 1/30/25 1:20 PM, Babu Moger wrote:
> 
> 
>>>>
>>>
>>> AMD also supports what is exposed to user space as "shareable_bits". According
>>> to APM:
>>> 	Depending on the implementation, some portions of the L3 Cache may be
>>> 	shared by other system functions or used for some other purpose not
>>> 	under the control of the PQOS feature set. The L3 Cache Allocation
>>> 	Sharing Mask returned by CPUID Fn0000_0010_EBX_x1[L3ShareAllocMask] is a
>>> 	bitmask that represents portions of the L3 that may be shared by those
>>> 	functions.
>>
>> Here is the complete text.
>>
>> The L3 Cache allocation sharing mask (L3ShareAllocMask) returned in EBX by
>> CPUID Fn0000_0010 with ECX=1 is a bit vector which represents portions of
>> the cache which may be shared with other system entities or used for some
>> other purpose not under the control of the QOS feature set. When software
>> sets a bit in one of the L3_MASK_n registers at the same bit positions a
>> bit in the L3ShareAllocMask, processors executing with the corresponding
>> COS will competitively share that portion of the cache with the other
>> function. If this mask is all 0’s, then there is no other entity in the
>> system competing with the processors for use of the L3 cache.
>>
>> The "L3ShareAllocMask" is always reported as 0 on AMD systems.
>>
>>> Could you please include what (if any) the relationship is between the CBM
>>> discoverable via Fn0000_0010_EBX_x1[L3ShareAllocMask] and the CBM of
>>> "highest-supported L3_MASK_n register" when SDCIAE is enabled?
>>
>> No. There is no relationship in here.
>>
>>>
>>> On the resctrl interface side the documentation currently states:
>>>
>>> 	"shareable_bits":
>>> 		Bitmask of shareable resource with other executing
>>> 		entities (e.g. I/O). User can use this when
>>> 		setting up exclusive cache partitions. Note that
>>> 		some platforms support devices that have their
>>> 		own settings for cache use which can over-ride
>>> 		these bits.
>>>
>>> Even though this was originally used to expose the content of
>>> Fn0000_0010_EBX_x1[L3ShareAllocMask] the intent of the content does
>>> seem to also apply to the "io_alloc" CBM also.
>>
>> It says "shared by other system functions or used for some other purpose
>> not under the control of the PQOS feature set".
> 
> This is a quote from the AMD spec, not the resctrl user interface documentation.
> 
> Please consider this from resctrl user interface perspective.
> 
>>
>> "io_alloc" is PQOS feature set. I feel it should not affect "shareable_bits".
> 
> When I read the resctrl user interface documentation for "shareable_bits" it
> sounds relevant to io_alloc. The "shareable_bits" contains a bitmask "of
> shareable resource with other executing entities (e.g. I/O)" ... is this
> not exactly io_alloc?

I agree the text is pretty generic. Actually, the whole bit mask 
(0xfffF) is shareable with io_alloc.

The 'shareable_bits' is coming from CPUID 
Fn0000_0010_EBX_x1[L3ShareAllocMask] which is always 0 on AMD systems.
It will be bit odd to manipulate these value. Not sure if we have to do it.

Thanks
Babu

Re: [PATCH v3 0/7] Support L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

[Reinette Chatre]

Hi Babu,

On 4/8/25 5:41 PM, Moger, Babu wrote:
> Hi Reinette,
> 
> On 4/8/2025 4:44 PM, Reinette Chatre wrote:
>> Hi Babu,
>>
>> On 4/7/25 1:12 PM, Moger, Babu wrote:
>>> On 3/21/25 17:50, Reinette Chatre wrote:
>>>> On 1/30/25 1:20 PM, Babu Moger wrote:
>>
>>
>>>>>
>>>>
>>>> AMD also supports what is exposed to user space as "shareable_bits". According
>>>> to APM:
>>>>     Depending on the implementation, some portions of the L3 Cache may be
>>>>     shared by other system functions or used for some other purpose not
>>>>     under the control of the PQOS feature set. The L3 Cache Allocation
>>>>     Sharing Mask returned by CPUID Fn0000_0010_EBX_x1[L3ShareAllocMask] is a
>>>>     bitmask that represents portions of the L3 that may be shared by those
>>>>     functions.
>>>
>>> Here is the complete text.
>>>
>>> The L3 Cache allocation sharing mask (L3ShareAllocMask) returned in EBX by
>>> CPUID Fn0000_0010 with ECX=1 is a bit vector which represents portions of
>>> the cache which may be shared with other system entities or used for some
>>> other purpose not under the control of the QOS feature set. When software
>>> sets a bit in one of the L3_MASK_n registers at the same bit positions a
>>> bit in the L3ShareAllocMask, processors executing with the corresponding
>>> COS will competitively share that portion of the cache with the other
>>> function. If this mask is all 0’s, then there is no other entity in the
>>> system competing with the processors for use of the L3 cache.
>>>
>>> The "L3ShareAllocMask" is always reported as 0 on AMD systems.
>>>
>>>> Could you please include what (if any) the relationship is between the CBM
>>>> discoverable via Fn0000_0010_EBX_x1[L3ShareAllocMask] and the CBM of
>>>> "highest-supported L3_MASK_n register" when SDCIAE is enabled?
>>>
>>> No. There is no relationship in here.
>>>
>>>>
>>>> On the resctrl interface side the documentation currently states:
>>>>
>>>>     "shareable_bits":
>>>>         Bitmask of shareable resource with other executing
>>>>         entities (e.g. I/O). User can use this when
>>>>         setting up exclusive cache partitions. Note that
>>>>         some platforms support devices that have their
>>>>         own settings for cache use which can over-ride
>>>>         these bits.
>>>>
>>>> Even though this was originally used to expose the content of
>>>> Fn0000_0010_EBX_x1[L3ShareAllocMask] the intent of the content does
>>>> seem to also apply to the "io_alloc" CBM also.
>>>
>>> It says "shared by other system functions or used for some other purpose
>>> not under the control of the PQOS feature set".
>>
>> This is a quote from the AMD spec, not the resctrl user interface documentation.
>>
>> Please consider this from resctrl user interface perspective.
>>
>>>
>>> "io_alloc" is PQOS feature set. I feel it should not affect "shareable_bits".
>>
>> When I read the resctrl user interface documentation for "shareable_bits" it
>> sounds relevant to io_alloc. The "shareable_bits" contains a bitmask "of
>> shareable resource with other executing entities (e.g. I/O)" ... is this
>> not exactly io_alloc?
> 
> I agree the text is pretty generic. Actually, the whole bit mask (0xfffF) is shareable with io_alloc.

I think the value of "shareable_bits" presented to user space could be the
actual io_alloc_cbm value. Thus, not the "possible IO bitmask" but the actual
value. This seems to be the best match of what "shareable_bits" represents, which
is the region currently used by IO devices. This partners well with the "bit_usage"
output, for example, "X" can be used to show which portions of cache are currently
used by both software (via schemata of resource groups) and hardware (via io_alloc_cbm).

> 
> The 'shareable_bits' is coming from CPUID Fn0000_0010_EBX_x1[L3ShareAllocMask] which is always 0 on AMD systems.
> It will be bit odd to manipulate these value. Not sure if we have to do it.

It is not clear to me what you mean with "manipulate". "shareable_bits" does currently
come from the existing register but AMD now provides more interfaces with which this data
can be obtained and it seems appropriate to use it.

Reinette

Re: [PATCH v3 0/7] Support L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

[Moger, Babu]

Hi Reinette,

On 4/8/2025 8:41 PM, Reinette Chatre wrote:
> Hi Babu,
> 
> On 4/8/25 5:41 PM, Moger, Babu wrote:
>> Hi Reinette,
>>
>> On 4/8/2025 4:44 PM, Reinette Chatre wrote:
>>> Hi Babu,
>>>
>>> On 4/7/25 1:12 PM, Moger, Babu wrote:
>>>> On 3/21/25 17:50, Reinette Chatre wrote:
>>>>> On 1/30/25 1:20 PM, Babu Moger wrote:
>>>
>>>
>>>>>>
>>>>>
>>>>> AMD also supports what is exposed to user space as "shareable_bits". According
>>>>> to APM:
>>>>>      Depending on the implementation, some portions of the L3 Cache may be
>>>>>      shared by other system functions or used for some other purpose not
>>>>>      under the control of the PQOS feature set. The L3 Cache Allocation
>>>>>      Sharing Mask returned by CPUID Fn0000_0010_EBX_x1[L3ShareAllocMask] is a
>>>>>      bitmask that represents portions of the L3 that may be shared by those
>>>>>      functions.
>>>>
>>>> Here is the complete text.
>>>>
>>>> The L3 Cache allocation sharing mask (L3ShareAllocMask) returned in EBX by
>>>> CPUID Fn0000_0010 with ECX=1 is a bit vector which represents portions of
>>>> the cache which may be shared with other system entities or used for some
>>>> other purpose not under the control of the QOS feature set. When software
>>>> sets a bit in one of the L3_MASK_n registers at the same bit positions a
>>>> bit in the L3ShareAllocMask, processors executing with the corresponding
>>>> COS will competitively share that portion of the cache with the other
>>>> function. If this mask is all 0’s, then there is no other entity in the
>>>> system competing with the processors for use of the L3 cache.
>>>>
>>>> The "L3ShareAllocMask" is always reported as 0 on AMD systems.
>>>>
>>>>> Could you please include what (if any) the relationship is between the CBM
>>>>> discoverable via Fn0000_0010_EBX_x1[L3ShareAllocMask] and the CBM of
>>>>> "highest-supported L3_MASK_n register" when SDCIAE is enabled?
>>>>
>>>> No. There is no relationship in here.
>>>>
>>>>>
>>>>> On the resctrl interface side the documentation currently states:
>>>>>
>>>>>      "shareable_bits":
>>>>>          Bitmask of shareable resource with other executing
>>>>>          entities (e.g. I/O). User can use this when
>>>>>          setting up exclusive cache partitions. Note that
>>>>>          some platforms support devices that have their
>>>>>          own settings for cache use which can over-ride
>>>>>          these bits.
>>>>>
>>>>> Even though this was originally used to expose the content of
>>>>> Fn0000_0010_EBX_x1[L3ShareAllocMask] the intent of the content does
>>>>> seem to also apply to the "io_alloc" CBM also.
>>>>
>>>> It says "shared by other system functions or used for some other purpose
>>>> not under the control of the PQOS feature set".
>>>
>>> This is a quote from the AMD spec, not the resctrl user interface documentation.
>>>
>>> Please consider this from resctrl user interface perspective.
>>>
>>>>
>>>> "io_alloc" is PQOS feature set. I feel it should not affect "shareable_bits".
>>>
>>> When I read the resctrl user interface documentation for "shareable_bits" it
>>> sounds relevant to io_alloc. The "shareable_bits" contains a bitmask "of
>>> shareable resource with other executing entities (e.g. I/O)" ... is this
>>> not exactly io_alloc?
>>
>> I agree the text is pretty generic. Actually, the whole bit mask (0xfffF) is shareable with io_alloc.
> 
> I think the value of "shareable_bits" presented to user space could be the
> actual io_alloc_cbm value. Thus, not the "possible IO bitmask" but the actual

Confused little bit here. The shareable_bits is resource property.
io_alloc_cbm is domain specific value. Not sure how to display it.

> value. This seems to be the best match of what "shareable_bits" represents, which
> is the region currently used by IO devices. This partners well with the "bit_usage"
> output, for example, "X" can be used to show which portions of cache are currently
> used by both software (via schemata of resource groups) and hardware (via io_alloc_cbm).

Haven't looked at this code much. Will look into it.
  >>
>> The 'shareable_bits' is coming from CPUID Fn0000_0010_EBX_x1[L3ShareAllocMask] which is always 0 on AMD systems.
>> It will be bit odd to manipulate these value. Not sure if we have to do it.
> 
> It is not clear to me what you mean with "manipulate". "shareable_bits" does currently
> come from the existing register but AMD now provides more interfaces with which this data
> can be obtained and it seems appropriate to use it.

Ok.

Thanks
Babu

Re: [PATCH v3 0/7] Support L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

[Reinette Chatre]

Hi Babu,

On 4/9/25 5:58 PM, Moger, Babu wrote:
> Hi Reinette,
> 
> On 4/8/2025 8:41 PM, Reinette Chatre wrote:
>> Hi Babu,
>>
>> On 4/8/25 5:41 PM, Moger, Babu wrote:
>>> Hi Reinette,
>>>
>>> On 4/8/2025 4:44 PM, Reinette Chatre wrote:
>>>> Hi Babu,
>>>>
>>>> On 4/7/25 1:12 PM, Moger, Babu wrote:
>>>>> On 3/21/25 17:50, Reinette Chatre wrote:
>>>>>> On 1/30/25 1:20 PM, Babu Moger wrote:
>>>>
>>>>
>>>>>>>
>>>>>>
>>>>>> AMD also supports what is exposed to user space as "shareable_bits". According
>>>>>> to APM:
>>>>>>      Depending on the implementation, some portions of the L3 Cache may be
>>>>>>      shared by other system functions or used for some other purpose not
>>>>>>      under the control of the PQOS feature set. The L3 Cache Allocation
>>>>>>      Sharing Mask returned by CPUID Fn0000_0010_EBX_x1[L3ShareAllocMask] is a
>>>>>>      bitmask that represents portions of the L3 that may be shared by those
>>>>>>      functions.
>>>>>
>>>>> Here is the complete text.
>>>>>
>>>>> The L3 Cache allocation sharing mask (L3ShareAllocMask) returned in EBX by
>>>>> CPUID Fn0000_0010 with ECX=1 is a bit vector which represents portions of
>>>>> the cache which may be shared with other system entities or used for some
>>>>> other purpose not under the control of the QOS feature set. When software
>>>>> sets a bit in one of the L3_MASK_n registers at the same bit positions a
>>>>> bit in the L3ShareAllocMask, processors executing with the corresponding
>>>>> COS will competitively share that portion of the cache with the other
>>>>> function. If this mask is all 0’s, then there is no other entity in the
>>>>> system competing with the processors for use of the L3 cache.
>>>>>
>>>>> The "L3ShareAllocMask" is always reported as 0 on AMD systems.
>>>>>
>>>>>> Could you please include what (if any) the relationship is between the CBM
>>>>>> discoverable via Fn0000_0010_EBX_x1[L3ShareAllocMask] and the CBM of
>>>>>> "highest-supported L3_MASK_n register" when SDCIAE is enabled?
>>>>>
>>>>> No. There is no relationship in here.
>>>>>
>>>>>>
>>>>>> On the resctrl interface side the documentation currently states:
>>>>>>
>>>>>>      "shareable_bits":
>>>>>>          Bitmask of shareable resource with other executing
>>>>>>          entities (e.g. I/O). User can use this when
>>>>>>          setting up exclusive cache partitions. Note that
>>>>>>          some platforms support devices that have their
>>>>>>          own settings for cache use which can over-ride
>>>>>>          these bits.
>>>>>>
>>>>>> Even though this was originally used to expose the content of
>>>>>> Fn0000_0010_EBX_x1[L3ShareAllocMask] the intent of the content does
>>>>>> seem to also apply to the "io_alloc" CBM also.
>>>>>
>>>>> It says "shared by other system functions or used for some other purpose
>>>>> not under the control of the PQOS feature set".
>>>>
>>>> This is a quote from the AMD spec, not the resctrl user interface documentation.
>>>>
>>>> Please consider this from resctrl user interface perspective.
>>>>
>>>>>
>>>>> "io_alloc" is PQOS feature set. I feel it should not affect "shareable_bits".
>>>>
>>>> When I read the resctrl user interface documentation for "shareable_bits" it
>>>> sounds relevant to io_alloc. The "shareable_bits" contains a bitmask "of
>>>> shareable resource with other executing entities (e.g. I/O)" ... is this
>>>> not exactly io_alloc?
>>>
>>> I agree the text is pretty generic. Actually, the whole bit mask (0xfffF) is shareable with io_alloc.
>>
>> I think the value of "shareable_bits" presented to user space could be the
>> actual io_alloc_cbm value. Thus, not the "possible IO bitmask" but the actual
> 
> Confused little bit here. The shareable_bits is resource property.
> io_alloc_cbm is domain specific value. Not sure how to display it.

ah, right. We should still aim to not let the new "io_alloc" interfaces cause
confusion with the existing "shareable_bits" that is already part of interface and
used to communicate IO allocation to user space. Perhaps we are just left with
needing a modification to the existing documentation surrounding "shareable_bits"?

> 
>> value. This seems to be the best match of what "shareable_bits" represents, which
>> is the region currently used by IO devices. This partners well with the "bit_usage"
>> output, for example, "X" can be used to show which portions of cache are currently
>> used by both software (via schemata of resource groups) and hardware (via io_alloc_cbm).
> 
> Haven't looked at this code much. Will look into it.

Thank you. This is per domain so should be a good fit ... but again the documentation
creates strong connection with "shareable_bits" that should be reworked.

Reinette

Re: [PATCH v3 0/7] Support L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

[Moger, Babu]

Hi Reinette,

On 4/9/2025 10:59 PM, Reinette Chatre wrote:
> Hi Babu,
> 
> On 4/9/25 5:58 PM, Moger, Babu wrote:
>> Hi Reinette,
>>
>> On 4/8/2025 8:41 PM, Reinette Chatre wrote:
>>> Hi Babu,
>>>
>>> On 4/8/25 5:41 PM, Moger, Babu wrote:
>>>> Hi Reinette,
>>>>
>>>> On 4/8/2025 4:44 PM, Reinette Chatre wrote:
>>>>> Hi Babu,
>>>>>
>>>>> On 4/7/25 1:12 PM, Moger, Babu wrote:
>>>>>> On 3/21/25 17:50, Reinette Chatre wrote:
>>>>>>> On 1/30/25 1:20 PM, Babu Moger wrote:
>>>>>
>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> AMD also supports what is exposed to user space as "shareable_bits". According
>>>>>>> to APM:
>>>>>>>       Depending on the implementation, some portions of the L3 Cache may be
>>>>>>>       shared by other system functions or used for some other purpose not
>>>>>>>       under the control of the PQOS feature set. The L3 Cache Allocation
>>>>>>>       Sharing Mask returned by CPUID Fn0000_0010_EBX_x1[L3ShareAllocMask] is a
>>>>>>>       bitmask that represents portions of the L3 that may be shared by those
>>>>>>>       functions.
>>>>>>
>>>>>> Here is the complete text.
>>>>>>
>>>>>> The L3 Cache allocation sharing mask (L3ShareAllocMask) returned in EBX by
>>>>>> CPUID Fn0000_0010 with ECX=1 is a bit vector which represents portions of
>>>>>> the cache which may be shared with other system entities or used for some
>>>>>> other purpose not under the control of the QOS feature set. When software
>>>>>> sets a bit in one of the L3_MASK_n registers at the same bit positions a
>>>>>> bit in the L3ShareAllocMask, processors executing with the corresponding
>>>>>> COS will competitively share that portion of the cache with the other
>>>>>> function. If this mask is all 0’s, then there is no other entity in the
>>>>>> system competing with the processors for use of the L3 cache.
>>>>>>
>>>>>> The "L3ShareAllocMask" is always reported as 0 on AMD systems.
>>>>>>
>>>>>>> Could you please include what (if any) the relationship is between the CBM
>>>>>>> discoverable via Fn0000_0010_EBX_x1[L3ShareAllocMask] and the CBM of
>>>>>>> "highest-supported L3_MASK_n register" when SDCIAE is enabled?
>>>>>>
>>>>>> No. There is no relationship in here.
>>>>>>
>>>>>>>
>>>>>>> On the resctrl interface side the documentation currently states:
>>>>>>>
>>>>>>>       "shareable_bits":
>>>>>>>           Bitmask of shareable resource with other executing
>>>>>>>           entities (e.g. I/O). User can use this when
>>>>>>>           setting up exclusive cache partitions. Note that
>>>>>>>           some platforms support devices that have their
>>>>>>>           own settings for cache use which can over-ride
>>>>>>>           these bits.
>>>>>>>
>>>>>>> Even though this was originally used to expose the content of
>>>>>>> Fn0000_0010_EBX_x1[L3ShareAllocMask] the intent of the content does
>>>>>>> seem to also apply to the "io_alloc" CBM also.
>>>>>>
>>>>>> It says "shared by other system functions or used for some other purpose
>>>>>> not under the control of the PQOS feature set".
>>>>>
>>>>> This is a quote from the AMD spec, not the resctrl user interface documentation.
>>>>>
>>>>> Please consider this from resctrl user interface perspective.
>>>>>
>>>>>>
>>>>>> "io_alloc" is PQOS feature set. I feel it should not affect "shareable_bits".
>>>>>
>>>>> When I read the resctrl user interface documentation for "shareable_bits" it
>>>>> sounds relevant to io_alloc. The "shareable_bits" contains a bitmask "of
>>>>> shareable resource with other executing entities (e.g. I/O)" ... is this
>>>>> not exactly io_alloc?
>>>>
>>>> I agree the text is pretty generic. Actually, the whole bit mask (0xfffF) is shareable with io_alloc.
>>>
>>> I think the value of "shareable_bits" presented to user space could be the
>>> actual io_alloc_cbm value. Thus, not the "possible IO bitmask" but the actual
>>
>> Confused little bit here. The shareable_bits is resource property.
>> io_alloc_cbm is domain specific value. Not sure how to display it.
> 
> ah, right. We should still aim to not let the new "io_alloc" interfaces cause
> confusion with the existing "shareable_bits" that is already part of interface and
> used to communicate IO allocation to user space. Perhaps we are just left with
> needing a modification to the existing documentation surrounding "shareable_bits"?

Yes. Agree. Will add the details. Lets go from there.
Thanks
Babu