[PATCH v7 00/10] x86,fs/resctrl: Support L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

[PATCH v7 00/10] x86,fs/resctrl: Support L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

[Babu Moger]

This series adds the support for L3 Smart Data Cache Injection Allocation
Enforcement (SDCIAE) to resctrl infrastructure. It is referred to as
"io_alloc" in resctrl subsystem.

Upcoming AMD hardware implements Smart Data Cache Injection (SDCI).
Smart Data Cache Injection (SDCI) is a mechanism that enables direct
insertion of data from I/O devices into the L3 cache. By directly caching
data from I/O devices rather than first storing the I/O data in DRAM, SDCI
reduces demands on DRAM bandwidth and reduces latency to the processor
consuming the I/O data.

The SDCIAE (SDCI Allocation Enforcement) PQE feature allows system software
to control the portion of the L3 cache used for SDCI devices.

When enabled, SDCIAE forces all SDCI lines to be placed into the L3 cache
partitions identified by the highest-supported L3_MASK_n register, where n
is the maximum supported CLOSID.

The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache
Injection Allocation Enforcement (SDCIAE)
Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537

The feature requires linux support of TPH (TLP Processing Hints).
The support is available in linux kernel after the commit
48d0fd2b903e3 ("PCI/TPH: Add TPH documentation")

The patches are based on top of commit (6.16.0-rc5)
commit b4ec95e3bc3f ("Merge x86/microcode into tip/master")

# Linux Implementation

Feature adds following interface files when the resctrl "io_alloc" feature
is supported on the resource:

/sys/fs/resctrl/info/L3/io_alloc: Report the feature status. Enable/disable the
				  feature by writing to the interface.

/sys/fs/resctrl/info/L3/io_alloc_cbm:  List the Capacity Bit Masks (CBMs) available
				       for I/O devices when io_alloc feature is enabled.
				       Configure the CBM by writing to the interface.

When CDP is enabled, these files will be created both in L3CODE and L3DATA.

# Examples:

a. Check if io_alloc feature is available
	# mount -t resctrl resctrl /sys/fs/resctrl/

	# cat /sys/fs/resctrl/info/L3/io_alloc
	disabled

b. Enable the io_alloc feature. 

	# echo 1 > /sys/fs/resctrl/info/L3/io_alloc 
	# cat /sys/fs/resctrl/info/L3/io_alloc
	enabled

c. Check the CBM values for the io_alloc feature.

	# cat /sys/fs/resctrl/info/L3/io_alloc_cbm 
	0=ffff;1=ffff

d. Change the CBM value for the domain 1:
	# echo 1=FF > /sys/fs/resctrl/info/L3/io_alloc_cbm

	# cat /sys/fs/resctrl/info/L3/io_alloc_cbm 
	0=ffff;1=00ff

d. Disable io_alloc feature and exit.

	# echo 0 > /sys/fs/resctrl/info/L3/io_alloc 
	# cat /sys/fs/resctrl/info/L3/io_alloc
	disabled

	# umount /sys/fs/resctrl/

---
v7:
  Fixed few conflicts in
  arch/x86/include/asm/cpufeatures.h
  arch/x86/kernel/cpu/scattered.c

  Updated the changelog in most patches. Removed the references of L3 in
  filesystem related changes.

  Removed the inline for resctrl_arch_get_io_alloc_enabled().
  Updated the code comment in resctrl.h.
  Changed the subject to x86,fs/resctrl where applicable.
 
  Split the patches based on the comment.
  https://lore.kernel.org/lkml/3bec3844-7fda-452b-988f-42b0de9d63ba@intel.com/
  Separated resctrl_io_alloc_show and bit_usage changes in two separate patches.

  Added new function resctrl_io_alloc_closid_supported() to verify io_alloc CLOSID.
 
  Added the code to initialize/update the schemata for both CDP_DATA and CDP_CODE when CDP is enabled.

  Rephrased the changelog and code comments in all the patches.

v6: 
   Sorry if you see this series duplicate. Messed up the
   emails linux-doc@vger.kernel.org and linux-kernel@vger.kernel.org.

   Sent v5 by mistake before completing all testing.
   Most of the changes are in resctrl.rst user doc.
   The resource name is no longer printed in io_alloc_cbms.
   Updated the related documentation accordingly.
   Resolved conflicts in cpufeatures.h
   Added lockdep_assert_cpus_held() in _resctrl_sdciae_enable() to protect
   r->ctrl_domains.

   Added more comments in include/linux/resctrl.h.

   Updated "io_alloc_cbm" details in user doc resctrl.rst. Resource name is
   not printed in CBM now.

   Updated subjects to fs/resctrl: where applicable.

v5: 
    Patches are created on top of recent resctrl FS/ARCH code restructure.
    The files monitor.c/rdtgroup.c have been split between FS and ARCH directories.
    Resolved the conflict due to the merge.

    Updated bit_usage to reflect the io_alloc CBM as discussed in the thread:
    https://lore.kernel.org/lkml/3ca0a5dc-ad9c-4767-9011-b79d986e1e8d@intel.com/
    Modified rdt_bit_usage_show() to read io_alloc_cbm in hw_shareable, ensuring
    that bit_usage accurately represents the CBMs.

    Moved prototypes of resctrl_arch_io_alloc_enable() and
    resctrl_arch_get_io_alloc_enabled() to include/linux/resctrl.h.

    Used rdt_kn_name to get the rdtgroup name instead of accesssing it directly
    while printing group name used by the io_alloc_closid.

    Updated show_doms() to print the resource if only it is valid. Pass NULL while
    printing io_alloc CBM.

    Changed the code to access io_alloc CBMs via either L3CODE or L3DATA resources.

v4: The "io_alloc" interface will report "enabled/disabled/not supported"
    instead of 0 or 1..

    Updated resctrl_io_alloc_closid_get() to verify the max closid availability
    using closids_supported().

    Updated the documentation for "shareable_bits" and "bit_usage".

    NOTE: io_alloc is about specific CLOS. rdt_bit_usage_show() is not designed
    handle bit_usage for specific CLOS. Its about overall system. So, we cannot
    really tell the user which CLOS is shared across both hardware and software.
    This is something we need to discuss.

    Introduced io_alloc_init() to initialize fflags.

    Printed the group name when io_alloc enablement fails to help user.
    
    Added rdtgroup_mutex before rdt_last_cmd_puts() in resctrl_io_alloc_cbm_show().
    Returned -ENODEV when resource type is CDP_DATA.

    Kept the resource name while printing the CBM (L3:0=ffff) that way we dont have
    to change show_doms() just for this feature and it is consistant across all the
    schemata display.

    Added new patch to call parse_cbm() directly to avoid code duplication.

    Checked all the series(v1-v3) again to verify if I missed any comment.

v3: Rewrote commit log for the last 3 patches. Changed the text to bit
    more generic than the AMD specific feature. Added AMD feature
    specifics in the end.

    Renamed the rdt_get_sdciae_alloc_cfg() to rdt_set_io_alloc_capable().
    Renamed the _resctrl_io_alloc_enable() to _resctrl_sdciae_enable()
    as it is arch specific.

    Changed the return to void in _resctrl_sdciae_enable() instead of int.
 
    The number of CLOSIDs is determined based on the minimum supported
    across all resources (in closid_init). It needs to match the max
    supported on the resource. Added the check to verify if MAX CLOSID
    availability on the system.

    Added CDP check to make sure io_alloc is configured in CDP_CODE.
    Highest CLOSID corresponds to CDP_CODE. 

    Added resctrl_io_alloc_closid_free() to free the io_alloc CLOSID.

    Added errors in few cases when CLOSID allocation fails.
    Fixes splat reported when info/L3/bit_usage is accesed when io_alloc is enabled.
    https://lore.kernel.org/lkml/SJ1PR11MB60837B532254E7B23BC27E84FC052@SJ1PR11MB6083.namprd11.prod.outlook.com/

v2: Added dependancy on X86_FEATURE_CAT_L3
    Removed the "" in CPU feature definition.

    Changed sdciae_capable to io_alloc_capable to make it as generic feature.
    Moved io_alloc_capable field in struct resctrl_cache.

    Changed the name of few arch functions similar to ABMC series.
    resctrl_arch_get_io_alloc_enabled()
    resctrl_arch_io_alloc_enable()

    Renamed the feature to "io_alloc".
    Added generic texts for the feature in commit log and resctrl.rst doc.
    Added resctrl_io_alloc_init_cat() to initialize io_alloc to default values
    when enabled.
    Fixed io_alloc interface to show only on L3 resource.
    Added the locks while processing io_alloc CBMs.

Previous versions:
v6: https://lore.kernel.org/lkml/cover.1749677012.git.babu.moger@amd.com/
v5: https://lore.kernel.org/lkml/cover.1747943499.git.babu.moger@amd.com/
v4: https://lore.kernel.org/lkml/cover.1745275431.git.babu.moger@amd.com/
v3: https://lore.kernel.org/lkml/cover.1738272037.git.babu.moger@amd.com/
v2: https://lore.kernel.org/lkml/cover.1734556832.git.babu.moger@amd.com/
v1: https://lore.kernel.org/lkml/cover.1723824984.git.babu.moger@amd.com/

Babu Moger (10):
  x86/cpufeatures: Add support for L3 Smart Data Cache Injection
    Allocation Enforcement
  x86/resctrl: Add SDCIAE feature in the command line options
  x86,fs/resctrl: Detect io_alloc feature
  x86,fs/resctrl: Implement "io_alloc" enable/disable handlers
  fs/resctrl: Update bit_usage to reflect io_alloc
  fs/resctrl: Introduce interface to display "io_alloc" support
  fs/resctrl: Add user interface to enable/disable io_alloc feature
  fs/resctrl: Introduce interface to display io_alloc CBMs
  fs/resctrl: Modify rdt_parse_data to pass mode and CLOSID
  fs/resctrl: Introduce interface to modify io_alloc Capacity Bit Masks

 .../admin-guide/kernel-parameters.txt         |   2 +-
 Documentation/filesystems/resctrl.rst         |  79 +++-
 arch/x86/include/asm/cpufeatures.h            |   1 +
 arch/x86/include/asm/msr-index.h              |   1 +
 arch/x86/kernel/cpu/cpuid-deps.c              |   1 +
 arch/x86/kernel/cpu/resctrl/core.c            |   9 +
 arch/x86/kernel/cpu/resctrl/internal.h        |   5 +
 arch/x86/kernel/cpu/resctrl/rdtgroup.c        |  40 ++
 arch/x86/kernel/cpu/scattered.c               |   1 +
 fs/resctrl/ctrlmondata.c                      |  41 +-
 fs/resctrl/internal.h                         |  10 +
 fs/resctrl/rdtgroup.c                         | 374 +++++++++++++++++-
 include/linux/resctrl.h                       |  24 ++
 13 files changed, 558 insertions(+), 30 deletions(-)

-- 
2.34.1

[PATCH v7 01/10] x86/cpufeatures: Add support for L3 Smart Data Cache Injection Allocation Enforcement

[Babu Moger]

Smart Data Cache Injection (SDCI) is a mechanism that enables direct
insertion of data from I/O devices into the L3 cache. By directly caching
data from I/O devices rather than first storing the I/O data in DRAM,
SDCI reduces demands on DRAM bandwidth and reduces latency to the processor
consuming the I/O data.

The SDCIAE (SDCI Allocation Enforcement) PQE feature allows system software
to control the portion of the L3 cache used for SDCI.

When enabled, SDCIAE forces all SDCI lines to be placed into the L3 cache
partitions identified by the highest-supported L3_MASK_n register, where n
is the maximum supported CLOSID.

Add CPUID feature bit that can be used to configure SDCIAE.

The feature details are documented in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache
Injection Allocation Enforcement (SDCIAE)

Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: No changes. Fixed few conflicts in
   arch/x86/include/asm/cpufeatures.h
   arch/x86/kernel/cpu/scattered.c

v6: Resolved conflicts in cpufeatures.h.

v5: No changes.

v4: Resolved a minor conflict in cpufeatures.h.

v3: No changes.

v2: Added dependancy on X86_FEATURE_CAT_L3
    Removed the "" in CPU feature definition.
    Minor text changes.
---
 arch/x86/include/asm/cpufeatures.h | 1 +
 arch/x86/kernel/cpu/cpuid-deps.c   | 1 +
 arch/x86/kernel/cpu/scattered.c    | 1 +
 3 files changed, 3 insertions(+)

diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index 602957dd2609..db020bd60edc 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -494,6 +494,7 @@
 #define X86_FEATURE_TSA_SQ_NO		(21*32+11) /* AMD CPU not vulnerable to TSA-SQ */
 #define X86_FEATURE_TSA_L1_NO		(21*32+12) /* AMD CPU not vulnerable to TSA-L1 */
 #define X86_FEATURE_CLEAR_CPU_BUF_VM	(21*32+13) /* Clear CPU buffers using VERW before VMRUN */
+#define X86_FEATURE_SDCIAE		(21*32+14) /* L3 Smart Data Cache Injection Allocation Enforcement */
 
 /*
  * BUG word(s)
diff --git a/arch/x86/kernel/cpu/cpuid-deps.c b/arch/x86/kernel/cpu/cpuid-deps.c
index 46efcbd6afa4..87e78586395b 100644
--- a/arch/x86/kernel/cpu/cpuid-deps.c
+++ b/arch/x86/kernel/cpu/cpuid-deps.c
@@ -72,6 +72,7 @@ static const struct cpuid_dep cpuid_deps[] = {
 	{ X86_FEATURE_CQM_MBM_LOCAL,		X86_FEATURE_CQM_LLC   },
 	{ X86_FEATURE_BMEC,			X86_FEATURE_CQM_MBM_TOTAL   },
 	{ X86_FEATURE_BMEC,			X86_FEATURE_CQM_MBM_LOCAL   },
+	{ X86_FEATURE_SDCIAE,			X86_FEATURE_CAT_L3    },
 	{ X86_FEATURE_AVX512_BF16,		X86_FEATURE_AVX512VL  },
 	{ X86_FEATURE_AVX512_FP16,		X86_FEATURE_AVX512BW  },
 	{ X86_FEATURE_ENQCMD,			X86_FEATURE_XSAVES    },
diff --git a/arch/x86/kernel/cpu/scattered.c b/arch/x86/kernel/cpu/scattered.c
index b4a1f6732a3a..b22bfe81d262 100644
--- a/arch/x86/kernel/cpu/scattered.c
+++ b/arch/x86/kernel/cpu/scattered.c
@@ -50,6 +50,7 @@ static const struct cpuid_bit cpuid_bits[] = {
 	{ X86_FEATURE_MBA,			CPUID_EBX,  6, 0x80000008, 0 },
 	{ X86_FEATURE_SMBA,			CPUID_EBX,  2, 0x80000020, 0 },
 	{ X86_FEATURE_BMEC,			CPUID_EBX,  3, 0x80000020, 0 },
+	{ X86_FEATURE_SDCIAE,			CPUID_EBX,  6, 0x80000020, 0 },
 	{ X86_FEATURE_TSA_SQ_NO,		CPUID_ECX,  1, 0x80000021, 0 },
 	{ X86_FEATURE_TSA_L1_NO,		CPUID_ECX,  2, 0x80000021, 0 },
 	{ X86_FEATURE_AMD_WORKLOAD_CLASS,	CPUID_EAX, 22, 0x80000021, 0 },
-- 
2.34.1

[PATCH v7 02/10] x86/resctrl: Add SDCIAE feature in the command line options

[Babu Moger]

Add the command line options to enable or disable the new resctrl feature
L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE).

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: No changes.

v6: No changes.

v5: No changes.

v4: No changes.

v3: No changes.

v2: No changes.
---
 Documentation/admin-guide/kernel-parameters.txt | 2 +-
 arch/x86/kernel/cpu/resctrl/core.c              | 2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 07e22ba5bfe3..aa1c5948bec2 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -6066,7 +6066,7 @@
 	rdt=		[HW,X86,RDT]
 			Turn on/off individual RDT features. List is:
 			cmt, mbmtotal, mbmlocal, l3cat, l3cdp, l2cat, l2cdp,
-			mba, smba, bmec.
+			mba, smba, bmec, sdciae.
 			E.g. to turn on cmt and turn off mba use:
 				rdt=cmt,!mba
 
diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
index 187d527ef73b..f6d84882cc4e 100644
--- a/arch/x86/kernel/cpu/resctrl/core.c
+++ b/arch/x86/kernel/cpu/resctrl/core.c
@@ -707,6 +707,7 @@ enum {
 	RDT_FLAG_MBA,
 	RDT_FLAG_SMBA,
 	RDT_FLAG_BMEC,
+	RDT_FLAG_SDCIAE,
 };
 
 #define RDT_OPT(idx, n, f)	\
@@ -732,6 +733,7 @@ static struct rdt_options rdt_options[]  __ro_after_init = {
 	RDT_OPT(RDT_FLAG_MBA,	    "mba",	X86_FEATURE_MBA),
 	RDT_OPT(RDT_FLAG_SMBA,	    "smba",	X86_FEATURE_SMBA),
 	RDT_OPT(RDT_FLAG_BMEC,	    "bmec",	X86_FEATURE_BMEC),
+	RDT_OPT(RDT_FLAG_SDCIAE,    "sdciae",	X86_FEATURE_SDCIAE),
 };
 #define NUM_RDT_OPTIONS ARRAY_SIZE(rdt_options)
 
-- 
2.34.1

[PATCH v7 03/10] x86,fs/resctrl: Detect io_alloc feature

[Babu Moger]

Smart Data Cache Injection (SDCI) is a mechanism that enables direct
insertion of data from I/O devices into the L3 cache. It can reduce the
demands on DRAM bandwidth and reduces latency to the processor consuming
the I/O data.

Introduce cache resource property "io_alloc_capable" that an architecture
can set if a portion of the cache can be allocated for I/O traffic.

Set this property on x86 systems that support SDCIAE (L3 Smart Data Cache
Injection Allocation Enforcement). This property is set only for the L3
cache resource on systems that support SDCIAE.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: Few text updates in changelog and resctrl.h.

v6: No changes.

v5: No changes.

v4: Updated the commit message and code comment based on feedback.

v3: Rewrote commit log. Changed the text to bit generic than the AMD specific.
    Renamed the rdt_get_sdciae_alloc_cfg() to rdt_set_io_alloc_capable().
    Removed leftover comment from v2.

v2: Changed sdciae_capable to io_alloc_capable to make it generic feature.
    Also moved the io_alloc_capable in struct resctrl_cache.
---
 arch/x86/kernel/cpu/resctrl/core.c | 7 +++++++
 include/linux/resctrl.h            | 3 +++
 2 files changed, 10 insertions(+)

diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
index f6d84882cc4e..1d1002526745 100644
--- a/arch/x86/kernel/cpu/resctrl/core.c
+++ b/arch/x86/kernel/cpu/resctrl/core.c
@@ -274,6 +274,11 @@ static void rdt_get_cdp_config(int level)
 	rdt_resources_all[level].r_resctrl.cdp_capable = true;
 }
 
+static void rdt_set_io_alloc_capable(struct rdt_resource *r)
+{
+	r->cache.io_alloc_capable = true;
+}
+
 static void rdt_get_cdp_l3_config(void)
 {
 	rdt_get_cdp_config(RDT_RESOURCE_L3);
@@ -842,6 +847,8 @@ static __init bool get_rdt_alloc_resources(void)
 		rdt_get_cache_alloc_cfg(1, r);
 		if (rdt_cpu_has(X86_FEATURE_CDP_L3))
 			rdt_get_cdp_l3_config();
+		if (rdt_cpu_has(X86_FEATURE_SDCIAE))
+			rdt_set_io_alloc_capable(r);
 		ret = true;
 	}
 	if (rdt_cpu_has(X86_FEATURE_CAT_L2)) {
diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h
index 6fb4894b8cfd..010f238843b2 100644
--- a/include/linux/resctrl.h
+++ b/include/linux/resctrl.h
@@ -191,6 +191,8 @@ struct rdt_mon_domain {
  * @arch_has_sparse_bitmasks:	True if a bitmask like f00f is valid.
  * @arch_has_per_cpu_cfg:	True if QOS_CFG register for this cache
  *				level has CPU scope.
+ * @io_alloc_capable:	True if portion of the cache can be configured
+ *			for I/O traffic.
  */
 struct resctrl_cache {
 	unsigned int	cbm_len;
@@ -198,6 +200,7 @@ struct resctrl_cache {
 	unsigned int	shareable_bits;
 	bool		arch_has_sparse_bitmasks;
 	bool		arch_has_per_cpu_cfg;
+	bool		io_alloc_capable;
 };
 
 /**
-- 
2.34.1

[PATCH v7 04/10] x86,fs/resctrl: Implement "io_alloc" enable/disable handlers

[Babu Moger]

"io_alloc" enables direct insertion of data from I/O devices into the
cache.

On AMD systems, "io_alloc" feature is backed by L3 Smart Data Cache
Injection Allocation Enforcement (SDCIAE). Change SDCIAE state by setting
(to enable) or clearing (to disable) bit 1 of MSR L3_QOS_EXT_CFG on all
logical processors within the cache domain.

Introduce architecture-specific call to enable and disable the feature.

The SDCIAE feature details are available in APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache
Injection Allocation Enforcement (SDCIAE)

Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: Removed the inline for resctrl_arch_get_io_alloc_enabled().
    Update code comment in resctrl.h.
    Changed the subject to x86,fs/resctrl.

v6: Added lockdep_assert_cpus_held() in _resctrl_sdciae_enable() to protect
    r->ctrl_domains.
    Added more comments in include/linux/resctrl.h.

v5: Resolved conflicts due to recent resctrl FS/ARCH code restructure.
    The files monitor.c/rdtgroup.c have been split between FS and ARCH directories.
    Moved prototypes of resctrl_arch_io_alloc_enable() and
    resctrl_arch_get_io_alloc_enabled() to include/linux/resctrl.h.

v4: Updated the commit log to address the feedback.

v3: Passed the struct rdt_resource to resctrl_arch_get_io_alloc_enabled() instead of resource id.
    Renamed the _resctrl_io_alloc_enable() to _resctrl_sdciae_enable() as it is arch specific.
    Changed the return to void in _resctrl_sdciae_enable() instead of int.
    Added more context in commit log and fixed few typos.

v2: Renamed the functions to simplify the code.
    Renamed sdciae_capable to io_alloc_capable.

    Changed the name of few arch functions similar to ABMC series.
    resctrl_arch_get_io_alloc_enabled()
    resctrl_arch_io_alloc_enable()
---
 arch/x86/include/asm/msr-index.h       |  1 +
 arch/x86/kernel/cpu/resctrl/internal.h |  5 ++++
 arch/x86/kernel/cpu/resctrl/rdtgroup.c | 40 ++++++++++++++++++++++++++
 include/linux/resctrl.h                | 21 ++++++++++++++
 4 files changed, 67 insertions(+)

diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
index 7490bb5c0776..c998cf0e1375 100644
--- a/arch/x86/include/asm/msr-index.h
+++ b/arch/x86/include/asm/msr-index.h
@@ -1222,6 +1222,7 @@
 /* - AMD: */
 #define MSR_IA32_MBA_BW_BASE		0xc0000200
 #define MSR_IA32_SMBA_BW_BASE		0xc0000280
+#define MSR_IA32_L3_QOS_EXT_CFG		0xc00003ff
 #define MSR_IA32_EVT_CFG_BASE		0xc0000400
 
 /* AMD-V MSRs */
diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
index 5e3c41b36437..70f5317f1ce4 100644
--- a/arch/x86/kernel/cpu/resctrl/internal.h
+++ b/arch/x86/kernel/cpu/resctrl/internal.h
@@ -37,6 +37,9 @@ struct arch_mbm_state {
 	u64	prev_msr;
 };
 
+/* Setting bit 1 in L3_QOS_EXT_CFG enables the SDCIAE feature. */
+#define SDCIAE_ENABLE_BIT		1
+
 /**
  * struct rdt_hw_ctrl_domain - Arch private attributes of a set of CPUs that share
  *			       a resource for a control function
@@ -102,6 +105,7 @@ struct msr_param {
  * @mon_scale:		cqm counter * mon_scale = occupancy in bytes
  * @mbm_width:		Monitor width, to detect and correct for overflow.
  * @cdp_enabled:	CDP state of this resource
+ * @sdciae_enabled:	SDCIAE feature (backing "io_alloc") is enabled.
  *
  * Members of this structure are either private to the architecture
  * e.g. mbm_width, or accessed via helpers that provide abstraction. e.g.
@@ -115,6 +119,7 @@ struct rdt_hw_resource {
 	unsigned int		mon_scale;
 	unsigned int		mbm_width;
 	bool			cdp_enabled;
+	bool			sdciae_enabled;
 };
 
 static inline struct rdt_hw_resource *resctrl_to_arch_res(struct rdt_resource *r)
diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
index 885026468440..c165ac333336 100644
--- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
+++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
@@ -229,6 +229,46 @@ bool resctrl_arch_get_cdp_enabled(enum resctrl_res_level l)
 	return rdt_resources_all[l].cdp_enabled;
 }
 
+bool resctrl_arch_get_io_alloc_enabled(struct rdt_resource *r)
+{
+	return resctrl_to_arch_res(r)->sdciae_enabled;
+}
+
+static void resctrl_sdciae_set_one_amd(void *arg)
+{
+	bool *enable = arg;
+
+	if (*enable)
+		msr_set_bit(MSR_IA32_L3_QOS_EXT_CFG, SDCIAE_ENABLE_BIT);
+	else
+		msr_clear_bit(MSR_IA32_L3_QOS_EXT_CFG, SDCIAE_ENABLE_BIT);
+}
+
+static void _resctrl_sdciae_enable(struct rdt_resource *r, bool enable)
+{
+	struct rdt_ctrl_domain *d;
+
+	/* Walking r->ctrl_domains, ensure it can't race with cpuhp */
+	lockdep_assert_cpus_held();
+
+	/* Update L3_QOS_EXT_CFG MSR on all the CPUs in all domains */
+	list_for_each_entry(d, &r->ctrl_domains, hdr.list)
+		on_each_cpu_mask(&d->hdr.cpu_mask, resctrl_sdciae_set_one_amd, &enable, 1);
+}
+
+int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable)
+{
+	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
+
+	if (hw_res->r_resctrl.cache.io_alloc_capable &&
+	    hw_res->sdciae_enabled != enable) {
+		_resctrl_sdciae_enable(r, enable);
+		hw_res->sdciae_enabled = enable;
+	}
+
+	return 0;
+}
+
 void resctrl_arch_reset_all_ctrls(struct rdt_resource *r)
 {
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h
index 010f238843b2..d98933ce77af 100644
--- a/include/linux/resctrl.h
+++ b/include/linux/resctrl.h
@@ -531,6 +531,27 @@ void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain *
  */
 void resctrl_arch_reset_all_ctrls(struct rdt_resource *r);
 
+/**
+ * resctrl_arch_io_alloc_enable() - Enable/disable io_alloc feature.
+ * @r:		The resctrl resource.
+ * @enable:	Enable (true) or disable (false) io_alloc on resource @r.
+ *
+ * This can be called from any CPU.
+ *
+ * Return:
+ * 0 on success, <0 on error.
+ */
+int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable);
+
+/**
+ * resctrl_arch_get_io_alloc_enabled() - Get io_alloc feature state.
+ * @r:		The resctrl resource.
+ *
+ * Return:
+ * true if io_alloc is enabled or false if disabled.
+ */
+bool resctrl_arch_get_io_alloc_enabled(struct rdt_resource *r);
+
 extern unsigned int resctrl_rmid_realloc_threshold;
 extern unsigned int resctrl_rmid_realloc_limit;
 
-- 
2.34.1

[PATCH v7 05/10] fs/resctrl: Update bit_usage to reflect io_alloc

[Babu Moger]

When the io_alloc feature is enabled, a portion of the cache can be
configured for shared use between hardware and software.

Update the bit_usage representation to reflect the io_alloc configuration.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: New patch split from earlier patch #5.
    Added resctrl_io_alloc_closid() to return max COSID.
---
 Documentation/filesystems/resctrl.rst | 20 ++++++++++-----
 fs/resctrl/rdtgroup.c                 | 37 +++++++++++++++++++++++++--
 2 files changed, 49 insertions(+), 8 deletions(-)

diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
index c7949dd44f2f..c3c412733632 100644
--- a/Documentation/filesystems/resctrl.rst
+++ b/Documentation/filesystems/resctrl.rst
@@ -89,12 +89,20 @@ related to allocation:
 		must be set when writing a mask.
 
 "shareable_bits":
-		Bitmask of shareable resource with other executing
-		entities (e.g. I/O). User can use this when
-		setting up exclusive cache partitions. Note that
-		some platforms support devices that have their
-		own settings for cache use which can over-ride
-		these bits.
+		Bitmask of shareable resource with other executing entities
+		(e.g. I/O). Applies to all instances of this resource. User
+		can use this when setting up exclusive cache partitions.
+		Note that some platforms support devices that have their
+		own settings for cache use which can over-ride these bits.
+
+		When "io_alloc" feature is enabled, a portion of the cache
+		can be configured for shared use between hardware and software.
+
+		"bit_usage" should be used to see which portions of each cache
+		instance is configured for hardware use via "io_alloc" feature
+		because every cache instance can have its "io_alloc" bitmask
+		configured independently.
+
 "bit_usage":
 		Annotated capacity bitmasks showing how all
 		instances of the resource are used. The legend is:
diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
index 77d08229d855..a2eea85aecc8 100644
--- a/fs/resctrl/rdtgroup.c
+++ b/fs/resctrl/rdtgroup.c
@@ -1030,6 +1030,20 @@ static int rdt_shareable_bits_show(struct kernfs_open_file *of,
 	return 0;
 }
 
+/*
+ * resctrl_io_alloc_closid() - io_alloc feature routes I/O traffic using
+ * the highest available CLOSID. Retrieve the maximum CLOSID supported by the
+ * resource. Note that if Code Data Prioritization (CDP) is enabled, the number
+ * of available CLOSIDs is reduced by half.
+ */
+static u32 resctrl_io_alloc_closid(struct rdt_resource *r)
+{
+	if (resctrl_arch_get_cdp_enabled(r->rid))
+		return resctrl_arch_get_num_closid(r) / 2  - 1;
+	else
+		return resctrl_arch_get_num_closid(r) - 1;
+}
+
 /*
  * rdt_bit_usage_show - Display current usage of resources
  *
@@ -1063,15 +1077,17 @@ static int rdt_bit_usage_show(struct kernfs_open_file *of,
 
 	cpus_read_lock();
 	mutex_lock(&rdtgroup_mutex);
-	hw_shareable = r->cache.shareable_bits;
 	list_for_each_entry(dom, &r->ctrl_domains, hdr.list) {
 		if (sep)
 			seq_putc(seq, ';');
+		hw_shareable = r->cache.shareable_bits;
 		sw_shareable = 0;
 		exclusive = 0;
 		seq_printf(seq, "%d=", dom->hdr.id);
 		for (i = 0; i < closids_supported(); i++) {
-			if (!closid_allocated(i))
+			if (!closid_allocated(i) ||
+			    (resctrl_arch_get_io_alloc_enabled(r) &&
+			     i == resctrl_io_alloc_closid(r)))
 				continue;
 			ctrl_val = resctrl_arch_get_config(r, dom, i,
 							   s->conf_type);
@@ -1099,6 +1115,23 @@ static int rdt_bit_usage_show(struct kernfs_open_file *of,
 				break;
 			}
 		}
+
+		/*
+		 * When the "io_alloc" feature is enabled, a portion of the
+		 * cache is configured for shared use between hardware and software.
+		 */
+		if (resctrl_arch_get_io_alloc_enabled(r)) {
+			if (resctrl_arch_get_cdp_enabled(r->rid))
+				ctrl_val = resctrl_arch_get_config(r, dom,
+								   resctrl_io_alloc_closid(r),
+								   CDP_CODE);
+			else
+				ctrl_val = resctrl_arch_get_config(r, dom,
+								   resctrl_io_alloc_closid(r),
+								   CDP_NONE);
+			hw_shareable |= ctrl_val;
+		}
+
 		for (i = r->cache.cbm_len - 1; i >= 0; i--) {
 			pseudo_locked = dom->plr ? dom->plr->cbm : 0;
 			hwb = test_bit(i, &hw_shareable);
-- 
2.34.1

[PATCH v7 06/10] fs/resctrl: Introduce interface to display "io_alloc" support

[Babu Moger]

"io_alloc" feature in resctrl allows direct insertion of data from I/O
devices into the cache.

Introduce the 'io_alloc' resctrl file to indicate the support for the
feature.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: Updated the changelog.
    Updated user doc for io_alloc in resctrl.rst.
    Added mutex rdtgroup_mutex in resctrl_io_alloc_show();

v6: Added "io_alloc_cbm" details in user doc resctrl.rst.
    Resource name is not printed in CBM now. Corrected the texts about it
    in resctrl.rst.

v5: Resolved conflicts due to recent resctrl FS/ARCH code restructure.
    Updated show_doms() to print the resource if only it is valid. Pass NULL while
    printing io_alloc CBM.
    Changed the code to access the CBMs via either L3CODE or L3DATA resources.

v4: Updated the change log.
    Added rdtgroup_mutex before rdt_last_cmd_puts().
    Returned -ENODEV when resource type is CDP_DATA.
    Kept the resource name while printing the CBM (L3:0=fff) that way
    I dont have to change show_doms() just for this feature and it is
    consistant across all the schemata display.

v3: Minor changes due to changes in resctrl_arch_get_io_alloc_enabled()
    and resctrl_io_alloc_closid_get().
    Added the check to verify CDP resource type.
    Updated the commit log.

v2: Fixed to display only on L3 resources.
    Added the locks while processing.
    Rename the displat to io_alloc_cbm (from sdciae_cmd).
---
 Documentation/filesystems/resctrl.rst | 25 +++++++++++++++++
 fs/resctrl/rdtgroup.c                 | 39 +++++++++++++++++++++++++++
 2 files changed, 64 insertions(+)

diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
index c3c412733632..354e6a00fa45 100644
--- a/Documentation/filesystems/resctrl.rst
+++ b/Documentation/filesystems/resctrl.rst
@@ -143,6 +143,31 @@ related to allocation:
 			"1":
 			      Non-contiguous 1s value in CBM is supported.
 
+"io_alloc":
+		"io_alloc" enables system software to configure the portion of
+		the cache allocated for I/O traffic. File may only exist if the
+		system supports this feature on some of its cache resources.
+
+			"disabled":
+			      Portions of cache used for allocation of I/O traffic
+			      cannot be configured.
+			"enabled":
+			      Portions of cache used for allocation of I/O traffic
+			      can be configured using "io_alloc_cbm".
+			"not supported":
+			      Support not available on the system.
+
+		The underlying implementation may reduce resources available to
+		general (CPU) cache allocation. See architecture specific notes
+		below. Depending on usage requirements the feature can be enabled
+		or disabled:
+
+		On AMD systems, the io_alloc feature is supported by the L3 Smart
+		Data Cache Injection Allocation Enforcement (SDCIAE). The CLOSID for
+		io_alloc is determined by the highest CLOSID supported by the resource.
+		When CDP is enabled, io_alloc routes I/O traffic using the highest
+		CLOSID allocated for the instruction cache (L3CODE).
+
 Memory bandwidth(MB) subdirectory contains the following files
 with respect to allocation:
 
diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
index a2eea85aecc8..d7c4417b4516 100644
--- a/fs/resctrl/rdtgroup.c
+++ b/fs/resctrl/rdtgroup.c
@@ -1836,6 +1836,28 @@ static ssize_t mbm_local_bytes_config_write(struct kernfs_open_file *of,
 	return ret ?: nbytes;
 }
 
+static int resctrl_io_alloc_show(struct kernfs_open_file *of,
+				 struct seq_file *seq, void *v)
+{
+	struct resctrl_schema *s = rdt_kn_parent_priv(of->kn);
+	struct rdt_resource *r = s->res;
+
+	mutex_lock(&rdtgroup_mutex);
+
+	if (r->cache.io_alloc_capable) {
+		if (resctrl_arch_get_io_alloc_enabled(r))
+			seq_puts(seq, "enabled\n");
+		else
+			seq_puts(seq, "disabled\n");
+	} else {
+		seq_puts(seq, "not supported\n");
+	}
+
+	mutex_unlock(&rdtgroup_mutex);
+
+	return 0;
+}
+
 /* rdtgroup information files for one cache resource. */
 static struct rftype res_common_files[] = {
 	{
@@ -1926,6 +1948,12 @@ static struct rftype res_common_files[] = {
 		.kf_ops		= &rdtgroup_kf_single_ops,
 		.seq_show	= rdt_thread_throttle_mode_show,
 	},
+	{
+		.name		= "io_alloc",
+		.mode		= 0444,
+		.kf_ops		= &rdtgroup_kf_single_ops,
+		.seq_show	= resctrl_io_alloc_show,
+	},
 	{
 		.name		= "max_threshold_occupancy",
 		.mode		= 0644,
@@ -2095,6 +2123,15 @@ static void thread_throttle_mode_init(void)
 				 RFTYPE_CTRL_INFO | RFTYPE_RES_MB);
 }
 
+static void io_alloc_init(void)
+{
+	struct rdt_resource *r = resctrl_arch_get_resource(RDT_RESOURCE_L3);
+
+	if (r->cache.io_alloc_capable)
+		resctrl_file_fflags_init("io_alloc", RFTYPE_CTRL_INFO |
+					 RFTYPE_RES_CACHE);
+}
+
 void resctrl_file_fflags_init(const char *config, unsigned long fflags)
 {
 	struct rftype *rft;
@@ -4282,6 +4319,8 @@ int resctrl_init(void)
 
 	thread_throttle_mode_init();
 
+	io_alloc_init();
+
 	ret = resctrl_mon_resource_init();
 	if (ret)
 		return ret;
-- 
2.34.1

[PATCH v7 07/10] fs/resctrl: Add user interface to enable/disable io_alloc feature

[Babu Moger]

"io_alloc" feature in resctrl enables direct insertion of data from I/O
devices into the cache.

Introduce user interface to enable/disable io_alloc feature.

On AMD systems, io_alloc feature is backed by SDCIAE (L3 Smart Data Cache
Injection Allocation Enforcement). When enabled, SDCIAE directs all SDCI
lines to be placed into the L3 cache partitions specified by the register
corresponding to the highest CLOSID supported by the resource. With CDP
enabled, io_alloc routes I/O traffic using the highest CLOSID assigned to
the instruction cache (L3CODE).

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: Separated resctrl_io_alloc_show and bit_usage changes in two separate
    patches.
    Added resctrl_io_alloc_closid_supported() to verify io_alloc CLOSID.
    Initialized the schema for both CDP_DATA and CDP_CODE when CDP is enabled.

v6: Changed the subject to fs/resctrl:

v5: Resolved conflicts due to recent resctrl FS/ARCH code restructure.
    Used rdt_kn_name to get the rdtgroup name instead of accesssing it directly
    while printing group name used by the io_alloc_closid.

    Updated bit_usage to reflect the io_alloc CBM as discussed in the thread:
    https://lore.kernel.org/lkml/3ca0a5dc-ad9c-4767-9011-b79d986e1e8d@intel.com/
    Modified rdt_bit_usage_show() to read io_alloc_cbm in hw_shareable, ensuring
    that bit_usage accurately represents the CBMs.

    Updated the code to modify io_alloc either with L3CODE or L3DATA.
    https://lore.kernel.org/lkml/c00c00ea-a9ac-4c56-961c-dc5bf633476b@intel.com/

v4: Updated the change log.
    Updated the user doc.
    The "io_alloc" interface will report "enabled/disabled/not supported".
    Updated resctrl_io_alloc_closid_get() to verify the max closid availability.
    Updated the documentation for "shareable_bits" and "bit_usage".
    Introduced io_alloc_init() to initialize fflags.
    Printed the group name when io_alloc enablement fails.

    NOTE: io_alloc is about specific CLOS. rdt_bit_usage_show() is not designed
    handle bit_usage for specific CLOS. Its about overall system. So, we cannot
    really tell the user which CLOS is shared across both hardware and software.
    We need to discuss this.

v3: Rewrote the change to make it generic.
    Rewrote the documentation in resctrl.rst to be generic and added
    AMD feature details in the end.
    Added the check to verify if MAX CLOSID availability on the system.
    Added CDP check to make sure io_alloc is configured in CDP_CODE.
    Added resctrl_io_alloc_closid_free() to free the io_alloc CLOSID.
    Added errors in few cases when CLOSID allocation fails.
    Fixes splat reported when info/L3/bit_usage is accesed when io_alloc
    is enabled.
    https://lore.kernel.org/lkml/SJ1PR11MB60837B532254E7B23BC27E84FC052@SJ1PR11MB6083.namprd11.prod.outlook.com/

v2: Renamed the feature to "io_alloc".
    Added generic texts for the feature in commit log and resctrl.rst doc.
    Added resctrl_io_alloc_init_cat() to initialize io_alloc to default
    values when enabled.
    Fixed io_alloc show functinality to display only on L3 resource.
---
 Documentation/filesystems/resctrl.rst |   8 ++
 fs/resctrl/rdtgroup.c                 | 149 +++++++++++++++++++++++++-
 2 files changed, 156 insertions(+), 1 deletion(-)

diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
index 354e6a00fa45..189c1ccb92d6 100644
--- a/Documentation/filesystems/resctrl.rst
+++ b/Documentation/filesystems/resctrl.rst
@@ -157,6 +157,14 @@ related to allocation:
 			"not supported":
 			      Support not available on the system.
 
+		The feature can be modified by writing to the interface, for example:
+
+		To enable:
+			# echo 1 > /sys/fs/resctrl/info/L3/io_alloc
+
+		To disable:
+			# echo 0 > /sys/fs/resctrl/info/L3/io_alloc
+
 		The underlying implementation may reduce resources available to
 		general (CPU) cache allocation. See architecture specific notes
 		below. Depending on usage requirements the feature can be enabled
diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
index d7c4417b4516..06c854caa55c 100644
--- a/fs/resctrl/rdtgroup.c
+++ b/fs/resctrl/rdtgroup.c
@@ -70,6 +70,7 @@ static struct seq_buf last_cmd_status;
 static char last_cmd_status_buf[512];
 
 static int rdtgroup_setup_root(struct rdt_fs_context *ctx);
+static int rdtgroup_init_cat(struct resctrl_schema *s, u32 closid);
 
 static void rdtgroup_destroy_root(void);
 
@@ -232,6 +233,16 @@ bool closid_allocated(unsigned int closid)
 	return !test_bit(closid, closid_free_map);
 }
 
+static bool resctrl_io_alloc_closid_alloc(u32 io_alloc_closid)
+{
+	return __test_and_clear_bit(io_alloc_closid, closid_free_map);
+}
+
+static void resctrl_io_alloc_closid_free(u32 io_alloc_closid)
+{
+	closid_free(io_alloc_closid);
+}
+
 /**
  * rdtgroup_mode_by_closid - Return mode of resource group with closid
  * @closid: closid if the resource group
@@ -1030,6 +1041,16 @@ static int rdt_shareable_bits_show(struct kernfs_open_file *of,
 	return 0;
 }
 
+/*
+ * resctrl_io_alloc_closid_supported() - io_alloc feature utilizes the
+ * highest CLOSID value to direct I/O traffic. Ensure that io_alloc_closid
+ * is in the supported range.
+ */
+static bool resctrl_io_alloc_closid_supported(u32 io_alloc_closid)
+{
+	return io_alloc_closid < closids_supported();
+}
+
 /*
  * resctrl_io_alloc_closid() - io_alloc feature routes I/O traffic using
  * the highest available CLOSID. Retrieve the maximum CLOSID supported by the
@@ -1858,6 +1879,131 @@ static int resctrl_io_alloc_show(struct kernfs_open_file *of,
 	return 0;
 }
 
+/*
+ * Initialize io_alloc CLOSID cache resource CBM with all usable (shared
+ * and unused) cache portions.
+ */
+static int resctrl_io_alloc_init_cat(struct rdt_resource *r,
+				     struct resctrl_schema *s, u32 closid)
+{
+	int ret;
+
+	rdt_staged_configs_clear();
+
+	ret = rdtgroup_init_cat(s, closid);
+	if (ret < 0)
+		goto out;
+
+	ret = resctrl_arch_update_domains(r, closid);
+
+out:
+	rdt_staged_configs_clear();
+	return ret;
+}
+
+static const char *rdtgroup_name_by_closid(int closid)
+{
+	struct rdtgroup *rdtgrp;
+
+	list_for_each_entry(rdtgrp, &rdt_all_groups, rdtgroup_list) {
+		if (rdtgrp->closid == closid)
+			return rdt_kn_name(rdtgrp->kn);
+	}
+
+	return NULL;
+}
+
+static struct resctrl_schema *resctrl_get_schema(enum resctrl_conf_type type)
+{
+	struct resctrl_schema *schema;
+
+	list_for_each_entry(schema, &resctrl_schema_all, list) {
+		if (schema->conf_type == type)
+			return schema;
+	}
+
+	return NULL;
+}
+
+static ssize_t resctrl_io_alloc_write(struct kernfs_open_file *of, char *buf,
+				      size_t nbytes, loff_t off)
+{
+	struct resctrl_schema *s = rdt_kn_parent_priv(of->kn);
+	enum resctrl_conf_type peer_type;
+	struct rdt_resource *r = s->res;
+	struct resctrl_schema *peer_s;
+	char const *grp_name;
+	u32 io_alloc_closid;
+	bool enable;
+	int ret;
+
+	ret = kstrtobool(buf, &enable);
+	if (ret)
+		return ret;
+
+	cpus_read_lock();
+	mutex_lock(&rdtgroup_mutex);
+
+	rdt_last_cmd_clear();
+
+	if (!r->cache.io_alloc_capable) {
+		rdt_last_cmd_printf("io_alloc is not supported on %s\n", s->name);
+		ret = -ENODEV;
+		goto out_unlock;
+	}
+
+	io_alloc_closid = resctrl_io_alloc_closid(r);
+	if (!resctrl_io_alloc_closid_supported(io_alloc_closid)) {
+		rdt_last_cmd_printf("io_alloc CLOSID (ctrl_hw_id) %d is not available\n",
+				    io_alloc_closid);
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+
+	/* If the feature is already up to date, no action is needed. */
+	if (resctrl_arch_get_io_alloc_enabled(r) == enable)
+		goto out_unlock;
+
+	if (enable) {
+		if (!resctrl_io_alloc_closid_alloc(io_alloc_closid)) {
+			grp_name = rdtgroup_name_by_closid(io_alloc_closid);
+			WARN_ON_ONCE(!grp_name);
+			rdt_last_cmd_printf("CLOSID (ctrl_hw_id) %d for io_alloc is used by %s group\n",
+					    io_alloc_closid, grp_name ? grp_name : "another");
+			ret = -ENOSPC;
+			goto out_unlock;
+		}
+
+		/* Initialize schema for both CDP_DATA and CDP_CODE when CDP is enabled */
+		if (resctrl_arch_get_cdp_enabled(r->rid)) {
+			peer_type = resctrl_peer_type(s->conf_type);
+			peer_s = resctrl_get_schema(peer_type);
+			if (peer_s)
+				ret = resctrl_io_alloc_init_cat(r, peer_s, io_alloc_closid);
+		}
+
+		if (!ret)
+			ret = resctrl_io_alloc_init_cat(r, s, io_alloc_closid);
+
+		if (ret) {
+			rdt_last_cmd_puts("Failed to initialize io_alloc allocations\n");
+			resctrl_io_alloc_closid_free(io_alloc_closid);
+			goto out_unlock;
+		}
+
+	} else {
+		resctrl_io_alloc_closid_free(io_alloc_closid);
+	}
+
+	ret = resctrl_arch_io_alloc_enable(r, enable);
+
+out_unlock:
+	mutex_unlock(&rdtgroup_mutex);
+	cpus_read_unlock();
+
+	return ret ?: nbytes;
+}
+
 /* rdtgroup information files for one cache resource. */
 static struct rftype res_common_files[] = {
 	{
@@ -1950,9 +2096,10 @@ static struct rftype res_common_files[] = {
 	},
 	{
 		.name		= "io_alloc",
-		.mode		= 0444,
+		.mode		= 0644,
 		.kf_ops		= &rdtgroup_kf_single_ops,
 		.seq_show	= resctrl_io_alloc_show,
+		.write          = resctrl_io_alloc_write,
 	},
 	{
 		.name		= "max_threshold_occupancy",
-- 
2.34.1

[PATCH v7 08/10] fs/resctrl: Introduce interface to display io_alloc CBMs

[Babu Moger]

The io_alloc feature in resctrl enables system software to configure
the portion of the cache allocated for I/O traffic.

Add "io_alloc_cbm" resctrl file to display CBMs (Capacity Bit Mask) o
fio_alloc feature.

The CBM interface file io_alloc_cbm will reside in the info directory
(e.g., /sys/fs/resctrl/info/L3/). Displaying the resource name is not
necessary. Pass the resource name to show_doms() and print it only if
the name is valid. For io_alloc, pass NULL to suppress printing the
resource name.

When CDP is enabled, io_alloc routes traffic using the highest CLOSID
associated with an L3CODE resource. However, CBMs can be accessed via
either L3CODE or L3DATA resources.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: Updated changelog.
    Updated use doc (resctrl.rst).
    Removed if (io_alloc_closid < 0) check. Not required anymore.

v6: Added "io_alloc_cbm" details in user doc resctrl.rst.
    Resource name is not printed in CBM now. Corrected the texts about it
    in resctrl.rst.

v5: Resolved conflicts due to recent resctrl FS/ARCH code restructure.
    Updated show_doms() to print the resource if only it is valid. Pass NULL while
    printing io_alloc CBM.
    Changed the code to access the CBMs via either L3CODE or L3DATA resources.

v4: Updated the change log.
    Added rdtgroup_mutex before rdt_last_cmd_puts().
    Returned -ENODEV when resource type is CDP_DATA.
    Kept the resource name while printing the CBM (L3:0=fff) that way
    I dont have to change show_doms() just for this feature and it is
    consistant across all the schemata display.

v3: Minor changes due to changes in resctrl_arch_get_io_alloc_enabled()
    and resctrl_io_alloc_closid_get().
    Added the check to verify CDP resource type.
    Updated the commit log.

v2: Fixed to display only on L3 resources.
    Added the locks while processing.
    Rename the displat to io_alloc_cbm (from sdciae_cmd).
---
 Documentation/filesystems/resctrl.rst | 18 +++++++++++
 fs/resctrl/ctrlmondata.c              |  8 +++--
 fs/resctrl/internal.h                 |  2 ++
 fs/resctrl/rdtgroup.c                 | 43 ++++++++++++++++++++++++++-
 4 files changed, 67 insertions(+), 4 deletions(-)

diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
index 189c1ccb92d6..c22a9dd667cd 100644
--- a/Documentation/filesystems/resctrl.rst
+++ b/Documentation/filesystems/resctrl.rst
@@ -176,6 +176,24 @@ related to allocation:
 		When CDP is enabled, io_alloc routes I/O traffic using the highest
 		CLOSID allocated for the instruction cache (L3CODE).
 
+"io_alloc_cbm":
+		CBMs(Capacity Bit Masks) that describe the portions of cache instances
+		to which I/O traffic from supported I/O devices are routed.
+
+		CBMs are displayed in the following format:
+
+			<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
+
+		Example::
+
+			# cat /sys/fs/resctrl/info/L3/io_alloc_cbm
+			0=ffff;1=ffff
+
+		When CDP is enabled "io_alloc_cbm" associated with the DATA and CODE
+		resources may reflect the same values. For example, values read from and
+		written to /sys/fs/resctrl/info/L3DATA/io_alloc_cbm may be reflected by
+		/sys/fs/resctrl/info/L3CODE/io_alloc_cbm and vice versa.
+
 Memory bandwidth(MB) subdirectory contains the following files
 with respect to allocation:
 
diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c
index d98e0d2de09f..e78828b0408a 100644
--- a/fs/resctrl/ctrlmondata.c
+++ b/fs/resctrl/ctrlmondata.c
@@ -381,7 +381,8 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
 	return ret ?: nbytes;
 }
 
-static void show_doms(struct seq_file *s, struct resctrl_schema *schema, int closid)
+void show_doms(struct seq_file *s, struct resctrl_schema *schema, char *resource_name,
+	       int closid)
 {
 	struct rdt_resource *r = schema->res;
 	struct rdt_ctrl_domain *dom;
@@ -391,7 +392,8 @@ static void show_doms(struct seq_file *s, struct resctrl_schema *schema, int clo
 	/* Walking r->domains, ensure it can't race with cpuhp */
 	lockdep_assert_cpus_held();
 
-	seq_printf(s, "%*s:", max_name_width, schema->name);
+	if (resource_name)
+		seq_printf(s, "%*s:", max_name_width, resource_name);
 	list_for_each_entry(dom, &r->ctrl_domains, hdr.list) {
 		if (sep)
 			seq_puts(s, ";");
@@ -437,7 +439,7 @@ int rdtgroup_schemata_show(struct kernfs_open_file *of,
 			closid = rdtgrp->closid;
 			list_for_each_entry(schema, &resctrl_schema_all, list) {
 				if (closid < schema->num_closid)
-					show_doms(s, schema, closid);
+					show_doms(s, schema, schema->name, closid);
 			}
 		}
 	} else {
diff --git a/fs/resctrl/internal.h b/fs/resctrl/internal.h
index 0a1eedba2b03..3d4a3d7696d0 100644
--- a/fs/resctrl/internal.h
+++ b/fs/resctrl/internal.h
@@ -374,6 +374,8 @@ void rdt_staged_configs_clear(void);
 bool closid_allocated(unsigned int closid);
 
 int resctrl_find_cleanest_closid(void);
+void show_doms(struct seq_file *s, struct resctrl_schema *schema,
+	       char *name, int closid);
 
 #ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK
 int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp);
diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
index 06c854caa55c..008657e43656 100644
--- a/fs/resctrl/rdtgroup.c
+++ b/fs/resctrl/rdtgroup.c
@@ -2004,6 +2004,38 @@ static ssize_t resctrl_io_alloc_write(struct kernfs_open_file *of, char *buf,
 	return ret ?: nbytes;
 }
 
+static int resctrl_io_alloc_cbm_show(struct kernfs_open_file *of,
+				     struct seq_file *seq, void *v)
+{
+	struct resctrl_schema *s = rdt_kn_parent_priv(of->kn);
+	struct rdt_resource *r = s->res;
+	int ret = 0;
+
+	cpus_read_lock();
+	mutex_lock(&rdtgroup_mutex);
+
+	rdt_last_cmd_clear();
+
+	if (!r->cache.io_alloc_capable) {
+		rdt_last_cmd_printf("io_alloc is not supported on %s\n", s->name);
+		ret = -ENODEV;
+		goto out_unlock;
+	}
+
+	if (!resctrl_arch_get_io_alloc_enabled(r)) {
+		rdt_last_cmd_printf("io_alloc is not enabled on %s\n", s->name);
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+
+	show_doms(seq, s, NULL, resctrl_io_alloc_closid(r));
+
+out_unlock:
+	mutex_unlock(&rdtgroup_mutex);
+	cpus_read_unlock();
+	return ret;
+}
+
 /* rdtgroup information files for one cache resource. */
 static struct rftype res_common_files[] = {
 	{
@@ -2101,6 +2133,12 @@ static struct rftype res_common_files[] = {
 		.seq_show	= resctrl_io_alloc_show,
 		.write          = resctrl_io_alloc_write,
 	},
+	{
+		.name		= "io_alloc_cbm",
+		.mode		= 0444,
+		.kf_ops		= &rdtgroup_kf_single_ops,
+		.seq_show	= resctrl_io_alloc_cbm_show,
+	},
 	{
 		.name		= "max_threshold_occupancy",
 		.mode		= 0644,
@@ -2274,9 +2312,12 @@ static void io_alloc_init(void)
 {
 	struct rdt_resource *r = resctrl_arch_get_resource(RDT_RESOURCE_L3);
 
-	if (r->cache.io_alloc_capable)
+	if (r->cache.io_alloc_capable) {
 		resctrl_file_fflags_init("io_alloc", RFTYPE_CTRL_INFO |
 					 RFTYPE_RES_CACHE);
+		resctrl_file_fflags_init("io_alloc_cbm",
+					 RFTYPE_CTRL_INFO | RFTYPE_RES_CACHE);
+	}
 }
 
 void resctrl_file_fflags_init(const char *config, unsigned long fflags)
-- 
2.34.1

[PATCH v7 09/10] fs/resctrl: Modify rdt_parse_data to pass mode and CLOSID

[Babu Moger]

The functions parse_cbm() and parse_bw() require mode and CLOSID to
validate the Capacity Bit Mask (CBM). It is passed through struct
rdtgroup in rdt_parse_data.

This can be simplified by passing the mode and closid directly, instead of
through the rdtgroup struct. Doing so also facilitates calling parse_cbm()
to verify the CBM within the io_alloc feature, since io_alloc does not
have rdtgroup context.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: Rephrase of changelog.

v6: Changed the subject line to fs/resctrl.

v5: Resolved conflicts due to recent resctrl FS/ARCH code restructure.

v4: New patch to call parse_cbm() directly to avoid code duplication.
---
 fs/resctrl/ctrlmondata.c | 29 +++++++++++++----------------
 fs/resctrl/internal.h    |  6 ++++++
 2 files changed, 19 insertions(+), 16 deletions(-)

diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c
index e78828b0408a..5c16557fb7a8 100644
--- a/fs/resctrl/ctrlmondata.c
+++ b/fs/resctrl/ctrlmondata.c
@@ -23,11 +23,6 @@
 
 #include "internal.h"
 
-struct rdt_parse_data {
-	struct rdtgroup		*rdtgrp;
-	char			*buf;
-};
-
 typedef int (ctrlval_parser_t)(struct rdt_parse_data *data,
 			       struct resctrl_schema *s,
 			       struct rdt_ctrl_domain *d);
@@ -77,8 +72,8 @@ static int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s,
 		    struct rdt_ctrl_domain *d)
 {
 	struct resctrl_staged_config *cfg;
-	u32 closid = data->rdtgrp->closid;
 	struct rdt_resource *r = s->res;
+	u32 closid = data->closid;
 	u32 bw_val;
 
 	cfg = &d->staged_config[s->conf_type];
@@ -156,9 +151,10 @@ static bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r)
 static int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 		     struct rdt_ctrl_domain *d)
 {
-	struct rdtgroup *rdtgrp = data->rdtgrp;
+	enum rdtgrp_mode mode = data->mode;
 	struct resctrl_staged_config *cfg;
 	struct rdt_resource *r = s->res;
+	u32 closid = data->closid;
 	u32 cbm_val;
 
 	cfg = &d->staged_config[s->conf_type];
@@ -171,7 +167,7 @@ static int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 	 * Cannot set up more than one pseudo-locked region in a cache
 	 * hierarchy.
 	 */
-	if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP &&
+	if (mode == RDT_MODE_PSEUDO_LOCKSETUP &&
 	    rdtgroup_pseudo_locked_in_hierarchy(d)) {
 		rdt_last_cmd_puts("Pseudo-locked region in hierarchy\n");
 		return -EINVAL;
@@ -180,9 +176,9 @@ static int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 	if (!cbm_validate(data->buf, &cbm_val, r))
 		return -EINVAL;
 
-	if ((rdtgrp->mode == RDT_MODE_EXCLUSIVE ||
-	     rdtgrp->mode == RDT_MODE_SHAREABLE) &&
-	    rdtgroup_cbm_overlaps_pseudo_locked(d, cbm_val)) {
+	if ((mode == RDT_MODE_EXCLUSIVE ||
+	     mode == RDT_MODE_SHAREABLE) &&
+	     rdtgroup_cbm_overlaps_pseudo_locked(d, cbm_val)) {
 		rdt_last_cmd_puts("CBM overlaps with pseudo-locked region\n");
 		return -EINVAL;
 	}
@@ -191,14 +187,14 @@ static int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
 	 * The CBM may not overlap with the CBM of another closid if
 	 * either is exclusive.
 	 */
-	if (rdtgroup_cbm_overlaps(s, d, cbm_val, rdtgrp->closid, true)) {
+	if (rdtgroup_cbm_overlaps(s, d, cbm_val, closid, true)) {
 		rdt_last_cmd_puts("Overlaps with exclusive group\n");
 		return -EINVAL;
 	}
 
-	if (rdtgroup_cbm_overlaps(s, d, cbm_val, rdtgrp->closid, false)) {
-		if (rdtgrp->mode == RDT_MODE_EXCLUSIVE ||
-		    rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP) {
+	if (rdtgroup_cbm_overlaps(s, d, cbm_val, closid, false)) {
+		if (mode == RDT_MODE_EXCLUSIVE ||
+		    mode == RDT_MODE_PSEUDO_LOCKSETUP) {
 			rdt_last_cmd_puts("Overlaps with other group\n");
 			return -EINVAL;
 		}
@@ -262,7 +258,8 @@ static int parse_line(char *line, struct resctrl_schema *s,
 	list_for_each_entry(d, &r->ctrl_domains, hdr.list) {
 		if (d->hdr.id == dom_id) {
 			data.buf = dom;
-			data.rdtgrp = rdtgrp;
+			data.closid = rdtgrp->closid;
+			data.mode = rdtgrp->mode;
 			if (parse_ctrlval(&data, s, d))
 				return -EINVAL;
 			if (rdtgrp->mode ==  RDT_MODE_PSEUDO_LOCKSETUP) {
diff --git a/fs/resctrl/internal.h b/fs/resctrl/internal.h
index 3d4a3d7696d0..ab76a1e2e679 100644
--- a/fs/resctrl/internal.h
+++ b/fs/resctrl/internal.h
@@ -201,6 +201,12 @@ struct rdtgroup {
 	struct pseudo_lock_region	*plr;
 };
 
+struct rdt_parse_data {
+	u32			closid;
+	enum rdtgrp_mode	mode;
+	char			*buf;
+};
+
 /* rdtgroup.flags */
 #define	RDT_DELETED		1
 
-- 
2.34.1

[PATCH v7 10/10] fs/resctrl: Introduce interface to modify io_alloc Capacity Bit Masks

[Babu Moger]

"io_alloc" feature enables direct insertion of data from I/O devices into
the cache. By directly caching data from I/O devices rather than first
storing the I/O data in DRAM, it reduces the demands on DRAM bandwidth and
reduces latency to the processor consuming the I/O data.

Provide the interface to modify io_alloc CBMs (Capacity Bit Mask) when
feature is enabled. Update the CBMs for both CDP_DATA and CDP_CODE when CDP
is enabled.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
v7: Updated changelog.
    Updated CBMs for both CDP_DATA and CDP_CODE when CDP is enabled.

v6: Updated the user doc restctr.doc for minor texts.
    Changed the subject to fs/resctrl.

v5: Changes due to FS/ARCH code restructure. The files monitor.c/rdtgroup.c
    have been split between FS and ARCH directories.
    Changed the code to access the CBMs via either L3CODE or L3DATA resources.

v4: Removed resctrl_io_alloc_parse_cbm and called parse_cbm() directly.

v3: Minor changes due to changes in resctrl_arch_get_io_alloc_enabled()
    and resctrl_io_alloc_closid_get().
    Taken care of handling the CBM update when CDP is enabled.
    Updated the commit log to make it generic.

v2: Added more generic text in documentation.
---
 Documentation/filesystems/resctrl.rst |   8 ++
 fs/resctrl/ctrlmondata.c              |   4 +-
 fs/resctrl/internal.h                 |   2 +
 fs/resctrl/rdtgroup.c                 | 112 +++++++++++++++++++++++++-
 4 files changed, 123 insertions(+), 3 deletions(-)

diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
index c22a9dd667cd..b9c3ffdec27f 100644
--- a/Documentation/filesystems/resctrl.rst
+++ b/Documentation/filesystems/resctrl.rst
@@ -189,6 +189,14 @@ related to allocation:
 			# cat /sys/fs/resctrl/info/L3/io_alloc_cbm
 			0=ffff;1=ffff
 
+		CBMs can be configured by writing to the interface.
+
+		Example::
+
+			# echo 1=FF > /sys/fs/resctrl/info/L3/io_alloc_cbm
+			# cat /sys/fs/resctrl/info/L3/io_alloc_cbm
+			0=ffff;1=00ff
+
 		When CDP is enabled "io_alloc_cbm" associated with the DATA and CODE
 		resources may reflect the same values. For example, values read from and
 		written to /sys/fs/resctrl/info/L3DATA/io_alloc_cbm may be reflected by
diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c
index 5c16557fb7a8..23c72a5ac043 100644
--- a/fs/resctrl/ctrlmondata.c
+++ b/fs/resctrl/ctrlmondata.c
@@ -148,8 +148,8 @@ static bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r)
  * Read one cache bit mask (hex). Check that it is valid for the current
  * resource type.
  */
-static int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
-		     struct rdt_ctrl_domain *d)
+int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
+	      struct rdt_ctrl_domain *d)
 {
 	enum rdtgrp_mode mode = data->mode;
 	struct resctrl_staged_config *cfg;
diff --git a/fs/resctrl/internal.h b/fs/resctrl/internal.h
index ab76a1e2e679..011564e69ed5 100644
--- a/fs/resctrl/internal.h
+++ b/fs/resctrl/internal.h
@@ -382,6 +382,8 @@ bool closid_allocated(unsigned int closid);
 int resctrl_find_cleanest_closid(void);
 void show_doms(struct seq_file *s, struct resctrl_schema *schema,
 	       char *name, int closid);
+int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
+	      struct rdt_ctrl_domain *d);
 
 #ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK
 int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp);
diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
index 008657e43656..1093ec5e385b 100644
--- a/fs/resctrl/rdtgroup.c
+++ b/fs/resctrl/rdtgroup.c
@@ -2036,6 +2036,115 @@ static int resctrl_io_alloc_cbm_show(struct kernfs_open_file *of,
 	return ret;
 }
 
+static int resctrl_io_alloc_parse_line(char *line,  struct rdt_resource *r,
+				       struct resctrl_schema *s, u32 closid)
+{
+	struct rdt_parse_data data;
+	struct rdt_ctrl_domain *d;
+	char *dom = NULL, *id;
+	unsigned long dom_id;
+
+next:
+	if (!line || line[0] == '\0')
+		return 0;
+
+	dom = strsep(&line, ";");
+	id = strsep(&dom, "=");
+	if (!dom || kstrtoul(id, 10, &dom_id)) {
+		rdt_last_cmd_puts("Missing '=' or non-numeric domain\n");
+		return -EINVAL;
+	}
+
+	dom = strim(dom);
+	list_for_each_entry(d, &r->ctrl_domains, hdr.list) {
+		if (d->hdr.id == dom_id) {
+			data.buf = dom;
+			data.mode = RDT_MODE_SHAREABLE;
+			data.closid = closid;
+			if (parse_cbm(&data, s, d))
+				return -EINVAL;
+			goto next;
+		}
+	}
+	return -EINVAL;
+}
+
+static ssize_t resctrl_io_alloc_cbm_write(struct kernfs_open_file *of,
+					  char *buf, size_t nbytes, loff_t off)
+{
+	struct resctrl_schema *s = rdt_kn_parent_priv(of->kn);
+	enum resctrl_conf_type peer_type;
+	struct rdt_resource *r = s->res;
+	struct resctrl_schema *peer_s;
+	u32 io_alloc_closid;
+	char *peer_buf;
+	int ret = 0;
+
+	/* Valid input requires a trailing newline */
+	if (nbytes == 0 || buf[nbytes - 1] != '\n')
+		return -EINVAL;
+
+	buf[nbytes - 1] = '\0';
+
+	cpus_read_lock();
+	mutex_lock(&rdtgroup_mutex);
+
+	if (!r->cache.io_alloc_capable) {
+		rdt_last_cmd_printf("io_alloc is not supported on %s\n", s->name);
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+
+	rdt_last_cmd_clear();
+	rdt_staged_configs_clear();
+
+	if (!resctrl_arch_get_io_alloc_enabled(r)) {
+		rdt_last_cmd_printf("io_alloc is not enabled on %s\n", s->name);
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+
+	io_alloc_closid = resctrl_io_alloc_closid(r);
+
+	/*
+	 * When CDP is enabled, update the schema for both CDP_DATA and CDP_CODE.
+	 * Since the buffer is altered during parsing, create a copy to handle
+	 * peer schemata separately.
+	 */
+	if (resctrl_arch_get_cdp_enabled(r->rid)) {
+		peer_type = resctrl_peer_type(s->conf_type);
+		peer_s = resctrl_get_schema(peer_type);
+		peer_buf = kmalloc(nbytes, GFP_KERNEL);
+		if (!peer_buf) {
+			rdt_last_cmd_puts("Out of Memory - io_alloc update failed\n");
+			ret = -ENOMEM;
+			goto out_unlock;
+		}
+
+		memcpy(peer_buf, buf, nbytes);
+
+		if (peer_s)
+			ret = resctrl_io_alloc_parse_line(peer_buf, r, peer_s, io_alloc_closid);
+
+		kfree(peer_buf);
+	}
+
+	if (!ret)
+		ret = resctrl_io_alloc_parse_line(buf, r, s, io_alloc_closid);
+
+	if (ret)
+		goto out_unlock;
+
+	ret = resctrl_arch_update_domains(r, io_alloc_closid);
+
+out_unlock:
+	rdt_staged_configs_clear();
+	mutex_unlock(&rdtgroup_mutex);
+	cpus_read_unlock();
+
+	return ret ?: nbytes;
+}
+
 /* rdtgroup information files for one cache resource. */
 static struct rftype res_common_files[] = {
 	{
@@ -2135,9 +2244,10 @@ static struct rftype res_common_files[] = {
 	},
 	{
 		.name		= "io_alloc_cbm",
-		.mode		= 0444,
+		.mode		= 0644,
 		.kf_ops		= &rdtgroup_kf_single_ops,
 		.seq_show	= resctrl_io_alloc_cbm_show,
+		.write		= resctrl_io_alloc_cbm_write,
 	},
 	{
 		.name		= "max_threshold_occupancy",
-- 
2.34.1

Re: [PATCH v7 01/10] x86/cpufeatures: Add support for L3 Smart Data Cache Injection Allocation Enforcement

[Borislav Petkov]

On Thu, Jul 10, 2025 at 12:16:15PM -0500, Babu Moger wrote:
> Smart Data Cache Injection (SDCI) is a mechanism that enables direct
> insertion of data from I/O devices into the L3 cache. By directly caching
> data from I/O devices rather than first storing the I/O data in DRAM,
> SDCI reduces demands on DRAM bandwidth and reduces latency to the processor
> consuming the I/O data.
> 
> The SDCIAE (SDCI Allocation Enforcement) PQE feature allows system software
> to control the portion of the L3 cache used for SDCI.
> 
> When enabled, SDCIAE forces all SDCI lines to be placed into the L3 cache
> partitions identified by the highest-supported L3_MASK_n register, where n
> is the maximum supported CLOSID.
> 
> Add CPUID feature bit that can be used to configure SDCIAE.
> 
> The feature details are documented in APM listed below [1].
> [1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
> Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache
> Injection Allocation Enforcement (SDCIAE)
> 
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---
> v7: No changes. Fixed few conflicts in
>    arch/x86/include/asm/cpufeatures.h
>    arch/x86/kernel/cpu/scattered.c
> 
> v6: Resolved conflicts in cpufeatures.h.
> 
> v5: No changes.
> 
> v4: Resolved a minor conflict in cpufeatures.h.
> 
> v3: No changes.
> 
> v2: Added dependancy on X86_FEATURE_CAT_L3
>     Removed the "" in CPU feature definition.
>     Minor text changes.
> ---
>  arch/x86/include/asm/cpufeatures.h | 1 +
>  arch/x86/kernel/cpu/cpuid-deps.c   | 1 +
>  arch/x86/kernel/cpu/scattered.c    | 1 +
>  3 files changed, 3 insertions(+)

Acked-by: Borislav Petkov (AMD) <bp@alien8.de>

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH v7 00/10] x86,fs/resctrl: Support L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

[Reinette Chatre]

Hi Babu,

On 7/10/25 10:16 AM, Babu Moger wrote:
> 
> This series adds the support for L3 Smart Data Cache Injection Allocation
> Enforcement (SDCIAE) to resctrl infrastructure. It is referred to as
> "io_alloc" in resctrl subsystem.
> 
> Upcoming AMD hardware implements Smart Data Cache Injection (SDCI).
> Smart Data Cache Injection (SDCI) is a mechanism that enables direct
> insertion of data from I/O devices into the L3 cache. By directly caching
> data from I/O devices rather than first storing the I/O data in DRAM, SDCI
> reduces demands on DRAM bandwidth and reduces latency to the processor
> consuming the I/O data.
> 
> The SDCIAE (SDCI Allocation Enforcement) PQE feature allows system software
> to control the portion of the L3 cache used for SDCI devices.
> 
> When enabled, SDCIAE forces all SDCI lines to be placed into the L3 cache
> partitions identified by the highest-supported L3_MASK_n register, where n
> is the maximum supported CLOSID.

Even though this CLOSID use is unique to AMD this implementation makes it part
of resctrl fs's support of io_alloc. It is confusing to have architectural
specific features be handled by resctrl fs so I think it will be useful to
add a snippet here to help folks trying to decipher this work. Consider
for example a snippet like:

  Since CLOSIDs are managed by resctrl fs it is least invasive to make
  the "io_alloc is supported by maximum supported CLOSID" part of the
  initial resctrl fs support for io_alloc. Take care not to expose this
  use of CLOSID for io_alloc to user space so that this is not required from
  other architectures that may support io_alloc differently in the future.

> 
> The feature details are documented in the APM listed below [1].
> [1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
> Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache
> Injection Allocation Enforcement (SDCIAE)
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
> 
> The feature requires linux support of TPH (TLP Processing Hints).
> The support is available in linux kernel after the commit
> 48d0fd2b903e3 ("PCI/TPH: Add TPH documentation")
> 
> The patches are based on top of commit (6.16.0-rc5)
> commit b4ec95e3bc3f ("Merge x86/microcode into tip/master")
> 
> # Linux Implementation
> 
> Feature adds following interface files when the resctrl "io_alloc" feature
> is supported on the resource:
> 
> /sys/fs/resctrl/info/L3/io_alloc: Report the feature status. Enable/disable the
> 				  feature by writing to the interface.
> 
> /sys/fs/resctrl/info/L3/io_alloc_cbm:  List the Capacity Bit Masks (CBMs) available
> 				       for I/O devices when io_alloc feature is enabled.
> 				       Configure the CBM by writing to the interface.
> 
> When CDP is enabled, these files will be created both in L3CODE and L3DATA.

"will be" -> "are"

> 
> # Examples:
> 
> a. Check if io_alloc feature is available
> 	# mount -t resctrl resctrl /sys/fs/resctrl/
> 
> 	# cat /sys/fs/resctrl/info/L3/io_alloc
> 	disabled
> 
> b. Enable the io_alloc feature. 
> 
> 	# echo 1 > /sys/fs/resctrl/info/L3/io_alloc 
> 	# cat /sys/fs/resctrl/info/L3/io_alloc
> 	enabled
> 
> c. Check the CBM values for the io_alloc feature.
> 
> 	# cat /sys/fs/resctrl/info/L3/io_alloc_cbm 
> 	0=ffff;1=ffff
> 
> d. Change the CBM value for the domain 1:

"for the domain 1" -> "of domain 1"?

(nit: inconsistent use of "." vs. ":" in items)

> 	# echo 1=FF > /sys/fs/resctrl/info/L3/io_alloc_cbm
> 
> 	# cat /sys/fs/resctrl/info/L3/io_alloc_cbm 
> 	0=ffff;1=00ff
> 
> d. Disable io_alloc feature and exit.
> 
> 	# echo 0 > /sys/fs/resctrl/info/L3/io_alloc 
> 	# cat /sys/fs/resctrl/info/L3/io_alloc
> 	disabled
> 
> 	# umount /sys/fs/resctrl/
> 
> ---

Reinette

Re: [PATCH v7 01/10] x86/cpufeatures: Add support for L3 Smart Data Cache Injection Allocation Enforcement

[Reinette Chatre]

Hi Babu,

On 7/10/25 10:16 AM, Babu Moger wrote:
> Smart Data Cache Injection (SDCI) is a mechanism that enables direct
> insertion of data from I/O devices into the L3 cache. By directly caching
> data from I/O devices rather than first storing the I/O data in DRAM,
> SDCI reduces demands on DRAM bandwidth and reduces latency to the processor
> consuming the I/O data.
> 
> The SDCIAE (SDCI Allocation Enforcement) PQE feature allows system software
> to control the portion of the L3 cache used for SDCI.
> 
> When enabled, SDCIAE forces all SDCI lines to be placed into the L3 cache
> partitions identified by the highest-supported L3_MASK_n register, where n
> is the maximum supported CLOSID.
> 
> Add CPUID feature bit that can be used to configure SDCIAE.
> 
> The feature details are documented in APM listed below [1].
> [1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
> Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache
> Injection Allocation Enforcement (SDCIAE)
> 
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---

Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>

Reinette

Re: [PATCH v7 02/10] x86/resctrl: Add SDCIAE feature in the command line options

[Reinette Chatre]

Hi Babu,

On 7/10/25 10:16 AM, Babu Moger wrote:
> Add the command line options to enable or disable the new resctrl feature
> L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE).
> 
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---
...
> ---
>  Documentation/admin-guide/kernel-parameters.txt | 2 +-
>  arch/x86/kernel/cpu/resctrl/core.c              | 2 ++
>  2 files changed, 3 insertions(+), 1 deletion(-)

Could you please also add associated update to
Documentation/filesystems/resctrl.rst ? For reference, the similar
ABMC change:
https://lore.kernel.org/lkml/00cd603997e3ee6a389f83aef066fe7313b1abaf.1752013061.git.babu.moger@amd.com/

Reinette

Re: [PATCH v7 03/10] x86,fs/resctrl: Detect io_alloc feature

[Reinette Chatre]

Hi Babu,

On 7/10/25 10:16 AM, Babu Moger wrote:
> Smart Data Cache Injection (SDCI) is a mechanism that enables direct
> insertion of data from I/O devices into the L3 cache. It can reduce the
> demands on DRAM bandwidth and reduces latency to the processor consuming
> the I/O data.
> 
> Introduce cache resource property "io_alloc_capable" that an architecture
> can set if a portion of the cache can be allocated for I/O traffic.
> 
> Set this property on x86 systems that support SDCIAE (L3 Smart Data Cache
> Injection Allocation Enforcement). This property is set only for the L3
> cache resource on systems that support SDCIAE.
> 
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---

Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>

Reinette

Re: [PATCH v7 04/10] x86,fs/resctrl: Implement "io_alloc" enable/disable handlers

[Reinette Chatre]

Hi Babu,

On 7/10/25 10:16 AM, Babu Moger wrote:
> "io_alloc" enables direct insertion of data from I/O devices into the
> cache.
> 
> On AMD systems, "io_alloc" feature is backed by L3 Smart Data Cache
> Injection Allocation Enforcement (SDCIAE). Change SDCIAE state by setting
> (to enable) or clearing (to disable) bit 1 of MSR L3_QOS_EXT_CFG on all
> logical processors within the cache domain.
> 
> Introduce architecture-specific call to enable and disable the feature.
> 
> The SDCIAE feature details are available in APM listed below [1].
> [1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
> Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache
> Injection Allocation Enforcement (SDCIAE)
> 
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---

...

>  
> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> index 885026468440..c165ac333336 100644
> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c

The code looks good to me but rdtgroup.c is becoming bloated by being the default place for
any resctrl changes. While I do know there are counter examples (and I admit I do
not have a clear understanding of original intent) ctrlmondata.c is documented and supported
by its content as the place for cache allocation code. Could you please move these changes
to arch/x86/kernel/cpu/resctrl/ctrlmondata.c? I will try to highlight the other places where
I believe the either fs/resctrl/ctrlmondata.c or arch/x86/kernel/cpu/resctrl/ctrlmondata.c is
more appropriate but please as a high level consider where appropriate to move the IO alloc
code to be with the rest of the cache allocation code. Looking at the series it will help keep
a couple of existing static functions static and keep data structures local to cache allocation
code.

Reinette

Re: [PATCH v7 05/10] fs/resctrl: Update bit_usage to reflect io_alloc

[Reinette Chatre]

Hi Babu,

On 7/10/25 10:16 AM, Babu Moger wrote:
> When the io_alloc feature is enabled, a portion of the cache can be
> configured for shared use between hardware and software.
> 
> Update the bit_usage representation to reflect the io_alloc configuration.

This patch is early in the series but also relies on capabilities added
later in the series. This cryptic changelog leaves a lot for the user
to decipher. For example:
- How is the "io_alloc CLOSID" chosen? This is mentioned in cover letter but
  not here. Doing so here may help explain the hardcoding of CDP_CODE done
  in this patch (which seem unnecessary, more later, but a lot depends on
  changes that follow this patch).
- No mention that when io_alloc is enabled then an associated CLOSID will
  be allocated. This is done later in series but assumed to be known here
  where rdt_bit_usage_show() implicitly assumes that when io_alloc is enabled
  then the "io_alloc CLOSID" is supported AND allocated (otherwise array overrun?)

> 
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---
> v7: New patch split from earlier patch #5.
>     Added resctrl_io_alloc_closid() to return max COSID.
> ---
>  Documentation/filesystems/resctrl.rst | 20 ++++++++++-----
>  fs/resctrl/rdtgroup.c                 | 37 +++++++++++++++++++++++++--
>  2 files changed, 49 insertions(+), 8 deletions(-)
> 
> diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
> index c7949dd44f2f..c3c412733632 100644
> --- a/Documentation/filesystems/resctrl.rst
> +++ b/Documentation/filesystems/resctrl.rst
> @@ -89,12 +89,20 @@ related to allocation:
>  		must be set when writing a mask.
>  
>  "shareable_bits":
> -		Bitmask of shareable resource with other executing
> -		entities (e.g. I/O). User can use this when
> -		setting up exclusive cache partitions. Note that
> -		some platforms support devices that have their
> -		own settings for cache use which can over-ride
> -		these bits.
> +		Bitmask of shareable resource with other executing entities
> +		(e.g. I/O). Applies to all instances of this resource. User
> +		can use this when setting up exclusive cache partitions.
> +		Note that some platforms support devices that have their
> +		own settings for cache use which can over-ride these bits.
> +
> +		When "io_alloc" feature is enabled, a portion of the cache
> +		can be configured for shared use between hardware and software.

To help distinguish how "io_alloc" is different from "shareable_bits" this can be:
		When "io_alloc" is enabled, a portion of each cache instance can
		be configured for shared use between hardware and software.

Please merge these two paragraphs.

> +		"bit_usage" should be used to see which portions of each cache
> +		instance is configured for hardware use via "io_alloc" feature
> +		because every cache instance can have its "io_alloc" bitmask
> +		configured independently.

append " via "io_alloc_cbm"."?
(but io_alloc_cbm does not exist at this point ... another motivation for this
patch to move later)

> +
>  "bit_usage":
>  		Annotated capacity bitmasks showing how all
>  		instances of the resource are used. The legend is:

Please note that this "bit_usage" section contain several references to "shareable_bits"
that should be updated to refer to "io_alloc_cbm" also.

With all these new terms introduced as common knowledge it is starting to look
like this patch would be easier to consume later in the series.

> diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
> index 77d08229d855..a2eea85aecc8 100644
> --- a/fs/resctrl/rdtgroup.c
> +++ b/fs/resctrl/rdtgroup.c
> @@ -1030,6 +1030,20 @@ static int rdt_shareable_bits_show(struct kernfs_open_file *of,
>  	return 0;
>  }
>  
> +/*
> + * resctrl_io_alloc_closid() - io_alloc feature routes I/O traffic using
> + * the highest available CLOSID. Retrieve the maximum CLOSID supported by the
> + * resource. Note that if Code Data Prioritization (CDP) is enabled, the number
> + * of available CLOSIDs is reduced by half.
> + */
> +static u32 resctrl_io_alloc_closid(struct rdt_resource *r)

Please move to ctrlmondata.c.

> +{
> +	if (resctrl_arch_get_cdp_enabled(r->rid))
> +		return resctrl_arch_get_num_closid(r) / 2  - 1;
> +	else
> +		return resctrl_arch_get_num_closid(r) - 1;
> +}
> +
>  /*
>   * rdt_bit_usage_show - Display current usage of resources
>   *
> @@ -1063,15 +1077,17 @@ static int rdt_bit_usage_show(struct kernfs_open_file *of,
>  
>  	cpus_read_lock();
>  	mutex_lock(&rdtgroup_mutex);
> -	hw_shareable = r->cache.shareable_bits;
>  	list_for_each_entry(dom, &r->ctrl_domains, hdr.list) {
>  		if (sep)
>  			seq_putc(seq, ';');
> +		hw_shareable = r->cache.shareable_bits;
>  		sw_shareable = 0;
>  		exclusive = 0;
>  		seq_printf(seq, "%d=", dom->hdr.id);
>  		for (i = 0; i < closids_supported(); i++) {
> -			if (!closid_allocated(i))
> +			if (!closid_allocated(i) ||
> +			    (resctrl_arch_get_io_alloc_enabled(r) &&
> +			     i == resctrl_io_alloc_closid(r)))
>  				continue;
>  			ctrl_val = resctrl_arch_get_config(r, dom, i,
>  							   s->conf_type);
> @@ -1099,6 +1115,23 @@ static int rdt_bit_usage_show(struct kernfs_open_file *of,
>  				break;
>  			}
>  		}
> +
> +		/*
> +		 * When the "io_alloc" feature is enabled, a portion of the
> +		 * cache is configured for shared use between hardware and software.
> +		 */
> +		if (resctrl_arch_get_io_alloc_enabled(r)) {

Here is undocumented implicit assumption that when io_alloc is enabled then
the "io_alloc CLOSID" is allocated. This is also outside the closids_supported()
loop which adds the other implicit assumption that if io_alloc is enabled then
its CLOSID is supported by resctrl fs. None of these concepts have been introduced
so far and is not mentioned in changelog.
It is not obvious here that an io_alloc CLOSID must be supported (this is just
something enforced by later patches) and also not obvious that an io_alloc CLOSID
must be allocated from same "pool" as other CLOSIDs. Without a good changelog and
context of later patches this is hard to understand.
These are motivations for this patch to move later in the series and then the
changelog can just refer to these assumptions as fact, making it all easier to follow.

> +			if (resctrl_arch_get_cdp_enabled(r->rid))
> +				ctrl_val = resctrl_arch_get_config(r, dom,
> +								   resctrl_io_alloc_closid(r),
> +								   CDP_CODE);
> +			else
> +				ctrl_val = resctrl_arch_get_config(r, dom,
> +								   resctrl_io_alloc_closid(r),
> +								   CDP_NONE);

This does not look necessary to me. Why not just:
		if (resctrl_arch_get_io_alloc_enabled(r)) {
			ctrl_val = resctrl_arch_get_config(r, dom,
							   resctrl_io_alloc_closid(r),
							   s->conf_type);
			hw_shareable |= ctrl_val;
		}

Since the later patches keep the CDP_CODE and CDP_DATA CBMs in sync it does not matter
if the io_alloc CBM is obtained from CDP_CODE or CDP_DATA and just providing the
schema's type to resctrl_arch_get_config() will have it retrieve the right CBM, no?

This may also be easier to understand/claim if this patch is later in the series.


> +			hw_shareable |= ctrl_val;
> +		}
> +
>  		for (i = r->cache.cbm_len - 1; i >= 0; i--) {
>  			pseudo_locked = dom->plr ? dom->plr->cbm : 0;
>  			hwb = test_bit(i, &hw_shareable);

Reinette

Re: [PATCH v7 06/10] fs/resctrl: Introduce interface to display "io_alloc" support

[Reinette Chatre]

Hi Babu,

On 7/10/25 10:16 AM, Babu Moger wrote:
> "io_alloc" feature in resctrl allows direct insertion of data from I/O
> devices into the cache.
> 
> Introduce the 'io_alloc' resctrl file to indicate the support for the
> feature.
> 
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---

...

> ---
>  Documentation/filesystems/resctrl.rst | 25 +++++++++++++++++
>  fs/resctrl/rdtgroup.c                 | 39 +++++++++++++++++++++++++++
>  2 files changed, 64 insertions(+)
> 
> diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
> index c3c412733632..354e6a00fa45 100644
> --- a/Documentation/filesystems/resctrl.rst
> +++ b/Documentation/filesystems/resctrl.rst
> @@ -143,6 +143,31 @@ related to allocation:
>  			"1":
>  			      Non-contiguous 1s value in CBM is supported.
>  
> +"io_alloc":
> +		"io_alloc" enables system software to configure the portion of
> +		the cache allocated for I/O traffic. File may only exist if the
> +		system supports this feature on some of its cache resources.
> +
> +			"disabled":
> +			      Portions of cache used for allocation of I/O traffic
> +			      cannot be configured.
> +			"enabled":
> +			      Portions of cache used for allocation of I/O traffic
> +			      can be configured using "io_alloc_cbm".
> +			"not supported":
> +			      Support not available on the system.

"Support not available on the system." -> "Support not available for this resource."?

> +
> +		The underlying implementation may reduce resources available to
> +		general (CPU) cache allocation. See architecture specific notes
> +		below. Depending on usage requirements the feature can be enabled
> +		or disabled:

"disabled:" -> "disabled."?

> +
> +		On AMD systems, the io_alloc feature is supported by the L3 Smart
> +		Data Cache Injection Allocation Enforcement (SDCIAE). The CLOSID for
> +		io_alloc is determined by the highest CLOSID supported by the resource.

"is determined by the" -> "is the"?

To make clear connection with previous paragraph you can append something like:
	When io_alloc is enabled on an AMD system the highest CLOSID is dedicated to
	io_alloc and no longer available for general (CPU) cache allocation.

> +		When CDP is enabled, io_alloc routes I/O traffic using the highest
> +		CLOSID allocated for the instruction cache (L3CODE).

To clear up what happens with L3DATA, what do you think of appending something like:
		, making this CLOSID no longer available for general (CPU) cache
		allocation for both the L3CODE and L3DATA resources.

> +
>  Memory bandwidth(MB) subdirectory contains the following files
>  with respect to allocation:
>  
> diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
> index a2eea85aecc8..d7c4417b4516 100644
> --- a/fs/resctrl/rdtgroup.c
> +++ b/fs/resctrl/rdtgroup.c
> @@ -1836,6 +1836,28 @@ static ssize_t mbm_local_bytes_config_write(struct kernfs_open_file *of,
>  	return ret ?: nbytes;
>  }
>  
> +static int resctrl_io_alloc_show(struct kernfs_open_file *of,

Please move to ctrlmondata.c


> +				 struct seq_file *seq, void *v)
> +{
> +	struct resctrl_schema *s = rdt_kn_parent_priv(of->kn);
> +	struct rdt_resource *r = s->res;
> +
> +	mutex_lock(&rdtgroup_mutex);
> +
> +	if (r->cache.io_alloc_capable) {
> +		if (resctrl_arch_get_io_alloc_enabled(r))
> +			seq_puts(seq, "enabled\n");
> +		else
> +			seq_puts(seq, "disabled\n");
> +	} else {
> +		seq_puts(seq, "not supported\n");
> +	}
> +
> +	mutex_unlock(&rdtgroup_mutex);
> +
> +	return 0;
> +}
> +
>  /* rdtgroup information files for one cache resource. */
>  static struct rftype res_common_files[] = {
>  	{
> @@ -1926,6 +1948,12 @@ static struct rftype res_common_files[] = {
>  		.kf_ops		= &rdtgroup_kf_single_ops,
>  		.seq_show	= rdt_thread_throttle_mode_show,
>  	},
> +	{
> +		.name		= "io_alloc",
> +		.mode		= 0444,
> +		.kf_ops		= &rdtgroup_kf_single_ops,
> +		.seq_show	= resctrl_io_alloc_show,
> +	},
>  	{
>  		.name		= "max_threshold_occupancy",
>  		.mode		= 0644,
> @@ -2095,6 +2123,15 @@ static void thread_throttle_mode_init(void)
>  				 RFTYPE_CTRL_INFO | RFTYPE_RES_MB);
>  }
>  
> +static void io_alloc_init(void)

This function's comment can benefit from a snippet that highlights that
even though this operates on hardcoded L3 resource it results in this file
being visible for *all* cache resources (eg. L2 cache also), whether they
support io_alloc or not.

> +{
> +	struct rdt_resource *r = resctrl_arch_get_resource(RDT_RESOURCE_L3);
> +
> +	if (r->cache.io_alloc_capable)
> +		resctrl_file_fflags_init("io_alloc", RFTYPE_CTRL_INFO |
> +					 RFTYPE_RES_CACHE);
> +}
> +
>  void resctrl_file_fflags_init(const char *config, unsigned long fflags)
>  {
>  	struct rftype *rft;
> @@ -4282,6 +4319,8 @@ int resctrl_init(void)
>  
>  	thread_throttle_mode_init();
>  
> +	io_alloc_init();
> +
>  	ret = resctrl_mon_resource_init();
>  	if (ret)
>  		return ret;

Reinette

Re: [PATCH v7 07/10] fs/resctrl: Add user interface to enable/disable io_alloc feature

[Reinette Chatre]

Hi Babu,

On 7/10/25 10:16 AM, Babu Moger wrote:
> "io_alloc" feature in resctrl enables direct insertion of data from I/O
> devices into the cache.
> 
> Introduce user interface to enable/disable io_alloc feature.

I think it is worth a mention *why* a user may want to disable this feature and
why is not just always enabled. Here it can be highlighted that this feature
may take resources (CLOSID) away from general (CPU) cache allocation and since
this may be scarce enabling user to disable this feature supports different use cases.

> 
> On AMD systems, io_alloc feature is backed by SDCIAE (L3 Smart Data Cache
> Injection Allocation Enforcement). When enabled, SDCIAE directs all SDCI
> lines to be placed into the L3 cache partitions specified by the register
> corresponding to the highest CLOSID supported by the resource. With CDP
> enabled, io_alloc routes I/O traffic using the highest CLOSID assigned to
> the instruction cache (L3CODE).

This is a lot of architecture specific text for a resctrl fs patch  ... I think
you are trying to motivate the resctrl fs implementation. Similar motivation
as proposed for cover letter can be used here to help explain the implementation
choices.

> 
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---

...

> ---
>  Documentation/filesystems/resctrl.rst |   8 ++
>  fs/resctrl/rdtgroup.c                 | 149 +++++++++++++++++++++++++-
>  2 files changed, 156 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
> index 354e6a00fa45..189c1ccb92d6 100644
> --- a/Documentation/filesystems/resctrl.rst
> +++ b/Documentation/filesystems/resctrl.rst
> @@ -157,6 +157,14 @@ related to allocation:
>  			"not supported":
>  			      Support not available on the system.
>  
> +		The feature can be modified by writing to the interface, for example:
> +
> +		To enable:
> +			# echo 1 > /sys/fs/resctrl/info/L3/io_alloc
> +
> +		To disable:
> +			# echo 0 > /sys/fs/resctrl/info/L3/io_alloc
> +
>  		The underlying implementation may reduce resources available to
>  		general (CPU) cache allocation. See architecture specific notes
>  		below. Depending on usage requirements the feature can be enabled
> diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
> index d7c4417b4516..06c854caa55c 100644
> --- a/fs/resctrl/rdtgroup.c
> +++ b/fs/resctrl/rdtgroup.c
> @@ -70,6 +70,7 @@ static struct seq_buf last_cmd_status;
>  static char last_cmd_status_buf[512];
>  
>  static int rdtgroup_setup_root(struct rdt_fs_context *ctx);
> +static int rdtgroup_init_cat(struct resctrl_schema *s, u32 closid);
>  
>  static void rdtgroup_destroy_root(void);
>  
> @@ -232,6 +233,16 @@ bool closid_allocated(unsigned int closid)
>  	return !test_bit(closid, closid_free_map);
>  }
>  
> +static bool resctrl_io_alloc_closid_alloc(u32 io_alloc_closid)
> +{
> +	return __test_and_clear_bit(io_alloc_closid, closid_free_map);
> +}
> +
> +static void resctrl_io_alloc_closid_free(u32 io_alloc_closid)
> +{
> +	closid_free(io_alloc_closid);
> +}

I do not think these should be helpers/wrappers with a separate
namespace. It will make the code easier to understand if it is clear that
the "io_alloc" CLOSID is allocated from the same "pool" as the CLOSID for
control groups. 

I would thus propose a specific, for example closid_alloc_fixed(u32 closid)
helper, and just call closid_free() directly.

> +
>  /**
>   * rdtgroup_mode_by_closid - Return mode of resource group with closid
>   * @closid: closid if the resource group
> @@ -1030,6 +1041,16 @@ static int rdt_shareable_bits_show(struct kernfs_open_file *of,
>  	return 0;
>  }
>  
> +/*
> + * resctrl_io_alloc_closid_supported() - io_alloc feature utilizes the
> + * highest CLOSID value to direct I/O traffic. Ensure that io_alloc_closid
> + * is in the supported range.
> + */
> +static bool resctrl_io_alloc_closid_supported(u32 io_alloc_closid)

Please move to ctrlmondata.c

> +{
> +	return io_alloc_closid < closids_supported();
> +}
> +
>  /*
>   * resctrl_io_alloc_closid() - io_alloc feature routes I/O traffic using
>   * the highest available CLOSID. Retrieve the maximum CLOSID supported by the
> @@ -1858,6 +1879,131 @@ static int resctrl_io_alloc_show(struct kernfs_open_file *of,
>  	return 0;
>  }
>  
> +/*
> + * Initialize io_alloc CLOSID cache resource CBM with all usable (shared
> + * and unused) cache portions.
> + */
> +static int resctrl_io_alloc_init_cat(struct rdt_resource *r,

Mixing two features (io_alloc and CAT) in the function name is confusing.
How about resctrl_io_alloc_init_cbm() and move to ctrlmondata.c?

> +				     struct resctrl_schema *s, u32 closid)

No need to provide resource as parameter, it can be determined from schema.

> +{
> +	int ret;
> +
> +	rdt_staged_configs_clear();
> +
> +	ret = rdtgroup_init_cat(s, closid);
> +	if (ret < 0)
> +		goto out;

More below, but I think this flow can be simplified by moving the logic
handling CDP here. If CDP is enabled for the resource then a successful
rdtgroup_init_cat() can just be followed by a snippet that copies the
staged config of the CDP type to the staged config of its peer type.
Their CBMs are supposed to be identical so there is no reason for all the
rdtgroup_init_cat() processing to be repeated. resctrl_arch_update_domains()
can handle updating both in a single call.

> +
> +	ret = resctrl_arch_update_domains(r, closid);
> +
> +out:
> +	rdt_staged_configs_clear();
> +	return ret;
> +}
> +
> +static const char *rdtgroup_name_by_closid(int closid)

This seems generic enough and appropriate for rdtgroup.c

> +{
> +	struct rdtgroup *rdtgrp;
> +
> +	list_for_each_entry(rdtgrp, &rdt_all_groups, rdtgroup_list) {
> +		if (rdtgrp->closid == closid)
> +			return rdt_kn_name(rdtgrp->kn);
> +	}
> +
> +	return NULL;
> +}
> +
> +static struct resctrl_schema *resctrl_get_schema(enum resctrl_conf_type type)

This also seems generic enough and appropriate for rdtgroup.c

> +{
> +	struct resctrl_schema *schema;
> +
> +	list_for_each_entry(schema, &resctrl_schema_all, list) {
> +		if (schema->conf_type == type)
> +			return schema;
> +	}
> +
> +	return NULL;
> +}
> +
> +static ssize_t resctrl_io_alloc_write(struct kernfs_open_file *of, char *buf,
> +				      size_t nbytes, loff_t off)
> +{
> +	struct resctrl_schema *s = rdt_kn_parent_priv(of->kn);
> +	enum resctrl_conf_type peer_type;
> +	struct rdt_resource *r = s->res;
> +	struct resctrl_schema *peer_s;
> +	char const *grp_name;
> +	u32 io_alloc_closid;
> +	bool enable;
> +	int ret;
> +
> +	ret = kstrtobool(buf, &enable);
> +	if (ret)
> +		return ret;
> +
> +	cpus_read_lock();
> +	mutex_lock(&rdtgroup_mutex);
> +
> +	rdt_last_cmd_clear();
> +
> +	if (!r->cache.io_alloc_capable) {
> +		rdt_last_cmd_printf("io_alloc is not supported on %s\n", s->name);
> +		ret = -ENODEV;
> +		goto out_unlock;
> +	}
> +
> +	io_alloc_closid = resctrl_io_alloc_closid(r);
> +	if (!resctrl_io_alloc_closid_supported(io_alloc_closid)) {
> +		rdt_last_cmd_printf("io_alloc CLOSID (ctrl_hw_id) %d is not available\n",
> +				    io_alloc_closid);
> +		ret = -EINVAL;
> +		goto out_unlock;
> +	}
> +
> +	/* If the feature is already up to date, no action is needed. */
> +	if (resctrl_arch_get_io_alloc_enabled(r) == enable)
> +		goto out_unlock;
> +
> +	if (enable) {
> +		if (!resctrl_io_alloc_closid_alloc(io_alloc_closid)) {
> +			grp_name = rdtgroup_name_by_closid(io_alloc_closid);
> +			WARN_ON_ONCE(!grp_name);
> +			rdt_last_cmd_printf("CLOSID (ctrl_hw_id) %d for io_alloc is used by %s group\n",
> +					    io_alloc_closid, grp_name ? grp_name : "another");
> +			ret = -ENOSPC;
> +			goto out_unlock;
> +		}
> +
> +		/* Initialize schema for both CDP_DATA and CDP_CODE when CDP is enabled */
> +		if (resctrl_arch_get_cdp_enabled(r->rid)) {

I think this block can be removed to simplify the flow by moving the CDP handling to 
resctrl_io_alloc_init_cat().

> +			peer_type = resctrl_peer_type(s->conf_type);
> +			peer_s = resctrl_get_schema(peer_type);
> +			if (peer_s)
> +				ret = resctrl_io_alloc_init_cat(r, peer_s, io_alloc_closid);
> +		}
> +
> +		if (!ret)
> +			ret = resctrl_io_alloc_init_cat(r, s, io_alloc_closid);
> +
> +		if (ret) {
> +			rdt_last_cmd_puts("Failed to initialize io_alloc allocations\n");
> +			resctrl_io_alloc_closid_free(io_alloc_closid);
> +			goto out_unlock;
> +		}
> +
> +	} else {
> +		resctrl_io_alloc_closid_free(io_alloc_closid);
> +	}
> +
> +	ret = resctrl_arch_io_alloc_enable(r, enable);
> +
> +out_unlock:
> +	mutex_unlock(&rdtgroup_mutex);
> +	cpus_read_unlock();
> +
> +	return ret ?: nbytes;
> +}
> +
>  /* rdtgroup information files for one cache resource. */
>  static struct rftype res_common_files[] = {
>  	{
> @@ -1950,9 +2096,10 @@ static struct rftype res_common_files[] = {
>  	},
>  	{
>  		.name		= "io_alloc",
> -		.mode		= 0444,
> +		.mode		= 0644,
>  		.kf_ops		= &rdtgroup_kf_single_ops,
>  		.seq_show	= resctrl_io_alloc_show,
> +		.write          = resctrl_io_alloc_write,
>  	},
>  	{
>  		.name		= "max_threshold_occupancy",

Reinette

Re: [PATCH v7 08/10] fs/resctrl: Introduce interface to display io_alloc CBMs

[Reinette Chatre]

Hi Babu,

On 7/10/25 10:16 AM, Babu Moger wrote:
> The io_alloc feature in resctrl enables system software to configure
> the portion of the cache allocated for I/O traffic.
> 
> Add "io_alloc_cbm" resctrl file to display CBMs (Capacity Bit Mask) o
> fio_alloc feature.

"o fio_alloc" -> "of the io_alloc"

> 
> The CBM interface file io_alloc_cbm will reside in the info directory

"will reside" -> "resides"

> (e.g., /sys/fs/resctrl/info/L3/). Displaying the resource name is not
> necessary. Pass the resource name to show_doms() and print it only if
> the name is valid. For io_alloc, pass NULL to suppress printing the
> resource name.
> 
> When CDP is enabled, io_alloc routes traffic using the highest CLOSID
> associated with an L3CODE resource. However, CBMs can be accessed via
> either L3CODE or L3DATA resources.

Seems like something is missing here since this is motivation for a behavior
but no mention of the behavior being motivated. Also please use imperative tone.

> 
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---

...

> ---
>  Documentation/filesystems/resctrl.rst | 18 +++++++++++
>  fs/resctrl/ctrlmondata.c              |  8 +++--
>  fs/resctrl/internal.h                 |  2 ++
>  fs/resctrl/rdtgroup.c                 | 43 ++++++++++++++++++++++++++-
>  4 files changed, 67 insertions(+), 4 deletions(-)
> 
> diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
> index 189c1ccb92d6..c22a9dd667cd 100644
> --- a/Documentation/filesystems/resctrl.rst
> +++ b/Documentation/filesystems/resctrl.rst
> @@ -176,6 +176,24 @@ related to allocation:
>  		When CDP is enabled, io_alloc routes I/O traffic using the highest
>  		CLOSID allocated for the instruction cache (L3CODE).
>  
> +"io_alloc_cbm":
> +		CBMs(Capacity Bit Masks) that describe the portions of cache instances
> +		to which I/O traffic from supported I/O devices are routed.

"are routed" -> "are routed when "io_alloc" is enabled"?

> +
> +		CBMs are displayed in the following format:
> +
> +			<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
> +
> +		Example::
> +
> +			# cat /sys/fs/resctrl/info/L3/io_alloc_cbm
> +			0=ffff;1=ffff
> +
> +		When CDP is enabled "io_alloc_cbm" associated with the DATA and CODE
> +		resources may reflect the same values. For example, values read from and
> +		written to /sys/fs/resctrl/info/L3DATA/io_alloc_cbm may be reflected by
> +		/sys/fs/resctrl/info/L3CODE/io_alloc_cbm and vice versa.
> +
>  Memory bandwidth(MB) subdirectory contains the following files
>  with respect to allocation:
>  
> diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c
> index d98e0d2de09f..e78828b0408a 100644
> --- a/fs/resctrl/ctrlmondata.c
> +++ b/fs/resctrl/ctrlmondata.c
> @@ -381,7 +381,8 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
>  	return ret ?: nbytes;
>  }
>  
> -static void show_doms(struct seq_file *s, struct resctrl_schema *schema, int closid)
> +void show_doms(struct seq_file *s, struct resctrl_schema *schema, char *resource_name,

show_doms() can remain static within ctrlmondata.c by moving resctrl_io_alloc_cbm_show()
to ctrlmondata.c

Reinette

Re: [PATCH v7 09/10] fs/resctrl: Modify rdt_parse_data to pass mode and CLOSID

[Reinette Chatre]

Hi Babu,

On 7/10/25 10:16 AM, Babu Moger wrote:
> The functions parse_cbm() and parse_bw() require mode and CLOSID to

No need to say "function" when using (). Also, drop parse_bw(), since it
does not validate CBMs.

> validate the Capacity Bit Mask (CBM). It is passed through struct

"passed through" -> "passed via"?

> rdtgroup in rdt_parse_data.

"rdt_parse_data" -> "struct rdt_parse_data"

> 
> This can be simplified by passing the mode and closid directly, instead of

closid -> CLOSID

> through the rdtgroup struct. Doing so also facilitates calling parse_cbm()
> to verify the CBM within the io_alloc feature, since io_alloc does not
> have rdtgroup context.

Above notes that "simplification" is the primary motivation but I do not think
this change qualifies as a "simplification". How about second paragraph changed
to something like:

	The io_alloc feature also uses CBMs to indicate which portions of
	cache are allocated for I/O traffic. The CBMs are provided by
	user space and need to be validated the same as CBMs provided for
	general (CPU) cache allocation. parse_cbm() cannot be used as-is
	since io_alloc does not have rdtgroup context.

	Pass the mode and CLOSID directly to parse_cbm() via struct rdt_parse_data
	instead of through the rdtgroup struct to facilitate calling parse_cbm() to
	verify the CBM of the io_alloc feature.

(please feel free to improve)

> 
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---



> ---
>  fs/resctrl/ctrlmondata.c | 29 +++++++++++++----------------
>  fs/resctrl/internal.h    |  6 ++++++
>  2 files changed, 19 insertions(+), 16 deletions(-)
> 
> diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c
> index e78828b0408a..5c16557fb7a8 100644
> --- a/fs/resctrl/ctrlmondata.c
> +++ b/fs/resctrl/ctrlmondata.c
> @@ -23,11 +23,6 @@
>  
>  #include "internal.h"
>  
> -struct rdt_parse_data {
> -	struct rdtgroup		*rdtgrp;
> -	char			*buf;
> -};
> -

This patch is only about replacing rdtgroup with mode and CLOSID, there is no
motivation for relocating the structure declaration. This looks to be a change
needed by following patch but is another change that becomes unnecessary if
the io_alloc code, specifically resctrl_io_alloc_cbm_write() and
resctrl_io_alloc_parse_line() from next patch, are moved to ctrlmondata.c.

...

> @@ -171,7 +167,7 @@ static int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
>  	 * Cannot set up more than one pseudo-locked region in a cache
>  	 * hierarchy.
>  	 */
> -	if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP &&
> +	if (mode == RDT_MODE_PSEUDO_LOCKSETUP &&
>  	    rdtgroup_pseudo_locked_in_hierarchy(d)) {
>  		rdt_last_cmd_puts("Pseudo-locked region in hierarchy\n");
>  		return -EINVAL;
> @@ -180,9 +176,9 @@ static int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
>  	if (!cbm_validate(data->buf, &cbm_val, r))
>  		return -EINVAL;
>  
> -	if ((rdtgrp->mode == RDT_MODE_EXCLUSIVE ||
> -	     rdtgrp->mode == RDT_MODE_SHAREABLE) &&
> -	    rdtgroup_cbm_overlaps_pseudo_locked(d, cbm_val)) {
> +	if ((mode == RDT_MODE_EXCLUSIVE ||
> +	     mode == RDT_MODE_SHAREABLE) &&
> +	     rdtgroup_cbm_overlaps_pseudo_locked(d, cbm_val)) {

Please fix alignment.

>  		rdt_last_cmd_puts("CBM overlaps with pseudo-locked region\n");
>  		return -EINVAL;
>  	}

Reinette

Re: [PATCH v7 10/10] fs/resctrl: Introduce interface to modify io_alloc Capacity Bit Masks

[Reinette Chatre]

Hi Babu,

On 7/10/25 10:16 AM, Babu Moger wrote:
> "io_alloc" feature enables direct insertion of data from I/O devices into
> the cache. By directly caching data from I/O devices rather than first
> storing the I/O data in DRAM, it reduces the demands on DRAM bandwidth and
> reduces latency to the processor consuming the I/O data.
> 
> Provide the interface to modify io_alloc CBMs (Capacity Bit Mask) when

"Provide the interface to modify io_alloc CBMs (Capacity Bit Mask) when feature is
enabled." ->
"Enable users to modify io_alloc CBMs (Capacity Bit Masks) via the io_alloc_cbm
resctrl file when io_alloc is enabled."?

> feature is enabled. Update the CBMs for both CDP_DATA and CDP_CODE when CDP
> is enabled.
> 
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---

...

> ---
>  Documentation/filesystems/resctrl.rst |   8 ++
>  fs/resctrl/ctrlmondata.c              |   4 +-
>  fs/resctrl/internal.h                 |   2 +
>  fs/resctrl/rdtgroup.c                 | 112 +++++++++++++++++++++++++-
>  4 files changed, 123 insertions(+), 3 deletions(-)
> 
> diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
> index c22a9dd667cd..b9c3ffdec27f 100644
> --- a/Documentation/filesystems/resctrl.rst
> +++ b/Documentation/filesystems/resctrl.rst
> @@ -189,6 +189,14 @@ related to allocation:
>  			# cat /sys/fs/resctrl/info/L3/io_alloc_cbm
>  			0=ffff;1=ffff
>  
> +		CBMs can be configured by writing to the interface.
> +
> +		Example::
> +
> +			# echo 1=FF > /sys/fs/resctrl/info/L3/io_alloc_cbm
> +			# cat /sys/fs/resctrl/info/L3/io_alloc_cbm
> +			0=ffff;1=00ff
> +
>  		When CDP is enabled "io_alloc_cbm" associated with the DATA and CODE
>  		resources may reflect the same values. For example, values read from and
>  		written to /sys/fs/resctrl/info/L3DATA/io_alloc_cbm may be reflected by
> diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c
> index 5c16557fb7a8..23c72a5ac043 100644
> --- a/fs/resctrl/ctrlmondata.c
> +++ b/fs/resctrl/ctrlmondata.c
> @@ -148,8 +148,8 @@ static bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r)
>   * Read one cache bit mask (hex). Check that it is valid for the current
>   * resource type.
>   */
> -static int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
> -		     struct rdt_ctrl_domain *d)
> +int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
> +	      struct rdt_ctrl_domain *d)

This can remain static by moving caller to ctrlmondata.c

>  {
>  	enum rdtgrp_mode mode = data->mode;
>  	struct resctrl_staged_config *cfg;
> diff --git a/fs/resctrl/internal.h b/fs/resctrl/internal.h
> index ab76a1e2e679..011564e69ed5 100644
> --- a/fs/resctrl/internal.h
> +++ b/fs/resctrl/internal.h
> @@ -382,6 +382,8 @@ bool closid_allocated(unsigned int closid);
>  int resctrl_find_cleanest_closid(void);
>  void show_doms(struct seq_file *s, struct resctrl_schema *schema,
>  	       char *name, int closid);
> +int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
> +	      struct rdt_ctrl_domain *d);
>  
>  #ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK
>  int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp);
> diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
> index 008657e43656..1093ec5e385b 100644
> --- a/fs/resctrl/rdtgroup.c
> +++ b/fs/resctrl/rdtgroup.c
> @@ -2036,6 +2036,115 @@ static int resctrl_io_alloc_cbm_show(struct kernfs_open_file *of,
>  	return ret;
>  }
>  
> +static int resctrl_io_alloc_parse_line(char *line,  struct rdt_resource *r,
> +				       struct resctrl_schema *s, u32 closid)

Please move to ctrlmondata.c

> +{
> +	struct rdt_parse_data data;
> +	struct rdt_ctrl_domain *d;
> +	char *dom = NULL, *id;
> +	unsigned long dom_id;
> +
> +next:
> +	if (!line || line[0] == '\0')
> +		return 0;
> +
> +	dom = strsep(&line, ";");
> +	id = strsep(&dom, "=");
> +	if (!dom || kstrtoul(id, 10, &dom_id)) {
> +		rdt_last_cmd_puts("Missing '=' or non-numeric domain\n");
> +		return -EINVAL;
> +	}
> +
> +	dom = strim(dom);
> +	list_for_each_entry(d, &r->ctrl_domains, hdr.list) {
> +		if (d->hdr.id == dom_id) {
> +			data.buf = dom;
> +			data.mode = RDT_MODE_SHAREABLE;
> +			data.closid = closid;
> +			if (parse_cbm(&data, s, d))
> +				return -EINVAL;
> +			goto next;
> +		}
> +	}
> +	return -EINVAL;
> +}
> +
> +static ssize_t resctrl_io_alloc_cbm_write(struct kernfs_open_file *of,
> +					  char *buf, size_t nbytes, loff_t off)

Please move to ctrlmondata.c

> +{
> +	struct resctrl_schema *s = rdt_kn_parent_priv(of->kn);
> +	enum resctrl_conf_type peer_type;
> +	struct rdt_resource *r = s->res;
> +	struct resctrl_schema *peer_s;
> +	u32 io_alloc_closid;
> +	char *peer_buf;
> +	int ret = 0;
> +
> +	/* Valid input requires a trailing newline */
> +	if (nbytes == 0 || buf[nbytes - 1] != '\n')
> +		return -EINVAL;
> +
> +	buf[nbytes - 1] = '\0';
> +
> +	cpus_read_lock();
> +	mutex_lock(&rdtgroup_mutex);
> +
> +	if (!r->cache.io_alloc_capable) {
> +		rdt_last_cmd_printf("io_alloc is not supported on %s\n", s->name);

rdt_last_cmd_clear() is required before any write to the buffer.

> +		ret = -EINVAL;
> +		goto out_unlock;
> +	}
> +
> +	rdt_last_cmd_clear();
> +	rdt_staged_configs_clear();
> +
> +	if (!resctrl_arch_get_io_alloc_enabled(r)) {
> +		rdt_last_cmd_printf("io_alloc is not enabled on %s\n", s->name);
> +		ret = -EINVAL;

Inconsistent error codes when io_alloc is not enabled.  resctrl_io_alloc_write()
and resctrl_io_alloc_cbm_show() returns -ENODEV.

> +		goto out_unlock;
> +	}
> +
> +	io_alloc_closid = resctrl_io_alloc_closid(r);
> +
> +	/*
> +	 * When CDP is enabled, update the schema for both CDP_DATA and CDP_CODE.
> +	 * Since the buffer is altered during parsing, create a copy to handle
> +	 * peer schemata separately.
> +	 */
> +	if (resctrl_arch_get_cdp_enabled(r->rid)) {
> +		peer_type = resctrl_peer_type(s->conf_type);
> +		peer_s = resctrl_get_schema(peer_type);
> +		peer_buf = kmalloc(nbytes, GFP_KERNEL);
> +		if (!peer_buf) {
> +			rdt_last_cmd_puts("Out of Memory - io_alloc update failed\n");
> +			ret = -ENOMEM;
> +			goto out_unlock;
> +		}
> +
> +		memcpy(peer_buf, buf, nbytes);
> +
> +		if (peer_s)
> +			ret = resctrl_io_alloc_parse_line(peer_buf, r, peer_s, io_alloc_closid);

Similar to resctrl_io_alloc_write() this looks unnecessary. I think this can be simplified
by moving the CDP check to resctrl_io_alloc_parse_line() where the provided CBM can be parsed
*once* and the staged config of a CDP type just copied to the staged config of its peer type.

> +
> +		kfree(peer_buf);
> +	}
> +
> +	if (!ret)
> +		ret = resctrl_io_alloc_parse_line(buf, r, s, io_alloc_closid);
> +
> +	if (ret)
> +		goto out_unlock;
> +
> +	ret = resctrl_arch_update_domains(r, io_alloc_closid);
> +
> +out_unlock:
> +	rdt_staged_configs_clear();
> +	mutex_unlock(&rdtgroup_mutex);
> +	cpus_read_unlock();
> +
> +	return ret ?: nbytes;
> +}
> +
>  /* rdtgroup information files for one cache resource. */
>  static struct rftype res_common_files[] = {
>  	{
> @@ -2135,9 +2244,10 @@ static struct rftype res_common_files[] = {
>  	},
>  	{
>  		.name		= "io_alloc_cbm",
> -		.mode		= 0444,
> +		.mode		= 0644,
>  		.kf_ops		= &rdtgroup_kf_single_ops,
>  		.seq_show	= resctrl_io_alloc_cbm_show,
> +		.write		= resctrl_io_alloc_cbm_write,
>  	},
>  	{
>  		.name		= "max_threshold_occupancy",

Reinette

Re: [PATCH v7 02/10] x86/resctrl: Add SDCIAE feature in the command line options

[Moger, Babu]

Hi Reinette,

On 7/21/25 18:29, Reinette Chatre wrote:
> Hi Babu,
> 
> On 7/10/25 10:16 AM, Babu Moger wrote:
>> Add the command line options to enable or disable the new resctrl feature
>> L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE).
>>
>> Signed-off-by: Babu Moger <babu.moger@amd.com>
>> ---
> ...
>> ---
>>  Documentation/admin-guide/kernel-parameters.txt | 2 +-
>>  arch/x86/kernel/cpu/resctrl/core.c              | 2 ++
>>  2 files changed, 3 insertions(+), 1 deletion(-)
> 
> Could you please also add associated update to
> Documentation/filesystems/resctrl.rst ? For reference, the similar
> ABMC change:
> https://lore.kernel.org/lkml/00cd603997e3ee6a389f83aef066fe7313b1abaf.1752013061.git.babu.moger@amd.com/
> 

Sure. Will take care of it.
-- 
Thanks
Babu Moger

Re: [PATCH v7 04/10] x86,fs/resctrl: Implement "io_alloc" enable/disable handlers

[Moger, Babu]

On 7/21/25 18:32, Reinette Chatre wrote:
> Hi Babu,
> 
> On 7/10/25 10:16 AM, Babu Moger wrote:
>> "io_alloc" enables direct insertion of data from I/O devices into the
>> cache.
>>
>> On AMD systems, "io_alloc" feature is backed by L3 Smart Data Cache
>> Injection Allocation Enforcement (SDCIAE). Change SDCIAE state by setting
>> (to enable) or clearing (to disable) bit 1 of MSR L3_QOS_EXT_CFG on all
>> logical processors within the cache domain.
>>
>> Introduce architecture-specific call to enable and disable the feature.
>>
>> The SDCIAE feature details are available in APM listed below [1].
>> [1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
>> Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache
>> Injection Allocation Enforcement (SDCIAE)
>>
>> Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
>> Signed-off-by: Babu Moger <babu.moger@amd.com>
>> ---
> 
> ...
> 
>>  
>> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>> index 885026468440..c165ac333336 100644
>> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> 
> The code looks good to me but rdtgroup.c is becoming bloated by being the default place for
> any resctrl changes. While I do know there are counter examples (and I admit I do
> not have a clear understanding of original intent) ctrlmondata.c is documented and supported
> by its content as the place for cache allocation code. Could you please move these changes
> to arch/x86/kernel/cpu/resctrl/ctrlmondata.c? I will try to highlight the other places where
> I believe the either fs/resctrl/ctrlmondata.c or arch/x86/kernel/cpu/resctrl/ctrlmondata.c is
> more appropriate but please as a high level consider where appropriate to move the IO alloc
> code to be with the rest of the cache allocation code. Looking at the series it will help keep
> a couple of existing static functions static and keep data structures local to cache allocation
> code.
> 

Sure. Moved it to arch/x86/kernel/cpu/resctrl/ctrlmondata.c.
-- 
Thanks
Babu Moger

Re: [PATCH v7 06/10] fs/resctrl: Introduce interface to display "io_alloc" support

[Moger, Babu]

Hi Reinette,

On 7/21/25 18:36, Reinette Chatre wrote:
> Hi Babu,
> 
> On 7/10/25 10:16 AM, Babu Moger wrote:
>> "io_alloc" feature in resctrl allows direct insertion of data from I/O
>> devices into the cache.
>>
>> Introduce the 'io_alloc' resctrl file to indicate the support for the
>> feature.
>>
>> Signed-off-by: Babu Moger <babu.moger@amd.com>
>> ---
> 
> ...
> 
>> ---
>>  Documentation/filesystems/resctrl.rst | 25 +++++++++++++++++
>>  fs/resctrl/rdtgroup.c                 | 39 +++++++++++++++++++++++++++
>>  2 files changed, 64 insertions(+)
>>
>> diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
>> index c3c412733632..354e6a00fa45 100644
>> --- a/Documentation/filesystems/resctrl.rst
>> +++ b/Documentation/filesystems/resctrl.rst
>> @@ -143,6 +143,31 @@ related to allocation:
>>  			"1":
>>  			      Non-contiguous 1s value in CBM is supported.
>>  
>> +"io_alloc":
>> +		"io_alloc" enables system software to configure the portion of
>> +		the cache allocated for I/O traffic. File may only exist if the
>> +		system supports this feature on some of its cache resources.
>> +
>> +			"disabled":
>> +			      Portions of cache used for allocation of I/O traffic
>> +			      cannot be configured.
>> +			"enabled":
>> +			      Portions of cache used for allocation of I/O traffic
>> +			      can be configured using "io_alloc_cbm".
>> +			"not supported":
>> +			      Support not available on the system.
> 
> "Support not available on the system." -> "Support not available for this resource."?

Sure.

> 
>> +
>> +		The underlying implementation may reduce resources available to
>> +		general (CPU) cache allocation. See architecture specific notes
>> +		below. Depending on usage requirements the feature can be enabled
>> +		or disabled:
> 
> "disabled:" -> "disabled."?

Sure.

> 
>> +
>> +		On AMD systems, the io_alloc feature is supported by the L3 Smart
>> +		Data Cache Injection Allocation Enforcement (SDCIAE). The CLOSID for
>> +		io_alloc is determined by the highest CLOSID supported by the resource.
> 
> "is determined by the" -> "is the"?
> 

Sure.

> To make clear connection with previous paragraph you can append something like:
> 	When io_alloc is enabled on an AMD system the highest CLOSID is dedicated to
> 	io_alloc and no longer available for general (CPU) cache allocation.

Sure.

> 
>> +		When CDP is enabled, io_alloc routes I/O traffic using the highest
>> +		CLOSID allocated for the instruction cache (L3CODE).
> 
> To clear up what happens with L3DATA, what do you think of appending something like:
> 		, making this CLOSID no longer available for general (CPU) cache
> 		allocation for both the L3CODE and L3DATA resources.
> 

Sure.

>> +
>>  Memory bandwidth(MB) subdirectory contains the following files
>>  with respect to allocation:
>>  
>> diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
>> index a2eea85aecc8..d7c4417b4516 100644
>> --- a/fs/resctrl/rdtgroup.c
>> +++ b/fs/resctrl/rdtgroup.c
>> @@ -1836,6 +1836,28 @@ static ssize_t mbm_local_bytes_config_write(struct kernfs_open_file *of,
>>  	return ret ?: nbytes;
>>  }
>>  
>> +static int resctrl_io_alloc_show(struct kernfs_open_file *of,
> 
> Please move to ctrlmondata.c

Yes.

> 
> 
>> +				 struct seq_file *seq, void *v)
>> +{
>> +	struct resctrl_schema *s = rdt_kn_parent_priv(of->kn);
>> +	struct rdt_resource *r = s->res;
>> +
>> +	mutex_lock(&rdtgroup_mutex);
>> +
>> +	if (r->cache.io_alloc_capable) {
>> +		if (resctrl_arch_get_io_alloc_enabled(r))
>> +			seq_puts(seq, "enabled\n");
>> +		else
>> +			seq_puts(seq, "disabled\n");
>> +	} else {
>> +		seq_puts(seq, "not supported\n");
>> +	}
>> +
>> +	mutex_unlock(&rdtgroup_mutex);
>> +
>> +	return 0;
>> +}
>> +
>>  /* rdtgroup information files for one cache resource. */
>>  static struct rftype res_common_files[] = {
>>  	{
>> @@ -1926,6 +1948,12 @@ static struct rftype res_common_files[] = {
>>  		.kf_ops		= &rdtgroup_kf_single_ops,
>>  		.seq_show	= rdt_thread_throttle_mode_show,
>>  	},
>> +	{
>> +		.name		= "io_alloc",
>> +		.mode		= 0444,
>> +		.kf_ops		= &rdtgroup_kf_single_ops,
>> +		.seq_show	= resctrl_io_alloc_show,
>> +	},
>>  	{
>>  		.name		= "max_threshold_occupancy",
>>  		.mode		= 0644,
>> @@ -2095,6 +2123,15 @@ static void thread_throttle_mode_init(void)
>>  				 RFTYPE_CTRL_INFO | RFTYPE_RES_MB);
>>  }
>>  
>> +static void io_alloc_init(void)
> 
> This function's comment can benefit from a snippet that highlights that
> even though this operates on hardcoded L3 resource it results in this file
> being visible for *all* cache resources (eg. L2 cache also), whether they
> support io_alloc or not.

Added the comment.

> 
>> +{
>> +	struct rdt_resource *r = resctrl_arch_get_resource(RDT_RESOURCE_L3);
>> +
>> +	if (r->cache.io_alloc_capable)
>> +		resctrl_file_fflags_init("io_alloc", RFTYPE_CTRL_INFO |
>> +					 RFTYPE_RES_CACHE);
>> +}
>> +
>>  void resctrl_file_fflags_init(const char *config, unsigned long fflags)
>>  {
>>  	struct rftype *rft;
>> @@ -4282,6 +4319,8 @@ int resctrl_init(void)
>>  
>>  	thread_throttle_mode_init();
>>  
>> +	io_alloc_init();
>> +
>>  	ret = resctrl_mon_resource_init();
>>  	if (ret)
>>  		return ret;
> 
> Reinette
> 

-- 
Thanks
Babu Moger

Re: [PATCH v7 07/10] fs/resctrl: Add user interface to enable/disable io_alloc feature

[Moger, Babu]

Hi Reinette,

On 7/21/2025 6:40 PM, Reinette Chatre wrote:
> Hi Babu,
> 
> On 7/10/25 10:16 AM, Babu Moger wrote:
>> "io_alloc" feature in resctrl enables direct insertion of data from I/O
>> devices into the cache.
>>
>> Introduce user interface to enable/disable io_alloc feature.
> 
> I think it is worth a mention *why* a user may want to disable this feature and
> why is not just always enabled. Here it can be highlighted that this feature
> may take resources (CLOSID) away from general (CPU) cache allocation and since
> this may be scarce enabling user to disable this feature supports different use cases.
> 

Sure.

>>
>> On AMD systems, io_alloc feature is backed by SDCIAE (L3 Smart Data Cache
>> Injection Allocation Enforcement). When enabled, SDCIAE directs all SDCI
>> lines to be placed into the L3 cache partitions specified by the register
>> corresponding to the highest CLOSID supported by the resource. With CDP
>> enabled, io_alloc routes I/O traffic using the highest CLOSID assigned to
>> the instruction cache (L3CODE).
> 
> This is a lot of architecture specific text for a resctrl fs patch  ... I think
> you are trying to motivate the resctrl fs implementation. Similar motivation
> as proposed for cover letter can be used here to help explain the implementation
> choices.

Updated the whole changelog.

fs/resctrl: Add user interface to enable/disable io_alloc feature

"io_alloc" feature in resctrl enables direct insertion of data from I/O
devices into the cache.

Introduce user interface to enable/disable io_alloc feature.

On AMD systems, when io_alloc is enabled, the highest CLOSID is reserved
exclusively for I/O allocation traffic and is no longer available for
general CPU cache allocation. This feature is disabled by default. Users
are encouraged to enable it only when running workloads that can benefit
from this functionality.

Since CLOSIDs are managed by resctrl fs, it is least invasive to make the
"io_alloc is supported by maximum supported CLOSID" part of the initial
resctrl fs support for io_alloc. Take care not to expose this use of
CLOSID for io_alloc to user space so that this is not required from other
architectures that may support io_alloc differently in the future.

Signed-off-by: Babu Moger <babu.moger@amd.com>

> 
>>
>> Signed-off-by: Babu Moger <babu.moger@amd.com>
>> ---
> 
> ...
> 
>> ---
>>   Documentation/filesystems/resctrl.rst |   8 ++
>>   fs/resctrl/rdtgroup.c                 | 149 +++++++++++++++++++++++++-
>>   2 files changed, 156 insertions(+), 1 deletion(-)
>>
>> diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
>> index 354e6a00fa45..189c1ccb92d6 100644
>> --- a/Documentation/filesystems/resctrl.rst
>> +++ b/Documentation/filesystems/resctrl.rst
>> @@ -157,6 +157,14 @@ related to allocation:
>>   			"not supported":
>>   			      Support not available on the system.
>>   
>> +		The feature can be modified by writing to the interface, for example:
>> +
>> +		To enable:
>> +			# echo 1 > /sys/fs/resctrl/info/L3/io_alloc
>> +
>> +		To disable:
>> +			# echo 0 > /sys/fs/resctrl/info/L3/io_alloc
>> +
>>   		The underlying implementation may reduce resources available to
>>   		general (CPU) cache allocation. See architecture specific notes
>>   		below. Depending on usage requirements the feature can be enabled
>> diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
>> index d7c4417b4516..06c854caa55c 100644
>> --- a/fs/resctrl/rdtgroup.c
>> +++ b/fs/resctrl/rdtgroup.c
>> @@ -70,6 +70,7 @@ static struct seq_buf last_cmd_status;
>>   static char last_cmd_status_buf[512];
>>   
>>   static int rdtgroup_setup_root(struct rdt_fs_context *ctx);
>> +static int rdtgroup_init_cat(struct resctrl_schema *s, u32 closid);
>>   
>>   static void rdtgroup_destroy_root(void);
>>   
>> @@ -232,6 +233,16 @@ bool closid_allocated(unsigned int closid)
>>   	return !test_bit(closid, closid_free_map);
>>   }
>>   
>> +static bool resctrl_io_alloc_closid_alloc(u32 io_alloc_closid)
>> +{
>> +	return __test_and_clear_bit(io_alloc_closid, closid_free_map);
>> +}
>> +
>> +static void resctrl_io_alloc_closid_free(u32 io_alloc_closid)
>> +{
>> +	closid_free(io_alloc_closid);
>> +}
> 
> I do not think these should be helpers/wrappers with a separate
> namespace. It will make the code easier to understand if it is clear that
> the "io_alloc" CLOSID is allocated from the same "pool" as the CLOSID for
> control groups.
> 
> I would thus propose a specific, for example closid_alloc_fixed(u32 closid)
> helper, and just call closid_free() directly.
> 

Sure.

>> +
>>   /**
>>    * rdtgroup_mode_by_closid - Return mode of resource group with closid
>>    * @closid: closid if the resource group
>> @@ -1030,6 +1041,16 @@ static int rdt_shareable_bits_show(struct kernfs_open_file *of,
>>   	return 0;
>>   }
>>   
>> +/*
>> + * resctrl_io_alloc_closid_supported() - io_alloc feature utilizes the
>> + * highest CLOSID value to direct I/O traffic. Ensure that io_alloc_closid
>> + * is in the supported range.
>> + */
>> +static bool resctrl_io_alloc_closid_supported(u32 io_alloc_closid)
> 
> Please move to ctrlmondata.c
> 

Sure.

>> +{
>> +	return io_alloc_closid < closids_supported();
>> +}
>> +
>>   /*
>>    * resctrl_io_alloc_closid() - io_alloc feature routes I/O traffic using
>>    * the highest available CLOSID. Retrieve the maximum CLOSID supported by the
>> @@ -1858,6 +1879,131 @@ static int resctrl_io_alloc_show(struct kernfs_open_file *of,
>>   	return 0;
>>   }
>>   
>> +/*
>> + * Initialize io_alloc CLOSID cache resource CBM with all usable (shared
>> + * and unused) cache portions.
>> + */
>> +static int resctrl_io_alloc_init_cat(struct rdt_resource *r,
> 
> Mixing two features (io_alloc and CAT) in the function name is confusing.
> How about resctrl_io_alloc_init_cbm() and move to ctrlmondata.c?

Yes.

> 
>> +				     struct resctrl_schema *s, u32 closid)
> 
> No need to provide resource as parameter, it can be determined from schema.

Sure.

> 
>> +{
>> +	int ret;
>> +
>> +	rdt_staged_configs_clear();
>> +
>> +	ret = rdtgroup_init_cat(s, closid);
>> +	if (ret < 0)
>> +		goto out;
> 
> More below, but I think this flow can be simplified by moving the logic
> handling CDP here. If CDP is enabled for the resource then a successful
> rdtgroup_init_cat() can just be followed by a snippet that copies the
> staged config of the CDP type to the staged config of its peer type.
> Their CBMs are supposed to be identical so there is no reason for all the
> rdtgroup_init_cat() processing to be repeated. resctrl_arch_update_domains()
> can handle updating both in a single call.

Yes.

> 
>> +
>> +	ret = resctrl_arch_update_domains(r, closid);
>> +
>> +out:
>> +	rdt_staged_configs_clear();
>> +	return ret;
>> +}
>> +
>> +static const char *rdtgroup_name_by_closid(int closid)
> 
> This seems generic enough and appropriate for rdtgroup.c
> 
>> +{
>> +	struct rdtgroup *rdtgrp;
>> +
>> +	list_for_each_entry(rdtgrp, &rdt_all_groups, rdtgroup_list) {
>> +		if (rdtgrp->closid == closid)
>> +			return rdt_kn_name(rdtgrp->kn);
>> +	}
>> +
>> +	return NULL;
>> +}
>> +
>> +static struct resctrl_schema *resctrl_get_schema(enum resctrl_conf_type type)
> 
> This also seems generic enough and appropriate for rdtgroup.c
> 
>> +{
>> +	struct resctrl_schema *schema;
>> +
>> +	list_for_each_entry(schema, &resctrl_schema_all, list) {
>> +		if (schema->conf_type == type)
>> +			return schema;
>> +	}
>> +
>> +	return NULL;
>> +}
>> +
>> +static ssize_t resctrl_io_alloc_write(struct kernfs_open_file *of, char *buf,
>> +				      size_t nbytes, loff_t off)
>> +{
>> +	struct resctrl_schema *s = rdt_kn_parent_priv(of->kn);
>> +	enum resctrl_conf_type peer_type;
>> +	struct rdt_resource *r = s->res;
>> +	struct resctrl_schema *peer_s;
>> +	char const *grp_name;
>> +	u32 io_alloc_closid;
>> +	bool enable;
>> +	int ret;
>> +
>> +	ret = kstrtobool(buf, &enable);
>> +	if (ret)
>> +		return ret;
>> +
>> +	cpus_read_lock();
>> +	mutex_lock(&rdtgroup_mutex);
>> +
>> +	rdt_last_cmd_clear();
>> +
>> +	if (!r->cache.io_alloc_capable) {
>> +		rdt_last_cmd_printf("io_alloc is not supported on %s\n", s->name);
>> +		ret = -ENODEV;
>> +		goto out_unlock;
>> +	}
>> +
>> +	io_alloc_closid = resctrl_io_alloc_closid(r);
>> +	if (!resctrl_io_alloc_closid_supported(io_alloc_closid)) {
>> +		rdt_last_cmd_printf("io_alloc CLOSID (ctrl_hw_id) %d is not available\n",
>> +				    io_alloc_closid);
>> +		ret = -EINVAL;
>> +		goto out_unlock;
>> +	}
>> +
>> +	/* If the feature is already up to date, no action is needed. */
>> +	if (resctrl_arch_get_io_alloc_enabled(r) == enable)
>> +		goto out_unlock;
>> +
>> +	if (enable) {
>> +		if (!resctrl_io_alloc_closid_alloc(io_alloc_closid)) {
>> +			grp_name = rdtgroup_name_by_closid(io_alloc_closid);
>> +			WARN_ON_ONCE(!grp_name);
>> +			rdt_last_cmd_printf("CLOSID (ctrl_hw_id) %d for io_alloc is used by %s group\n",
>> +					    io_alloc_closid, grp_name ? grp_name : "another");
>> +			ret = -ENOSPC;
>> +			goto out_unlock;
>> +		}
>> +
>> +		/* Initialize schema for both CDP_DATA and CDP_CODE when CDP is enabled */
>> +		if (resctrl_arch_get_cdp_enabled(r->rid)) {
> 
> I think this block can be removed to simplify the flow by moving the CDP handling to
> resctrl_io_alloc_init_cat().

Sure. Also moved the whole function to fs/resctrl/ctrlmondata.c.

> 
>> +			peer_type = resctrl_peer_type(s->conf_type);
>> +			peer_s = resctrl_get_schema(peer_type);
>> +			if (peer_s)
>> +				ret = resctrl_io_alloc_init_cat(r, peer_s, io_alloc_closid);
>> +		}
>> +
>> +		if (!ret)
>> +			ret = resctrl_io_alloc_init_cat(r, s, io_alloc_closid);
>> +
>> +		if (ret) {
>> +			rdt_last_cmd_puts("Failed to initialize io_alloc allocations\n");
>> +			resctrl_io_alloc_closid_free(io_alloc_closid);
>> +			goto out_unlock;
>> +		}
>> +
>> +	} else {
>> +		resctrl_io_alloc_closid_free(io_alloc_closid);
>> +	}
>> +
>> +	ret = resctrl_arch_io_alloc_enable(r, enable);
>> +
>> +out_unlock:
>> +	mutex_unlock(&rdtgroup_mutex);
>> +	cpus_read_unlock();
>> +
>> +	return ret ?: nbytes;
>> +}
>> +
>>   /* rdtgroup information files for one cache resource. */
>>   static struct rftype res_common_files[] = {
>>   	{
>> @@ -1950,9 +2096,10 @@ static struct rftype res_common_files[] = {
>>   	},
>>   	{
>>   		.name		= "io_alloc",
>> -		.mode		= 0444,
>> +		.mode		= 0644,
>>   		.kf_ops		= &rdtgroup_kf_single_ops,
>>   		.seq_show	= resctrl_io_alloc_show,
>> +		.write          = resctrl_io_alloc_write,
>>   	},
>>   	{
>>   		.name		= "max_threshold_occupancy",
> 
> Reinette
> 

Thanks
Babu

Re: [PATCH v7 07/10] fs/resctrl: Add user interface to enable/disable io_alloc feature

[Reinette Chatre]

Hi Babu,

On 7/31/25 3:52 PM, Moger, Babu wrote:
> Hi Reinette,
> 
> On 7/21/2025 6:40 PM, Reinette Chatre wrote:
>> Hi Babu,
>>
>> On 7/10/25 10:16 AM, Babu Moger wrote:
>>> "io_alloc" feature in resctrl enables direct insertion of data from I/O
>>> devices into the cache.
>>>
>>> Introduce user interface to enable/disable io_alloc feature.
>>
>> I think it is worth a mention *why* a user may want to disable this feature and
>> why is not just always enabled. Here it can be highlighted that this feature
>> may take resources (CLOSID) away from general (CPU) cache allocation and since
>> this may be scarce enabling user to disable this feature supports different use cases.
>>
> 
> Sure.
> 
>>>
>>> On AMD systems, io_alloc feature is backed by SDCIAE (L3 Smart Data Cache
>>> Injection Allocation Enforcement). When enabled, SDCIAE directs all SDCI
>>> lines to be placed into the L3 cache partitions specified by the register
>>> corresponding to the highest CLOSID supported by the resource. With CDP
>>> enabled, io_alloc routes I/O traffic using the highest CLOSID assigned to
>>> the instruction cache (L3CODE).
>>
>> This is a lot of architecture specific text for a resctrl fs patch  ... I think
>> you are trying to motivate the resctrl fs implementation. Similar motivation
>> as proposed for cover letter can be used here to help explain the implementation
>> choices.
> 
> Updated the whole changelog.
> 
> fs/resctrl: Add user interface to enable/disable io_alloc feature
> 
> "io_alloc" feature in resctrl enables direct insertion of data from I/O
> devices into the cache.
> 
> Introduce user interface to enable/disable io_alloc feature.

The solution should be at end of changelog after description of problem it
solves.

> On AMD systems, when io_alloc is enabled, the highest CLOSID is reserved
> exclusively for I/O allocation traffic and is no longer available for
> general CPU cache allocation. This feature is disabled by default. Users

Changelog should always be in imperative tone and problem and solution should
be in separate paragraphs (above paragraph mixes problem and solution).  

For example, "Disable "io_alloc" feature by default to ensure all resources are
available for general CPU cache allocation. ..." Although I do not think this is
accurate since this patch does not do this?

> are encouraged to enable it only when running workloads that can benefit
> from this functionality.
> 
> Since CLOSIDs are managed by resctrl fs, it is least invasive to make the
> "io_alloc is supported by maximum supported CLOSID" part of the initial
> resctrl fs support for io_alloc. Take care not to expose this use of
> CLOSID for io_alloc to user space so that this is not required from other
> architectures that may support io_alloc differently in the future.
> 

The changelog requirements I refer to are documented in "Changelog" section
of Documentation/process/maintainer-tip.rst. I feel like this should be
familiar to you by now.

Reinette

Re: [PATCH v7 07/10] fs/resctrl: Add user interface to enable/disable io_alloc feature

[Moger, Babu]

Hi Reinette,

On 8/4/25 11:07, Reinette Chatre wrote:
> Hi Babu,
> 
> On 7/31/25 3:52 PM, Moger, Babu wrote:
>> Hi Reinette,
>>
>> On 7/21/2025 6:40 PM, Reinette Chatre wrote:
>>> Hi Babu,
>>>
>>> On 7/10/25 10:16 AM, Babu Moger wrote:
>>>> "io_alloc" feature in resctrl enables direct insertion of data from I/O
>>>> devices into the cache.
>>>>
>>>> Introduce user interface to enable/disable io_alloc feature.
>>>
>>> I think it is worth a mention *why* a user may want to disable this feature and
>>> why is not just always enabled. Here it can be highlighted that this feature
>>> may take resources (CLOSID) away from general (CPU) cache allocation and since
>>> this may be scarce enabling user to disable this feature supports different use cases.
>>>
>>
>> Sure.
>>
>>>>
>>>> On AMD systems, io_alloc feature is backed by SDCIAE (L3 Smart Data Cache
>>>> Injection Allocation Enforcement). When enabled, SDCIAE directs all SDCI
>>>> lines to be placed into the L3 cache partitions specified by the register
>>>> corresponding to the highest CLOSID supported by the resource. With CDP
>>>> enabled, io_alloc routes I/O traffic using the highest CLOSID assigned to
>>>> the instruction cache (L3CODE).
>>>
>>> This is a lot of architecture specific text for a resctrl fs patch  ... I think
>>> you are trying to motivate the resctrl fs implementation. Similar motivation
>>> as proposed for cover letter can be used here to help explain the implementation
>>> choices.
>>
>> Updated the whole changelog.
>>
>> fs/resctrl: Add user interface to enable/disable io_alloc feature
>>
>> "io_alloc" feature in resctrl enables direct insertion of data from I/O
>> devices into the cache.
>>
>> Introduce user interface to enable/disable io_alloc feature.
> 
> The solution should be at end of changelog after description of problem it
> solves.

Sure. Moved this text below.

> 
>> On AMD systems, when io_alloc is enabled, the highest CLOSID is reserved
>> exclusively for I/O allocation traffic and is no longer available for
>> general CPU cache allocation. This feature is disabled by default. Users
> 
> Changelog should always be in imperative tone and problem and solution should
> be in separate paragraphs (above paragraph mixes problem and solution).  
> 
> For example, "Disable "io_alloc" feature by default to ensure all resources are
> available for general CPU cache allocation. ..." Although I do not think this is
> accurate since this patch does not do this?

Yes. Removed the text "This feature is disabled by default."
> 
>> are encouraged to enable it only when running workloads that can benefit
>> from this functionality.
>>
>> Since CLOSIDs are managed by resctrl fs, it is least invasive to make the
>> "io_alloc is supported by maximum supported CLOSID" part of the initial
>> resctrl fs support for io_alloc. Take care not to expose this use of
>> CLOSID for io_alloc to user space so that this is not required from other
>> architectures that may support io_alloc differently in the future.
>>
> 
> The changelog requirements I refer to are documented in "Changelog" section
> of Documentation/process/maintainer-tip.rst. I feel like this should be
> familiar to you by now.

Sure,  Thank you.

-- 
Thanks
Babu Moger

Re: [PATCH v7 08/10] fs/resctrl: Introduce interface to display io_alloc CBMs

[Moger, Babu]

Hi Reinette,

On 7/21/25 18:42, Reinette Chatre wrote:
> Hi Babu,
> 
> On 7/10/25 10:16 AM, Babu Moger wrote:
>> The io_alloc feature in resctrl enables system software to configure
>> the portion of the cache allocated for I/O traffic.
>>
>> Add "io_alloc_cbm" resctrl file to display CBMs (Capacity Bit Mask) o
>> fio_alloc feature.
> 
> "o fio_alloc" -> "of the io_alloc"
> 

Sure.

>>
>> The CBM interface file io_alloc_cbm will reside in the info directory
> 
> "will reside" -> "resides"
> 

Sure.

>> (e.g., /sys/fs/resctrl/info/L3/). Displaying the resource name is not
>> necessary. Pass the resource name to show_doms() and print it only if
>> the name is valid. For io_alloc, pass NULL to suppress printing the
>> resource name.
>>
>> When CDP is enabled, io_alloc routes traffic using the highest CLOSID
>> associated with an L3CODE resource. However, CBMs can be accessed via
>> either L3CODE or L3DATA resources.
> 
> Seems like something is missing here since this is motivation for a behavior
> but no mention of the behavior being motivated. Also please use imperative tone.

Sure. Added motivation for this.

> 
>>
>> Signed-off-by: Babu Moger <babu.moger@amd.com>
>> ---
> 
> ...
> 
>> ---
>>  Documentation/filesystems/resctrl.rst | 18 +++++++++++
>>  fs/resctrl/ctrlmondata.c              |  8 +++--
>>  fs/resctrl/internal.h                 |  2 ++
>>  fs/resctrl/rdtgroup.c                 | 43 ++++++++++++++++++++++++++-
>>  4 files changed, 67 insertions(+), 4 deletions(-)
>>
>> diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
>> index 189c1ccb92d6..c22a9dd667cd 100644
>> --- a/Documentation/filesystems/resctrl.rst
>> +++ b/Documentation/filesystems/resctrl.rst
>> @@ -176,6 +176,24 @@ related to allocation:
>>  		When CDP is enabled, io_alloc routes I/O traffic using the highest
>>  		CLOSID allocated for the instruction cache (L3CODE).
>>  
>> +"io_alloc_cbm":
>> +		CBMs(Capacity Bit Masks) that describe the portions of cache instances
>> +		to which I/O traffic from supported I/O devices are routed.
> 
> "are routed" -> "are routed when "io_alloc" is enabled"?
> 

Sure.

>> +
>> +		CBMs are displayed in the following format:
>> +
>> +			<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
>> +
>> +		Example::
>> +
>> +			# cat /sys/fs/resctrl/info/L3/io_alloc_cbm
>> +			0=ffff;1=ffff
>> +
>> +		When CDP is enabled "io_alloc_cbm" associated with the DATA and CODE
>> +		resources may reflect the same values. For example, values read from and
>> +		written to /sys/fs/resctrl/info/L3DATA/io_alloc_cbm may be reflected by
>> +		/sys/fs/resctrl/info/L3CODE/io_alloc_cbm and vice versa.
>> +
>>  Memory bandwidth(MB) subdirectory contains the following files
>>  with respect to allocation:
>>  
>> diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c
>> index d98e0d2de09f..e78828b0408a 100644
>> --- a/fs/resctrl/ctrlmondata.c
>> +++ b/fs/resctrl/ctrlmondata.c
>> @@ -381,7 +381,8 @@ ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
>>  	return ret ?: nbytes;
>>  }
>>  
>> -static void show_doms(struct seq_file *s, struct resctrl_schema *schema, int closid)
>> +void show_doms(struct seq_file *s, struct resctrl_schema *schema, char *resource_name,
> 
> show_doms() can remain static within ctrlmondata.c by moving resctrl_io_alloc_cbm_show()
> to ctrlmondata.c
> 

Yes. Moved it to ctrlmondata.c.
Thanks
Babu Moger

Re: [PATCH v7 09/10] fs/resctrl: Modify rdt_parse_data to pass mode and CLOSID

[Moger, Babu]

Hi Reinette,

On 7/21/25 22:30, Reinette Chatre wrote:
> Hi Babu,
> 
> On 7/10/25 10:16 AM, Babu Moger wrote:
>> The functions parse_cbm() and parse_bw() require mode and CLOSID to
> 
> No need to say "function" when using (). Also, drop parse_bw(), since it
> does not validate CBMs.

Sure.
> 
>> validate the Capacity Bit Mask (CBM). It is passed through struct
> 
> "passed through" -> "passed via"?
> 

Sure.

>> rdtgroup in rdt_parse_data.
> 
> "rdt_parse_data" -> "struct rdt_parse_data"
> 

Sure.

>>
>> This can be simplified by passing the mode and closid directly, instead of
> 
> closid -> CLOSID
> 
Sure,

>> through the rdtgroup struct. Doing so also facilitates calling parse_cbm()
>> to verify the CBM within the io_alloc feature, since io_alloc does not
>> have rdtgroup context.
> 
> Above notes that "simplification" is the primary motivation but I do not think
> this change qualifies as a "simplification". How about second paragraph changed
> to something like:
> 
> 	The io_alloc feature also uses CBMs to indicate which portions of
> 	cache are allocated for I/O traffic. The CBMs are provided by
> 	user space and need to be validated the same as CBMs provided for
> 	general (CPU) cache allocation. parse_cbm() cannot be used as-is
> 	since io_alloc does not have rdtgroup context.
> 
> 	Pass the mode and CLOSID directly to parse_cbm() via struct rdt_parse_data
> 	instead of through the rdtgroup struct to facilitate calling parse_cbm() to
> 	verify the CBM of the io_alloc feature.
> 
> (please feel free to improve)
> 

Looks good.

>>
>> Signed-off-by: Babu Moger <babu.moger@amd.com>
>> ---
> 
> 
> 
>> ---
>>  fs/resctrl/ctrlmondata.c | 29 +++++++++++++----------------
>>  fs/resctrl/internal.h    |  6 ++++++
>>  2 files changed, 19 insertions(+), 16 deletions(-)
>>
>> diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c
>> index e78828b0408a..5c16557fb7a8 100644
>> --- a/fs/resctrl/ctrlmondata.c
>> +++ b/fs/resctrl/ctrlmondata.c
>> @@ -23,11 +23,6 @@
>>  
>>  #include "internal.h"
>>  
>> -struct rdt_parse_data {
>> -	struct rdtgroup		*rdtgrp;
>> -	char			*buf;
>> -};
>> -
> 
> This patch is only about replacing rdtgroup with mode and CLOSID, there is no
> motivation for relocating the structure declaration. This looks to be a change
> needed by following patch but is another change that becomes unnecessary if
> the io_alloc code, specifically resctrl_io_alloc_cbm_write() and
> resctrl_io_alloc_parse_line() from next patch, are moved to ctrlmondata.c.
> 

Yes. Not required to move now.

> ...
> 
>> @@ -171,7 +167,7 @@ static int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
>>  	 * Cannot set up more than one pseudo-locked region in a cache
>>  	 * hierarchy.
>>  	 */
>> -	if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP &&
>> +	if (mode == RDT_MODE_PSEUDO_LOCKSETUP &&
>>  	    rdtgroup_pseudo_locked_in_hierarchy(d)) {
>>  		rdt_last_cmd_puts("Pseudo-locked region in hierarchy\n");
>>  		return -EINVAL;
>> @@ -180,9 +176,9 @@ static int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
>>  	if (!cbm_validate(data->buf, &cbm_val, r))
>>  		return -EINVAL;
>>  
>> -	if ((rdtgrp->mode == RDT_MODE_EXCLUSIVE ||
>> -	     rdtgrp->mode == RDT_MODE_SHAREABLE) &&
>> -	    rdtgroup_cbm_overlaps_pseudo_locked(d, cbm_val)) {
>> +	if ((mode == RDT_MODE_EXCLUSIVE ||
>> +	     mode == RDT_MODE_SHAREABLE) &&
>> +	     rdtgroup_cbm_overlaps_pseudo_locked(d, cbm_val)) {
> 
> Please fix alignment.

Sure.

> 
>>  		rdt_last_cmd_puts("CBM overlaps with pseudo-locked region\n");
>>  		return -EINVAL;
>>  	}
> 
> Reinette
> 

-- 
Thanks
Babu Moger

Re: [PATCH v7 10/10] fs/resctrl: Introduce interface to modify io_alloc Capacity Bit Masks

[Moger, Babu]

Hi Reinette,

On 7/21/25 22:32, Reinette Chatre wrote:
> Hi Babu,
> 
> On 7/10/25 10:16 AM, Babu Moger wrote:
>> "io_alloc" feature enables direct insertion of data from I/O devices into
>> the cache. By directly caching data from I/O devices rather than first
>> storing the I/O data in DRAM, it reduces the demands on DRAM bandwidth and
>> reduces latency to the processor consuming the I/O data.
>>
>> Provide the interface to modify io_alloc CBMs (Capacity Bit Mask) when
> 
> "Provide the interface to modify io_alloc CBMs (Capacity Bit Mask) when feature is
> enabled." ->
> "Enable users to modify io_alloc CBMs (Capacity Bit Masks) via the io_alloc_cbm
> resctrl file when io_alloc is enabled."?

Sure.

> 
>> feature is enabled. Update the CBMs for both CDP_DATA and CDP_CODE when CDP
>> is enabled.
>>
>> Signed-off-by: Babu Moger <babu.moger@amd.com>
>> ---
> 
> ...
> 
>> ---
>>  Documentation/filesystems/resctrl.rst |   8 ++
>>  fs/resctrl/ctrlmondata.c              |   4 +-
>>  fs/resctrl/internal.h                 |   2 +
>>  fs/resctrl/rdtgroup.c                 | 112 +++++++++++++++++++++++++-
>>  4 files changed, 123 insertions(+), 3 deletions(-)
>>
>> diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
>> index c22a9dd667cd..b9c3ffdec27f 100644
>> --- a/Documentation/filesystems/resctrl.rst
>> +++ b/Documentation/filesystems/resctrl.rst
>> @@ -189,6 +189,14 @@ related to allocation:
>>  			# cat /sys/fs/resctrl/info/L3/io_alloc_cbm
>>  			0=ffff;1=ffff
>>  
>> +		CBMs can be configured by writing to the interface.
>> +
>> +		Example::
>> +
>> +			# echo 1=FF > /sys/fs/resctrl/info/L3/io_alloc_cbm
>> +			# cat /sys/fs/resctrl/info/L3/io_alloc_cbm
>> +			0=ffff;1=00ff
>> +
>>  		When CDP is enabled "io_alloc_cbm" associated with the DATA and CODE
>>  		resources may reflect the same values. For example, values read from and
>>  		written to /sys/fs/resctrl/info/L3DATA/io_alloc_cbm may be reflected by
>> diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c
>> index 5c16557fb7a8..23c72a5ac043 100644
>> --- a/fs/resctrl/ctrlmondata.c
>> +++ b/fs/resctrl/ctrlmondata.c
>> @@ -148,8 +148,8 @@ static bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r)
>>   * Read one cache bit mask (hex). Check that it is valid for the current
>>   * resource type.
>>   */
>> -static int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
>> -		     struct rdt_ctrl_domain *d)
>> +int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
>> +	      struct rdt_ctrl_domain *d)
> 
> This can remain static by moving caller to ctrlmondata.c

Yes.

> 
>>  {
>>  	enum rdtgrp_mode mode = data->mode;
>>  	struct resctrl_staged_config *cfg;
>> diff --git a/fs/resctrl/internal.h b/fs/resctrl/internal.h
>> index ab76a1e2e679..011564e69ed5 100644
>> --- a/fs/resctrl/internal.h
>> +++ b/fs/resctrl/internal.h
>> @@ -382,6 +382,8 @@ bool closid_allocated(unsigned int closid);
>>  int resctrl_find_cleanest_closid(void);
>>  void show_doms(struct seq_file *s, struct resctrl_schema *schema,
>>  	       char *name, int closid);
>> +int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s,
>> +	      struct rdt_ctrl_domain *d);
>>  
>>  #ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK
>>  int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp);
>> diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
>> index 008657e43656..1093ec5e385b 100644
>> --- a/fs/resctrl/rdtgroup.c
>> +++ b/fs/resctrl/rdtgroup.c
>> @@ -2036,6 +2036,115 @@ static int resctrl_io_alloc_cbm_show(struct kernfs_open_file *of,
>>  	return ret;
>>  }
>>  
>> +static int resctrl_io_alloc_parse_line(char *line,  struct rdt_resource *r,
>> +				       struct resctrl_schema *s, u32 closid)
> 
> Please move to ctrlmondata.c

Yes.

> 
>> +{
>> +	struct rdt_parse_data data;
>> +	struct rdt_ctrl_domain *d;
>> +	char *dom = NULL, *id;
>> +	unsigned long dom_id;
>> +
>> +next:
>> +	if (!line || line[0] == '\0')
>> +		return 0;
>> +
>> +	dom = strsep(&line, ";");
>> +	id = strsep(&dom, "=");
>> +	if (!dom || kstrtoul(id, 10, &dom_id)) {
>> +		rdt_last_cmd_puts("Missing '=' or non-numeric domain\n");
>> +		return -EINVAL;
>> +	}
>> +
>> +	dom = strim(dom);
>> +	list_for_each_entry(d, &r->ctrl_domains, hdr.list) {
>> +		if (d->hdr.id == dom_id) {
>> +			data.buf = dom;
>> +			data.mode = RDT_MODE_SHAREABLE;
>> +			data.closid = closid;
>> +			if (parse_cbm(&data, s, d))
>> +				return -EINVAL;
>> +			goto next;
>> +		}
>> +	}
>> +	return -EINVAL;
>> +}
>> +
>> +static ssize_t resctrl_io_alloc_cbm_write(struct kernfs_open_file *of,
>> +					  char *buf, size_t nbytes, loff_t off)
> 
> Please move to ctrlmondata.c

Yes.

> 
>> +{
>> +	struct resctrl_schema *s = rdt_kn_parent_priv(of->kn);
>> +	enum resctrl_conf_type peer_type;
>> +	struct rdt_resource *r = s->res;
>> +	struct resctrl_schema *peer_s;
>> +	u32 io_alloc_closid;
>> +	char *peer_buf;
>> +	int ret = 0;
>> +
>> +	/* Valid input requires a trailing newline */
>> +	if (nbytes == 0 || buf[nbytes - 1] != '\n')
>> +		return -EINVAL;
>> +
>> +	buf[nbytes - 1] = '\0';
>> +
>> +	cpus_read_lock();
>> +	mutex_lock(&rdtgroup_mutex);
>> +
>> +	if (!r->cache.io_alloc_capable) {
>> +		rdt_last_cmd_printf("io_alloc is not supported on %s\n", s->name);
> 
> rdt_last_cmd_clear() is required before any write to the buffer.

Added now.

> 
>> +		ret = -EINVAL;
>> +		goto out_unlock;
>> +	}
>> +
>> +	rdt_last_cmd_clear();
>> +	rdt_staged_configs_clear();
>> +
>> +	if (!resctrl_arch_get_io_alloc_enabled(r)) {
>> +		rdt_last_cmd_printf("io_alloc is not enabled on %s\n", s->name);
>> +		ret = -EINVAL;
> 
> Inconsistent error codes when io_alloc is not enabled.  resctrl_io_alloc_write()
> and resctrl_io_alloc_cbm_show() returns -ENODEV.

Changed it to -ENODEV.

> 
>> +		goto out_unlock;
>> +	}
>> +
>> +	io_alloc_closid = resctrl_io_alloc_closid(r);
>> +
>> +	/*
>> +	 * When CDP is enabled, update the schema for both CDP_DATA and CDP_CODE.
>> +	 * Since the buffer is altered during parsing, create a copy to handle
>> +	 * peer schemata separately.
>> +	 */
>> +	if (resctrl_arch_get_cdp_enabled(r->rid)) {
>> +		peer_type = resctrl_peer_type(s->conf_type);
>> +		peer_s = resctrl_get_schema(peer_type);
>> +		peer_buf = kmalloc(nbytes, GFP_KERNEL);
>> +		if (!peer_buf) {
>> +			rdt_last_cmd_puts("Out of Memory - io_alloc update failed\n");
>> +			ret = -ENOMEM;
>> +			goto out_unlock;
>> +		}
>> +
>> +		memcpy(peer_buf, buf, nbytes);
>> +
>> +		if (peer_s)
>> +			ret = resctrl_io_alloc_parse_line(peer_buf, r, peer_s, io_alloc_closid);
> 
> Similar to resctrl_io_alloc_write() this looks unnecessary. I think this can be simplified
> by moving the CDP check to resctrl_io_alloc_parse_line() where the provided CBM can be parsed
> *once* and the staged config of a CDP type just copied to the staged config of its peer type.

Yes. Changed it now.

> 
>> +
>> +		kfree(peer_buf);
>> +	}
>> +
>> +	if (!ret)
>> +		ret = resctrl_io_alloc_parse_line(buf, r, s, io_alloc_closid);
>> +
>> +	if (ret)
>> +		goto out_unlock;
>> +
>> +	ret = resctrl_arch_update_domains(r, io_alloc_closid);
>> +
>> +out_unlock:
>> +	rdt_staged_configs_clear();
>> +	mutex_unlock(&rdtgroup_mutex);
>> +	cpus_read_unlock();
>> +
>> +	return ret ?: nbytes;
>> +}
>> +
>>  /* rdtgroup information files for one cache resource. */
>>  static struct rftype res_common_files[] = {
>>  	{
>> @@ -2135,9 +2244,10 @@ static struct rftype res_common_files[] = {
>>  	},
>>  	{
>>  		.name		= "io_alloc_cbm",
>> -		.mode		= 0444,
>> +		.mode		= 0644,
>>  		.kf_ops		= &rdtgroup_kf_single_ops,
>>  		.seq_show	= resctrl_io_alloc_cbm_show,
>> +		.write		= resctrl_io_alloc_cbm_write,
>>  	},
>>  	{
>>  		.name		= "max_threshold_occupancy",
> 
> Reinette
> 

-- 
Thanks
Babu Moger

Re: [PATCH v7 05/10] fs/resctrl: Update bit_usage to reflect io_alloc

[Moger, Babu]

Hi Reinette,

On 7/21/25 18:35, Reinette Chatre wrote:
> Hi Babu,
> 
> On 7/10/25 10:16 AM, Babu Moger wrote:
>> When the io_alloc feature is enabled, a portion of the cache can be
>> configured for shared use between hardware and software.
>>
>> Update the bit_usage representation to reflect the io_alloc configuration.
> 
> This patch is early in the series but also relies on capabilities added
> later in the series. This cryptic changelog leaves a lot for the user
> to decipher. For example:
> - How is the "io_alloc CLOSID" chosen? This is mentioned in cover letter but
>   not here. Doing so here may help explain the hardcoding of CDP_CODE done
>   in this patch (which seem unnecessary, more later, but a lot depends on
>   changes that follow this patch).
> - No mention that when io_alloc is enabled then an associated CLOSID will
>   be allocated. This is done later in series but assumed to be known here
>   where rdt_bit_usage_show() implicitly assumes that when io_alloc is enabled
>   then the "io_alloc CLOSID" is supported AND allocated (otherwise array overrun?)

Make sense. Moved this patch to the last.

> 
>>
>> Signed-off-by: Babu Moger <babu.moger@amd.com>
>> ---
>> v7: New patch split from earlier patch #5.
>>     Added resctrl_io_alloc_closid() to return max COSID.
>> ---
>>  Documentation/filesystems/resctrl.rst | 20 ++++++++++-----
>>  fs/resctrl/rdtgroup.c                 | 37 +++++++++++++++++++++++++--
>>  2 files changed, 49 insertions(+), 8 deletions(-)
>>
>> diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
>> index c7949dd44f2f..c3c412733632 100644
>> --- a/Documentation/filesystems/resctrl.rst
>> +++ b/Documentation/filesystems/resctrl.rst
>> @@ -89,12 +89,20 @@ related to allocation:
>>  		must be set when writing a mask.
>>  
>>  "shareable_bits":
>> -		Bitmask of shareable resource with other executing
>> -		entities (e.g. I/O). User can use this when
>> -		setting up exclusive cache partitions. Note that
>> -		some platforms support devices that have their
>> -		own settings for cache use which can over-ride
>> -		these bits.
>> +		Bitmask of shareable resource with other executing entities
>> +		(e.g. I/O). Applies to all instances of this resource. User
>> +		can use this when setting up exclusive cache partitions.
>> +		Note that some platforms support devices that have their
>> +		own settings for cache use which can over-ride these bits.
>> +
>> +		When "io_alloc" feature is enabled, a portion of the cache
>> +		can be configured for shared use between hardware and software.
> 
> To help distinguish how "io_alloc" is different from "shareable_bits" this can be:
> 		When "io_alloc" is enabled, a portion of each cache instance can
> 		be configured for shared use between hardware and software.

Sure.

> 
> Please merge these two paragraphs.

Sure.

> 
>> +		"bit_usage" should be used to see which portions of each cache
>> +		instance is configured for hardware use via "io_alloc" feature
>> +		because every cache instance can have its "io_alloc" bitmask
>> +		configured independently.
> 
> append " via "io_alloc_cbm"."?

sure.

> (but io_alloc_cbm does not exist at this point ... another motivation for this
> patch to move later)
> 

Sure.

>> +
>>  "bit_usage":
>>  		Annotated capacity bitmasks showing how all
>>  		instances of the resource are used. The legend is:
> 
> Please note that this "bit_usage" section contain several references to "shareable_bits"
> that should be updated to refer to "io_alloc_cbm" also.
> 
> With all these new terms introduced as common knowledge it is starting to look
> like this patch would be easier to consume later in the series.

Sure.

> 
>> diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
>> index 77d08229d855..a2eea85aecc8 100644
>> --- a/fs/resctrl/rdtgroup.c
>> +++ b/fs/resctrl/rdtgroup.c
>> @@ -1030,6 +1030,20 @@ static int rdt_shareable_bits_show(struct kernfs_open_file *of,
>>  	return 0;
>>  }
>>  
>> +/*
>> + * resctrl_io_alloc_closid() - io_alloc feature routes I/O traffic using
>> + * the highest available CLOSID. Retrieve the maximum CLOSID supported by the
>> + * resource. Note that if Code Data Prioritization (CDP) is enabled, the number
>> + * of available CLOSIDs is reduced by half.
>> + */
>> +static u32 resctrl_io_alloc_closid(struct rdt_resource *r)
> 
> Please move to ctrlmondata.c.

Yes.

> 
>> +{
>> +	if (resctrl_arch_get_cdp_enabled(r->rid))
>> +		return resctrl_arch_get_num_closid(r) / 2  - 1;
>> +	else
>> +		return resctrl_arch_get_num_closid(r) - 1;
>> +}
>> +
>>  /*
>>   * rdt_bit_usage_show - Display current usage of resources
>>   *
>> @@ -1063,15 +1077,17 @@ static int rdt_bit_usage_show(struct kernfs_open_file *of,
>>  
>>  	cpus_read_lock();
>>  	mutex_lock(&rdtgroup_mutex);
>> -	hw_shareable = r->cache.shareable_bits;
>>  	list_for_each_entry(dom, &r->ctrl_domains, hdr.list) {
>>  		if (sep)
>>  			seq_putc(seq, ';');
>> +		hw_shareable = r->cache.shareable_bits;
>>  		sw_shareable = 0;
>>  		exclusive = 0;
>>  		seq_printf(seq, "%d=", dom->hdr.id);
>>  		for (i = 0; i < closids_supported(); i++) {
>> -			if (!closid_allocated(i))
>> +			if (!closid_allocated(i) ||
>> +			    (resctrl_arch_get_io_alloc_enabled(r) &&
>> +			     i == resctrl_io_alloc_closid(r)))
>>  				continue;
>>  			ctrl_val = resctrl_arch_get_config(r, dom, i,
>>  							   s->conf_type);
>> @@ -1099,6 +1115,23 @@ static int rdt_bit_usage_show(struct kernfs_open_file *of,
>>  				break;
>>  			}
>>  		}
>> +
>> +		/*
>> +		 * When the "io_alloc" feature is enabled, a portion of the
>> +		 * cache is configured for shared use between hardware and software.
>> +		 */
>> +		if (resctrl_arch_get_io_alloc_enabled(r)) {
> 
> Here is undocumented implicit assumption that when io_alloc is enabled then
> the "io_alloc CLOSID" is allocated. This is also outside the closids_supported()
> loop which adds the other implicit assumption that if io_alloc is enabled then
> its CLOSID is supported by resctrl fs. None of these concepts have been introduced
> so far and is not mentioned in changelog.
> It is not obvious here that an io_alloc CLOSID must be supported (this is just
> something enforced by later patches) and also not obvious that an io_alloc CLOSID
> must be allocated from same "pool" as other CLOSIDs. Without a good changelog and
> context of later patches this is hard to understand.
> These are motivations for this patch to move later in the series and then the
> changelog can just refer to these assumptions as fact, making it all easier to follow.

Sure.

> 
>> +			if (resctrl_arch_get_cdp_enabled(r->rid))
>> +				ctrl_val = resctrl_arch_get_config(r, dom,
>> +								   resctrl_io_alloc_closid(r),
>> +								   CDP_CODE);
>> +			else
>> +				ctrl_val = resctrl_arch_get_config(r, dom,
>> +								   resctrl_io_alloc_closid(r),
>> +								   CDP_NONE);
> 
> This does not look necessary to me. Why not just:
> 		if (resctrl_arch_get_io_alloc_enabled(r)) {
> 			ctrl_val = resctrl_arch_get_config(r, dom,
> 							   resctrl_io_alloc_closid(r),
> 							   s->conf_type);
> 			hw_shareable |= ctrl_val;
> 		}
> 
> Since the later patches keep the CDP_CODE and CDP_DATA CBMs in sync it does not matter
> if the io_alloc CBM is obtained from CDP_CODE or CDP_DATA and just providing the
> schema's type to resctrl_arch_get_config() will have it retrieve the right CBM, no?

Yes. That is correct.

> 
> This may also be easier to understand/claim if this patch is later in the series.

Sure.

> 
> 
>> +			hw_shareable |= ctrl_val;
>> +		}
>> +
>>  		for (i = r->cache.cbm_len - 1; i >= 0; i--) {
>>  			pseudo_locked = dom->plr ? dom->plr->cbm : 0;
>>  			hwb = test_bit(i, &hw_shareable);
> 
> Reinette
> 

-- 
Thanks
Babu Moger

Re: [PATCH v7 00/10] x86,fs/resctrl: Support L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

[Moger, Babu]

Hi Reinette,


On 7/21/25 18:28, Reinette Chatre wrote:
> Hi Babu,
> 
> On 7/10/25 10:16 AM, Babu Moger wrote:
>>
>> This series adds the support for L3 Smart Data Cache Injection Allocation
>> Enforcement (SDCIAE) to resctrl infrastructure. It is referred to as
>> "io_alloc" in resctrl subsystem.
>>
>> Upcoming AMD hardware implements Smart Data Cache Injection (SDCI).
>> Smart Data Cache Injection (SDCI) is a mechanism that enables direct
>> insertion of data from I/O devices into the L3 cache. By directly caching
>> data from I/O devices rather than first storing the I/O data in DRAM, SDCI
>> reduces demands on DRAM bandwidth and reduces latency to the processor
>> consuming the I/O data.
>>
>> The SDCIAE (SDCI Allocation Enforcement) PQE feature allows system software
>> to control the portion of the L3 cache used for SDCI devices.
>>
>> When enabled, SDCIAE forces all SDCI lines to be placed into the L3 cache
>> partitions identified by the highest-supported L3_MASK_n register, where n
>> is the maximum supported CLOSID.
> 
> Even though this CLOSID use is unique to AMD this implementation makes it part
> of resctrl fs's support of io_alloc. It is confusing to have architectural
> specific features be handled by resctrl fs so I think it will be useful to
> add a snippet here to help folks trying to decipher this work. Consider
> for example a snippet like:
> 
>   Since CLOSIDs are managed by resctrl fs it is least invasive to make
>   the "io_alloc is supported by maximum supported CLOSID" part of the
>   initial resctrl fs support for io_alloc. Take care not to expose this
>   use of CLOSID for io_alloc to user space so that this is not required from
>   other architectures that may support io_alloc differently in the future.
> 

Sure.

>>
>> The feature details are documented in the APM listed below [1].
>> [1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
>> Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache
>> Injection Allocation Enforcement (SDCIAE)
>> Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
>>
>> The feature requires linux support of TPH (TLP Processing Hints).
>> The support is available in linux kernel after the commit
>> 48d0fd2b903e3 ("PCI/TPH: Add TPH documentation")
>>
>> The patches are based on top of commit (6.16.0-rc5)
>> commit b4ec95e3bc3f ("Merge x86/microcode into tip/master")
>>
>> # Linux Implementation
>>
>> Feature adds following interface files when the resctrl "io_alloc" feature
>> is supported on the resource:
>>
>> /sys/fs/resctrl/info/L3/io_alloc: Report the feature status. Enable/disable the
>> 				  feature by writing to the interface.
>>
>> /sys/fs/resctrl/info/L3/io_alloc_cbm:  List the Capacity Bit Masks (CBMs) available
>> 				       for I/O devices when io_alloc feature is enabled.
>> 				       Configure the CBM by writing to the interface.
>>
>> When CDP is enabled, these files will be created both in L3CODE and L3DATA.
> 
> "will be" -> "are"
> 

Sure.

>>
>> # Examples:
>>
>> a. Check if io_alloc feature is available
>> 	# mount -t resctrl resctrl /sys/fs/resctrl/
>>
>> 	# cat /sys/fs/resctrl/info/L3/io_alloc
>> 	disabled
>>
>> b. Enable the io_alloc feature. 
>>
>> 	# echo 1 > /sys/fs/resctrl/info/L3/io_alloc 
>> 	# cat /sys/fs/resctrl/info/L3/io_alloc
>> 	enabled
>>
>> c. Check the CBM values for the io_alloc feature.
>>
>> 	# cat /sys/fs/resctrl/info/L3/io_alloc_cbm 
>> 	0=ffff;1=ffff
>>
>> d. Change the CBM value for the domain 1:
> 
> "for the domain 1" -> "of domain 1"?

Sure.

> 
> (nit: inconsistent use of "." vs. ":" in items)

Fixed it.

> 
>> 	# echo 1=FF > /sys/fs/resctrl/info/L3/io_alloc_cbm
>>
>> 	# cat /sys/fs/resctrl/info/L3/io_alloc_cbm 
>> 	0=ffff;1=00ff
>>
>> d. Disable io_alloc feature and exit.
>>
>> 	# echo 0 > /sys/fs/resctrl/info/L3/io_alloc 
>> 	# cat /sys/fs/resctrl/info/L3/io_alloc
>> 	disabled
>>
>> 	# umount /sys/fs/resctrl/
>>
>> ---
> 
> Reinette
> 
> 

-- 
Thanks
Babu Moger