Re: [PATCH v5 0/8] x86/resctrl: Support L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE)

["Moger, Babu" <babu.moger@amd.com>]

Please disregard this series. Need to resend this series. Had some issues
with my git-send and while testing sent the series by mistake.

On 5/22/25 14:51, Babu Moger wrote:
> 
> This series adds the support for L3 Smart Data Cache Injection Allocation
> Enforcement (SDCIAE) to resctrl infrastructure. It is refered to as "io_alloc"
> in resctrl subsystem.
> 
> Upcoming AMD hardware implements Smart Data Cache Injection (SDCI).
> Smart Data Cache Injection (SDCI) is a mechanism that enables direct
> insertion of data from I/O devices into the L3 cache. By directly caching
> data from I/O devices rather than first storing the I/O data in DRAM, SDCI
> reduces demands on DRAM bandwidth and reduces latency to the processor
> consuming the I/O data.
> 
> The SDCIAE (SDCI Allocation Enforcement) PQE feature allows system software
> to control the portion of the L3 cache used for SDCI devices.
> 
> When enabled, SDCIAE forces all SDCI lines to be placed into the L3 cache
> partitions identified by the highest-supported L3_MASK_n register, where n
> is the maximum supported CLOSID.
> 
> The feature details are documented in the APM listed below [1].
> [1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
> Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache
> Injection Allocation Enforcement (SDCIAE)
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
> 
> The feature requires linux support of TPH (TLP Processing Hints).
> The support is available in linux kernel after the commit
> 48d0fd2b903e3 ("PCI/TPH: Add TPH documentation")
> 
> The patches are based on top of commit
> 54d14f25664bbb (tip/x86/cache) ("MAINTAINERS: Add reviewers for fs/resctrl")
> 
> # Linux Implementation
> 
> Feature adds following interface files when the resctrl "io_alloc" feature is
> supported on L3 resource:
> 
> /sys/fs/resctrl/info/L3/io_alloc: Report the feature status. Enable/disable the
> 				  feature by writing to the interface.
> 
> /sys/fs/resctrl/info/L3/io_alloc_cbm:  List the Capacity Bit Masks (CBMs) available
> 				       for I/O devices when io_alloc feature is enabled.
> 				       Configure the CBM by writing to the interface.
> 
> When CDP is enabled, these files will be created both in L3CODE and L3DATA.
> 
> # Examples:
> 
> a. Check if io_alloc feature is available
> 	#mount -t resctrl resctrl /sys/fs/resctrl/
> 
> 	# cat /sys/fs/resctrl/info/L3/io_alloc
> 	disabled
> 
> b. Enable the io_alloc feature. 
> 
> 	# echo 1 > /sys/fs/resctrl/info/L3/io_alloc 
> 	# cat /sys/fs/resctrl/info/L3/io_alloc
> 	enabled
> 
> c. Check the CBM values for the io_alloc feature.
> 
> 	# cat /sys/fs/resctrl/info/L3/io_alloc_cbm 
> 	L3:0=ffff;1=ffff
> 
> d. Change the CBM value for the domain 1:
> 	# echo L3:1=FF > /sys/fs/resctrl/info/L3/io_alloc_cbm
> 
> 	# cat /sys/fs/resctrl/info/L3/io_alloc_cbm 
> 	L3:0=ffff;1=00ff
> 
> d. Disable io_alloc feature and exit.
> 
> 	# echo 0 > /sys/fs/resctrl/info/L3/io_alloc 
> 	# cat /sys/fs/resctrl/info/L3/io_alloc
> 	disabled
> 
> 	#umount /sys/fs/resctrl/
> 
> ---
> v5: 
>     Patches are created on top of recent resctrl FS/ARCH code restructure.
>     The files monitor.c/rdtgroup.c have been split between FS and ARCH directories.
>     Resolved the conflict due to the merge.
> 
>     Updated bit_usage to reflect the io_alloc CBM as discussed in the thread:
>     https://lore.kernel.org/lkml/3ca0a5dc-ad9c-4767-9011-b79d986e1e8d@intel.com/
>     Modified rdt_bit_usage_show() to read io_alloc_cbm in hw_shareable, ensuring
>     that bit_usage accurately represents the CBMs.
> 
>     Moved prototypes of resctrl_arch_io_alloc_enable() and
>     resctrl_arch_get_io_alloc_enabled() to include/linux/resctrl.h.
> 
>     Used rdt_kn_name to get the rdtgroup name instead of accesssing it directly
>     while printing group name used by the io_alloc_closid.
> 
>     Updated show_doms() to print the resource if only it is valid. Pass NULL while
>     printing io_alloc CBM.
> 
>     Changed the code to access io_alloc CBMs via either L3CODE or L3DATA resources.
> 
> v4: The "io_alloc" interface will report "enabled/disabled/not supported"
>     instead of 0 or 1..
> 
>     Updated resctrl_io_alloc_closid_get() to verify the max closid availability
>     using closids_supported().
> 
>     Updated the documentation for "shareable_bits" and "bit_usage".
> 
>     NOTE: io_alloc is about specific CLOS. rdt_bit_usage_show() is not designed
>     handle bit_usage for specific CLOS. Its about overall system. So, we cannot
>     really tell the user which CLOS is shared across both hardware and software.
>     This is something we need to discuss.
> 
>     Introduced io_alloc_init() to initialize fflags.
> 
>     Printed the group name when io_alloc enablement fails to help user.
>     
>     Added rdtgroup_mutex before rdt_last_cmd_puts() in resctrl_io_alloc_cbm_show().
>     Returned -ENODEV when resource type is CDP_DATA.
> 
>     Kept the resource name while printing the CBM (L3:0=ffff) that way we dont have
>     to change show_doms() just for this feature and it is consistant across all the
>     schemata display.
> 
>     Added new patch to call parse_cbm() directly to avoid code duplication.
> 
>     Checked all the series(v1-v3) again to verify if I missed any comment.
> 
> v3: Rewrote commit log for the last 3 patches. Changed the text to bit
>     more generic than the AMD specific feature. Added AMD feature
>     specifics in the end.
> 
>     Renamed the rdt_get_sdciae_alloc_cfg() to rdt_set_io_alloc_capable().
>     Renamed the _resctrl_io_alloc_enable() to _resctrl_sdciae_enable()
>     as it is arch specific.
> 
>     Changed the return to void in _resctrl_sdciae_enable() instead of int.
>  
>     The number of CLOSIDs is determined based on the minimum supported
>     across all resources (in closid_init). It needs to match the max
>     supported on the resource. Added the check to verify if MAX CLOSID
>     availability on the system.
> 
>     Added CDP check to make sure io_alloc is configured in CDP_CODE.
>     Highest CLOSID corresponds to CDP_CODE. 
> 
>     Added resctrl_io_alloc_closid_free() to free the io_alloc CLOSID.
> 
>     Added errors in few cases when CLOSID allocation fails.
>     Fixes splat reported when info/L3/bit_usage is accesed when io_alloc is enabled.
>     https://lore.kernel.org/lkml/SJ1PR11MB60837B532254E7B23BC27E84FC052@SJ1PR11MB6083.namprd11.prod.outlook.com/
> 
> v2: Added dependancy on X86_FEATURE_CAT_L3
>     Removed the "" in CPU feature definition.
> 
>     Changed sdciae_capable to io_alloc_capable to make it as generic feature.
>     Moved io_alloc_capable field in struct resctrl_cache.
> 
>     Changed the name of few arch functions similar to ABMC series.
>     resctrl_arch_get_io_alloc_enabled()
>     resctrl_arch_io_alloc_enable()
> 
>     Renamed the feature to "io_alloc".
>     Added generic texts for the feature in commit log and resctrl.rst doc.
>     Added resctrl_io_alloc_init_cat() to initialize io_alloc to default values
>     when enabled.
>     Fixed io_alloc interface to show only on L3 resource.
>     Added the locks while processing io_alloc CBMs.
> 
> Previous versions:
> v4: https://lore.kernel.org/lkml/cover.1745275431.git.babu.moger@amd.com/
> v3: https://lore.kernel.org/lkml/cover.1738272037.git.babu.moger@amd.com/
> v2: https://lore.kernel.org/lkml/cover.1734556832.git.babu.moger@amd.com/
> v1: https://lore.kernel.org/lkml/cover.1723824984.git.babu.moger@amd.com/
> 
> 
> Babu Moger (8):
>   x86/cpufeatures: Add support for L3 Smart Data Cache Injection
>     Allocation Enforcement
>   x86/resctrl: Add SDCIAE feature in the command line options
>   x86/resctrl: Detect io_alloc feature
>   x86/resctrl: Implement "io_alloc" enable/disable handlers
>   x86/resctrl: Add user interface to enable/disable io_alloc feature
>   x86/resctrl: Introduce interface to display io_alloc CBMs
>   x86/resctrl: Modify rdt_parse_data to pass mode and CLOSID
>   x86/resctrl: Introduce interface to modify io_alloc Capacity Bit Masks
> 
>  .../admin-guide/kernel-parameters.txt         |   2 +-
>  Documentation/filesystems/resctrl.rst         |  57 +++
>  arch/x86/include/asm/cpufeatures.h            |   1 +
>  arch/x86/include/asm/msr-index.h              |   1 +
>  arch/x86/kernel/cpu/cpuid-deps.c              |   1 +
>  arch/x86/kernel/cpu/resctrl/core.c            |   9 +
>  arch/x86/kernel/cpu/resctrl/internal.h        |   5 +
>  arch/x86/kernel/cpu/resctrl/rdtgroup.c        |  37 ++
>  arch/x86/kernel/cpu/scattered.c               |   1 +
>  fs/resctrl/ctrlmondata.c                      |  41 +-
>  fs/resctrl/internal.h                         |  10 +
>  fs/resctrl/rdtgroup.c                         | 350 +++++++++++++++++-
>  include/linux/resctrl.h                       |  18 +
>  13 files changed, 510 insertions(+), 23 deletions(-)
> 

-- 
Thanks
Babu Moger