[RESEND PATCH v4 00/15] x86/resctrl: Add kernel-mode (e.g., PLZA) support to the resctrl subsystem

[Babu Moger <babu.moger@amd.com>]

Hi All,

Had some problem sending the patches earlier. Resending it again.

This series adds support for AMD's Privilege-Level Zero Association
(PLZA) so kernel work can be assigned to a resctrl group, and wires it
up through a small generic "kernel mode" (kmode) layer in fs/resctrl
so future architectures can plug in without touching core resctrl.

The features are documented in:

   AMD64 Zen6 Platform Quality of Service (PQOS) Extensions,
   Publication # 69193 Revision 1.00, Issue Date March 2026

available at https://bugzilla.kernel.org/show_bug.cgi?id=206537

The patches are based on top of commit (tip/master v7.2-rc2):

  f705c6728b35 Merge branch into tip/master: 'x86/msr'

Background
==========

Customers have identified an issue while using the QoS resource Control
feature. If a memory bandwidth associated with a CLOSID is aggressively
throttled, and it moves into Kernel mode, the Kernel operations are also
aggressively throttled. This can stall forward progress and eventually
degrade overall system performance.

Privilege-Level Zero Association (PLZA) allows the user to specify a CLOSID
and/or RMID for execution at Privilege Level Zero. When PLZA is enabled on
a CPU, kernel work at PL0 uses the CLOSID and/or RMID from MSR
PQR_PLZA_ASSOC; otherwise, the CPU uses the CLOSID and RMID from PQR_ASSOC.


Design
======

A new sysfs file, info/kernel_mode, holds the global policy for resource
allocation and monitoring of kernel work and the resource group (when
applicable) associated with the policy.  Reads list the supported modes
and the currently active binding; writes change the policy or rebind to a
different group.  Look at the thread below for design discussion.
https://lore.kernel.org/lkml/14a8ad0a-e842-4268-871a-0762f1169e03@intel.com/

Per-rdtgroup files kmode_cpus and kmode_cpus_list scope the binding
to a subset of online CPUs without unbind/rebind churn.  They are
visible only on the group that is currently the active kernel-mode
binding.

The arch hook, resctrl_arch_configure_kmode keep the fs/resctrl layer
arch-neutral.

Introduced `resctrl_set_kmode_support()` so architecture code
can register supported kernel-mode policies during resctrl initialization.

Only AMD PLZA is wired up here; Intel and ARM can add their own
support later by implementing the hooks.

Examples
========

(See Documentation/filesystems/resctrl.rst, "kernel_mode",
"kmode_cpus", and "Examples on working with kernel_mode", for the full
UAPI.)

  # Mount resctrl
  # mount -t resctrl resctrl /sys/fs/resctrl
  # cd /sys/fs/resctrl

  # Read the supported modes.  The active mode is bracketed for display
  # only.  Inactive global-assign modes report ":group=uninitialized".
  # cat info/kernel_mode
  [inherit_ctrl_and_mon]
  global_assign_ctrl_inherit_mon_per_cpu:group=uninitialized
  global_assign_ctrl_assign_mon_per_cpu:group=uninitialized

  # Create a CTRL_MON group and bind kernel-mode allocation to it.
  # mkdir ctrl1
  # echo "global_assign_ctrl_inherit_mon_per_cpu:group=ctrl1//" \
          > info/kernel_mode
  # cat info/kernel_mode
  inherit_ctrl_and_mon
  [global_assign_ctrl_inherit_mon_per_cpu:group=ctrl1//]
  global_assign_ctrl_assign_mon_per_cpu:group=uninitialized

  # kmode_cpus and kmode_cpus_list are visible only on the bound group.
  # ls ctrl1/kmode_cpus*
  ctrl1/kmode_cpus  ctrl1/kmode_cpus_list

  # Restrict the binding to a CPU subset; the write is incremental.
  # echo 0-3 > ctrl1/kmode_cpus_list
  # cat ctrl1/kmode_cpus
  f
  # cat ctrl1/kmode_cpus_list
  0-3

  # Return to the default inherit mode.
  # echo "inherit_ctrl_and_mon" > info/kernel_mode
  # cat info/kernel_mode
  [inherit_ctrl_and_mon]
  global_assign_ctrl_inherit_mon_per_cpu:group=uninitialized
  global_assign_ctrl_assign_mon_per_cpu:group=uninitialized

Tested on AMD with PLZA; the generic bits build clean on x86 without
PLZA support and are no-ops at runtime.

Layout
======

  01-03  x86: PLZA CPU feature, command-line option, and MSR/data-structure
         plumbing.
  04-07  fs/resctrl + x86: kmode data structures, arch hooks, supported-mode
         initialization, and PLZA policy advertisement.
  08     fs/resctrl: info/kernel_mode read-only introspection.
  09-10  fs/resctrl: per-rdtgroup kmode_cpus[_list] exposure and reset of
         the binding when the bound rdtgroup is removed.
  11     fs/resctrl: program kernel-mode binding when a CPU comes online.
  12     resctrl: hide kmode_cpus[_list] on groups not bound to kernel-mode.
  13-14  fs/resctrl: info/kernel_mode write and incremental kmode_cpus[_list]
         writes.
  15     fs/resctrl: documentation and end-to-end examples.


Changelog
=========
v4:
  - Reorder and split the series into 15 patches: separate read-only
    info/kernel_mode display (08) from the write path (13);
    add hotplug support when a CPU comes online (11); add an end-to-end
    documentation/examples patch (15).

  - Introduced `resctrl_set_kmode_support()` so architecture code can
    register supported kernel-mode policies during resctrl initialization.

  - info/kernel_mode UAPI: inactive global-assign modes report
    ":group=uninitialized" (not ":group=none"); the inherit mode has
    no ":group=" suffix; square brackets mark the active mode on read
    only and must not be included when writing.

  - info/kernel_mode write: validate group type (assign_mon may bind an
    RDTMON_GROUP or RDTCTRL_GROUP; inherit_mon only RDTCTRL_GROUP);
    run fail paths before tearing down the active binding so errors
    retain the old binding; reset any active global-assign binding
    before reprogramming when rebinding.

  - kmode_cpus / kmode_cpus_list: writable with incremental
    enable/disable deltas; empty masks are allowed; offline CPUs and
    writes through stale file descriptors (opened before an
    info/kernel_mode change) are rejected; pseudo-locked and
    pseudo-lock-setup groups are rejected.

  - Hotplug: newly online CPUs are added to the bound group's
    kmode_cpu_mask and programmed when a global-assign policy is
    active.

  - Documentation updated alongside the UAPI changes, including an
    "Examples on working with kernel_mode" walk-through.

v3:
  - Generalise the layer beyond AMD: rename "PLZA mode" to "kernel
    mode" (kmode) in code, sysfs, and Documentation.  The public
    interface is now info/kernel_mode and per-group kmode_cpus[_list].
  - info/kernel_mode UAPI cleanups: designated initialisers +
    static_assert for the mode-name table; strim() the input; clearer
    error messages via last_cmd_status.
  - kmode_cpus / kmode_cpus_list: expose read-only, gate visibility on
    the bound group, and allow incremental writes.
  - Reset the binding when the bound rdtgroup is removed, instead of
    leaving stale state.
  - Kerneldoc/comment cleanups across the series.

v2:
  - Similar to RFC with a new proposal; interface names were not final.
  - Separated Global Bandwidth Enforcement (GLBE) from PLZA; this
    series only adds PLZA support.
  - Used "kmode" instead of "PLZA" in the generic layer.

Previous versions:
v3: https://lore.kernel.org/lkml/cover.1777591496.git.babu.moger@amd.com/
v2: https://lore.kernel.org/lkml/cover.1773347820.git.babu.moger@amd.com/
v1: https://lore.kernel.org/lkml/cover.1769029977.git.babu.moger@amd.com/


Babu Moger (15):
  x86/resctrl: Support Privilege Level Zero Association (PLZA)
  x86/resctrl: Add PLZA support to command-line options
  x86/resctrl: Add data structures and definitions for PLZA
    configuration
  fs/resctrl: Introduce kernel mode (kmode) data structures
  x86,fs/resctrl: Introduce architecture hooks to program kernel-mode
  fs/resctrl: Introduce resctrl_set_kmode_support() to initialize
    supported modes
  x86/resctrl: Expose the supported PLZA kernel-mode policies during
    init
  fs/resctrl: Add interface to display supported and active kernel-mode
    policy
  fs/resctrl: Introduce kmode_cpus/kmode_cpus_list per rdtgroup
  fs/resctrl: Reset the kernel-mode binding when an rdtgroup is removed
  fs/resctrl: Program kernel-mode binding when CPU comes online
  fs/resctrl: Hide kmode_cpus[_list] on groups not bound to kernel-mode
  fs/resctrl: Add interface to modify kernel-mode via info/kernel_mode
  fs/resctrl: Allow user space to write kmode_cpus/kmode_cpus_list
  fs/resctrl: Add documentation on kernel_mode with example

 .../admin-guide/kernel-parameters.txt         |   2 +-
 Documentation/filesystems/resctrl.rst         | 195 ++++++
 arch/x86/include/asm/cpufeatures.h            |   1 +
 arch/x86/include/asm/msr-index.h              |   1 +
 arch/x86/kernel/cpu/resctrl/core.c            |   8 +
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c     |  36 +
 arch/x86/kernel/cpu/resctrl/internal.h        |  37 +
 arch/x86/kernel/cpu/scattered.c               |   1 +
 drivers/resctrl/mpam_resctrl.c                |   5 +
 fs/resctrl/internal.h                         |  16 +
 fs/resctrl/rdtgroup.c                         | 663 ++++++++++++++++++
 include/linux/resctrl.h                       |  56 ++
 12 files changed, 1020 insertions(+), 1 deletion(-)

-- 
2.43.0