[PATCH RFC 00/23] resctrl: MBA control emulation and ARM MPAM MB_NODE support

[Fenghua Yu <fenghuay@nvidia.com>]

This series enables MBA bandwidth control emulation on ARM MPAM when the
default MB control is disabled, using a node-scoped MB_NODE control as
the native backend.  It extends memory-bandwidth monitoring and
control to memory-level MSCs or CPU-less NUMA nodes, and adds
documentation and tests.

Background

On x86, MBA bandwidth control is typically associated with the L3 cache:
schemata lines use L3 cache ids and monitor domains appear as mon_L3_XX.
On some ARM MPAM systems the MBA control is backed by a memory-level MSC
rather than the L3 cache MSC.  A memory-level MSC is represented as a
NUMA node, and that node commonly has no CPUs of its own (a memory-only
node that still participates in bandwidth control).  For such a resource
the control scope is NODE, schemata identifiers are NUMA node ids,
monitor domains are named mon_NODE_XX, and a node-scoped MB_NODE control
may be the only control with usable bandwidth hardware.

Because the memory-level MSC lives on a CPU-less NUMA node, the driver
must handle nodes with no local CPUs.  mpam_ris_get_affinity() derives
affinity from the component's NUMA node id, so a memory-only node would
otherwise end up with an empty CPU mask and never be registered.  When
the derived mask is empty, the driver falls back to cpu_possible_mask
(masked by MSC accessibility) so the node still gets a resctrl domain.
The borrowed affinity is kept out of class->affinity: those CPUs already
contribute through CPU-ful nodes in the class, and subtracting the
borrowed mask on teardown would remove CPUs that other live nodes still
depend on.  Domain setup iterates all MSC components per CPU (keyed by
component), so each component — including these CPU-less memory nodes —
gets its own control and monitor domain.

Examples of NODE and emulation interfaces on MPAM.

**MB disabled, MB_NODE enabled** (``MB`` emulated by ``MB_NODE``)::

        info/MB/resource_schemata/
        ├── mode			  # [legacy] native
        └── MB/
            ├── scope                     # NODE
            ├── status                    # disabled
            └── MB_NODE/
                ├── scope                 # NODE
                └── status                # enabled
schemata:   MB_NODE and MB keeps same values
        MB_NODE:0=100;1=100;2=100;10=100;18=100;26=100;34=100;35=100
             MB:0=100;1=100;2=100;10=100;18=100;26=100;34=100;35=100
             L3:1=ffff;2=ffff

mon_data/
mon_L3_01  mon_NODE_00  mon_NODE_02  mon_NODE_18  mon_NODE_34
mon_L3_02  mon_NODE_01  mon_NODE_10  mon_NODE_26  mon_NODE_35

mon_data/mon_L3_01/llc_occupancy
mon_data/mon_NODE_01/mbm_total_bytes

Switch to native by writing to "mode":

        info/MB/resource_schemata/
        ├── mode			  # legacy [native]
        ├── MB/
        │   └── scope                     # NODE
        │   └── status                    # disabled
        └── MB_NODE/			  # MB_NODE is moved to here
                ├── scope                 # NODE
                └── status                # enabled

schemata:   Only show MB_NODE. No "MB:" is emulated.
        MB_NODE:0=100;1=100;2=100;10=100;18=100;26=100;34=100;35=100
             L3:1=ffff;2=ffff

mon_data/
mon_L3_01  mon_NODE_00  mon_NODE_02  mon_NODE_18  mon_NODE_34
mon_L3_02  mon_NODE_01  mon_NODE_10  mon_NODE_26  mon_NODE_35

mon_data/mon_L3_01/llc_occupancy
mon_data/mon_NODE_01/mbm_total_bytes

**MB enabled, MB_NODE enabled**::

Same for legacy and native:

        info/MB/resource_schemata/
        ├── mode			  # [legacy] native
        ├── MB/
        │   ├── scope                     # L3
        │   └── status                    # enabled
        └── MB_NODE/
            ├── scope                     # NODE
            └── status                    # enabled
schemata:
        MB_NODE:0=100;1=100;2=100;10=100;18=100;26=100;34=100;35=100
             MB:1=100;2=100
             L3:1=ffff;2=ffff

mon_data/
mon_L3_01  mon_NODE_00  mon_NODE_02  mon_NODE_18  mon_NODE_34
mon_L3_02  mon_NODE_01  mon_NODE_10  mon_NODE_26  mon_NODE_35

mon_data/mon_L3_01/llc_occupancy
mon_data/mon_L3_01/mbm_total_bytes
mon_data/mon_NODE_01/mbm_total_bytes

The patches can be compiled but not tested on x86.

The 23 patches are grouped as follows:

1. Fixes for previous issues (patches 1–2)

   Correct resource_schemata control subdirectory ownership on non-root
   resctrl mounts, and fix a NULL address access in the ARM MPAM driver.

2. resctrl control emulation (patches 3–8)

   Introduce the MBA emulation framework in the resctrl core: a writable
   legacy/native mode under info/MB/resource_schemata/, per-control status
   and nested schemata layout, schemata mirroring for controls without MBW
   hardware, and subdir rebuild when the mode changes.  Patch 8 documents
   the user-visible behaviour in Documentation/filesystems/resctrl.rst.

   This group and group 4 are trying to implement the concept in
   discussion [1].

   The main purpose of this group is to make CPU-less NUMA Node patches
   work.

   This implementation is only limited to MPAM CPU-less NUMA Node
   emulation. It needs to be enhanced to make another other emulation
   work, X86 memory region or RISC-V emulation etc.

   It's totally fine for me to either replace this implementation with
   another implementation or continue to enhance this implementition
   for control emulations.

3. Memory bandwidth monitoring and control on memory-level MSCs
   (patches 9–17)

   Generalise the monitor infrastructure beyond hard-wired L3 paths:
   de-hardcode L3 monitor plumbing, expose MBA MBM counter assignment,
   name node-scoped monitor domains mon_NODE_<id>, add a node-scope MBM
   total event, and make MBM handling resource-aware.

   On the ARM MPAM side, add memory-level MSC and ABMC support, refine
   L3 topology and class selection, refactor domain setup to iterate all
   MSC components per CPU (keyed by component), and handle CPU-less NUMA
   memory nodes that have no local CPUs but still participate in MPAM
   bandwidth control.

4. MB_NODE emulates MB (patch 18)

   Wire MB_NODE as the native backend that emulates a disabled MB control
   in the ARM MPAM driver.

5. Documentation (patches 19–21)

   Document memory-level MB control and NUMA nodes in the arm64 MPAM
   guide (patch 19), NODE-scoped MBA domains and mon_NODE_* monitoring
   in the resctrl documentation (patch 20), and an MB_NODE emulation
   example on ARM MPAM (patch 21).

6. Tests (patches 22–23)

   Add a KUnit test for CPU-less NUMA node affinity handling and a
   kselftest that exercises MB emulation mode switching, resource_schemata
   hierarchy, and schemata mirroring on ARM MPAM.

To apply the patches, on top of [2],

Testing

  # KUnit (CPU-less affinity)
  ./tools/testing/kunit/kunit.py run test_mpam_devices

  # kselftest (MB emulation; requires ARM MPAM with disabled MB control)
  cd tools/testing/selftests/resctrl
  ./resctrl_tests -a mb_emulation_test

The branch: https://github.com/fyu1/linux/  emul.cpu_less.rfc/
which is on top of [2].

[1] [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
https://lore.kernel.org/lkml/cd96bda7-675d-451e-b6d9-9fe4a405edbe@intel.com/
[2] Reinette's resctrl control branch: git://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git branch resctrl/controls_rfc_v1


Fenghua Yu (23):
  resctrl: Fix ownership of resource_schemata control subdirectories
  arm_mpam: Fix NULL address access issue
  resctrl: Expose MBA resource_schemata mode sysfs
  resctrl: Expose per-control status in resource_schemata
  resctrl: Add nested resource_schemata support for emulated controls
  resctrl: Mirror schemata for controls without MBW hardware
  resctrl: Rebuild resource_schemata subdirs on MBA mode change
  Documentation: resctrl: document MBA control emulation
  resctrl: De-hardcode L3 monitor infrastructure
  resctrl: Expose MBA MBM counter assignment sysfs
  resctrl: name node-scoped monitor domains mon_NODE_<id>
  resctrl: Add node-scope MBM total event
  resctrl: Make MBM paths resource-aware
  arm_mpam: Support memory-level MSCs and ABMC per class
  arm_mpam: Refine L3 topology and class selection
  arm_mpam: Include all MSC components during domain setup
  arm_mpam: Handle CPU-less numa nodes
  arm_mpam: Emulate MB control with node-scoped MB_NODE control
  Documentation: arm64: mpam: document memory-level MB control and NUMA nodes
  Documentation: resctrl: document NODE-scoped MBA domains and mon_NODE monitoring
  Documentation: resctrl: document MB_NODE emulation example on ARM MPAM
  arm_mpam: Add KUnit test for CPU-less NUMA node affinity
  selftests/resctrl: Add MB emulation test for ARM MPAM

 Documentation/arch/arm64/mpam.rst             | 105 ++--
 Documentation/filesystems/resctrl.rst         | 184 ++++++-
 arch/x86/kernel/cpu/resctrl/core.c            |   2 +
 drivers/resctrl/mpam_devices.c                |  59 +-
 drivers/resctrl/mpam_internal.h               |   8 +
 drivers/resctrl/mpam_resctrl.c                | 488 ++++++++++++-----
 drivers/resctrl/test_mpam_devices.c           |  67 +++
 fs/resctrl/ctrlmondata.c                      |  99 +++-
 fs/resctrl/internal.h                         |   9 +-
 fs/resctrl/monitor.c                          | 225 +++++---
 fs/resctrl/rdtgroup.c                         | 507 +++++++++++++++---
 include/linux/resctrl.h                       |  62 ++-
 include/linux/resctrl_types.h                 |  15 +-
 tools/testing/selftests/resctrl/config        |   1 +
 .../selftests/resctrl/mb_emulation_test.c     | 460 ++++++++++++++++
 tools/testing/selftests/resctrl/resctrl.h     |   1 +
 .../testing/selftests/resctrl/resctrl_tests.c |   1 +
 17 files changed, 1942 insertions(+), 351 deletions(-)
 create mode 100644 tools/testing/selftests/resctrl/mb_emulation_test.c

-- 
2.43.0