[RFC v2] arm,x86,fs/resctrl: Generic schema description Proof of Concept

[Reinette Chatre <reinette.chatre@intel.com>]

Hi Everybody,                                                                   
                                                                                
I understand that folks are eager for this work to settle so that features that
depend on it can make progress.
                                                                                
I do have a request if you have a feature that depends on this PoC:
                                                                                
	Please do not fork this PoC and keep changes to the PoC code in your fork
        to meet the requirements of the feature you are working on. This is resulting
        in duplicate (even triplicate) changes to PoC floating around. Instead,
        please collaborate on this PoC by responding to the patches that need changes/fixes
        to support your feature. If the patches do not exist yet, please participate in
        the discussions that plan these changes so the work adding the discussed features
	is not duplicated. If you already have the code, please feel free to send as fixups.
        Doing so will help this PoC to settle faster and help meet the requirements of the
        feature(s) a couple of folks are working on.
                                                                                
RFC v1: https://lore.kernel.org/lkml/aab804b9-e8b5-40ad-a85b-af7033391243@intel.com/
	git://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git branch resctrl/controls_rfc_v1

RFC v2 of this PoC can be found at:
git://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git branch resctrl/controls_rfc_v2

The list of changes is long. For an overview of the new interfaces, please see the new
documentation patch:
https://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git/patch/?id=2e2d0140e857eae2ba3e71e8d9c595897649599a

Remaining Opens:
---------------
- MPAM only compile tested.
- Only "SAMPLE" code provided for x86 to demonstrate emulated controls.
- Architecture should take care to restore control value of legacy control when
  switching mode from "native" to "legacy". This may be complicated by a
  legacy control backed by multiple native controls and thus best handled by
  architecture. This adds a complication that resctrl cannot promise that
  switching between control modes will not be destructive since a reverse mapping
  may not exist.
- When switching control mode to native the top level control info files
  stop returning information. This could be considered "work as intended"?
- Add relationships between controls. For example, cannot set "MIN" control
  value to be larger than "MAX. Unclear how to do this in resctrl fs when
  considering that the relationship may span between enabled and disabled
  controls. This is potentially something that needs to be managed by architecture
  and thus potentially without a consistent failure to user space.
                                                                                
Changes since RFC v1:
--------------------
There are many changes since RFC v1 since I aimed to address all feedback
(but please see the deferred/dropped list below). If I missed anything it
was not intentional. Please feel free to point it out to me.

User visible changes:
--------------------
- Expose bitmap control properties to user space.
    -- Only expose two properties:
       - resctrl_ctrl_bitmap::min_cbm_bits exposed via new file "min_bits".
         This drops the familiar "cbm" term to not make the control specific
         to capacity.
       - resctrl_ctrl_bitmap::cbm_len exposed via "max" file
    -- While "shareable_bits" is technically a bitmap control property this does
       seem an opportunity to deprecate it since the IO alloc support proved
       that it does not accurately represent shared allocations, "bit_usage"
       is the best for that.
- Add support for control flags with "linear" as first flag of the "scalar"
  control and "sparse" the first flag of the "bitmap" control.
  Include flags in output of the "type" file.
  Suggested by Chenyu: https://lore.kernel.org/lkml/90ae82da-02b2-4058-8c1e-f16df3226d8a@intel.com/
- Add a "default" field to a control using the "reset_val" name suggested
  by Drew and return that instead of always using "max" for the scalar control.
  The "MIN" control can return its minimum value as default.
  Suggested by Ben in https://lore.kernel.org/lkml/29c95b69-e1a4-46b1-ab8b-45c09308b924@arm.com/
  Use case from Drew: https://lore.kernel.org/lkml/aiCBratZchVFVhws@gen8/
  Drew suggested name as "reset_val" https://lore.kernel.org/lkml/aiOrznnZjTGywMna@thelio>
- Change "resource_schemata" to "schemata".
  Fenghua: https://lore.kernel.org/lkml/d0f84cef-58f5-4b84-8171-a0dfe8fa6829@nvidia.com/
- Fix the control alignment. (Chenyu and Tony)
- Drop the "per control" scope. Move the "scope" back to being per-resource and
  do not expose a per control "scope" file.  This helps to support new resources
  like node scoped MBA that cannot emulate the legacy MB control. Also
  supports emulated controls by ensuring that all controls associated with
  a resource have the same scope.
- Support emulating legacy control.
  - Support multiple backing controls for a legacy control.
  - Expose new per-resource "control_mode" file that user space can use to
    switch between legacy and native controls. Only the enabled controls
    are shown to user in schemata file.
  - The new "native" control mode is only supported for the MBA resource.
    The solution is generic but there are implications for, for example, the
    resctrl features that rely on the legacy cache control (cache pseudo-locking
    and IO alloc). Defer support for other resource since there are no immediate
    plans to have, for example, the legacy cache control backed by something else.
  - Architecture differences imply a lot of architecture flexibility in
    support for emulating controls.
    Architecture can expect resctrl fs to only stage control values in
    controls that are enabled. Within architecture there could be rules
    that determine what to program based on the control's emulated controls
    or it could dynamically program update callbacks when the user changes
    the control mode. RFC includes sample code identified by "SAMPLE" for the
    latter that demonstrates one way to solve the x86 variant where the hardware
    self supports both legacy and finer grained controls.
  - The top level legacy control property files stop returning data when
    the mode switches to "native".
- Draft of documentation that describes multiple controls and emulated
  controls.                                                                     
- Fix Intel MBA tolerance to be 10 to match the "round-up" implementation.

Changes not user visible:
-------------------------
- Rebase on tip x86/cache with following applied on top:
  "x86,fs/resctrl: Improve resctrl quality and consistency"
    https://lore.kernel.org/lkml/cover.1782857711.git.reinette.chatre@intel.com/
  "x86,fs/resctrl,arm_mpam: Factor MBA parse-time conversion to be per-arch"
    https://lore.kernel.org/lkml/20260709093111.367851-1-ben.horgan@arm.com/
- Fix initialization of controls list in MPAM. (Ben)
- Dan Carpenter reported an issue with missing unlocks in
  rdt_bit_usage_show() that was resolved as part of rebase on top of the
  new info_kn_lock()/info_kn_unlock() helpers.
- Drop "mpam,x86,fs/resctrl: Make memory bandwidth delay a resource
  property" so that memory bandwidth delay linear flag can be a control flag
  instead.
- Separate adding the control to struct msr_param from the change that
  makes hardware update functions part of control.
  ("x86/resctrl: Make control update properties part of the control") to
  create new patch that is moved earlier in series to support bandwidth
  linear flag property being part of control.
  New patch: "x86/resctrl: Include control with information controlling hardware change"
- Rename the controls to match their type to reduce confusion.
  Suggested by Chenyu: https://lore.kernel.org/lkml/773353e8-b2a7-4c20-b6fa-195b0b301107@intel.com/
  New names:
  struct resctrl_membw -> struct resctrl_ctrl_scalar
  struct resctrl_cache -> struct resctrl_ctrl_bitmap
  resctrl_ctrl::membw -> resctrl_ctrl::scalar
  resctrl_ctrl::cache -> resctrl_ctrl::bitmap
- Switch subject prefix to be "arm,x86,fs/resctrl: ..." when changing both
  architectures and resctrl fs code.
- Switch to MPAM subject prefix of "arm_mpam: resctrl:" when only changing
  MPAM code.

Changes deferred/dropped:
------------------------
- Let "scope" form part of the control name.
  Drop this since the current plan is to have the scope be part of the resource
  name for new resources.
- Introduce new per-control "status" that can be "enabled" or "disabled".
  Drop this. While controls now have an "enabled" or "disabled" state internally,
  this is not exposed to user space via a file. Instead, the "enabled" or
  "disabled" state is exposed to user space via the entries appearing in schemata
  file. This simplifies the subtleties when the hardware self supports both
  legacy and finer grained interfaces (aka RDT's region-aware).
- Defer emulated control support for cache controls.
- Supporting different monitoring scope is not part of this PoC.
                                                                                
Any feedback is appreciated.

Reinette