Re: [RFC v2] arm,x86,fs/resctrl: Generic schema description Proof of Concept

[Fenghua Yu <fenghuay@nvidia.com>]

Hi, Reinette,

On 8/3/26 22:26, Reinette Chatre wrote:
> Hi Everybody,
>                                                                                  
> I understand that folks are eager for this work to settle so that features that
> depend on it can make progress.
>                                                                                  
> I do have a request if you have a feature that depends on this PoC:
>                                                                                  
> 	Please do not fork this PoC and keep changes to the PoC code in your fork
>          to meet the requirements of the feature you are working on. This is resulting
>          in duplicate (even triplicate) changes to PoC floating around. Instead,
>          please collaborate on this PoC by responding to the patches that need changes/fixes
>          to support your feature. If the patches do not exist yet, please participate in
>          the discussions that plan these changes so the work adding the discussed features
> 	is not duplicated. If you already have the code, please feel free to send as fixups.
>          Doing so will help this PoC to settle faster and help meet the requirements of the
>          feature(s) a couple of folks are working on.
>                                                                                  
> RFC v1: https://lore.kernel.org/lkml/aab804b9-e8b5-40ad-a85b-af7033391243@intel.com/
> 	git://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git branch resctrl/controls_rfc_v1
> 
> RFC v2 of this PoC can be found at:
> git://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git branch resctrl/controls_rfc_v2
> 
> The list of changes is long. For an overview of the new interfaces, please see the new
> documentation patch:
> https://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git/patch/?id=2e2d0140e857eae2ba3e71e8d9c595897649599a
> 
> Remaining Opens:
> ---------------
> - MPAM only compile tested.
> - Only "SAMPLE" code provided for x86 to demonstrate emulated controls.
> - Architecture should take care to restore control value of legacy control when
>    switching mode from "native" to "legacy". This may be complicated by a
>    legacy control backed by multiple native controls and thus best handled by
>    architecture. This adds a complication that resctrl cannot promise that
>    switching between control modes will not be destructive since a reverse mapping
>    may not exist.
> - When switching control mode to native the top level control info files
>    stop returning information. This could be considered "work as intended"?
> - Add relationships between controls. For example, cannot set "MIN" control
>    value to be larger than "MAX. Unclear how to do this in resctrl fs when
>    considering that the relationship may span between enabled and disabled
>    controls. This is potentially something that needs to be managed by architecture
>    and thus potentially without a consistent failure to user space.
>                                                                                  
> Changes since RFC v1:
> --------------------
> There are many changes since RFC v1 since I aimed to address all feedback
> (but please see the deferred/dropped list below). If I missed anything it
> was not intentional. Please feel free to point it out to me.
> 
> User visible changes:
> --------------------
> - Expose bitmap control properties to user space.
>      -- Only expose two properties:
>         - resctrl_ctrl_bitmap::min_cbm_bits exposed via new file "min_bits".
>           This drops the familiar "cbm" term to not make the control specific
>           to capacity.
>         - resctrl_ctrl_bitmap::cbm_len exposed via "max" file
>      -- While "shareable_bits" is technically a bitmap control property this does
>         seem an opportunity to deprecate it since the IO alloc support proved
>         that it does not accurately represent shared allocations, "bit_usage"
>         is the best for that.
> - Add support for control flags with "linear" as first flag of the "scalar"
>    control and "sparse" the first flag of the "bitmap" control.
>    Include flags in output of the "type" file.
>    Suggested by Chenyu: https://lore.kernel.org/lkml/90ae82da-02b2-4058-8c1e-f16df3226d8a@intel.com/
> - Add a "default" field to a control using the "reset_val" name suggested
>    by Drew and return that instead of always using "max" for the scalar control.
>    The "MIN" control can return its minimum value as default.
>    Suggested by Ben in https://lore.kernel.org/lkml/29c95b69-e1a4-46b1-ab8b-45c09308b924@arm.com/
>    Use case from Drew: https://lore.kernel.org/lkml/aiCBratZchVFVhws@gen8/
>    Drew suggested name as "reset_val" https://lore.kernel.org/lkml/aiOrznnZjTGywMna@thelio>
> - Change "resource_schemata" to "schemata".
>    Fenghua: https://lore.kernel.org/lkml/d0f84cef-58f5-4b84-8171-a0dfe8fa6829@nvidia.com/
> - Fix the control alignment. (Chenyu and Tony)
> - Drop the "per control" scope. Move the "scope" back to being per-resource and
>    do not expose a per control "scope" file.  This helps to support new resources
>    like node scoped MBA that cannot emulate the legacy MB control. Also
>    supports emulated controls by ensuring that all controls associated with
>    a resource have the same scope.
> - Support emulating legacy control.
>    - Support multiple backing controls for a legacy control.
>    - Expose new per-resource "control_mode" file that user space can use to
>      switch between legacy and native controls. Only the enabled controls
>      are shown to user in schemata file.
>    - The new "native" control mode is only supported for the MBA resource.
>      The solution is generic but there are implications for, for example, the
>      resctrl features that rely on the legacy cache control (cache pseudo-locking
>      and IO alloc). Defer support for other resource since there are no immediate
>      plans to have, for example, the legacy cache control backed by something else.
>    - Architecture differences imply a lot of architecture flexibility in
>      support for emulating controls.
>      Architecture can expect resctrl fs to only stage control values in
>      controls that are enabled. Within architecture there could be rules
>      that determine what to program based on the control's emulated controls
>      or it could dynamically program update callbacks when the user changes
>      the control mode. RFC includes sample code identified by "SAMPLE" for the
>      latter that demonstrates one way to solve the x86 variant where the hardware
>      self supports both legacy and finer grained controls.
>    - The top level legacy control property files stop returning data when
>      the mode switches to "native".
> - Draft of documentation that describes multiple controls and emulated
>    controls.
> - Fix Intel MBA tolerance to be 10 to match the "round-up" implementation.
> 
> Changes not user visible:
> -------------------------
> - Rebase on tip x86/cache with following applied on top:
>    "x86,fs/resctrl: Improve resctrl quality and consistency"
>      https://lore.kernel.org/lkml/cover.1782857711.git.reinette.chatre@intel.com/
>    "x86,fs/resctrl,arm_mpam: Factor MBA parse-time conversion to be per-arch"
>      https://lore.kernel.org/lkml/20260709093111.367851-1-ben.horgan@arm.com/
> - Fix initialization of controls list in MPAM. (Ben)
> - Dan Carpenter reported an issue with missing unlocks in
>    rdt_bit_usage_show() that was resolved as part of rebase on top of the
>    new info_kn_lock()/info_kn_unlock() helpers.
> - Drop "mpam,x86,fs/resctrl: Make memory bandwidth delay a resource
>    property" so that memory bandwidth delay linear flag can be a control flag
>    instead.
> - Separate adding the control to struct msr_param from the change that
>    makes hardware update functions part of control.
>    ("x86/resctrl: Make control update properties part of the control") to
>    create new patch that is moved earlier in series to support bandwidth
>    linear flag property being part of control.
>    New patch: "x86/resctrl: Include control with information controlling hardware change"
> - Rename the controls to match their type to reduce confusion.
>    Suggested by Chenyu: https://lore.kernel.org/lkml/773353e8-b2a7-4c20-b6fa-195b0b301107@intel.com/
>    New names:
>    struct resctrl_membw -> struct resctrl_ctrl_scalar
>    struct resctrl_cache -> struct resctrl_ctrl_bitmap
>    resctrl_ctrl::membw -> resctrl_ctrl::scalar
>    resctrl_ctrl::cache -> resctrl_ctrl::bitmap
> - Switch subject prefix to be "arm,x86,fs/resctrl: ..." when changing both
>    architectures and resctrl fs code.
> - Switch to MPAM subject prefix of "arm_mpam: resctrl:" when only changing
>    MPAM code.
> 
> Changes deferred/dropped:
> ------------------------
> - Let "scope" form part of the control name.
>    Drop this since the current plan is to have the scope be part of the resource
>    name for new resources.
> - Introduce new per-control "status" that can be "enabled" or "disabled".
>    Drop this. While controls now have an "enabled" or "disabled" state internally,
>    this is not exposed to user space via a file. Instead, the "enabled" or
>    "disabled" state is exposed to user space via the entries appearing in schemata
>    file. This simplifies the subtleties when the hardware self supports both
>    legacy and finer grained interfaces (aka RDT's region-aware).
> - Defer emulated control support for cache controls.
> - Supporting different monitoring scope is not part of this PoC.
>                                                                                  
> Any feedback is appreciated.
With minor MPAM changes, I can do simple tests on MPAM:
-               INIT_LIST_HEAD(&hw_ctrl->r_ctrl.emulated_by);
+               INIT_LIST_HEAD(&mpam_ctrl->r_ctrl.emulated_by);

-               INIT_LIST_HEAD(&hw_ctrl->r_ctrl.emulated_by);
+               INIT_LIST_HEAD(&mpam_ctrl->r_ctrl.emulated_by);

There is a branch resctrl/controls_rfc_v2.1. Seems no diff from 
controls_rfc_v2. Is either branch OK for base code?

  * @emulated_by:List of controls that emulate this control. When set 
the containing
  *              struct resctrl_ctrl is likely a legacy control and 
@emulated_by
  *              are the finer grained hardware controls used to back 
the legacy
  *              control. This emulation is hidden from user in schemata 
file when
  *              rdt_resource::ctrl_mode is RESCTRL_CTRL_MODE_LEGACY and
  *              exposed when rdt_resource::ctrl_mode is
  *              RESCTRL_CTRL_MODE_NATIVE.
  */
struct resctrl_ctrl {
...
         struct list_head        emulated_by;
};

Does this mean a legacy resctrl control can be emulated by a few resctrl 
controls (which is stored in the list)?

e.g. legacy MB control can be emulated by a few controls? I thought 
"MB:" line is emulated by only one native control. e.g. "MB:" line is 
emulated by "MB_NODE:" line.

Thanks.

-Fenghua