Re: [RFC PATCH 13/19] x86/resctrl: Add PLZA state tracking and context switch handling

["Moger, Babu" <bmoger@amd.com>]

Hi Reinette,

On 2/23/2026 11:12 AM, Reinette Chatre wrote:
> Hi Babu,
> 
> On 2/20/26 2:44 PM, Moger, Babu wrote:
>> On 2/19/2026 8:53 PM, Reinette Chatre wrote:
> 
>>> Summary of considerations surrounding CLOSID/RMID (PARTID/PMG) assignment for kernel work
>>> =========================================================================================
>>>
>>> - PLZA currently only supports global assignment (only PLZA_EN of
>>>     MSR_IA32_PQR_PLZA_ASSOC may differ on logical processors). Even so, current
>>>     speculation is that RMID_EN=0 implies that user space RMID is used to monitor
>>>     kernel work that could appear to user as "kernel mode" supporting multiple RMIDs.
>>>     https://lore.kernel.org/lkml/abb049fa-3a3d-4601-9ae3-61eeb7fd8fcf@amd.com/
>>
>> Yes. RMID_EN=0 means dont use separate RMID for plza.
> 
> Thank you very much for confirming.
> 
> ...
> 
>>> How can resctrl support the requirements?
>>> =========================================
>>>
>>> New global resctrl fs files
>>> ===========================
>>> info/kernel_mode (always visible)
>>> info/kernel_mode_assignment (visibility and content depends on active setting in info/kernel_mode)
>>
>> Probably good idea to drop "assign" for this work. We already have mbm_assign mode and related work.
> 
> hmmm ... I think "assign" is generic enough of a word that
> it cannot be claimed by a single feature.
> 
> 
>> info/kernel_mode_assoc or info/kernel_mode_association? Or We can wait later to rename appropriately.
> 
> yes, naming can be settled later.


Sure.

> 
>>
>>>
>>> info/kernel_mode
>>> ================
>>> - Displays the currently active as well as possible features available to user
>>>     space.
>>> - Single place where user can query "kernel mode" behavior and capabilities of the
>>>     system.
>>> - Some possible values:
>>>     - inherit_ctrl_and_mon <=== previously named "match_user", just renamed for consistency with other names
>>>        When active, kernel and user space use the same CLOSID/RMID. The current status
>>>        quo for x86.
>>>     - global_assign_ctrl_inherit_mon
>>>        When active, CLOSID/control group can be assigned for *all* (hence, "global")
>>>        kernel work while all kernel work uses same RMID as user space.
>>>        Can only be supported on architecture where CLOSID and RMID are independent.
>>>        An arch may support this in hardware (RMID_EN=0?) or this can be done by resctrl during
>>>        context switch if the RMID is independent and the context switches cost is
>>>        considered "reasonable".
>>>        This supports use case https://lore.kernel.org/lkml/CABPqkBSq=cgn-am4qorA_VN0vsbpbfDePSi7gubicpROB1=djw@mail.gmail.com/
>>>        for PLZA.
>>>     - global_assign_ctrl_assign_mon
>>>        When active the same resource group (CLOSID and RMID) can be assigned to
>>>        *all* kernel work. This could be any group, including the default group.
>>>        There may not be a use case for this but it could be useful as an intemediate
>>>        step of the mode that follow (more later).
>>>     - per_group_assign_ctrl_assign_mon
>>>        When active every resource group can be associated with another (or the same)
>>>        resource group. This association maps the resource group for user space work
>>>        to resource group for kernel work. This is similar to the "kernel_group" idea
>>>        presented in:
>>>        https://lore.kernel.org/lkml/aYyxAPdTFejzsE42@e134344.arm.com/
>>>        This addresses use case https://lore.kernel.org/lkml/CABPqkBSq=cgn-am4qorA_VN0vsbpbfDePSi7gubicpROB1=djw@mail.gmail.com/
>>>        for MPAM.
>>
>> All these new names and related information will go in global structure.
>>
>> Something like this..
>>
>> Struct kern_mode {
>>         enum assoc_mode;
>>         struct rdtgroup *k_rdtgrp;
>>         ...
>> };
>>
>> Not sure what other information will be required here. Will know once I stared working on it.
>>
>> This structure will be updated based on user echo's in "kernel_mode" and "kernel_mode_assignment".
> 
> This looks to be a good start. I think keeping the rdtgroup association is good since
> it helps to easily display the name to user space while also providing access to the CLOSID
> and RMID that is assigned to the tasks.
> By placing them in their own structure instead of just globals it does make it easier to
> build on when some modes have different requirements wrt rdtgroup management.

I am not clear on this comment. Can you please elaborate little bit?

Thanks
Babu


> You may encounter that certain arrangements work better to support interactions with the
> task structure that are not clear at this time.
> 
> 
>>
>>
>>> - Additional values can be added as new requirements arise, for example "per_task"
>>>     assignment. Connecting visibility of info/kernel_mode_assignment to mode in
>>>     info/kernel_mode enables resctrl to later support additional modes that may require
>>>     different configuration files, potentially per-resource group like the "tasks_kernel"
>>>     (or perhaps rather "kernel_mode_tasks" to have consistent prefix for this feature)
>>>     and "cpus_kernel" ("kernel_mode_cpus"?) discussed in these threads.
>>
>> So, per resource group file "kernel_mode_tasks" and "kernel_mode_cpus" are not required right now. Correct?
> 
> Correct. The way I see it the baseline implementation to support PLZA should be
> straightforward. We'll probably spend a bit extra time on the supporting documentation
> to pave the way for possible additions.
> 
>>>     User can view active and supported modes:
>>>
>>>      # cat info/kernel_mode
>>>      [inherit_ctrl_and_mon]
>>>      global_assign_ctrl_inherit_mon
>>>      global_assign_ctrl_assign_mon
>>>
>>> User can switch modes:
>>>      # echo global_assign_ctrl_inherit_mon > kernel_mode
>>>      # cat kernel_mode
>>>      inherit_ctrl_and_mon
>>>      [global_assign_ctrl_inherit_mon]
>>>      global_assign_ctrl_assign_mon
>>>
>>>
>>> info/kernel_mode_assignment
>>> ===========================
>>> - Visibility depends on active mode in info/kernel_mode.
>>> - Content depends on active mode in info/kernel_mode
>>> - Syntax to identify resource groups can use the syntax created as part of earlier ABMC work
>>>     that supports default group https://lore.kernel.org/lkml/cover.1737577229.git.babu.moger@amd.com/
>>> - Default CTRL_MON group and if relevant, the default MON group, can be the default
>>>     assignment when user just changes the kernel_mode without setting the assignment.
>>>
>>> info/kernel_mode_assignment when mode is global_assign_ctrl_inherit_mon
>>> -----------------------------------------------------------------------
>>> - info/kernel_mode_assignment contains single value that is the name of the control group
>>>     used for all kernel work.
>>> - CLOSID/PARTID used for kernel work is determined from the control group assigned
>>> - default value is default CTRL_MON group
>>> - no monitor group assignment, kernel work inherits user space RMID
>>> - syntax is
>>>       <CTRL_MON group> with "/" meaning default.
>>>
>>> info/kernel_mode_assignment when mode is global_assign_ctrl_assign_mon
>>> -----------------------------------------------------------------------
>>> - info/kernel_mode_assignment contains single value that is the name of the resource group
>>>     used for all kernel work.
>>> - Combined CLOSID/RMID or combined PARTID/PMG is set globally to be associated with all
>>>     kernel work.
>>> - default value is default CTRL_MON group
>>> - syntax is
>>>       <CTRL_MON group>/MON group>/ with "//" meaning default control and default monitoring group.
>>>
>>> info/kernel_mode_assignment when mode is per_group_assign_ctrl_assign_mon
>>> -------------------------------------------------------------------------
>>> - this presents the information proposed in https://lore.kernel.org/lkml/aYyxAPdTFejzsE42@e134344.arm.com/
>>>     within a single file for convenience and potential optimization when user space needs to make changes.
>>>     Interface proposed in https://lore.kernel.org/lkml/aYyxAPdTFejzsE42@e134344.arm.com/ is also an option
>>>     and as an alternative a per-resource group "kernel_group" can be made visible when user space enables
>>>     this mode.
>>> - info/kernel_mode_assignment contains a mapping of every resource group to another resource group:
>>>     <resource group for user space work>:<resource group for kernel work>
>>> - all resource groups must be present in first field of this file
>>> - Even though this is a "per group" setting expectation is that this will set the
>>>     kernel work CLOSID/RMID for every task. This implies that writing to this file would need
>>>     to access the tasklist_lock that, when taking for too long, may impact other parts of system.
>>>     See https://lore.kernel.org/lkml/CALPaoCh0SbG1+VbbgcxjubE7Cc2Pb6QqhG3NH6X=WwsNfqNjtA@mail.gmail.com/
>>
>> This mode is currently not supported in AMD PLZA implementation. But we have to keep the options open for future enhancement for MPAM. I am still learning on MPM requirement.
>>
>>>
>>> Scenarios supported
>>> ===================
>>>
>>> Default
>>> -------
>>> For x86 I understand kernel work and user work to be done with same CLOSID/RMID which
>>> implies that info/kernel_mode can always be visible and at least display:
>>>      # cat info/kernel_mode
>>>      [inherit_ctrl_and_mon]
>>>
>>> info/kernel_mode_assignment is not visible in this mode.
>>>
>>> I understand MPAM may have different defaults here so would like to understand better.
>>>
>>> Dedicated global allocations for kernel work, monitoring same for user space and kernel (PLZA)
>>> ----------------------------------------------------------------------------------------------
>>> Possible scenario with PLZA, not MPAM (see later):
>>> 1. Create group(s) to manage allocations associated with user space work
>>>      and assign tasks/CPUs to these groups.
>>> 2. Create group to manage allocations associated with all kernel work.
>>>      - For example,
>>>      # mkdir /sys/fs/resctrl/unthrottled
>>>      - No constraints from resctrl fs on interactions with files in this group. From resctrl
>>>        fs perspective it is not "dedicated" to kernel work but just another resource group.
>>
>> That is correct. We dont need to handle the group special for kernel_mode while creating the group. However, there will some handling required when kernel_mode group is deleted. We need to move the tasks/cpus back to default group and update the global kernel_mode structure.
> 
> Good point, yes.
> 
> 
> ...
> 
>>> Dedicated global allocations for kernel work, monitoring same for user space and kernel (MPAM)
>>> ----------------------------------------------------------------------------------------------
>>> 1. User space creates resource and monitoring groups for user tasks:
>>>        /sys/fs/resctrl <= User space default allocations
>>>      /sys/fs/resctrl/g1 <= User space allocations g1
>>>      /sys/fs/resctrl/g1/mon_groups/g1m1 <= User space monitoring group g1m1
>>>      /sys/fs/resctrl/g1/mon_groups/g1m2 <= User space monitoring group g1m2
>>>      /sys/fs/resctrl/g2 <= User space allocations g2
>>>      /sys/fs/resctrl/g2/mon_groups/g2m1 <= User space monitoring group g2m1
>>>      /sys/fs/resctrl/g2/mon_groups/g2m2 <= User space monitoring group g2m2
>>>
>>> 2. User space creates resource and monitoring groups for kernel work (system has two PMG):
>>>      /sys/fs/resctrl/kernel <= Kernel space allocations
>>>      /sys/fs/resctrl/kernel/mon_data               <= Kernel space monitoring for all of default and g1
>>>      /sys/fs/resctrl/kernel/mon_groups/kernel_g2   <= Kernel space monitoring for all of g2
>>> 3. Set kernel mode to per_group_assign_ctrl_assign_mon:
>>>      # echo per_group_assign_ctrl_assign_mon > info/kernel_mode
>>>      - info/kernel_mode_assignment becomes visible and contains
>>>      # cat info/kernel_mode_assignment
>>>      //://
>>>      g1//://
>>>      g1/g1m1/://
>>>      g1/g1m2/://
>>>      g2//://
>>>      g2/g2m1/://
>>>      g2/g2m2/://
>>>      - An optimization here may be to have the change to per_group_assign_ctrl_assign_mon mode be implemented
>>>        similar to the change to global_assign_ctrl_assign_mon that initializes a global default. This can
>>>        avoid keeping tasklist_lock for a long time to set all tasks' kernel CLOSID/RMID to default just for
>>>        user space to likely change it.
>>> 4. Set groups to be used for kernel work:
>>>      # echo '//:kernel//\ng1//:kernel//\ng1/g1m1/:kernel//\ng1/g1m2/:kernel//\ng2//:kernel/kernel_g2/\ng2/g2m1/:kernel/kernel_g2/\ng2/g2m2/:kernel/kernel_g2/\n' > info/kernel_mode_assignment
>>>
>>
>> Currently, this is not supported in AMD's PLZA implimentation. But we need to keep this option open for MPAM.
> 
> Right. I expect PLZA to at least support "global_assign_ctrl_inherit_mon" mode
> since that is the one we know somebody is waiting for. I am not actually sure about
> "global_assign_ctrl_assign_mon" for PLZA. It is the variant intended to be implemented
> by this RFC submission and does not seem difficult to implement but I have not really heard
> any requests around it. Please do correct me if I missed anything here.
> 
>>
>>> The interfaces proposed aim to maintain compatibility with existing user space tools while
>>> adding support for all requirements expressed thus far in an efficient way. For an existing
>>> user space tool there is no change in meaning of any existing file and no existing known
>>> resource group files are made to disappear. There is a global configuration that lets user space
>>> manage allocations without needing to check and configure each control group, even per-resource
>>> group allocations can be managed from user space with a single read/write to support
>>> making changes in most efficient way.
>>>
>>> What do you think?
>>>
>>
>> I will start planning this work. Feel free to add more details.
>> I Will have more questions as I start working on it.
>>
>> I will separate GMBA work from this work.
>>
>> Will send both series separately.
>>
>> Thanks for details and summary.
>>
> 
> Thank you very much.
> 
> Reinette
> 
>