Linux Documentation
 help / color / mirror / Atom feed
From: Babu Moger <babu.moger@amd.com>
To: tony.luck@intel.com, reinette.chatre@intel.com,
	ben.horgan@arm.com, Dave.Martin@arm.com, james.morse@arm.com,
	bp@alien8.de
Cc: corbet@lwn.net, skhan@linuxfoundation.org, rdunlap@infradead.org,
	x86@kernel.org, fustini@kernel.org, tglx@kernel.org,
	mingo@redhat.com, dave.hansen@linux.intel.com, hpa@zytor.com,
	fenghuay@nvidia.com, rppt@kernel.org, arnd@arndb.de,
	kmehltretter@gmail.com, dapeng1.mi@linux.intel.com,
	elver@google.com, enelsonmoore@gmail.com, jlayton@kernel.org,
	seanjc@google.com, ewanhai-oc@zhaoxin.com,
	binbin.wu@linux.intel.com, nikunj@amd.com, ashish.kalra@amd.com,
	peterz@infradead.org, superm1@kernel.org,
	thomas.lendacky@amd.com, prathyushi.nangia@amd.com,
	kim.phillips@amd.com, ricardo.neri-calderon@linux.intel.com,
	darwi@linutronix.de, linux-doc@vger.kernel.org,
	linux-kernel@vger.kernel.org,
	linux-arm-kernel@lists.infradead.org
Subject: Re: [PATCH v7 13/19] x86,fs/resctrl: Add CPU hotplug support for kernel mode
Date: Mon, 16 Nov 2026 14:08:01 -0600	[thread overview]
Message-ID: <79684ccf-9b77-4c2c-8f86-0fee306a8745@amd.com> (raw)
In-Reply-To: <a46c34d2cb1cf11daaece5ef49f7e7936082d026.1794437052.git.babu.moger@amd.com>



On 11/11/26 16:44, Babu Moger wrote:
> The RESCTRL_GLOBAL_ENABLE_PER_CPU kernel mode allows the CPU's CLOSID
> and/or RMID association to change when transitioning from user mode to
> kernel mode. The mode is enabled or disabled on a per-CPU basis through a
> backing resource group. CPUs that come online are brought into a state
> consistent with the active kernel mode, while CPUs that go offline are
> removed from its CPU mask.
> 
> Integrate RESCTRL_GLOBAL_ENABLE_PER_CPU mode with CPU hotplug handling so
> that CPUs automatically join the active kernel mode when they come online
> and are removed from it when they go offline. Also, ensure the register
> state remains in a consistent reset state across kexec boot transitions.
> This ensures that kernel mode behavior remains consistent across CPU
> hotplug events and kexec transitions.
> 
> Introduce resctrl_kmode_global_online_cpu() and
> resctrl_kmode_global_offline_cpu(), and hook them into the existing
> resctrl_online_cpu() and resctrl_offline_cpu() paths. Also update
> clear_closid_rmid() to reset MSR_IA32_PQR_PLZA_ASSOC register across kexec
> transitions.
> 
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---
> v7: Changelog update.
>      Code comment update.
>      Added changes to reset the PLZA register during the bootup.
> 
> v6: Update the changelog in non-bugfix form.
>      Update the code to remove rdtgroup->kmode.
>      Introduce new wrapper resctrl_configure_kmode_global() to consolidate
>      architecture calls.
> 
> v5: Added resctrl_kmode_offline_cpu() to disable the kernel mode
>      association and update kmode_cpu_mask when an associated CPU goes
>      offline.
>      This addresses sashiko comments:
>      https://sashiko.dev/#/patchset/cover.1783461016.git.babu.moger%40amd.com?part=11
>      https://sashiko.dev/#/patchset/cover.1783461016.git.babu.moger%40amd.com?part=10
>      Also moved the patch earlier than patch 10.
>      [RESEND PATCH v4 10/15] fs/resctrl: Reset the kernel mode binding when an rdtgroup is removed
>      That seemed much more clear.
> 
> v4: New patch in the series. Patch taken from:
>      https://lore.kernel.org/lkml/20260611111706.1981788-5-qinyuntan@linux.alibaba.com/
>      Updated the code to enable kernel mode by default whenever a CPU comes
>      online when one of global-assign mode is enabled.
> ---
>   arch/x86/kernel/cpu/resctrl/core.c |  3 ++
>   fs/resctrl/rdtgroup.c              | 56 ++++++++++++++++++++++++++++++
>   2 files changed, 59 insertions(+)
> 
> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
> index 14530c11bbd7..d7faf30c2885 100644
> --- a/arch/x86/kernel/cpu/resctrl/core.c
> +++ b/arch/x86/kernel/cpu/resctrl/core.c
> @@ -704,6 +704,9 @@ static void clear_closid_rmid(int cpu)
>   	state->cur_closid = RESCTRL_RESERVED_CLOSID;
>   	state->cur_rmid = RESCTRL_RESERVED_RMID;
>   	wrmsrq(MSR_IA32_PQR_ASSOC, val.q);
> +
> +	if (boot_cpu_has(X86_FEATURE_PLZA))
> +		wrmsrq(MSR_IA32_PQR_PLZA_ASSOC, 0);
>   }
>   

Sashiko reported:

https://sashiko.dev/#/patchset/cover.1794437052.git.babu.moger%40amd.com?part=13

=============================================================
commit 87e9e9cf13d544344f9797647860bcba9ffab37d
Author: Babu Moger <babu.moger@amd.com>

x86,fs/resctrl: Add CPU hotplug support for kernel mode

This commit integrates RESCTRL_GLOBAL_ENABLE_PER_CPU kernel mode with CPU
hotplug handling and resets MSR_IA32_PQR_PLZA_ASSOC in clear_closid_rmid().

> diff --git a/arch/x86/kernel/cpu/resctrl/core.c 
b/arch/x86/kernel/cpu/resctrl/core.c
> index fc59d66f77de1..33965b760aed0 100644
> --- a/arch/x86/kernel/cpu/resctrl/core.c
> +++ b/arch/x86/kernel/cpu/resctrl/core.c
> @@ -735,6 +735,9 @@ static void clear_closid_rmid(int cpu)
>  	state->cur_closid = RESCTRL_RESERVED_CLOSID;
>  	state->cur_rmid = RESCTRL_RESERVED_RMID;
>  	wrmsrq(MSR_IA32_PQR_ASSOC, val.q);
> +
> +	if (boot_cpu_has(X86_FEATURE_PLZA))
> +		wrmsrq(MSR_IA32_PQR_PLZA_ASSOC, 0);
>  }

Does zeroing MSR_IA32_PQR_PLZA_ASSOC in clear_closid_rmid() clobber the
domain's active PLZA configuration during CPU hotplug?

When a CPU goes offline while RESCTRL_GLOBAL_ENABLE_PER_CPU is active,
resctrl_arch_offline_cpu() first runs resctrl_offline_cpu() ->
resctrl_kmode_global_offline_cpu(), which clears plza_en while 
preserving rmid, rmid_en, closid, and closid_en in 
MSR_IA32_PQR_PLZA_ASSOC (or leaves the MSR untouched if the CPU was not 
in kmode_cpu_mask). Immediately afterward, resctrl_arch_offline_cpu() 
calls clear_closid_rmid() and writes 0 to MSR_IA32_PQR_PLZA_ASSOC:

arch/x86/kernel/cpu/resctrl/core.c:resctrl_arch_offline_cpu() {
	...
	resctrl_offline_cpu(cpu);

	mutex_lock(&domain_list_lock);
	for_each_capable_rdt_resource(r)
		domain_remove_cpu(cpu, r);
	mutex_unlock(&domain_list_lock);

	clear_closid_rmid(cpu);
	...
}

That overwrites the MSR value just programmed by
resctrl_kmode_global_offline_cpu() and zeroes rmid, rmid_en, closid, and
closid_en on the offlined CPU while other CPUs in the same QoS domain
still have the active kernel mode configuration programmed, violating 
the hardware requirement documented for union msr_pqr_plza_assoc that 
all fields except plza_en must be programmed identically across every 
CPU in a QoS domain. During CPU online, resctrl_arch_online_cpu() 
likewise calls clear_closid_rmid() after domain_add_cpu() and before 
resctrl_online_cpu() takes rdtgroup_mutex to program the active 
configuration.
=======================================================================

Yes. This is an issue. We cannot write 0 to MSR_IA32_PQR_PLZA_ASSOC in 
hotplug handler.

Couple of options:

1. Introduce the reboot notifier for resctrl and clear 
MSR_IA32_PQR_PLZA_ASSOC. Reboot notifier is called during kexec reboot.

or

2. Just clear the PLZA_EN bit in clear_closid_rmid().
    if (boot_cpu_has(X86_FEATURE_PLZA))
        msr_clear_bit(MSR_IA32_PQR_PLZA_ASSOC, PLZA_EN);

Thoughts?

Thanks
Babu

  reply	other threads:[~2026-11-16 20:08 UTC|newest]

Thread overview: 24+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-11 22:43 [PATCH v7 00/19] x86,fs/resctrl: Add kernel-mode (e.g., PLZA) support to the resctrl subsystem Babu Moger
2026-11-11 22:43 ` [PATCH v7 01/19] x86/cpufeatures: Support Privilege Level Zero Association (PLZA) Babu Moger
2026-11-11 22:43 ` [PATCH v7 02/19] x86/resctrl: Add PLZA support to command-line options Babu Moger
2026-11-11 22:43 ` [PATCH v7 03/19] x86/resctrl: Add PLZA MSR definition and field layout Babu Moger
2026-11-11 22:43 ` [PATCH v7 04/19] fs/resctrl: Introduce kernel mode enum Babu Moger
2026-11-16 20:07   ` Babu Moger
2026-11-11 22:43 ` [PATCH v7 05/19] arm,x86,fs/resctrl: Introduce arch hook to program global per-CPU kernel mode Babu Moger
2026-11-11 22:43 ` [PATCH v7 06/19] fs/resctrl: Introduce data structures for kernel mode configuration Babu Moger
2026-11-11 22:44 ` [PATCH v7 07/19] fs/resctrl: Introduce resctrl_set_kmode_support() to register supported modes Babu Moger
2026-11-11 22:44 ` [PATCH v7 08/19] x86/resctrl: Register RESCTRL_GLOBAL_ENABLE_PER_CPU when PLZA is available Babu Moger
2026-11-11 22:44 ` [PATCH v7 09/19] fs/resctrl: Rearrange top-level files in info directory Babu Moger
2026-11-11 22:44 ` [PATCH v7 10/19] fs/resctrl: Add interface to display kernel mode status Babu Moger
2026-11-11 22:44 ` [PATCH v7 11/19] fs/resctrl: Add support for hidden resource group files Babu Moger
2026-11-11 22:44 ` [PATCH v7 12/19] fs/resctrl: Introduce kmode_cpus/kmode_cpus_list per rdtgroup Babu Moger
2026-11-11 22:44 ` [PATCH v7 13/19] x86,fs/resctrl: Add CPU hotplug support for kernel mode Babu Moger
2026-11-16 20:08   ` Babu Moger [this message]
2026-11-11 22:44 ` [PATCH v7 14/19] fs/resctrl: Deactivate kernel mode when the group is removed Babu Moger
2026-11-11 22:44 ` [PATCH v7 15/19] fs/resctrl: Extend info/kernel_mode to configure the kernel mode Babu Moger
2026-11-16 20:09   ` Babu Moger
2026-11-11 22:44 ` [PATCH v7 16/19] fs/resctrl: Reject mode changes for groups backing " Babu Moger
2026-11-11 22:44 ` [PATCH v7 17/19] fs/resctrl: Control visibility of global per-CPU kernel mode group files Babu Moger
2026-11-16 20:10   ` Babu Moger
2026-11-11 22:44 ` [PATCH v7 18/19] fs/resctrl: Allow userspace to write kmode_cpus/kmode_cpus_list Babu Moger
2026-11-11 22:44 ` [PATCH v7 19/19] fs/resctrl: Add kernel mode documentation and examples Babu Moger

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=79684ccf-9b77-4c2c-8f86-0fee306a8745@amd.com \
    --to=babu.moger@amd.com \
    --cc=Dave.Martin@arm.com \
    --cc=arnd@arndb.de \
    --cc=ashish.kalra@amd.com \
    --cc=ben.horgan@arm.com \
    --cc=binbin.wu@linux.intel.com \
    --cc=bp@alien8.de \
    --cc=corbet@lwn.net \
    --cc=dapeng1.mi@linux.intel.com \
    --cc=darwi@linutronix.de \
    --cc=dave.hansen@linux.intel.com \
    --cc=elver@google.com \
    --cc=enelsonmoore@gmail.com \
    --cc=ewanhai-oc@zhaoxin.com \
    --cc=fenghuay@nvidia.com \
    --cc=fustini@kernel.org \
    --cc=hpa@zytor.com \
    --cc=james.morse@arm.com \
    --cc=jlayton@kernel.org \
    --cc=kim.phillips@amd.com \
    --cc=kmehltretter@gmail.com \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@redhat.com \
    --cc=nikunj@amd.com \
    --cc=peterz@infradead.org \
    --cc=prathyushi.nangia@amd.com \
    --cc=rdunlap@infradead.org \
    --cc=reinette.chatre@intel.com \
    --cc=ricardo.neri-calderon@linux.intel.com \
    --cc=rppt@kernel.org \
    --cc=seanjc@google.com \
    --cc=skhan@linuxfoundation.org \
    --cc=superm1@kernel.org \
    --cc=tglx@kernel.org \
    --cc=thomas.lendacky@amd.com \
    --cc=tony.luck@intel.com \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox