Re: [RESEND PATCH v4 13/15] fs/resctrl: Add interface to modify kernel-mode via info/kernel_mode

["Moger, Babu" <bmoger@amd.com>]

On 7/7/2026 4:50 PM, Babu Moger wrote:
> info/kernel_mode reports which kernel-mode policies the platform supports
> and which one is active, but it is read-only and does not show the rdtgroup
> bound to the active kernel-mode policy.
> 
> User space needs both pieces to manage kernel-mode assignment: it must be
> able to select a policy and bind the global-assign modes to a specific
> resctrl group, and it must be able to read back which group is currently
> bound.
> 
> Make info/kernel_mode writable and extend the read format to identify the
> bound group for active global-assign modes.  Inactive global-assign modes
> are reported as "group=uninitialized"; INHERIT has no group suffix because
> it does not bind an rdtgroup.  Square brackets mark the active mode on read
> only and must not be included when writing.
> 
> Document the interface in Documentation/filesystems/resctrl.rst.
> 
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---
> v4: Rewrote the changelog.
>      Moved all the fail path earlier so that failures will retain the old
>      binding.
>      Taken care of requirement
>      assign_mon can be  RDTMON_GROUP or RDTCTRL_GROUP.
>      inherit_mon can only be RDTCTRL_GROUP.
> 
> v3: New patch to handle the changed interface file info/kernel_mode.
> ---
>   Documentation/filesystems/resctrl.rst |  80 +++++++++
>   fs/resctrl/rdtgroup.c                 | 238 +++++++++++++++++++++++++-
>   2 files changed, 315 insertions(+), 3 deletions(-)
> 
> diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
> index f3e941404967..5a13814d1325 100644
> --- a/Documentation/filesystems/resctrl.rst
> +++ b/Documentation/filesystems/resctrl.rst
> @@ -528,6 +528,86 @@ conveyed in the error returns from file operations. E.g.
>   	# cat info/last_cmd_status
>   	mask f7 has non-consecutive 1-bits
>   
> +"kernel_mode":
> +	In the top level of the "info" directory, "kernel_mode" controls how
> +	resource allocation and monitoring work in kernel mode. This is used on
> +	some platforms to assign dedicated allocation and/or monitoring to
> +	kernel-mode work.
> +
> +	Reading the file lists supported kernel modes, one per line.  The
> +	currently active mode is wrapped in square brackets for display only.
> +
> +	The modes are displayed in the following format:
> +	"<mode>:group=<ctrl>/<mon>/"
> +
> +	The inherit_ctrl_and_mon mode is shown as "<mode>" with no ":group=" suffix.
> +	Inactive global-assign modes report "group=uninitialized".
> +
> +	The bound group path uses empty components when they do not apply:
> +
> +	"//":
> +		The default (root) control group.
> +	"<ctrl>//":
> +		The control group named <ctrl>.
> +	"/<mon>/":
> +		The monitor group named <mon> under the default control group.
> +	"<ctrl>/<mon>/":
> +		The monitor group named <mon> under the control group named <ctrl>.
> +
> +	Modes:
> +
> +	- "inherit_ctrl_and_mon": Kernel work inherits allocation and monitoring
> +	  from the current user-space task (default).
> +	- "global_assign_ctrl_inherit_mon_per_cpu": A single allocation is
> +	  assigned for all kernel work; monitoring is still inherited from user
> +	  space.  Requires a CTRL_MON group.
> +	- "global_assign_ctrl_assign_mon_per_cpu": A single resource group
> +	  supplies both allocation and monitoring for all kernel work.  May be
> +	  a CTRL_MON or MON group.
> +
> +	Only modes supported by the platform are listed on read.
> +	Example::
> +
> +	  # mount -t resctrl resctrl /sys/fs/resctrl
> +	  # cd /sys/fs/resctrl
> +	  # cat info/kernel_mode
> +	  [inherit_ctrl_and_mon]
> +	  global_assign_ctrl_inherit_mon_per_cpu:group=uninitialized
> +	  global_assign_ctrl_assign_mon_per_cpu:group=uninitialized
> +
> +	The modes and binding can be modified by writing to the interface. Writing
> +	one line (terminated by a newline) selects the active mode and binds it to
> +	a resctrl group.
> +
> +	Writes must follow the format:
> +	"<mode>:group=<ctrl>/<mon>/"
> +
> +	The ":group=<spec>" suffix is optional; when omitted the default (root)
> +	control group is used for global-assign modes. Selecting a new mode or group
> +	tears down any active global-assign binding before programming the new one,
> +	including when switching between global-assign modes on the same group.
> +	The inherit_ctrl_and_mon mode ignores any supplied group and clears the
> +	active kernel-mode binding.  The mode must match one of the supported names
> +	exactly, and modes not advertised by the platform cannot be set. The
> +	display-only "group=uninitialized" form is rejected. Errors are reported in
> +	"info/last_cmd_status".
> +	Example::
> +
> +	  # mkdir ctrl1
> +	  # echo "global_assign_ctrl_assign_mon_per_cpu:group=ctrl1//" \
> +	         > info/kernel_mode
> +
> +	  # cat info/kernel_mode
> +	  inherit_ctrl_and_mon
> +	  global_assign_ctrl_inherit_mon_per_cpu:group=uninitialized
> +	  [global_assign_ctrl_assign_mon_per_cpu:group=ctrl1//]
> +
> +	  # echo "inherit_ctrl_and_mon" > info/kernel_mode
> +	  # cat info/kernel_mode
> +	  [inherit_ctrl_and_mon]
> +	  global_assign_ctrl_inherit_mon_per_cpu:group=uninitialized
> +	  global_assign_ctrl_assign_mon_per_cpu:group=uninitialized
> +
>   Resource alloc and monitor groups
>   =================================
>   
> diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
> index c537846d9264..7b06c3b3f00e 100644
> --- a/fs/resctrl/rdtgroup.c
> +++ b/fs/resctrl/rdtgroup.c
> @@ -1059,7 +1059,8 @@ static_assert(ARRAY_SIZE(resctrl_mode_str) == RESCTRL_NUM_KERNEL_MODES);
>    * @v: unused
>    *
>    * Displays one line per mode set in resctrl_kcfg.kmode. Bracket the active
> - * policy (resctrl_kcfg.kmode_cur).
> + * policy (resctrl_kcfg.kmode_cur). Square brackets are display-only; writes
> + * to info/kernel_mode must not include them.
>    *
>    * INHERIT_CTRL_AND_MON is displayed as "[inherit_ctrl_and_mon]" when active
>    * or "inherit_ctrl_and_mon" when supported but inactive, with no :group=
> @@ -1068,7 +1069,7 @@ static_assert(ARRAY_SIZE(resctrl_mode_str) == RESCTRL_NUM_KERNEL_MODES);
>    * Global-assign modes append :group=. An inactive mode is emitted as
>    * "<mode>:group=uninitialized". An active mode with a bound group is emitted
>    * as "[<mode>:group=<ctrl>/<mon>/]", where <ctrl>/<mon>/ is derived from
> - * resctrl_kcfg.k_rdtgrp.
> + * resctrl_kcfg.k_rdtgrp. The bracketed form is not accepted on write.
>    *
>    * Return: 0 on success, or -ENOENT on error.
>    */
> @@ -1164,6 +1165,8 @@ static void resctrl_kmode_files_set_visible(struct rdtgroup *rdtgrp, bool visibl
>    * @kmode:	Kernel-mode policy currently active on @rdtgrp.
>    *
>    * Reset the kernel-mode binding on the CPUs in @rdtgrp's @kmode_cpu_mask.
> + * Called from resctrl_kernel_mode_write() whenever an active global-assign
> + * policy is replaced, including a mode change on the same group.
>    */
>   static void rdtgroup_config_kmode_reset(struct rdtgroup *rdtgrp,
>   					enum resctrl_kernel_mode kmode)
> @@ -1212,6 +1215,234 @@ static void rdtgroup_kmode_detach(struct rdtgroup *rdtgrp)
>   	resctrl_kcfg.kmode_cur = INHERIT_CTRL_AND_MON;
>   }
>   
> +/**
> + * rdtgroup_config_kmode() - Push @rdtgrp's kernel CLOSID/RMID to hardware
> + * @rdtgrp:	Resctrl group whose CLOSID/RMID should be programmed.
> + * @kmode:	Kernel-mode policy to program for @rdtgrp.
> + *
> + * @rdtgrp carries the CLOSID/RMID to program. For monitor groups, the CLOSID
> + * matches the parent control group while the RMID belongs to the monitor group.
> + *
> + * The caller (resctrl_kernel_mode_write()) is responsible for validating that
> + * the (kmode, group type) pair is permitted before invoking this helper.
> + * This helper records the current online CPUs in @rdtgrp->kmode_cpu_mask and
> + * programs those CPUs with @rdtgrp's CLOSID/RMID.
> + */
> +static void rdtgroup_config_kmode(struct rdtgroup *rdtgrp, enum resctrl_kernel_mode kmode)
> +{
> +	bool assign_mon = (kmode == GLOBAL_ASSIGN_CTRL_ASSIGN_MON_PER_CPU);
> +
> +	/* A new binding starts with all currently online CPUs in scope. */
> +	cpumask_copy(&rdtgrp->kmode_cpu_mask, cpu_online_mask);
> +
> +	resctrl_arch_configure_kmode(&rdtgrp->kmode_cpu_mask, rdtgrp->closid,
> +				     rdtgrp->mon.rmid, assign_mon, true);
> +
> +	rdtgrp->kmode = true;
> +	resctrl_kmode_files_set_visible(rdtgrp, true);
> +}
> +
> +/**
> + * rdtgroup_by_kmode_path() - Resolve a "<ctrl>/<mon>/" path to an rdtgroup
> + * @ctrl_name:	Control-group name, or "" for the default control group.
> + * @mon_name:	Monitor-group name, or "" to select the control group itself.
> + *
> + * Matches the path syntax emitted by resctrl_kernel_mode_show():
> + *   "//"            - the default control group
> + *   "<ctrl>//"      - control group @ctrl_name
> + *   "/<mon>/"       - monitor group @mon_name under the default control group
> + *   "<ctrl>/<mon>/" - monitor group @mon_name under control group @ctrl_name
> + *
> + * An empty @ctrl_name selects &rdtgroup_default. Otherwise @ctrl_name must
> + * match an existing control group. If @mon_name is empty, the selected control
> + * group is returned; otherwise @mon_name is looked up in the selected control
> + * group's monitor children.
> + *
> + * Return: Pointer to the matching rdtgroup, or NULL if no such group exists.
> + */
> +static struct rdtgroup *rdtgroup_by_kmode_path(const char *ctrl_name,
> +					       const char *mon_name)
> +{
> +	struct rdtgroup *rdtg, *parent = &rdtgroup_default;
> +
> +	if (*ctrl_name) {
> +		parent = NULL;
> +		list_for_each_entry(rdtg, &rdt_all_groups, rdtgroup_list) {
> +			if (rdtg->type != RDTCTRL_GROUP)
> +				continue;
> +			if (!strcmp(rdt_kn_name(rdtg->kn), ctrl_name)) {
> +				parent = rdtg;
> +				break;
> +			}
> +		}
> +	}
> +	if (!parent)
> +		return NULL;
> +
> +	if (!*mon_name)
> +		return parent;
> +
> +	list_for_each_entry(rdtg, &parent->mon.crdtgrp_list, mon.crdtgrp_list)
> +		if (!strcmp(rdt_kn_name(rdtg->kn), mon_name))
> +			return rdtg;
> +	return NULL;
> +}
> +
> +/**
> + * resctrl_kernel_mode_write() - Select kernel mode and bind group via info/kernel_mode
> + * @of:		kernfs file handle.
> + * @buf:	One line of the form "<mode>[:group=<ctrl>/<mon>/]"; must end
> + *		with a newline. Do not include the square brackets used to mark
> + *		the active mode in resctrl_kernel_mode_show(). Leading and trailing
> + *		whitespace is ignored, as is whitespace between the mode name and
> + *		an optional ":group=" suffix. The ":group=<spec>" suffix is
> + *		optional; when omitted the default control group
> + *		(&rdtgroup_default) is used.
> + * @nbytes:	Length of @buf.
> + * @off:	File offset (unused).
> + *
> + * Parses @buf, validates that <mode> is listed in resctrl_mode_str[] and is
> + * supported by the platform (resctrl_kcfg.kmode), resolves <ctrl>/<mon>/ to
> + * an existing rdtgroup (or picks &rdtgroup_default if no group was specified),
> + * treats INHERIT as an unbound mode, tears down any active global-assign
> + * binding via rdtgroup_config_kmode_reset(), programs hardware via
> + * rdtgroup_config_kmode() when the new mode is not INHERIT_CTRL_AND_MON, and
> + * on success updates resctrl_kcfg.k_rdtgrp and resctrl_kcfg.kmode_cur. The
> + * display-only "group=uninitialized" form is rejected.  Errors are reported
> + * in last_cmd_status.
> + *
> + * Return: @nbytes on success, negative errno with last_cmd_status set on error.
> + */
> +static ssize_t resctrl_kernel_mode_write(struct kernfs_open_file *of,
> +					 char *buf, size_t nbytes, loff_t off)
> +{
> +	enum resctrl_kernel_mode mode;
> +	char *mode_str, *group_str, *slash;
> +	const char *ctrl_name, *mon_name;
> +	struct rdtgroup *rdtgrp;
> +	int ret = 0;
> +
> +	if (nbytes == 0 || buf[nbytes - 1] != '\n')
> +		return -EINVAL;
> +	buf[nbytes - 1] = '\0';
> +
> +	/* Tolerate surrounding whitespace before mode parsing. */
> +	buf = strim(buf);
> +
> +	/*
> +	 * Split "<mode>:group=<spec>"; the ":group=<spec>" suffix is optional
> +	 * and when omitted the default control group (&rdtgroup_default) is used.
> +	 * Trim mode_str after the split so whitespace before ":group=" is ignored.
> +	 * Square brackets from resctrl_kernel_mode_show() are not accepted.
> +	 */
> +	group_str = strstr(buf, ":group=");
> +	if (group_str) {
> +		*group_str = '\0';
> +		group_str += strlen(":group=");
> +	}
> +	mode_str = strim(buf);
> +
> +	mutex_lock(&rdtgroup_mutex);
> +	rdt_last_cmd_clear();
> +
> +	for (mode = 0; mode < RESCTRL_NUM_KERNEL_MODES; mode++)
> +		if (!strcmp(mode_str, resctrl_mode_str[mode]))
> +			break;
> +
> +	if (mode == RESCTRL_NUM_KERNEL_MODES) {
> +		rdt_last_cmd_puts("Unknown kernel mode\n");
> +		ret = -EINVAL;
> +		goto out_unlock;
> +	}
> +
> +	if (!(test_bit(mode, &resctrl_kcfg.kmode))) {
> +		rdt_last_cmd_puts("Kernel mode not available\n");
> +		ret = -EINVAL;
> +		goto out_unlock;
> +	}
> +
> +	/*
> +	 * INHERIT mode binds no rdtgroup. Ignore any user-supplied :group=
> +	 * suffix and let show print the mode without a group.
> +	 */
> +	if (mode == INHERIT_CTRL_AND_MON) {
> +		rdtgrp = NULL;
> +		goto update_mode;
> +	}
> +
> +	if (!group_str) {
> +		/* No ":group=" suffix: fall back to the default control group. */
> +		rdtgrp = &rdtgroup_default;
> +	} else if (!strcmp(group_str, "uninitialized")) {
> +		/* Display-only placeholder emitted by show; not selectable. */
> +		rdt_last_cmd_puts("Cannot bind to 'uninitialized' group\n");
> +		ret = -EINVAL;
> +		goto out_unlock;
> +	} else {
> +		/* Require exactly "<ctrl>/<mon>/" format */
> +		slash = strchr(group_str, '/');
> +		if (!slash) {
> +			rdt_last_cmd_puts("Group must be <ctrl>/<mon>/\n");
> +			ret = -EINVAL;
> +			goto out_unlock;
> +		}
> +		*slash = '\0';
> +		ctrl_name = group_str;
> +		mon_name = slash + 1;
> +		slash = strchr(mon_name, '/');
> +		if (!slash || slash[1] != '\0') {
> +			rdt_last_cmd_puts("Group must be <ctrl>/<mon>/\n");
> +			ret = -EINVAL;
> +			goto out_unlock;
> +		}
> +		*slash = '\0';
> +
> +		rdtgrp = rdtgroup_by_kmode_path(ctrl_name, mon_name);
> +		if (!rdtgrp) {
> +			rdt_last_cmd_puts("Group not found\n");
> +			ret = -EINVAL;
> +			goto out_unlock;
> +		}
> +	}
> +
> +	if (mode == GLOBAL_ASSIGN_CTRL_INHERIT_MON_PER_CPU &&
> +	    rdtgrp->type != RDTCTRL_GROUP) {
> +		rdt_last_cmd_puts("global_assign_ctrl_inherit_mon_per_cpu requires a control group\n");
> +		ret = -EINVAL;
> +		goto out_unlock;
> +	}
> +
> +	if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKED ||
> +	    rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP) {
> +		rdt_last_cmd_puts("Pseudo-locking in progress\n");
> +		ret = -EINVAL;
> +		goto out_unlock;
> +	}
> +
> +update_mode:
> +	/* No-op if the same mode is already active on the same group. */
> +	if (resctrl_kcfg.kmode_cur == mode && resctrl_kcfg.k_rdtgrp == rdtgrp)
> +		goto out_unlock;
> +
> +	/*
> +	 * Tear down any active global-assign binding before applying the new
> +	 * policy, including when only the mode changes on the same group.
> +	 */
> +	if (resctrl_kcfg.kmode_cur != INHERIT_CTRL_AND_MON)
> +		rdtgroup_config_kmode_reset(resctrl_kcfg.k_rdtgrp,
> +					    resctrl_kcfg.kmode_cur);
> +
> +	if (mode != INHERIT_CTRL_AND_MON)
> +		rdtgroup_config_kmode(rdtgrp, mode);
> +
> +	resctrl_kcfg.k_rdtgrp = rdtgrp;
> +	resctrl_kcfg.kmode_cur = mode;
> +
> +out_unlock:
> +	mutex_unlock(&rdtgroup_mutex);
> +	return ret ?: nbytes;
> +}
> +
>   void *rdt_kn_parent_priv(struct kernfs_node *kn)
>   {
>   	/*
> @@ -2117,9 +2348,10 @@ static struct rftype res_common_files[] = {
>   	},
>   	{
>   		.name		= "kernel_mode",
> -		.mode		= 0444,
> +		.mode		= 0644,
>   		.kf_ops		= &rdtgroup_kf_single_ops,
>   		.seq_show	= resctrl_kernel_mode_show,
> +		.write		= resctrl_kernel_mode_write,
>   		.fflags		= RFTYPE_TOP_INFO,
>   	},
>   	{


Sashiko reported:

https://sashiko.dev/#/patchset/cover.1783461016.git.babu.moger%40amd.com?part=13

--- Patch [13]: [RESEND PATCH v4 13/15] fs/resctrl: Add interface to 
modify kernel-mode via info/kernel_mode ---
commit f22e9f0418857d31e6764f0d2604e8d9fd48827e
Author: Babu Moger <babu.moger@amd.com>
fs/resctrl: Add interface to modify kernel-mode via info/kernel_mode

info/kernel_mode reports which kernel-mode policies the platform supports
and which one is active, but it is read-only and does not show the rdtgroup
bound to the active kernel-mode policy. This patch makes it writable and
extends the read format to identify the bound group.
 > diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
 > index c537846d92648..7b06c3b3f00e0 100644
 > --- a/fs/resctrl/rdtgroup.c
 > +++ b/fs/resctrl/rdtgroup.c
[ ... ]
 > @@ -1212,6 +1215,234 @@ static void rdtgroup_kmode_detach(struct 
rdtgroup *rdtgrp)
 > +static void rdtgroup_config_kmode(struct rdtgroup *rdtgrp, enum 
resctrl_kernel_mode kmode)
 > +{
 > +	bool assign_mon = (kmode == GLOBAL_ASSIGN_CTRL_ASSIGN_MON_PER_CPU);
 > +
 > +	/* A new binding starts with all currently online CPUs in scope. */
 > +	cpumask_copy(&rdtgrp->kmode_cpu_mask, cpu_online_mask);

In the resctrl_kernel_mode_write() call path, cpu_online_mask is 
accessed here without holding cpus_read_lock(). Could a concurrent CPU 
hotplug operation remove a CPU from cpu_online_mask immediately after 
this copy executes, causing on_each_cpu_mask() to issue a warning for 
offline CPUs?
---------------------------------------------------------------------

Yes. Need to hold cpus_read_lock() while accessing cpu_online_mask.
Will add it in next revision.

---------------------------------------------------------------------


Also, if a CPU is brought offline while a kernel-mode policy is active,
is it removed from kmode_cpu_mask? Looking at resctrl_offline_cpu(), it
clears the CPU from the standard rdtgrp->cpu_mask but appears to leave
kmode_cpu_mask unchanged. Will this accumulate offline CPU references,
leading to warnings during later policy resets or changes?

----------------------------------------------------------------------

Yes. This is another issue. This will be fixed as part of

[RESEND PATCH v4 11/15] fs/resctrl: Program kernel-mode binding when CPU 
comes online

----------------------------------------------------------------------

 > +
 > +	resctrl_arch_configure_kmode(&rdtgrp->kmode_cpu_mask, rdtgrp->closid,
 > +				     rdtgrp->mon.rmid, assign_mon, true);
 > +
 > +	rdtgrp->kmode = true;
 > +	resctrl_kmode_files_set_visible(rdtgrp, true);
 > +}
[ ... ]
 > +static ssize_t resctrl_kernel_mode_write(struct kernfs_open_file *of,
 > +					 char *buf, size_t nbytes, loff_t off)
 > +{
[ ... ]
 > +	if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKED ||
 > +	    rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP) {

This prevents assigning a kernel mode to a group that is already
pseudo-locked, but what prevents a group that is already bound to a 
kernel mode from entering pseudo-locksetup mode?
Looking at rdtgroup_locksetup_enter(), it checks if rdtgrp->cpu_mask is
empty but does not check rdtgrp->kmode. If the group successfully enters
pseudo-locked mode, rdtgroup_pseudo_lock_create() calls closid_free().
Could this allow the closid to be reallocated to a new group while the
hardware remains actively programmed for the kernel-mode CPUs?

-------------------------------------------------------------------
Yes. Needs a check for rdtgrp->kmode in rdtgroup_locksetup_enter().
Will add it.
---------------------------------------------------------------------


 > +		rdt_last_cmd_puts("Pseudo-locking in progress\n");
 > +		ret = -EINVAL;
 > +		goto out_unlock;
 > +	}
 > +
 > +update_mode:
 > +	/* No-op if the same mode is already active on the same group. */
 > +	if (resctrl_kcfg.kmode_cur == mode && resctrl_kcfg.k_rdtgrp == rdtgrp)
 > +		goto out_unlock;
 > +
 > +	/*
 > +	 * Tear down any active global-assign binding before applying the new
 > +	 * policy, including when only the mode changes on the same group.
 > +	 */
 > +	if (resctrl_kcfg.kmode_cur != INHERIT_CTRL_AND_MON)
 > +		rdtgroup_config_kmode_reset(resctrl_kcfg.k_rdtgrp,
 > +					    resctrl_kcfg.kmode_cur);
 > +
 > +	if (mode != INHERIT_CTRL_AND_MON)
 > +		rdtgroup_config_kmode(rdtgrp, mode);

If a monitor group is bound to a kernel mode policy here, what happens 
if it is subsequently reparented via rename()? The existing 
rdtgroup_rename() logic allows the rename if rdtgrp->cpu_mask
is empty, ignoring kmode_cpu_mask. Since mongrp_reparent() updates the
software closid but does not reprogram hardware MSRs for the CPUs in
kmode_cpu_mask, might the kernel-mode CPUs continue using the old 
parent's closid even after it is freed and reallocated?

---------------------------------------------------------------------
Yes. This needs a fix. Will add a check in rdtgroup_rename() for 
kmode_cpu_mask.

Thanks

Babu