From: Ben Horgan <ben.horgan@arm.com>
To: Jinjie Ruan <ruanjinjie@huawei.com>,
tony.luck@intel.com, reinette.chatre@intel.com,
Dave.Martin@arm.com, james.morse@arm.com, babu.moger@amd.com,
corbet@lwn.net, skhan@linuxfoundation.org, rdunlap@infradead.org,
bp@alien8.de, tan.shaopeng@fujitsu.com, gshan@redhat.com,
jic23@kernel.org, fenghuay@nvidia.com, zengheng4@huawei.com,
x86@kernel.org, linux-kernel@vger.kernel.org,
linux-doc@vger.kernel.org
Subject: Re: [PATCH v6 5/6] fs/resctrl: Release the limbo RMIDs when the CDP state changes
Date: Thu, 19 Nov 2026 16:49:27 +0000 [thread overview]
Message-ID: <6dd79d09-576b-4086-ba5c-7f8663292a6e@arm.com> (raw)
In-Reply-To: <20261119094858.813874-6-ruanjinjie@huawei.com>
Hi Jinjie,
On 19/11/2026 09:48, Jinjie Ruan wrote:
> On MPAM the counter of an RMID is selected by the PARTIDs of the CLOSID
> that owned it. Without CDP, CLOSID c is PARTID c; with CDP it is
> PARTIDs 2c and 2c+1. An RMID that stays in limbo across an unmount and
> a mount that changes the CDP state is read through the PARTIDs of the
> new state, which are not the PARTIDs its cache lines are tagged with.
>
> With 5 PARTIDs:
>
> mount CDP off, CLOSID c -> PARTID c
> mkdir g1..g4 CLOSIDs 1..4
> umount RMIDs enter limbo with CLOSIDs 1..4
> mount -o cdp CDP on, CLOSID c -> PARTIDs 2c, 2c+1
> limbo handler reads CLOSID 2 as PARTID 5 > PARTID_MAX 4
> MSC error interrupt, mpam_disable()
>
> The other direction is worse because nothing reports it. If resctrl is
> mounted with CDP and mounted again without it, CLOSID c is read as
> PARTID c, which is in range, so the limbo handler silently tests the
> occupancy of the PARTID of a different CLOSID and frees or keeps the
> RMIDs based on it.
>
> Where the RMID depends on the CLOSID, as on MPAM, release the RMIDs in
> limbo without reading them when a mount changes the CDP state, instead
> of restarting the limbo handler. Cache lines that still carry those
> RMIDs are accounted to the next user of each RMID. Where the RMID does
> not depend on the CLOSID, as on x86, the CDP state does not change how
> an RMID is read and the limbo handler is restarted as before.
>
> Compare the CDP state of every resource, not just the L3: on MPAM, CDP
> enabled on any resource gives every CLOSID two PARTIDs in all MSCs, so
> mounting with -o cdpl2 alone also changes how the L3 monitors read a
> CLOSID, and cache levels beyond the L3 would too.
>
> Reproduced on QEMU with the sequence above:
>
> mpam:__mpam_irq_handler: error irq from msc:1 'MSMONCFG_ID_RANGE', partid:5, pmg: 0, ris: 0
> mpam:mpam_disable: MPAM disabled due to hardware error interrupt
>
> With the change the sequence completes without an error.
>
> Fixes: 6789fb99282c ("arm_mpam: resctrl: Add CDP emulation")
> Suggested-by: Ben Horgan <ben.horgan@arm.com>
> Suggested-by: Reinette Chatre <reinette.chatre@intel.com>
> Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
> Cc: <stable@vger.kernel.org>
> ---
> v6:
> - Drop resctrl_arch_mon_cdp_enabled() and compare the CDP state of
> every resouce's cdp state, with a comment on why they are
> checked (Reinette, Ben).
> - Restrict the release to the systems where the RMID depends on the
> CLOSID, in the changelog and the comments, and keep the hardware IDs
> out of the file system comments (Reinette).
> - Reword the limbo_cdp_enabled comment (Reinette).
> - Drop the stable version range, the Fixes tag already gives it.
> v5:
> - Split the change of the limbo handler lifetime into the previous
> patch. The limbo handler now stops while resctrl is unmounted on every
> architecture, and only the release of the RMIDs depends on
> CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID.
> - Drop the cqm_work_cpu changes. resctrl_limbo_resume() cancels the
> worker before it queues it again or releases the RMIDs.
> - Move the rdt_disable_ctx() reordering to a separate patch.
> - Add resctrl_arch_mon_cdp_enabled() instead of comparing the CDP state
> of L2 in the file system.
> - Rename limbo_cdp to limbo_cdp_enabled and document that it is only set
> with CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID.
> - Check QOS_L3_OCCUP_EVENT_ID first and drop the rmid_busy_llc check.
> - Use lockdep_assert_cpus_held() at the start of resctrl_limbo_resume().
> - Describe the reverse sequence, from CDP on to CDP off.
> - Link to v4: https://lore.kernel.org/all/20261105070934.2258613-4-ruanjinjie@huawei.com/
> v4:
> - Use resctrl_mounted instead of a separate limbo_paused flag.
> - Record the CDP state once when resctrl is mounted.
> - Add resctrl_limbo_resume(), called once resctrl_mounted is set.
> - Previous Link: https://lore.kernel.org/all/20261031094616.1691026-1-ruanjinjie@huawei.com/
> ---
> fs/resctrl/monitor.c | 35 +++++++++++++++++++++++++++++++++--
> 1 file changed, 33 insertions(+), 2 deletions(-)
>
> diff --git a/fs/resctrl/monitor.c b/fs/resctrl/monitor.c
> index ede1708b920e..2c8c60138768 100644
> --- a/fs/resctrl/monitor.c
> +++ b/fs/resctrl/monitor.c
> @@ -70,6 +70,13 @@ static u32 *closid_num_dirty_rmid;
> */
> static unsigned int rmid_limbo_count;
>
> +/*
> + * @limbo_cdp_enabled - True if the RMIDs in limbo were placed there while CDP
> + * was enabled. Only used when CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID is
> + * defined.
> + */
> +static bool limbo_cdp_enabled;
> +
> /*
> * @rmid_entry - The entry in the limbo and free lists.
> */
> @@ -1044,12 +1051,15 @@ int setup_rmid_lru_list(void)
> * cqm_resume_limbo_handler() - Restart the limbo handler after a mount.
> *
> * The limbo handler stops while resctrl is unmounted. Restart it on every
> - * domain that still has RMIDs in limbo.
> + * domain that still has RMIDs in limbo. Where the RMID depends on the CLOSID
> + * and the mount changed the RMID index a CLOSID is monitored with, the RMIDs
> + * in limbo can no longer be read: release them instead.
> */
> void cqm_resume_limbo_handler(void)
> {
> struct rdt_resource *r = resctrl_arch_get_resource(RDT_RESOURCE_L3);
> struct rdt_l3_mon_domain *d;
> + bool release = false;
>
> lockdep_assert_cpus_held();
> lockdep_assert_held(&rdtgroup_mutex);
> @@ -1057,6 +1067,24 @@ void cqm_resume_limbo_handler(void)
> if (!r->mon_capable || !resctrl_is_mon_event_enabled(QOS_L3_OCCUP_EVENT_ID))
> return;
>
> + /*
> + * Where the RMID depends on the CLOSID, the RMIDs in limbo can only
> + * be read with the CDP state they were freed under. Check every
> + * resource because CDP enabled on any resource changes how every
> + * CLOSID is monitored, so CDP on another resource alone also changes
> + * how the L3 monitors are read.
> + */
> + if (IS_ENABLED(CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID)) {
> + struct rdt_resource *res;
> + bool cdp_enabled = false;
> +
> + for_each_rdt_resource(res)
> + cdp_enabled |= resctrl_arch_get_cdp_enabled(res->rid);
> +
> + release = cdp_enabled != limbo_cdp_enabled;
> + limbo_cdp_enabled = cdp_enabled;
> + }
> +
> list_for_each_entry_rcu(d, &r->mon_domains, hdr.list,
> lockdep_is_cpus_held()) {
> if (!has_busy_rmid(d))
> @@ -1067,7 +1095,10 @@ void cqm_resume_limbo_handler(void)
> * it is queued again on a CPU picked now.
> */
> cancel_delayed_work(&d->cqm_limbo);
> - cqm_setup_limbo_handler(d, 0, RESCTRL_PICK_ANY_CPU);
> + if (release)
> + release_limbo_rmids(d);
Rather than release all the rmids don't we want to check them all to see which are busy?
Thanks,
Ben
> + else
> + cqm_setup_limbo_handler(d, 0, RESCTRL_PICK_ANY_CPU);
> }
> }
>
next prev parent reply other threads:[~2026-11-19 16:49 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-19 9:48 [PATCH v6 0/6] fs/resctrl: Fix some resctl bugs related to mpam Jinjie Ruan
2026-11-19 9:48 ` [PATCH v6 1/6] fs/resctrl: Report a temporary RMID shortage per CLOSID Jinjie Ruan
2026-11-19 9:48 ` [PATCH v6 2/6] fs/resctrl: Do not allocate a monitor context when releasing limbo RMIDs Jinjie Ruan
2026-11-19 9:48 ` [PATCH v6 3/6] fs/resctrl: Clear the CDP state after the cleanup Jinjie Ruan
2026-11-19 11:21 ` Ben Horgan
2026-11-19 12:15 ` Jinjie Ruan
2026-11-19 9:48 ` [PATCH v6 4/6] fs/resctrl: Stop the limbo handler while resctrl is unmounted Jinjie Ruan
2026-11-19 12:34 ` Ben Horgan
2026-11-19 9:48 ` [PATCH v6 5/6] fs/resctrl: Release the limbo RMIDs when the CDP state changes Jinjie Ruan
2026-11-19 16:49 ` Ben Horgan [this message]
2026-11-19 9:48 ` [PATCH v6 6/6] fs/resctrl: Disallow monitor group move when the RMID index changes Jinjie Ruan
2026-11-19 16:57 ` Ben Horgan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=6dd79d09-576b-4086-ba5c-7f8663292a6e@arm.com \
--to=ben.horgan@arm.com \
--cc=Dave.Martin@arm.com \
--cc=babu.moger@amd.com \
--cc=bp@alien8.de \
--cc=corbet@lwn.net \
--cc=fenghuay@nvidia.com \
--cc=gshan@redhat.com \
--cc=james.morse@arm.com \
--cc=jic23@kernel.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=rdunlap@infradead.org \
--cc=reinette.chatre@intel.com \
--cc=ruanjinjie@huawei.com \
--cc=skhan@linuxfoundation.org \
--cc=tan.shaopeng@fujitsu.com \
--cc=tony.luck@intel.com \
--cc=x86@kernel.org \
--cc=zengheng4@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox