From: Shrikanth Hegde <sshegde@linux.ibm.com>
To: Yury Norov <ynorov@nvidia.com>,
Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@redhat.com>, Thomas Gleixner <tglx@kernel.org>,
Juri Lelli <juri.lelli@redhat.com>,
Vincent Guittot <vincent.guittot@linaro.org>
Cc: Yury Norov <yury.norov@gmail.com>,
"Rafael J. Wysocki" <rafael@kernel.org>,
Alexander Gordeev <agordeev@linux.ibm.com>,
Andrea Righi <arighi@nvidia.com>, Ben Segall <bsegall@google.com>,
Christian Borntraeger <borntraeger@linux.ibm.com>,
Danilo Krummrich <dakr@kernel.org>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
Heiko Carstens <hca@linux.ibm.com>,
Ilya Leoshkevich <iii@linux.ibm.com>,
Johan Hovold <johan@kernel.org>, Jonathan Corbet <corbet@lwn.net>,
K Prateek Nayak <kprateek.nayak@amd.com>,
Maxim Kuvyrkov <maxim.kuvyrkov@grdncc.com>,
Mel Gorman <mgorman@suse.de>, Mete Durlu <meted@linux.ibm.com>,
Randy Dunlap <rdunlap@infradead.org>,
Rasmus Villemoes <linux@rasmusvillemoes.dk>,
Shuah Khan <skhan@linuxfoundation.org>,
Steven Rostedt <rostedt@goodmis.org>,
Sven Schnelle <svens@linux.ibm.com>,
Valentin Schneider <vschneid@redhat.com>,
Vasily Gorbik <gor@linux.ibm.com>,
linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org,
driver-core@lists.linux.dev, linux-s390@vger.kernel.org
Subject: Re: [PATCH 6/7] virt: steal_governor: restart sampling immediately after CPU hotplug
Date: Thu, 19 Nov 2026 11:36:46 +0530 [thread overview]
Message-ID: <37a30a91-bd86-4dbb-ad12-8cbb509f7c4c@linux.ibm.com> (raw)
In-Reply-To: <20261113184257.621707-7-ynorov@nvidia.com>
Hi Yury.
On 11/14/26 12:12 AM, Yury Norov wrote:
> CPU hotplug between samples can distort the calculated steal ratio because
> the governor divides accumulated steal time by the current active CPU
> count, even when that count changed during the sampling interval.
>
> For example, assume a constant 3% steal rate on each CPU, with four active
> CPUs for 900ms followed by two active CPUs for 100ms. The accumulated
> steal time is:
>
> 4 * 900ms * 3% + 2 * 100ms * 3% = 108ms + 6ms = 114ms
This example is not realistic. If you had 3% steal with 4 CPUs, and if they become 2,
your steal time will definitely increase. It will likely double. So
governor likely calculates 6%.
But yes, you can have realistic examples, where governor may calculate
the values which goes slightly above or below. But IMHO those examples are very
limited. Even in the above examples, the steal governor was supposed to trigger
high threshold since the steal time with 2 CPUs is 6%. So if not now, in the next
delta it would have triggered it anyways.
Some examples are really corner cases. Maybe like
60 CPUs 3% steal, and then 40 CPUs 5.1% steal = 4.55%
(Below threshold, but should have been high). One interval missed. But if you see
large duration was in low threshold, so it is agreeable to miss it.
>
> The elapsed time is 1000ms. Using the final count of two active CPUs, the
> governor calculates:
>
> 114ms / (2 * 1000ms) = 5.7%
>
> This exceeds the default 5% high threshold, even though the actual steal
> rate stayed at 3%. Using the CPU counts over the whole interval gives:
>
> 114ms / (4 * 900ms + 2 * 100ms) = 3%
>
> Checking num_online_cpus() doesn't account for cases where CPUs go online
> and offline such that the total number of online CPUs doesn't change. So,
> implement the sampling restart on hotplug event.
>
I am not sure if we really need to stress about hotplug cases. IIUC, when a user
does hotplug, systems are usually idle. (Atleast that is what we have observed in
real life use cases). And hotplug is really user driven, so user knows system is going
through some transitions and will be stable soon.
> Fixes: 27d47ebce4d6 ("virt/steal_governor: Implement steal_governor policy loop")
> Signed-off-by: Yury Norov <ynorov@nvidia.com>
> ---
> drivers/virt/steal_governor.c | 37 ++++++++++++++++++++++++++++++++---
> 1 file changed, 34 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/virt/steal_governor.c b/drivers/virt/steal_governor.c
> index c34d91267ec1..f71dbf27e4a9 100644
> --- a/drivers/virt/steal_governor.c
> +++ b/drivers/virt/steal_governor.c
> @@ -14,6 +14,7 @@
> #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
>
> #include <linux/cleanup.h>
> +#include <linux/cpuhotplug.h>
> #include <linux/cpuhplock.h>
> #include <linux/cpumask.h>
> #include <linux/init.h>
> @@ -42,6 +43,8 @@ struct steal_governor {
> unsigned long delay;
> unsigned int high_threshold;
> unsigned int low_threshold;
> + int hp_state;
> + bool sample_invalidated;
> struct delayed_work work;
> };
>
> @@ -217,6 +220,13 @@ static bool preferred_cpus_valid(void)
> return true;
> }
>
> +static int hotplug_callback(unsigned int cpu)
> +{
> + sg_ctx.sample_invalidated = true;
> + mod_delayed_work(system_wq, &sg_ctx.work, 0);
> + return 0;
> +}
> +
> static void steal_governor_loop(struct work_struct *work)
> {
> u64 curr_steal, delta_steal, delta_ns, steal_ratio;
> @@ -233,6 +243,11 @@ static void steal_governor_loop(struct work_struct *work)
> sg_ctx.steal = curr_steal;
> sg_ctx.time = now;
>
> + if (sg_ctx.sample_invalidated) {
> + sg_ctx.sample_invalidated = false;
> + goto requeue_work;
> + }
> +
> /*
> * steal_ratio = (delta_steal * 100*100)/(delta_ns * num_cpus())
> * To avoid possible overflow, divide the denominator early.
> @@ -254,11 +269,14 @@ static void steal_governor_loop(struct work_struct *work)
> return;
> }
>
> - schedule_delayed_work(&sg_ctx.work, sg_ctx.delay);
> +requeue_work:
> + mod_delayed_work(system_wq, &sg_ctx.work, sg_ctx.delay);
> }
>
> static int __init steal_governor_init(void)
> {
> + int ret;
> +
> #ifdef CONFIG_XEN
> if (xen_initial_domain()) {
> pr_err("Cannot load in Xen Dom0 (Host OS). Driver is for guests only.\n");
> @@ -273,8 +291,20 @@ static int __init steal_governor_init(void)
> }
>
> INIT_DELAYED_WORK(&sg_ctx.work, steal_governor_loop);
> - sg_ctx.steal = get_system_steal_time();
> - sg_ctx.time = ktime_get();
> + /* Lock to prevent unobserved hotplug event during setup. */
> + scoped_guard(cpus_read_lock) {
> + sg_ctx.steal = get_system_steal_time();
> + sg_ctx.time = ktime_get();
> + sg_ctx.sample_invalidated = false;
> + ret = cpuhp_setup_state_nocalls_cpuslocked(CPUHP_AP_ONLINE_DYN,
> + "virt/steal_governor:online",
> + hotplug_callback,
> + hotplug_callback);
> + }
> + if (ret < 0)
> + return ret;
> + sg_ctx.hp_state = ret;
> +
> schedule_delayed_work(&sg_ctx.work, sg_ctx.delay);
> pr_info("enabled. interval: %ums, high_threshold: %u, low_threshold: %u\n",
> jiffies_to_msecs(sg_ctx.delay), sg_ctx.high_threshold, sg_ctx.low_threshold);
> @@ -285,6 +315,7 @@ static int __init steal_governor_init(void)
> static void __exit steal_governor_exit(void)
> {
> disable_delayed_work_sync(&sg_ctx.work);
> + cpuhp_remove_state_nocalls(sg_ctx.hp_state);
> guard(cpus_read_lock)();
> guard(mutex)(&cpu_preferred_mutex);
>
next prev parent reply other threads:[~2026-11-19 6:07 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-13 18:42 [PATCH 0/7] Harden preferred CPUs management Yury Norov
2026-11-13 18:42 ` [PATCH 1/7] cpumask: add reset_preferred_cpumask() helper Yury Norov
2026-11-13 18:42 ` [PATCH 2/7] sched: restore preferred CPUs when hotplug removes the last one Yury Norov
2026-11-19 6:13 ` Shrikanth Hegde
2026-11-13 18:42 ` [PATCH 3/7] virt: steal_governor: remove the minimum elapsed-time check Yury Norov
2026-11-13 18:42 ` [PATCH 4/7] virt: steal_governor: serialize sampling against CPU hotplug Yury Norov
2026-11-19 6:22 ` Shrikanth Hegde
2026-11-13 18:42 ` [PATCH 5/7] virt: steal_governor: warn on unexpected preferred mask states Yury Norov
2026-11-13 18:42 ` [PATCH 6/7] virt: steal_governor: restart sampling immediately after CPU hotplug Yury Norov
2026-11-16 2:13 ` Yury Norov
2026-11-19 6:06 ` Shrikanth Hegde [this message]
2026-11-13 18:42 ` [PATCH 7/7] cpu: make preferred CPU policy writers mutually exclusive Yury Norov
2026-11-19 6:10 ` Shrikanth Hegde
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=37a30a91-bd86-4dbb-ad12-8cbb509f7c4c@linux.ibm.com \
--to=sshegde@linux.ibm.com \
--cc=agordeev@linux.ibm.com \
--cc=arighi@nvidia.com \
--cc=borntraeger@linux.ibm.com \
--cc=bsegall@google.com \
--cc=corbet@lwn.net \
--cc=dakr@kernel.org \
--cc=dietmar.eggemann@arm.com \
--cc=driver-core@lists.linux.dev \
--cc=gor@linux.ibm.com \
--cc=gregkh@linuxfoundation.org \
--cc=hca@linux.ibm.com \
--cc=iii@linux.ibm.com \
--cc=johan@kernel.org \
--cc=juri.lelli@redhat.com \
--cc=kprateek.nayak@amd.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-s390@vger.kernel.org \
--cc=linux@rasmusvillemoes.dk \
--cc=maxim.kuvyrkov@grdncc.com \
--cc=meted@linux.ibm.com \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=rafael@kernel.org \
--cc=rdunlap@infradead.org \
--cc=rostedt@goodmis.org \
--cc=skhan@linuxfoundation.org \
--cc=svens@linux.ibm.com \
--cc=tglx@kernel.org \
--cc=vincent.guittot@linaro.org \
--cc=vschneid@redhat.com \
--cc=ynorov@nvidia.com \
--cc=yury.norov@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox