* [PATCH 1/2] KVM: enable halt polling shrink parameter by default
2023-11-02 15:46 [PATCH 0/2] KVM: enable halt poll shrink parameter Parshuram Sangle
@ 2023-11-02 15:46 ` Parshuram Sangle
2023-11-02 15:46 ` [PATCH 2/2] KVM: documentation update to halt polling Parshuram Sangle
` (3 subsequent siblings)
4 siblings, 0 replies; 7+ messages in thread
From: Parshuram Sangle @ 2023-11-02 15:46 UTC (permalink / raw)
To: kvm, pbonzini; +Cc: linux-kernel, jaishankar.rajendran, parshuram.sangle
Default halt_poll_ns_shrink value of 0 always resets polling interval
to 0 on an un-successful poll where vcpu wakeup is not received. This is
mostly to avoid pointless polling for more number of shorter intervals. But
disabled shrink assumes vcpu wakeup is less likely to be received in
subsequent shorter polling intervals. Another side effect of 0 shrink value
is that, even on a successful poll if total block time was greater than
current polling interval, the polling interval starts over from 0 instead
of shrinking by a factor.
Enabling shrink with value of 2 allows the polling interval to gradually
decrement in case of un-successful poll events as well. This gives a fair
chance for successful polling events in subsequent polling intervals rather
than resetting it to 0 and starting over from grow_start.
Below kvm stat log snippet shows interleaved growth and shrinking of
polling interval:
87162647182125: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 10000 (grow 0)
87162647637763: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 20000 (grow 10000)
87162649627943: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 40000 (grow 20000)
87162650892407: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 20000 (shrink 40000)
87162651540378: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 40000 (grow 20000)
87162652276768: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 20000 (shrink 40000)
87162652515037: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 40000 (grow 20000)
87162653383787: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 20000 (shrink 40000)
87162653627670: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 10000 (shrink 20000)
87162653796321: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 20000 (grow 10000)
87162656171645: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 10000 (shrink 20000)
87162661607487: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 0 (shrink 10000)
Having both grow and shrink enabled creates a balance in polling interval
growth and shrink behavior. Tests show improved successful polling attempt
ratio which contribute to VM performance. Power penalty is quite negligible
as shrunk polling intervals create bursts of very short durations.
Performance assessment results show 3-6% improvements in CPU+GPU, Memory
and Storage Android VM workloads whereas 5-9% improvement in average FPS of
gaming VM workloads.
Power penalty is below 1% where host OS is either idle or running a
native workload having 2 VMs enabled. CPU/GPU intensive gaming workloads
as well do not show any increased power overhead with shrink enabled.
Co-developed-by: Rajendran Jaishankar <jaishankar.rajendran@intel.com>
Signed-off-by: Rajendran Jaishankar <jaishankar.rajendran@intel.com>
Signed-off-by: Parshuram Sangle <parshuram.sangle@intel.com>
---
Documentation/virt/kvm/halt-polling.rst | 2 +-
virt/kvm/kvm_main.c | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/Documentation/virt/kvm/halt-polling.rst b/Documentation/virt/kvm/halt-polling.rst
index c82a04b709b4..64f32a81133f 100644
--- a/Documentation/virt/kvm/halt-polling.rst
+++ b/Documentation/virt/kvm/halt-polling.rst
@@ -105,7 +105,7 @@ powerpc kvm-hv case.
| | grow_halt_poll_ns() | |
| | function. | |
+-----------------------+---------------------------+-------------------------+
-|halt_poll_ns_shrink | The value by which the | 0 |
+|halt_poll_ns_shrink | The value by which the | 2 |
| | halt polling interval is | |
| | divided in the | |
| | shrink_halt_poll_ns() | |
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 486800a7024b..cfc474558660 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -91,8 +91,8 @@ unsigned int halt_poll_ns_grow_start = 10000; /* 10us */
module_param(halt_poll_ns_grow_start, uint, 0644);
EXPORT_SYMBOL_GPL(halt_poll_ns_grow_start);
-/* Default resets per-vcpu halt_poll_ns . */
-unsigned int halt_poll_ns_shrink;
+/* Default halves per-vcpu halt_poll_ns. */
+unsigned int halt_poll_ns_shrink = 2;
module_param(halt_poll_ns_shrink, uint, 0644);
EXPORT_SYMBOL_GPL(halt_poll_ns_shrink);
--
2.17.1
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 2/2] KVM: documentation update to halt polling
2023-11-02 15:46 [PATCH 0/2] KVM: enable halt poll shrink parameter Parshuram Sangle
2023-11-02 15:46 ` [PATCH 1/2] KVM: enable halt polling shrink parameter by default Parshuram Sangle
@ 2023-11-02 15:46 ` Parshuram Sangle
2024-06-04 23:20 ` Sean Christopherson
2023-12-11 15:28 ` [PATCH 0/2] KVM: enable halt poll shrink parameter Parshuram Sangle
` (2 subsequent siblings)
4 siblings, 1 reply; 7+ messages in thread
From: Parshuram Sangle @ 2023-11-02 15:46 UTC (permalink / raw)
To: kvm, pbonzini; +Cc: linux-kernel, jaishankar.rajendran, parshuram.sangle
Corrected kvm documentation to reflect current handling of
polling interval on successful and un-successful polling events.
Also updated the description about newly added halt_poll_ns_grow_start
parameter.
Co-developed-by: Rajendran Jaishankar <jaishankar.rajendran@intel.com>
Signed-off-by: Rajendran Jaishankar <jaishankar.rajendran@intel.com>
Signed-off-by: Parshuram Sangle <parshuram.sangle@intel.com>
---
Documentation/virt/kvm/halt-polling.rst | 24 +++++++++++++-----------
1 file changed, 13 insertions(+), 11 deletions(-)
diff --git a/Documentation/virt/kvm/halt-polling.rst b/Documentation/virt/kvm/halt-polling.rst
index 64f32a81133f..cff388d9dc72 100644
--- a/Documentation/virt/kvm/halt-polling.rst
+++ b/Documentation/virt/kvm/halt-polling.rst
@@ -44,12 +44,14 @@ or in the case of powerpc kvm-hv, in the vcore struct:
Thus this is a per vcpu (or vcore) value.
-During polling if a wakeup source is received within the halt polling interval,
-the interval is left unchanged. In the event that a wakeup source isn't
-received during the polling interval (and thus schedule is invoked) there are
-two options, either the polling interval and total block time[0] were less than
-the global max polling interval (see module params below), or the total block
-time was greater than the global max polling interval.
+During polling if a wakeup source is not received within the halt polling
+interval (and thus schedule is invoked), the interval is always shrunk in
+shrink_halt_poll_ns(). In the event that a wakeup source is received during
+the polling interval, polling interval is left unchanged if total block time
+is below or equal to current interval value otherwise there are two options,
+either the polling interval and total block time[0] were less than the global
+max polling interval (see module params below), or the total block time was
+greater than the global max polling interval.
In the event that both the polling interval and total block time were less than
the global max polling interval then the polling interval can be increased in
@@ -79,11 +81,11 @@ adjustment of the polling interval.
Module Parameters
=================
-The kvm module has 3 tuneable module parameters to adjust the global max
-polling interval as well as the rate at which the polling interval is grown and
-shrunk. These variables are defined in include/linux/kvm_host.h and as module
-parameters in virt/kvm/kvm_main.c, or arch/powerpc/kvm/book3s_hv.c in the
-powerpc kvm-hv case.
+The kvm module has 4 tunable module parameters to adjust the global max
+polling interval, initial value (to grow from 0) as well as the rate at which
+the polling interval is grown and shrunk. These variables are defined in
+include/linux/kvm_host.h and as module parameters in virt/kvm/kvm_main.c, or
+arch/powerpc/kvm/book3s_hv.c in the powerpc kvm-hv case.
+-----------------------+---------------------------+-------------------------+
|Module Parameter | Description | Default Value |
--
2.17.1
^ permalink raw reply related [flat|nested] 7+ messages in thread* Re: [PATCH 2/2] KVM: documentation update to halt polling
2023-11-02 15:46 ` [PATCH 2/2] KVM: documentation update to halt polling Parshuram Sangle
@ 2024-06-04 23:20 ` Sean Christopherson
0 siblings, 0 replies; 7+ messages in thread
From: Sean Christopherson @ 2024-06-04 23:20 UTC (permalink / raw)
To: Parshuram Sangle; +Cc: kvm, pbonzini, linux-kernel, jaishankar.rajendran
On Thu, Nov 02, 2023, Parshuram Sangle wrote:
> Corrected kvm documentation to reflect current handling of
> polling interval on successful and un-successful polling events.
> Also updated the description about newly added halt_poll_ns_grow_start
> parameter.
>
> Co-developed-by: Rajendran Jaishankar <jaishankar.rajendran@intel.com>
> Signed-off-by: Rajendran Jaishankar <jaishankar.rajendran@intel.com>
> Signed-off-by: Parshuram Sangle <parshuram.sangle@intel.com>
> ---
> Documentation/virt/kvm/halt-polling.rst | 24 +++++++++++++-----------
> 1 file changed, 13 insertions(+), 11 deletions(-)
>
> diff --git a/Documentation/virt/kvm/halt-polling.rst b/Documentation/virt/kvm/halt-polling.rst
> index 64f32a81133f..cff388d9dc72 100644
> --- a/Documentation/virt/kvm/halt-polling.rst
> +++ b/Documentation/virt/kvm/halt-polling.rst
> @@ -44,12 +44,14 @@ or in the case of powerpc kvm-hv, in the vcore struct:
>
> Thus this is a per vcpu (or vcore) value.
>
> -During polling if a wakeup source is received within the halt polling interval,
> -the interval is left unchanged. In the event that a wakeup source isn't
> -received during the polling interval (and thus schedule is invoked) there are
> -two options, either the polling interval and total block time[0] were less than
> -the global max polling interval (see module params below), or the total block
> -time was greater than the global max polling interval.
> +During polling if a wakeup source is not received within the halt polling
> +interval (and thus schedule is invoked), the interval is always shrunk in
> +shrink_halt_poll_ns().
This is wrong. KVM grows the interval if the wakeup event arrives after KVM
stops polling if the total halt time is less than the max allowed interval.
The existing wording is a bit odd, and it fails to document the per-VM capability,
but I didn't find anything in the existing documentation that is incorrect (ignoring
the Module Parameters section below).
If we wanted to clean things up, I would vote for something like the below. For
this patch, I'm going to drop this update and just keep the Module Parameters fixes.
During polling if a wakeup source is received within the halt polling interval,
the interval is left unchanged (the total block time[0] is less than or equal
to the current interval). If the block time is greater than the interval,
there are two possibilities: either the polling interval and total block time
were less than the VM's max polling interval (see module params and
KVM_CAP_HALT_POLL below), or the total block time was greater than the VM's max
polling interval.
In the event that both the polling interval and total block time were less than
the max polling interval then the polling interval can be increased in the hope
that next time during the longer polling interval the wake up source will be
received while the host is polling and the latency benefits will be received.
The polling interval is grown in the function grow_halt_poll_ns() and is
multiplied by the module parameters halt_poll_ns_grow and
halt_poll_ns_grow_start.
In the event that the total block time was greater than the VM's max polling
interval then the host will never poll for long enough to wakeup during the
polling interval (KVM will always poll for less time than the block time), so
the polling internal is shrunk in order to avoid pointless polling. The polling
interval is shrunk in the function shrink_halt_poll_ns() and is divided by the
module parameter halt_poll_ns_shrink, or set to 0 iff halt_poll_ns_shrink == 0.
It is worth noting that this adjustment process attempts to hone in on some
steady state polling interval but will only really do a good job for wakeups
which come at an approximately constant rate, otherwise there will be constant
adjustment of the polling interval.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 0/2] KVM: enable halt poll shrink parameter
2023-11-02 15:46 [PATCH 0/2] KVM: enable halt poll shrink parameter Parshuram Sangle
2023-11-02 15:46 ` [PATCH 1/2] KVM: enable halt polling shrink parameter by default Parshuram Sangle
2023-11-02 15:46 ` [PATCH 2/2] KVM: documentation update to halt polling Parshuram Sangle
@ 2023-12-11 15:28 ` Parshuram Sangle
2024-05-03 21:47 ` Sean Christopherson
2024-06-04 23:29 ` Sean Christopherson
4 siblings, 0 replies; 7+ messages in thread
From: Parshuram Sangle @ 2023-12-11 15:28 UTC (permalink / raw)
To: kvm, pbonzini; +Cc: linux-kernel, jaishankar.rajendran, parshuram.sangle
Soft reminder for patch review
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 0/2] KVM: enable halt poll shrink parameter
2023-11-02 15:46 [PATCH 0/2] KVM: enable halt poll shrink parameter Parshuram Sangle
` (2 preceding siblings ...)
2023-12-11 15:28 ` [PATCH 0/2] KVM: enable halt poll shrink parameter Parshuram Sangle
@ 2024-05-03 21:47 ` Sean Christopherson
2024-06-04 23:29 ` Sean Christopherson
4 siblings, 0 replies; 7+ messages in thread
From: Sean Christopherson @ 2024-05-03 21:47 UTC (permalink / raw)
To: Parshuram Sangle; +Cc: kvm, pbonzini, linux-kernel, jaishankar.rajendran
On Thu, Nov 02, 2023, Parshuram Sangle wrote:
> KVM halt polling interval growth and shrink behavior has evolved since its
> inception. The current mechanism adjusts the polling interval based on whether
> vcpu wakeup was received or not during polling interval using grow and shrink
> parameter values. Though grow parameter is logically set to 2 by default,
> shrink parameter is kept disabled (set to 0).
>
> Disabled shrink has two issues:
> 1) Resets polling interval to 0 on every un-successful poll assuming it is
> less likely to receive a vcpu wakeup in further shrunk intervals.
> 2) Even on successful poll, if total block time is greater or equal to current
> poll_ns value, polling interval is reset to 0 instead shrinking gradually.
>
> These aspects reduce the chances receiving valid wakeup during polling and
> lose potential performance benefits for VM workloads.
>
> Below is the summary of experiments conducted to assess performance and power
> impact by enabling the halt_poll_ns_shrink parameter(value set to 2).
>
> Performance Test Summary: (Higher is better)
> --------------------------------------------
> Platform Details: Chrome Brya platform
> CPU - Alder Lake (12th Gen Intel CPU i7-1255U)
> Host kernel version - 5.15.127-20371-g710a1611ad33
>
> Android VM workload (Score) Base Shrink Enabled (value 2) Delta
> ---------------------------------------------------------------------------
> GeekBench Multi-core(CPU) 5754 5856 2%
> 3D Mark Slingshot(CPU+GPU) 15486 15885 3%
> Stream (handopt)(Memory) 20566 21594 5%
> fio seq-read (Storage) 727 747 3%
> fio seq-write (Storage) 331 343 3%
> fio rand-read (Storage) 690 732 6%
> fio rand-write (Storage) 299 300 1%
>
> Steam Gaming VM (Avg FPS) Base Shrink Enabled (value 2) Delta
> ---------------------------------------------------------------------------
> Metro Redux (OpenGL) 54.80 59.60 9%
> Dota 2 (Open GL) 48.74 51.40 5%
> Dota 2 (Vulkan) 20.80 21.10 1%
> SpaceShip (Vulkan) 20.40 21.52 6%
>
> With Shrink enabled, majority of workloads show higher % of successful polling.
> Reduced latency of returning control back to VM and avoided overhead of vm_exit
> contribute to these performance gains.
>
> Power Impact Assessment Summary: (Lower is better)
> --------------------------------------------------
> Method : DAQ measurements of CPU and Memory rails
>
> CPU+Memory (Watt) Base Shrink Enabled (value 2) Delta
> ---------------------------------------------------------------------------
> Idle* (Host) 0.636 0.631 -0.8%
> Video Playback (Host) 2.225 2.210 -0.7%
> Tomb Raider (VM) 17.261 17.175 -0.5%
> SpaceShip Benchmark(VM) 17.079 17.123 0.3%
>
> *Idle power - Idle system with no application running, Android and Borealis
> VMs enabled running no workload. Duration 180 sec.
>
> Power measurements done for Chrome idle scenario and active Gaming VM
> workload show negligible power overhead since additional polling creates
> very short duration bursts which are less likely to have gone to a
> complete idle CPU state.
>
> NOTE: No tests are conducted on non-x86 platform with this changed config
>
> The default values of grow and shrink parameters get commonly used by
> various VM deployments unless specifically tuned for performance. Hence
> referring to performance and power measurements results shown above, it is
> recommended to have shrink enabled (with value 2) by default so that there
> is no need to explicitly set this parameter through kernel cmdline or by
> other means.
I am by no means an expert on halt polling or power management, but all of this
seems like a reasonable tradeoff. And even without the numbers you provided,
starting from scratch after a single failure is rather odd.
So unless someone objects, I'll plan on applying this for 6.11 in a few weeks
(after the 6.10 merge window closes).
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH 0/2] KVM: enable halt poll shrink parameter
2023-11-02 15:46 [PATCH 0/2] KVM: enable halt poll shrink parameter Parshuram Sangle
` (3 preceding siblings ...)
2024-05-03 21:47 ` Sean Christopherson
@ 2024-06-04 23:29 ` Sean Christopherson
4 siblings, 0 replies; 7+ messages in thread
From: Sean Christopherson @ 2024-06-04 23:29 UTC (permalink / raw)
To: Sean Christopherson, kvm, pbonzini, Parshuram Sangle
Cc: linux-kernel, jaishankar.rajendran
On Thu, 02 Nov 2023 21:16:26 +0530, Parshuram Sangle wrote:
> KVM halt polling interval growth and shrink behavior has evolved since its
> inception. The current mechanism adjusts the polling interval based on whether
> vcpu wakeup was received or not during polling interval using grow and shrink
> parameter values. Though grow parameter is logically set to 2 by default,
> shrink parameter is kept disabled (set to 0).
>
> Disabled shrink has two issues:
> 1) Resets polling interval to 0 on every un-successful poll assuming it is
> less likely to receive a vcpu wakeup in further shrunk intervals.
> 2) Even on successful poll, if total block time is greater or equal to current
> poll_ns value, polling interval is reset to 0 instead shrinking gradually.
>
> [...]
Applied to kvm-x86 generic, with a reduced version of the doc update as
described in response to patch 2. Thanks!
[1/2] KVM: enable halt polling shrink parameter by default
https://github.com/kvm-x86/linux/commit/aeb1b22a3ac8
[2/2] KVM: documentation update to halt polling
https://github.com/kvm-x86/linux/commit/f8aadead1971
--
https://github.com/kvm-x86/linux/tree/next
^ permalink raw reply [flat|nested] 7+ messages in thread