[PATCH 0/2] KVM: enable halt poll shrink parameter

[PATCH 0/2] KVM: enable halt poll shrink parameter

[Parshuram Sangle]

KVM halt polling interval growth and shrink behavior has evolved since its
inception. The current mechanism adjusts the polling interval based on whether
vcpu wakeup was received or not during polling interval using grow and shrink
parameter values. Though grow parameter is logically set to 2 by default,
shrink parameter is kept disabled (set to 0).

Disabled shrink has two issues:
1) Resets polling interval to 0 on every un-successful poll assuming it is
less likely to receive a vcpu wakeup in further shrunk intervals.
2) Even on successful poll, if total block time is greater or equal to current
poll_ns value, polling interval is reset to 0 instead shrinking gradually.

These aspects reduce the chances receiving valid wakeup during polling and
lose potential performance benefits for VM workloads.

Below is the summary of experiments conducted to assess performance and power
impact by enabling the halt_poll_ns_shrink parameter(value set to 2).

Performance Test Summary: (Higher is better)
--------------------------------------------
Platform Details: Chrome Brya platform
CPU - Alder Lake (12th Gen Intel CPU i7-1255U)
Host kernel version - 5.15.127-20371-g710a1611ad33

Android VM workload (Score)   Base      Shrink Enabled (value 2)    Delta
---------------------------------------------------------------------------
GeekBench Multi-core(CPU)     5754      5856                        2%
3D Mark Slingshot(CPU+GPU)    15486     15885                       3%
Stream (handopt)(Memory)      20566     21594                       5%
fio seq-read (Storage)        727       747                         3%
fio seq-write (Storage)       331       343                         3%
fio rand-read (Storage)       690       732                         6%
fio rand-write (Storage)      299       300                         1%

Steam Gaming VM (Avg FPS)     Base      Shrink Enabled (value 2)    Delta
---------------------------------------------------------------------------
Metro Redux (OpenGL)          54.80     59.60                       9%
Dota 2 (Open GL)              48.74     51.40                       5%
Dota 2 (Vulkan)               20.80     21.10                       1%
SpaceShip (Vulkan)            20.40     21.52                       6%

With Shrink enabled, majority of workloads show higher % of successful polling.
Reduced latency of returning control back to VM and avoided overhead of vm_exit
contribute to these performance gains.

Power Impact Assessment Summary: (Lower is better)
--------------------------------------------------
Method : DAQ measurements of CPU and Memory rails

CPU+Memory (Watt)             Base      Shrink Enabled (value 2)    Delta
---------------------------------------------------------------------------
Idle* (Host)                  0.636     0.631                       -0.8%
Video Playback (Host)         2.225     2.210                       -0.7%
Tomb Raider (VM)              17.261    17.175                      -0.5%
SpaceShip Benchmark(VM)       17.079    17.123                       0.3%

*Idle power - Idle system with no application running, Android and Borealis
VMs enabled running no workload. Duration 180 sec.

Power measurements done for Chrome idle scenario and active Gaming VM 
workload show negligible power overhead since additional polling creates
very short duration bursts which are less likely to have gone to a
complete idle CPU state.

NOTE: No tests are conducted on non-x86 platform with this changed config

The default values of grow and shrink parameters get commonly used by
various VM deployments unless specifically tuned for performance. Hence
referring to performance and power measurements results shown above, it is
recommended to have shrink enabled (with value 2) by default so that there
is no need to explicitly set this parameter through kernel cmdline or by
other means.

Parshuram Sangle (2):
  KVM: enable halt polling shrink parameter by default
  KVM: documentation update to halt polling

 Documentation/virt/kvm/halt-polling.rst | 26 +++++++++++++------------
 virt/kvm/kvm_main.c                     |  4 ++--
 2 files changed, 16 insertions(+), 14 deletions(-)


base-commit: 2b3f2325e71f09098723727d665e2e8003d455dc
-- 
2.17.1

[PATCH 1/2] KVM: enable halt polling shrink parameter by default

[Parshuram Sangle]

Default halt_poll_ns_shrink value of 0 always resets polling interval
to 0 on an un-successful poll where vcpu wakeup is not received. This is
mostly to avoid pointless polling for more number of shorter intervals. But
disabled shrink assumes vcpu wakeup is less likely to be received in
subsequent shorter polling intervals. Another side effect of 0 shrink value
is that, even on a successful poll if total block time was greater than
current polling interval, the polling interval starts over from 0 instead
of shrinking by a factor.

Enabling shrink with value of 2 allows the polling interval to gradually
decrement in case of un-successful poll events as well. This gives a fair
chance for successful polling events in subsequent polling intervals rather
than resetting it to 0 and starting over from grow_start.

Below kvm stat log snippet shows interleaved growth and shrinking of
polling interval:
87162647182125: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 10000 (grow 0)
87162647637763: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 20000 (grow 10000)
87162649627943: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 40000 (grow 20000)
87162650892407: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 20000 (shrink 40000)
87162651540378: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 40000 (grow 20000)
87162652276768: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 20000 (shrink 40000)
87162652515037: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 40000 (grow 20000)
87162653383787: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 20000 (shrink 40000)
87162653627670: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 10000 (shrink 20000)
87162653796321: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 20000 (grow 10000)
87162656171645: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 10000 (shrink 20000)
87162661607487: kvm_halt_poll_ns: vcpu 0: halt_poll_ns 0 (shrink 10000)

Having both grow and shrink enabled creates a balance in polling interval
growth and shrink behavior. Tests show improved successful polling attempt
ratio which contribute to VM performance. Power penalty is quite negligible
as shrunk polling intervals create bursts of very short durations.

Performance assessment results show 3-6% improvements in CPU+GPU, Memory
and Storage Android VM workloads whereas 5-9% improvement in average FPS of
gaming VM workloads.

Power penalty is below 1% where host OS is either idle or running a
native workload having 2 VMs enabled. CPU/GPU intensive gaming workloads
as well do not show any increased power overhead with shrink enabled.

Co-developed-by: Rajendran Jaishankar <jaishankar.rajendran@intel.com>
Signed-off-by: Rajendran Jaishankar <jaishankar.rajendran@intel.com>
Signed-off-by: Parshuram Sangle <parshuram.sangle@intel.com>
---
 Documentation/virt/kvm/halt-polling.rst | 2 +-
 virt/kvm/kvm_main.c                     | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/Documentation/virt/kvm/halt-polling.rst b/Documentation/virt/kvm/halt-polling.rst
index c82a04b709b4..64f32a81133f 100644
--- a/Documentation/virt/kvm/halt-polling.rst
+++ b/Documentation/virt/kvm/halt-polling.rst
@@ -105,7 +105,7 @@ powerpc kvm-hv case.
 |			| grow_halt_poll_ns()	    |			      |
 |			| function.		    |			      |
 +-----------------------+---------------------------+-------------------------+
-|halt_poll_ns_shrink	| The value by which the    | 0			      |
+|halt_poll_ns_shrink	| The value by which the    | 2			      |
 |			| halt polling interval is  |			      |
 |			| divided in the	    |			      |
 |			| shrink_halt_poll_ns()	    |			      |
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 486800a7024b..cfc474558660 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -91,8 +91,8 @@ unsigned int halt_poll_ns_grow_start = 10000; /* 10us */
 module_param(halt_poll_ns_grow_start, uint, 0644);
 EXPORT_SYMBOL_GPL(halt_poll_ns_grow_start);
 
-/* Default resets per-vcpu halt_poll_ns . */
-unsigned int halt_poll_ns_shrink;
+/* Default halves per-vcpu halt_poll_ns. */
+unsigned int halt_poll_ns_shrink = 2;
 module_param(halt_poll_ns_shrink, uint, 0644);
 EXPORT_SYMBOL_GPL(halt_poll_ns_shrink);
 
-- 
2.17.1

[PATCH 2/2] KVM: documentation update to halt polling

[Parshuram Sangle]

Corrected kvm documentation to reflect current handling of
polling interval on successful and un-successful polling events.
Also updated the description about newly added halt_poll_ns_grow_start
parameter.

Co-developed-by: Rajendran Jaishankar <jaishankar.rajendran@intel.com>
Signed-off-by: Rajendran Jaishankar <jaishankar.rajendran@intel.com>
Signed-off-by: Parshuram Sangle <parshuram.sangle@intel.com>
---
 Documentation/virt/kvm/halt-polling.rst | 24 +++++++++++++-----------
 1 file changed, 13 insertions(+), 11 deletions(-)

diff --git a/Documentation/virt/kvm/halt-polling.rst b/Documentation/virt/kvm/halt-polling.rst
index 64f32a81133f..cff388d9dc72 100644
--- a/Documentation/virt/kvm/halt-polling.rst
+++ b/Documentation/virt/kvm/halt-polling.rst
@@ -44,12 +44,14 @@ or in the case of powerpc kvm-hv, in the vcore struct:
 
 Thus this is a per vcpu (or vcore) value.
 
-During polling if a wakeup source is received within the halt polling interval,
-the interval is left unchanged. In the event that a wakeup source isn't
-received during the polling interval (and thus schedule is invoked) there are
-two options, either the polling interval and total block time[0] were less than
-the global max polling interval (see module params below), or the total block
-time was greater than the global max polling interval.
+During polling if a wakeup source is not received within the halt polling
+interval (and thus schedule is invoked), the interval is always shrunk in
+shrink_halt_poll_ns(). In the event that a wakeup source is received during
+the polling interval, polling interval is left unchanged if total block time
+is below or equal to current interval value otherwise there are two options,
+either the polling interval and total block time[0] were less than the global
+max polling interval (see module params below), or the total block time was
+greater than the global max polling interval.
 
 In the event that both the polling interval and total block time were less than
 the global max polling interval then the polling interval can be increased in
@@ -79,11 +81,11 @@ adjustment of the polling interval.
 Module Parameters
 =================
 
-The kvm module has 3 tuneable module parameters to adjust the global max
-polling interval as well as the rate at which the polling interval is grown and
-shrunk. These variables are defined in include/linux/kvm_host.h and as module
-parameters in virt/kvm/kvm_main.c, or arch/powerpc/kvm/book3s_hv.c in the
-powerpc kvm-hv case.
+The kvm module has 4 tunable module parameters to adjust the global max
+polling interval, initial value (to grow from 0) as well as the rate at which
+the polling interval is grown and shrunk. These variables are defined in
+include/linux/kvm_host.h and as module parameters in virt/kvm/kvm_main.c, or
+arch/powerpc/kvm/book3s_hv.c in the powerpc kvm-hv case.
 
 +-----------------------+---------------------------+-------------------------+
 |Module Parameter	|   Description		    |	     Default Value    |
-- 
2.17.1

Re: [PATCH 0/2] KVM: enable halt poll shrink parameter

[Parshuram Sangle]

Soft reminder for patch review

Re: [PATCH 0/2] KVM: enable halt poll shrink parameter

[Sean Christopherson]

On Thu, Nov 02, 2023, Parshuram Sangle wrote:
> KVM halt polling interval growth and shrink behavior has evolved since its
> inception. The current mechanism adjusts the polling interval based on whether
> vcpu wakeup was received or not during polling interval using grow and shrink
> parameter values. Though grow parameter is logically set to 2 by default,
> shrink parameter is kept disabled (set to 0).
> 
> Disabled shrink has two issues:
> 1) Resets polling interval to 0 on every un-successful poll assuming it is
> less likely to receive a vcpu wakeup in further shrunk intervals.
> 2) Even on successful poll, if total block time is greater or equal to current
> poll_ns value, polling interval is reset to 0 instead shrinking gradually.
> 
> These aspects reduce the chances receiving valid wakeup during polling and
> lose potential performance benefits for VM workloads.
> 
> Below is the summary of experiments conducted to assess performance and power
> impact by enabling the halt_poll_ns_shrink parameter(value set to 2).
> 
> Performance Test Summary: (Higher is better)
> --------------------------------------------
> Platform Details: Chrome Brya platform
> CPU - Alder Lake (12th Gen Intel CPU i7-1255U)
> Host kernel version - 5.15.127-20371-g710a1611ad33
> 
> Android VM workload (Score)   Base      Shrink Enabled (value 2)    Delta
> ---------------------------------------------------------------------------
> GeekBench Multi-core(CPU)     5754      5856                        2%
> 3D Mark Slingshot(CPU+GPU)    15486     15885                       3%
> Stream (handopt)(Memory)      20566     21594                       5%
> fio seq-read (Storage)        727       747                         3%
> fio seq-write (Storage)       331       343                         3%
> fio rand-read (Storage)       690       732                         6%
> fio rand-write (Storage)      299       300                         1%
> 
> Steam Gaming VM (Avg FPS)     Base      Shrink Enabled (value 2)    Delta
> ---------------------------------------------------------------------------
> Metro Redux (OpenGL)          54.80     59.60                       9%
> Dota 2 (Open GL)              48.74     51.40                       5%
> Dota 2 (Vulkan)               20.80     21.10                       1%
> SpaceShip (Vulkan)            20.40     21.52                       6%
> 
> With Shrink enabled, majority of workloads show higher % of successful polling.
> Reduced latency of returning control back to VM and avoided overhead of vm_exit
> contribute to these performance gains.
> 
> Power Impact Assessment Summary: (Lower is better)
> --------------------------------------------------
> Method : DAQ measurements of CPU and Memory rails
> 
> CPU+Memory (Watt)             Base      Shrink Enabled (value 2)    Delta
> ---------------------------------------------------------------------------
> Idle* (Host)                  0.636     0.631                       -0.8%
> Video Playback (Host)         2.225     2.210                       -0.7%
> Tomb Raider (VM)              17.261    17.175                      -0.5%
> SpaceShip Benchmark(VM)       17.079    17.123                       0.3%
> 
> *Idle power - Idle system with no application running, Android and Borealis
> VMs enabled running no workload. Duration 180 sec.
> 
> Power measurements done for Chrome idle scenario and active Gaming VM 
> workload show negligible power overhead since additional polling creates
> very short duration bursts which are less likely to have gone to a
> complete idle CPU state.
> 
> NOTE: No tests are conducted on non-x86 platform with this changed config
> 
> The default values of grow and shrink parameters get commonly used by
> various VM deployments unless specifically tuned for performance. Hence
> referring to performance and power measurements results shown above, it is
> recommended to have shrink enabled (with value 2) by default so that there
> is no need to explicitly set this parameter through kernel cmdline or by
> other means.

I am by no means an expert on halt polling or power management, but all of this
seems like a reasonable tradeoff.  And even without the numbers you provided,
starting from scratch after a single failure is rather odd.

So unless someone objects, I'll plan on applying this for 6.11 in a few weeks
(after the 6.10 merge window closes).

Re: [PATCH 2/2] KVM: documentation update to halt polling

[Sean Christopherson]

On Thu, Nov 02, 2023, Parshuram Sangle wrote:
> Corrected kvm documentation to reflect current handling of
> polling interval on successful and un-successful polling events.
> Also updated the description about newly added halt_poll_ns_grow_start
> parameter.
> 
> Co-developed-by: Rajendran Jaishankar <jaishankar.rajendran@intel.com>
> Signed-off-by: Rajendran Jaishankar <jaishankar.rajendran@intel.com>
> Signed-off-by: Parshuram Sangle <parshuram.sangle@intel.com>
> ---
>  Documentation/virt/kvm/halt-polling.rst | 24 +++++++++++++-----------
>  1 file changed, 13 insertions(+), 11 deletions(-)
> 
> diff --git a/Documentation/virt/kvm/halt-polling.rst b/Documentation/virt/kvm/halt-polling.rst
> index 64f32a81133f..cff388d9dc72 100644
> --- a/Documentation/virt/kvm/halt-polling.rst
> +++ b/Documentation/virt/kvm/halt-polling.rst
> @@ -44,12 +44,14 @@ or in the case of powerpc kvm-hv, in the vcore struct:
>  
>  Thus this is a per vcpu (or vcore) value.
>  
> -During polling if a wakeup source is received within the halt polling interval,
> -the interval is left unchanged. In the event that a wakeup source isn't
> -received during the polling interval (and thus schedule is invoked) there are
> -two options, either the polling interval and total block time[0] were less than
> -the global max polling interval (see module params below), or the total block
> -time was greater than the global max polling interval.
> +During polling if a wakeup source is not received within the halt polling
> +interval (and thus schedule is invoked), the interval is always shrunk in
> +shrink_halt_poll_ns().

This is wrong.  KVM grows the interval if the wakeup event arrives after KVM
stops polling if the total halt time is less than the max allowed interval.

The existing wording is a bit odd, and it fails to document the per-VM capability,
but I didn't find anything in the existing documentation that is incorrect (ignoring
the Module Parameters section below).

If we wanted to clean things up, I would vote for something like the below.  For
this patch, I'm going to drop this update and just keep the Module Parameters fixes.

  During polling if a wakeup source is received within the halt polling interval,
  the interval is left unchanged (the total block time[0] is less than or equal
  to the current interval).  If the block time is greater than the interval,
  there are two possibilities: either the polling interval and total block time
  were less than the VM's max polling interval (see module params and
  KVM_CAP_HALT_POLL below), or the total block time was greater than the VM's max
  polling interval.
  
  In the event that both the polling interval and total block time were less than
  the max polling interval then the polling interval can be increased in the hope
  that next time during the longer polling interval the wake up source will be
  received while the host is polling and the latency benefits will be received.
  The polling interval is grown in the function grow_halt_poll_ns() and is
  multiplied by the module parameters halt_poll_ns_grow and
  halt_poll_ns_grow_start.
  
  In the event that the total block time was greater than the VM's max polling
  interval then the host will never poll for long enough to wakeup during the
  polling interval (KVM will always poll for less time than the block time), so
  the polling internal is shrunk in order to avoid pointless polling. The polling
  interval is shrunk in the function shrink_halt_poll_ns() and is divided by the
  module parameter halt_poll_ns_shrink, or set to 0 iff halt_poll_ns_shrink == 0.
  
  It is worth noting that this adjustment process attempts to hone in on some
  steady state polling interval but will only really do a good job for wakeups
  which come at an approximately constant rate, otherwise there will be constant
  adjustment of the polling interval.

Re: [PATCH 0/2] KVM: enable halt poll shrink parameter

[Sean Christopherson]

On Thu, 02 Nov 2023 21:16:26 +0530, Parshuram Sangle wrote:
> KVM halt polling interval growth and shrink behavior has evolved since its
> inception. The current mechanism adjusts the polling interval based on whether
> vcpu wakeup was received or not during polling interval using grow and shrink
> parameter values. Though grow parameter is logically set to 2 by default,
> shrink parameter is kept disabled (set to 0).
> 
> Disabled shrink has two issues:
> 1) Resets polling interval to 0 on every un-successful poll assuming it is
> less likely to receive a vcpu wakeup in further shrunk intervals.
> 2) Even on successful poll, if total block time is greater or equal to current
> poll_ns value, polling interval is reset to 0 instead shrinking gradually.
> 
> [...]

Applied to kvm-x86 generic, with a reduced version of the doc update as
described in response to patch 2.  Thanks!

[1/2] KVM: enable halt polling shrink parameter by default
      https://github.com/kvm-x86/linux/commit/aeb1b22a3ac8
[2/2] KVM: documentation update to halt polling
      https://github.com/kvm-x86/linux/commit/f8aadead1971

--
https://github.com/kvm-x86/linux/tree/next