[RFC PATCH 0/3] Avoid soft lockup message when KVM is stopped by host

[RFC PATCH 0/3] Avoid soft lockup message when KVM is stopped by host

[Eric B Munson]

This set is just a rough first pass at avoiding soft lockup warnings when a host
pauses the execution of a guest.  A flag is set by the host in the shared page
used for the pvclock when the host goes to stop the guest.  When the guest
resumes and detects a soft lockup, this flag is checked and cleared and the soft
lockup message is skipped.

This currently breaks the build for non-x86 architectures but part of what I am
looking for here is how to go about adding the function stubs for everything
else.

Eric B Munson (3):
  Add flag to indicate that a vm was stopped by the host
  Add functions to check if the host has stopped the vm
  Add check for suspended vm in softlockup detector

 arch/x86/include/asm/pvclock-abi.h |    1 +
 arch/x86/include/asm/pvclock.h     |    3 +++
 arch/x86/kernel/kvmclock.c         |   12 ++++++++++++
 kernel/watchdog.c                  |   11 +++++++++++
 4 files changed, 27 insertions(+), 0 deletions(-)

-- 
1.7.4.1

[RFC PATCH 1/3] Add flag to indicate that a vm was stopped by the host

[Eric B Munson]

This flag will be used to check if the vm was stopped by the host when a soft
lockup was detected.

Signed-off-by: Eric B Munson <emunson@mgebm.net>
---
 arch/x86/include/asm/pvclock-abi.h |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/arch/x86/include/asm/pvclock-abi.h b/arch/x86/include/asm/pvclock-abi.h
index 35f2d19..10e738a 100644
--- a/arch/x86/include/asm/pvclock-abi.h
+++ b/arch/x86/include/asm/pvclock-abi.h
@@ -40,5 +40,6 @@ struct pvclock_wall_clock {
 } __attribute__((__packed__));
 
 #define PVCLOCK_TSC_STABLE_BIT	(1 << 0)
+#define PVCLOCK_STOPPED		(1 << 1)
 #endif /* __ASSEMBLY__ */
 #endif /* _ASM_X86_PVCLOCK_ABI_H */
-- 
1.7.4.1

[RFC PATCH 2/3] Add functions to check if the host has stopped the vm

[Eric B Munson]

When a host stops or suspends a VM it will set a flag to show this.  The
watchdog will use these functions to determine if a softlockup is real, or the
result of a suspended VM.

Signed-off-by: Eric B Munson <emunson@mgebm.net>
---
 arch/x86/include/asm/pvclock.h |    3 +++
 arch/x86/kernel/kvmclock.c     |   12 ++++++++++++
 2 files changed, 15 insertions(+), 0 deletions(-)

diff --git a/arch/x86/include/asm/pvclock.h b/arch/x86/include/asm/pvclock.h
index a518c0a..a9e9b4f 100644
--- a/arch/x86/include/asm/pvclock.h
+++ b/arch/x86/include/asm/pvclock.h
@@ -13,6 +13,9 @@ void pvclock_read_wallclock(struct pvclock_wall_clock *wall,
 			    struct timespec *ts);
 void pvclock_resume(void);
 
+bool kvm_check_host_stopped(void);
+void kvm_clear_host_stopped(void);
+
 /*
  * Scale a 64-bit delta by scaling and multiplying by a 32-bit fraction,
  * yielding a 64-bit result.
diff --git a/arch/x86/kernel/kvmclock.c b/arch/x86/kernel/kvmclock.c
index c1a0188..5a42582 100644
--- a/arch/x86/kernel/kvmclock.c
+++ b/arch/x86/kernel/kvmclock.c
@@ -113,6 +113,18 @@ static void kvm_get_preset_lpj(void)
 	preset_lpj = lpj;
 }
 
+bool kvm_check_host_stopped()
+{
+	struct pvclock_vcpu_time_info *src = &per_cpu(hv_clock, 0);
+	return (src->flags & PVCLOCK_STOPPED) != 0;
+}
+
+void kvm_clear_host_stopped()
+{
+	struct pvclock_vcpu_time_info *src = &per_cpu(hv_clock, 0);
+	src->flags = src->flags & (~PVCLOCK_STOPPED);
+}
+
 static struct clocksource kvm_clock = {
 	.name = "kvm-clock",
 	.read = kvm_clock_get_cycles,
-- 
1.7.4.1

[RFC PATCH 3/3] Add check for suspended vm in softlockup detector

[Eric B Munson]

A suspended VM can cause spurious soft lockup warnings.  To avoid these, the
watchdog now checks if the kernel knows it was stopped by the host and skips
the warning if so.

Signed-off-by: Eric B Munson <emunson@mgebm.net>
---
 kernel/watchdog.c |   11 +++++++++++
 1 files changed, 11 insertions(+), 0 deletions(-)

diff --git a/kernel/watchdog.c b/kernel/watchdog.c
index 36491cd..9cd03b3 100644
--- a/kernel/watchdog.c
+++ b/kernel/watchdog.c
@@ -25,6 +25,7 @@
 #include <linux/sysctl.h>
 
 #include <asm/irq_regs.h>
+#include <asm/pvclock.h>
 #include <linux/perf_event.h>
 
 int watchdog_enabled = 1;
@@ -292,6 +293,16 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
 	 */
 	duration = is_softlockup(touch_ts);
 	if (unlikely(duration)) {
+		/*
+		 * If a virtual machine is stopped by the host it can look to
+		 * the watchdog like a soft lockup, check to see if the host
+		 * stopped the vm before we issue the warning
+		 */
+		if (kvm_check_host_stopped()) {
+			kvm_clear_host_stopped();
+			return HRTIMER_RESTART;
+		}
+
 		/* only warn once */
 		if (__this_cpu_read(soft_watchdog_warn) == true)
 			return HRTIMER_RESTART;
-- 
1.7.4.1

Re: [RFC PATCH 0/3] Avoid soft lockup message when KVM is stopped by host

[Dor Laor]

On 08/03/2011 05:24 PM, Eric B Munson wrote:
> This set is just a rough first pass at avoiding soft lockup warnings when a host
> pauses the execution of a guest.  A flag is set by the host in the shared page
> used for the pvclock when the host goes to stop the guest.  When the guest
> resumes and detects a soft lockup, this flag is checked and cleared and the soft
> lockup message is skipped.

While this will cover the case were the host stops a guest, there will 
be other plain cases where the host is just over committed and will 
cause a softlockup false positive on the guest.

Softlockup should use stolen time that makes use of the guest running 
info would cover both cases

>
> This currently breaks the build for non-x86 architectures but part of what I am
> looking for here is how to go about adding the function stubs for everything
> else.
>
> Eric B Munson (3):
>    Add flag to indicate that a vm was stopped by the host
>    Add functions to check if the host has stopped the vm
>    Add check for suspended vm in softlockup detector
>
>   arch/x86/include/asm/pvclock-abi.h |    1 +
>   arch/x86/include/asm/pvclock.h     |    3 +++
>   arch/x86/kernel/kvmclock.c         |   12 ++++++++++++
>   kernel/watchdog.c                  |   11 +++++++++++
>   4 files changed, 27 insertions(+), 0 deletions(-)
>

Re: [RFC PATCH 0/3] Avoid soft lockup message when KVM is stopped by host

[Marcelo Tosatti]

On Wed, Aug 03, 2011 at 10:24:58AM -0400, Eric B Munson wrote:
> This set is just a rough first pass at avoiding soft lockup warnings when a host
> pauses the execution of a guest.  A flag is set by the host in the shared page
> used for the pvclock when the host goes to stop the guest.  When the guest
> resumes and detects a soft lockup, this flag is checked and cleared and the soft
> lockup message is skipped.
> 
> This currently breaks the build for non-x86 architectures but part of what I am
> looking for here is how to go about adding the function stubs for everything
> else.

To fix this, you can save/restore the offset added to kvmclock on
vmstop/vmstart. See hw/kvmclock.c in the qemu source tree.

Re: [RFC PATCH 0/3] Avoid soft lockup message when KVM is stopped by host

[Anthony Liguori]

On 08/04/2011 03:37 AM, Dor Laor wrote:
> On 08/03/2011 05:24 PM, Eric B Munson wrote:
>> This set is just a rough first pass at avoiding soft lockup warnings
>> when a host
>> pauses the execution of a guest. A flag is set by the host in the
>> shared page
>> used for the pvclock when the host goes to stop the guest. When the guest
>> resumes and detects a soft lockup, this flag is checked and cleared
>> and the soft
>> lockup message is skipped.
>
> While this will cover the case were the host stops a guest, there will
> be other plain cases where the host is just over committed and will
> cause a softlockup false positive on the guest.
>
> Softlockup should use stolen time that makes use of the guest running
> info would cover both cases

At least in the current steal time implementation, there are numerous 
cases where steal time is not accounted but you'd hit a soft lockup.

Pausing an idle guest via (qemu) stop is an example.  Likewise, a guest 
that is descheduled while idle but then not scheduled for prolonged 
periods of time would also not be accounted as steal time.

Regards,

Anthony Liguori