[Qemu-devel] [PATCH v2 0/2] savevm odness related to kvmclock

[Qemu-devel] [PATCH v2 0/2] savevm odness related to kvmclock

[Glauber Costa]

Some users told me that savevm path is behaving oddly wrt kvmclock.
The first oddness is that a guarantee we never made (AFAIK) is being broken:
two consecutive "savevm" operations, with the machine stopped in between
produces different results, due to the call to KVM_GET_CLOCK ioctl.
I believe the assumption that if the vm does not run, its saveable
state won't change is fairly reasonable. Maybe we should formally
guarantee that?

Also, this patch deals with the fact that this happens even if
kvmclock is disabled in cpuid: its savevm section is registered
nevertheless. Here, I try to register it only if it's enabled at
machine start.

v2: improvements suggested by Paolo, and patch reordering.

Glauber Costa (2):
  Do not register kvmclock savevm section if kvmclock is disabled.
  make kvmclock value idempotent for stopped machine

 cpus.c            |    3 +++
 qemu-kvm-x86.c    |   23 +++++++++++++++--------
 qemu-kvm.h        |    3 +++
 target-i386/kvm.c |    7 +++++++
 4 files changed, 28 insertions(+), 8 deletions(-)

-- 
1.7.2.3

[Qemu-devel] [PATCH v2 1/2] Do not register kvmclock savevm section if kvmclock is disabled.

[Glauber Costa]

Usually nobody usually thinks about that scenario (me included and specially),
but kvmclock can be actually disabled in the host.

It happens in two scenarios:
 1. host too old.
 2. we passed -kvmclock to our -cpu parameter.

In both cases, we should not register kvmclock savevm section. This patch
achives that by registering this section only if kvmclock is actually
currently enabled in cpuid.

The only caveat is that we have to register the savevm section a little bit
later, since we won't know the final kvmclock state before cpuid gets parsed.

Signed-off-by: Glauber Costa <glommer@redhat.com>
---
 cpus.c            |    3 +++
 qemu-kvm-x86.c    |   19 +++++++++++++------
 qemu-kvm.h        |    3 +++
 target-i386/kvm.c |    7 +++++++
 4 files changed, 26 insertions(+), 6 deletions(-)

diff --git a/cpus.c b/cpus.c
index a55c330..a24098e 100644
--- a/cpus.c
+++ b/cpus.c
@@ -97,6 +97,9 @@ void cpu_synchronize_all_post_init(void)
     for (cpu = first_cpu; cpu; cpu = cpu->next_cpu) {
         cpu_synchronize_post_init(cpu);
     }
+    if (kvm_enabled()) {
+        kvmclock_register_savevm();
+    }
 }
 
 int cpu_is_stopped(CPUState *env)
diff --git a/qemu-kvm-x86.c b/qemu-kvm-x86.c
index 20b7d6d..14a52ce 100644
--- a/qemu-kvm-x86.c
+++ b/qemu-kvm-x86.c
@@ -504,6 +504,9 @@ static void kvmclock_pre_save(void *opaque)
 {
     struct kvm_clock_data *cl = opaque;
 
+    if (!kvmclock_enabled)
+        return;
+
     kvm_vm_ioctl(kvm_state, KVM_GET_CLOCK, cl);
 }
 
@@ -528,6 +531,16 @@ static const VMStateDescription vmstate_kvmclock= {
 };
 #endif
 
+/* This has to happen after vcpu setup*/
+void kvmclock_register_savevm(void)
+{
+#ifdef KVM_CAP_ADJUST_CLOCK
+    if (kvmclock_enabled && kvm_check_extension(kvm_state, KVM_CAP_ADJUST_CLOCK)) {
+        vmstate_register(NULL, 0, &vmstate_kvmclock, &kvmclock_data);
+    }
+#endif
+}
+
 int kvm_arch_qemu_create_context(void)
 {
     int r;
@@ -545,12 +558,6 @@ int kvm_arch_qemu_create_context(void)
         return -1;
     }
 
-#ifdef KVM_CAP_ADJUST_CLOCK
-    if (kvm_check_extension(kvm_state, KVM_CAP_ADJUST_CLOCK)) {
-        vmstate_register(NULL, 0, &vmstate_kvmclock, &kvmclock_data);
-    }
-#endif
-
     r = kvm_set_boot_cpu_id(0);
     if (r < 0 && r != -ENOSYS) {
         return r;
diff --git a/qemu-kvm.h b/qemu-kvm.h
index 0f3fb50..0a104ef 100644
--- a/qemu-kvm.h
+++ b/qemu-kvm.h
@@ -752,6 +752,9 @@ int handle_tpr_access(void *opaque, CPUState *env, uint64_t rip,
 #define qemu_kvm_cpu_stop(env) do {} while(0)
 #endif
 
+extern int kvmclock_enabled;
+void kvmclock_register_savevm(void);
+
 #ifdef CONFIG_KVM
 
 typedef struct KVMSlot {
diff --git a/target-i386/kvm.c b/target-i386/kvm.c
index 95e5d02..5443765 100644
--- a/target-i386/kvm.c
+++ b/target-i386/kvm.c
@@ -293,6 +293,7 @@ void kvm_inject_x86_mce(CPUState *cenv, int bank, uint64_t status,
 }
 
 static int _kvm_arch_init_vcpu(CPUState *env);
+int kvmclock_enabled = 1;
 
 int kvm_arch_init_vcpu(CPUState *env)
 {
@@ -350,6 +351,12 @@ int kvm_arch_init_vcpu(CPUState *env)
     memset(c, 0, sizeof(*c));
     c->function = KVM_CPUID_FEATURES;
     c->eax = env->cpuid_kvm_features & get_para_features(env);
+
+    if (!(c->eax & (1 << KVM_FEATURE_CLOCKSOURCE))) {
+        /* In theory cpuid is per-cpu, and this is a global variable,
+         * but we don't expect kvmclock enabled in some cpus only */
+        kvmclock_enabled = 0;
+    }
 #endif
 
     cpu_x86_cpuid(env, 0, 0, &limit, &unused, &unused, &unused);
-- 
1.7.2.3

[Qemu-devel] Re: [PATCH v2 1/2] Do not register kvmclock savevm section if kvmclock is disabled.

[Marcelo Tosatti]

On Mon, Dec 06, 2010 at 09:03:46AM -0500, Glauber Costa wrote:
> Usually nobody usually thinks about that scenario (me included and specially),
> but kvmclock can be actually disabled in the host.
> 
> It happens in two scenarios:
>  1. host too old.
>  2. we passed -kvmclock to our -cpu parameter.
> 
> In both cases, we should not register kvmclock savevm section. This patch
> achives that by registering this section only if kvmclock is actually
> currently enabled in cpuid.
> 
> The only caveat is that we have to register the savevm section a little bit
> later, since we won't know the final kvmclock state before cpuid gets parsed.

What is the problem of registering the section? Restoring the value if
the host does not support it returns an error?

Can't you ignore the error if kvmclock is not reported in cpuid, in the
restore handler?

Also, please generate against uq/master.

[Qemu-devel] Re: [PATCH v2 1/2] Do not register kvmclock savevm section if kvmclock is disabled.

[Marcelo Tosatti]

On Mon, Dec 06, 2010 at 07:04:01PM -0200, Marcelo Tosatti wrote:
> On Mon, Dec 06, 2010 at 09:03:46AM -0500, Glauber Costa wrote:
> > Usually nobody usually thinks about that scenario (me included and specially),
> > but kvmclock can be actually disabled in the host.
> > 
> > It happens in two scenarios:
> >  1. host too old.
> >  2. we passed -kvmclock to our -cpu parameter.
> > 
> > In both cases, we should not register kvmclock savevm section. This patch
> > achives that by registering this section only if kvmclock is actually
> > currently enabled in cpuid.
> > 
> > The only caveat is that we have to register the savevm section a little bit
> > later, since we won't know the final kvmclock state before cpuid gets parsed.
> 
> What is the problem of registering the section? Restoring the value if
> the host does not support it returns an error?
> 
> Can't you ignore the error if kvmclock is not reported in cpuid, in the
> restore handler?
> 
> Also, please generate against uq/master.

Sorry, this is not upstream yet, so ignore the uq/master part.

[Qemu-devel] Re: [PATCH v2 1/2] Do not register kvmclock savevm section if kvmclock is disabled.

[Glauber Costa]

On Mon, 2010-12-06 at 19:04 -0200, Marcelo Tosatti wrote:
> On Mon, Dec 06, 2010 at 09:03:46AM -0500, Glauber Costa wrote:
> > Usually nobody usually thinks about that scenario (me included and specially),
> > but kvmclock can be actually disabled in the host.
> > 
> > It happens in two scenarios:
> >  1. host too old.
> >  2. we passed -kvmclock to our -cpu parameter.
> > 
> > In both cases, we should not register kvmclock savevm section. This patch
> > achives that by registering this section only if kvmclock is actually
> > currently enabled in cpuid.
> > 
> > The only caveat is that we have to register the savevm section a little bit
> > later, since we won't know the final kvmclock state before cpuid gets parsed.
> 
> What is the problem of registering the section? Restoring the value if
> the host does not support it returns an error?
> 
> Can't you ignore the error if kvmclock is not reported in cpuid, in the
> restore handler?

We can change the restore handler, but not the restore handler of
binaries that are already out there. The motivation here is precisely to
address migration to hosts without kvmclock, so it's better to have
a way to disable, than to count on the fact that the other side will be
able to ignore it.

[Qemu-devel] Re: [PATCH v2 1/2] Do not register kvmclock savevm section if kvmclock is disabled.

[Marcelo Tosatti]

On Tue, Dec 07, 2010 at 03:12:36PM -0200, Glauber Costa wrote:
> On Mon, 2010-12-06 at 19:04 -0200, Marcelo Tosatti wrote:
> > On Mon, Dec 06, 2010 at 09:03:46AM -0500, Glauber Costa wrote:
> > > Usually nobody usually thinks about that scenario (me included and specially),
> > > but kvmclock can be actually disabled in the host.
> > > 
> > > It happens in two scenarios:
> > >  1. host too old.
> > >  2. we passed -kvmclock to our -cpu parameter.
> > > 
> > > In both cases, we should not register kvmclock savevm section. This patch
> > > achives that by registering this section only if kvmclock is actually
> > > currently enabled in cpuid.
> > > 
> > > The only caveat is that we have to register the savevm section a little bit
> > > later, since we won't know the final kvmclock state before cpuid gets parsed.
> > 
> > What is the problem of registering the section? Restoring the value if
> > the host does not support it returns an error?
> > 
> > Can't you ignore the error if kvmclock is not reported in cpuid, in the
> > restore handler?
> 
> We can change the restore handler, but not the restore handler of
> binaries that are already out there. The motivation here is precisely to
> address migration to hosts without kvmclock, so it's better to have
> a way to disable, than to count on the fact that the other side will be
> able to ignore it.

OK. Can't you register conditionally on kvmclock cpuid bit at the end of
kvm_arch_init_vcpu, in target-i386/kvm.c?

[Qemu-devel] Re: [PATCH v2 1/2] Do not register kvmclock savevm section if kvmclock is disabled.

[Glauber Costa]

On Wed, 2010-12-08 at 17:31 -0200, Marcelo Tosatti wrote:
> On Tue, Dec 07, 2010 at 03:12:36PM -0200, Glauber Costa wrote:
> > On Mon, 2010-12-06 at 19:04 -0200, Marcelo Tosatti wrote:
> > > On Mon, Dec 06, 2010 at 09:03:46AM -0500, Glauber Costa wrote:
> > > > Usually nobody usually thinks about that scenario (me included and specially),
> > > > but kvmclock can be actually disabled in the host.
> > > > 
> > > > It happens in two scenarios:
> > > >  1. host too old.
> > > >  2. we passed -kvmclock to our -cpu parameter.
> > > > 
> > > > In both cases, we should not register kvmclock savevm section. This patch
> > > > achives that by registering this section only if kvmclock is actually
> > > > currently enabled in cpuid.
> > > > 
> > > > The only caveat is that we have to register the savevm section a little bit
> > > > later, since we won't know the final kvmclock state before cpuid gets parsed.
> > > 
> > > What is the problem of registering the section? Restoring the value if
> > > the host does not support it returns an error?
> > > 
> > > Can't you ignore the error if kvmclock is not reported in cpuid, in the
> > > restore handler?
> > 
> > We can change the restore handler, but not the restore handler of
> > binaries that are already out there. The motivation here is precisely to
> > address migration to hosts without kvmclock, so it's better to have
> > a way to disable, than to count on the fact that the other side will be
> > able to ignore it.
> 
> OK. Can't you register conditionally on kvmclock cpuid bit at the end of
> kvm_arch_init_vcpu, in target-i386/kvm.c?

Haven't looked at it, but will today. Actually, tsc has (obviously) the
same problem and I plan to respin the patch today including a fix for it
as well.

Thanks!

[Qemu-devel] [PATCH v2 2/2] make kvmclock value idempotent for stopped machine

[Glauber Costa]

Although we never made such commitment clear (well, to the best
of my knowledge), some people expect that two savevm issued in sequence
in a stopped machine will yield the same results. This is not a crazy
requirement, since we don't expect a stopped machine to be updating its state,
for any device.

With kvmclock, this is not the case, since the .pre_save hook will issue an
ioctl to the host to acquire a timestamp, which is always changing.

This patch moves the value acquisition to vm state change handlers, conditional
on not being run. This could mean mean our get clock ioctl is issued more times,
but this should be fine since vm_stop is not a hot path.

When we do migrate, we'll transfer that value along.

Signed-off-by: Glauber Costa <glommer@redhat.com>
CC: Paolo Bonzini <pbonzini@redhat.com>

---
 qemu-kvm-x86.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/qemu-kvm-x86.c b/qemu-kvm-x86.c
index 14a52ce..0e357ac 100644
--- a/qemu-kvm-x86.c
+++ b/qemu-kvm-x86.c
@@ -500,11 +500,11 @@ static int kvm_enable_tpr_access_reporting(CPUState *env)
 #ifdef KVM_CAP_ADJUST_CLOCK
 static struct kvm_clock_data kvmclock_data;
 
-static void kvmclock_pre_save(void *opaque)
+static void kvmclock_update_clock(void *opaque, int running, int reason)
 {
     struct kvm_clock_data *cl = opaque;
 
-    if (!kvmclock_enabled)
+    if ((!kvmclock_enabled) || running)
         return;
 
     kvm_vm_ioctl(kvm_state, KVM_GET_CLOCK, cl);
@@ -522,7 +522,6 @@ static const VMStateDescription vmstate_kvmclock= {
     .version_id = 1,
     .minimum_version_id = 1,
     .minimum_version_id_old = 1,
-    .pre_save = kvmclock_pre_save,
     .post_load = kvmclock_post_load,
     .fields      = (VMStateField []) {
         VMSTATE_U64(clock, struct kvm_clock_data),
@@ -537,6 +536,7 @@ void kvmclock_register_savevm(void)
 #ifdef KVM_CAP_ADJUST_CLOCK
     if (kvmclock_enabled && kvm_check_extension(kvm_state, KVM_CAP_ADJUST_CLOCK)) {
         vmstate_register(NULL, 0, &vmstate_kvmclock, &kvmclock_data);
+        qemu_add_vm_change_state_handler(kvmclock_update_clock, &kvmclock_data);
     }
 #endif
 }
-- 
1.7.2.3

[Qemu-devel] Re: [PATCH v2 0/2] savevm odness related to kvmclock

[Paolo Bonzini]

On 12/06/2010 03:03 PM, Glauber Costa wrote:
> Some users told me that savevm path is behaving oddly wrt kvmclock.
> The first oddness is that a guarantee we never made (AFAIK) is being broken:
> two consecutive "savevm" operations, with the machine stopped in between
> produces different results, due to the call to KVM_GET_CLOCK ioctl.
> I believe the assumption that if the vm does not run, its saveable
> state won't change is fairly reasonable. Maybe we should formally
> guarantee that?
>
> Also, this patch deals with the fact that this happens even if
> kvmclock is disabled in cpuid: its savevm section is registered
> nevertheless. Here, I try to register it only if it's enabled at
> machine start.
>
> v2: improvements suggested by Paolo, and patch reordering.
>
> Glauber Costa (2):
>    Do not register kvmclock savevm section if kvmclock is disabled.
>    make kvmclock value idempotent for stopped machine
>
>   cpus.c            |    3 +++
>   qemu-kvm-x86.c    |   23 +++++++++++++++--------
>   qemu-kvm.h        |    3 +++
>   target-i386/kvm.c |    7 +++++++
>   4 files changed, 28 insertions(+), 8 deletions(-)

ACK

Paolo