[Qemu-devel] [RFC 0/2] Reduce the VM downtime about 300us

[Qemu-devel] [RFC 0/2] Reduce the VM downtime about 300us

[Liang Li]

This patch is for kvm live migration optimization, it fixes the issue which
commit 317b0a6d8ba tries to fix in another way, and it can reduce the live 
migration VM downtime about 300us.

*This patch is not tested for the issue commit 317b0a6d8ba tries to fix*

Liang Li (2):
  kvmclock: remove cpu_synchronize_all_states call
  Revert "Introduce cpu_clean_all_dirty"

 cpus.c                | 17 ++++++++---------
 hw/i386/kvm/clock.c   | 17 -----------------
 include/sysemu/cpus.h |  1 -
 include/sysemu/kvm.h  |  8 --------
 kvm-all.c             |  5 -----
 5 files changed, 8 insertions(+), 40 deletions(-)

-- 
1.9.1

[Qemu-devel] [RFC 1/2] kvmclock: remove cpu_synchronize_all_states call

[Liang Li]

The commit 317b0a6d8 fixed an issue which caused by the outdated env->tsc
value, but the fix lead to cpu_synchronize_all_states called twice during
live migration. The cpu_synchronize_all_states takes about 300us, which is
a bit expensive.
This patch tries to fix the issue in another way, and there is only one
cpu_synchronize_all_states call during live migration.

Signed-off-by: Liang Li <liang.z.li@intel.com>
---
 cpus.c              |  8 ++++++++
 hw/i386/kvm/clock.c | 17 -----------------
 2 files changed, 8 insertions(+), 17 deletions(-)

diff --git a/cpus.c b/cpus.c
index c1e74d9..cd5abef 100644
--- a/cpus.c
+++ b/cpus.c
@@ -634,6 +634,14 @@ static int do_vm_stop(RunState state)
         pause_all_vcpus();
         runstate_set(state);
         vm_state_notify(0, state);
+        switch (state) {
+        case RUN_STATE_PAUSED:
+        case RUN_STATE_SUSPENDED:
+            cpu_synchronize_all_states();
+            break;
+        default:
+            break;
+        }
         qapi_event_send_stop(&error_abort);
     }
 
diff --git a/hw/i386/kvm/clock.c b/hw/i386/kvm/clock.c
index efdf165..a9db073 100644
--- a/hw/i386/kvm/clock.c
+++ b/hw/i386/kvm/clock.c
@@ -17,7 +17,6 @@
 #include "qemu/host-utils.h"
 #include "sysemu/sysemu.h"
 #include "sysemu/kvm.h"
-#include "sysemu/cpus.h"
 #include "hw/sysbus.h"
 #include "hw/kvm/clock.h"
 
@@ -125,22 +124,6 @@ static void kvmclock_vm_state_change(void *opaque, int running,
             return;
         }
 
-        cpu_synchronize_all_states();
-        /* In theory, the cpu_synchronize_all_states() call above wouldn't
-         * affect the rest of the code, as the VCPU state inside CPUState
-         * is supposed to always match the VCPU state on the kernel side.
-         *
-         * In practice, calling cpu_synchronize_state() too soon will load the
-         * kernel-side APIC state into X86CPU.apic_state too early, APIC state
-         * won't be reloaded later because CPUState.vcpu_dirty==true, and
-         * outdated APIC state may be migrated to another host.
-         *
-         * The real fix would be to make sure outdated APIC state is read
-         * from the kernel again when necessary. While this is not fixed, we
-         * need the cpu_clean_all_dirty() call below.
-         */
-        cpu_clean_all_dirty();
-
         ret = kvm_vm_ioctl(kvm_state, KVM_GET_CLOCK, &data);
         if (ret < 0) {
             fprintf(stderr, "KVM_GET_CLOCK failed: %s\n", strerror(ret));
-- 
1.9.1

[Qemu-devel] [RFC 2/2] Revert "Introduce cpu_clean_all_dirty"

[Liang Li]

This reverts commit de9d61e83d43be9069e6646fa9d57a3f47779d28.
---
 cpus.c                | 9 ---------
 include/sysemu/cpus.h | 1 -
 include/sysemu/kvm.h  | 8 --------
 kvm-all.c             | 5 -----
 4 files changed, 23 deletions(-)

diff --git a/cpus.c b/cpus.c
index cd5abef..ffa81cb 100644
--- a/cpus.c
+++ b/cpus.c
@@ -616,15 +616,6 @@ void cpu_synchronize_all_post_init(void)
     }
 }
 
-void cpu_clean_all_dirty(void)
-{
-    CPUState *cpu;
-
-    CPU_FOREACH(cpu) {
-        cpu_clean_state(cpu);
-    }
-}
-
 static int do_vm_stop(RunState state)
 {
     int ret = 0;
diff --git a/include/sysemu/cpus.h b/include/sysemu/cpus.h
index 3f162a9..4f79081 100644
--- a/include/sysemu/cpus.h
+++ b/include/sysemu/cpus.h
@@ -10,7 +10,6 @@ void cpu_stop_current(void);
 void cpu_synchronize_all_states(void);
 void cpu_synchronize_all_post_reset(void);
 void cpu_synchronize_all_post_init(void);
-void cpu_clean_all_dirty(void);
 
 void qtest_clock_warp(int64_t dest);
 
diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
index 983e99e..083fca6 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -380,7 +380,6 @@ int kvm_physical_memory_addr_from_host(KVMState *s, void *ram_addr,
 void kvm_cpu_synchronize_state(CPUState *cpu);
 void kvm_cpu_synchronize_post_reset(CPUState *cpu);
 void kvm_cpu_synchronize_post_init(CPUState *cpu);
-void kvm_cpu_clean_state(CPUState *cpu);
 
 /* generic hooks - to be moved/refactored once there are more users */
 
@@ -405,13 +404,6 @@ static inline void cpu_synchronize_post_init(CPUState *cpu)
     }
 }
 
-static inline void cpu_clean_state(CPUState *cpu)
-{
-    if (kvm_enabled()) {
-        kvm_cpu_clean_state(cpu);
-    }
-}
-
 int kvm_irqchip_add_msi_route(KVMState *s, MSIMessage msg);
 int kvm_irqchip_update_msi_route(KVMState *s, int virq, MSIMessage msg);
 void kvm_irqchip_release_virq(KVMState *s, int virq);
diff --git a/kvm-all.c b/kvm-all.c
index 06e06f2..f4df584 100644
--- a/kvm-all.c
+++ b/kvm-all.c
@@ -1779,11 +1779,6 @@ void kvm_cpu_synchronize_post_init(CPUState *cpu)
     run_on_cpu(cpu, do_kvm_cpu_synchronize_post_init, cpu);
 }
 
-void kvm_cpu_clean_state(CPUState *cpu)
-{
-    cpu->kvm_vcpu_dirty = false;
-}
-
 int kvm_cpu_exec(CPUState *cpu)
 {
     struct kvm_run *run = cpu->kvm_run;
-- 
1.9.1

Re: [Qemu-devel] [RFC 0/2] Reduce the VM downtime about 300us

[Marcin Gibuła]

W dniu 2015-08-25 o 07:52, Liang Li pisze:
> This patch is for kvm live migration optimization, it fixes the issue which
> commit 317b0a6d8ba tries to fix in another way, and it can reduce the live
> migration VM downtime about 300us.
>
> *This patch is not tested for the issue commit 317b0a6d8ba tries to fix*

I'll try to test it within next few days.

-- 
mg

Re: [Qemu-devel] [RFC 0/2] Reduce the VM downtime about 300us

[Li, Liang Z]

> W dniu 2015-08-25 o 07:52, Liang Li pisze:
> > This patch is for kvm live migration optimization, it fixes the issue
> > which commit 317b0a6d8ba tries to fix in another way, and it can
> > reduce the live migration VM downtime about 300us.
> >
> > *This patch is not tested for the issue commit 317b0a6d8ba tries to
> > fix*
> 
> I'll try to test it within next few days.

Mg, thanks very much!

Liang 
> 
> --
> mg

Re: [Qemu-devel] [RFC 0/2] Reduce the VM downtime about 300us

[Li, Liang Z]

Hi mg,

Have you tested this patch? Can it fix the kvm-clock issue?

Liang


> -----Original Message-----
> From: Marcin Gibuła [mailto:m.gibula@beyond.pl]
> Sent: Wednesday, August 26, 2015 3:26 AM
> To: Li, Liang Z; qemu-devel@nongnu.org
> Cc: pbonzini@redhat.com; mtosatti@redhat.com
> Subject: Re: [RFC 0/2] Reduce the VM downtime about 300us
> 
> W dniu 2015-08-25 o 07:52, Liang Li pisze:
> > This patch is for kvm live migration optimization, it fixes the issue
> > which commit 317b0a6d8ba tries to fix in another way, and it can
> > reduce the live migration VM downtime about 300us.
> >
> > *This patch is not tested for the issue commit 317b0a6d8ba tries to
> > fix*
> 
> I'll try to test it within next few days.
> 
> --
> mg