* [Qemu-devel] [v2 1/2] kvmclock: add a new function to update env->tsc.
2015-11-05 3:51 [Qemu-devel] [v2 0/2] Optimize the env->tsc update operation Liang Li
@ 2015-11-05 3:51 ` Liang Li
2015-11-05 3:51 ` [Qemu-devel] [v2 2/2] Revert "Introduce cpu_clean_all_dirty" Liang Li
2015-11-05 10:28 ` [Qemu-devel] [v2 0/2] Optimize the env->tsc update operation Paolo Bonzini
2 siblings, 0 replies; 4+ messages in thread
From: Liang Li @ 2015-11-05 3:51 UTC (permalink / raw)
To: qemu-devel
Cc: Liang Li, ehabkost, mst, mtosatti, quintela, m.gibula, amit.shah,
pbonzini, rth
The commit 317b0a6d8 fixed an issue which caused by the outdated
env->tsc value, but the fix lead to 'cpu_synchronize_all_states()'
called twice during live migration. The 'cpu_synchronize_all_states()'
takes about 130us for a VM which has 4 vcpus, it's a bit expensive.
Synchronize the whole CPU context just for updating env->tsc is too
wasting, this patch use a new function to update the env->tsc.
Comparing to 'cpu_synchronize_all_states()', it only takes about 20us.
Signed-off-by: Liang Li <liang.z.li@intel.com>
---
hw/i386/kvm/clock.c | 18 ++----------------
target-i386/kvm.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
target-i386/kvm_i386.h | 1 +
3 files changed, 48 insertions(+), 16 deletions(-)
diff --git a/hw/i386/kvm/clock.c b/hw/i386/kvm/clock.c
index efdf165..0593a3f 100644
--- a/hw/i386/kvm/clock.c
+++ b/hw/i386/kvm/clock.c
@@ -17,7 +17,7 @@
#include "qemu/host-utils.h"
#include "sysemu/sysemu.h"
#include "sysemu/kvm.h"
-#include "sysemu/cpus.h"
+#include "kvm_i386.h"
#include "hw/sysbus.h"
#include "hw/kvm/clock.h"
@@ -125,21 +125,7 @@ static void kvmclock_vm_state_change(void *opaque, int running,
return;
}
- cpu_synchronize_all_states();
- /* In theory, the cpu_synchronize_all_states() call above wouldn't
- * affect the rest of the code, as the VCPU state inside CPUState
- * is supposed to always match the VCPU state on the kernel side.
- *
- * In practice, calling cpu_synchronize_state() too soon will load the
- * kernel-side APIC state into X86CPU.apic_state too early, APIC state
- * won't be reloaded later because CPUState.vcpu_dirty==true, and
- * outdated APIC state may be migrated to another host.
- *
- * The real fix would be to make sure outdated APIC state is read
- * from the kernel again when necessary. While this is not fixed, we
- * need the cpu_clean_all_dirty() call below.
- */
- cpu_clean_all_dirty();
+ kvm_synchronize_all_tsc();
ret = kvm_vm_ioctl(kvm_state, KVM_GET_CLOCK, &data);
if (ret < 0) {
diff --git a/target-i386/kvm.c b/target-i386/kvm.c
index 64046cb..2a9953b 100644
--- a/target-i386/kvm.c
+++ b/target-i386/kvm.c
@@ -111,6 +111,51 @@ bool kvm_allows_irq0_override(void)
return !kvm_irqchip_in_kernel() || kvm_has_gsi_routing();
}
+static int kvm_get_tsc(CPUState *cs)
+{
+ X86CPU *cpu = X86_CPU(cs);
+ CPUX86State *env = &cpu->env;
+ struct {
+ struct kvm_msrs info;
+ struct kvm_msr_entry entries[1];
+ } msr_data;
+ int ret;
+
+ if (env->tsc_valid) {
+ return 0;
+ }
+
+ msr_data.info.nmsrs = 1;
+ msr_data.entries[0].index = MSR_IA32_TSC;
+ env->tsc_valid = !runstate_is_running();
+
+ ret = kvm_vcpu_ioctl(CPU(cpu), KVM_GET_MSRS, &msr_data);
+ if (ret < 0) {
+ return ret;
+ }
+
+ env->tsc = msr_data.entries[0].data;
+ return 0;
+}
+
+static inline void do_kvm_synchronize_tsc(void *arg)
+{
+ CPUState *cpu = arg;
+
+ kvm_get_tsc(cpu);
+}
+
+void kvm_synchronize_all_tsc(void)
+{
+ CPUState *cpu;
+
+ if (kvm_enabled()) {
+ CPU_FOREACH(cpu) {
+ run_on_cpu(cpu, do_kvm_synchronize_tsc, cpu);
+ }
+ }
+}
+
static struct kvm_cpuid2 *try_get_cpuid(KVMState *s, int max)
{
struct kvm_cpuid2 *cpuid;
diff --git a/target-i386/kvm_i386.h b/target-i386/kvm_i386.h
index e557e94..c1b312b 100644
--- a/target-i386/kvm_i386.h
+++ b/target-i386/kvm_i386.h
@@ -15,6 +15,7 @@
bool kvm_allows_irq0_override(void);
bool kvm_has_smm(void);
+void kvm_synchronize_all_tsc(void);
void kvm_arch_reset_vcpu(X86CPU *cs);
void kvm_arch_do_init_vcpu(X86CPU *cs);
--
1.9.1
^ permalink raw reply related [flat|nested] 4+ messages in thread* [Qemu-devel] [v2 2/2] Revert "Introduce cpu_clean_all_dirty"
2015-11-05 3:51 [Qemu-devel] [v2 0/2] Optimize the env->tsc update operation Liang Li
2015-11-05 3:51 ` [Qemu-devel] [v2 1/2] kvmclock: add a new function to update env->tsc Liang Li
@ 2015-11-05 3:51 ` Liang Li
2015-11-05 10:28 ` [Qemu-devel] [v2 0/2] Optimize the env->tsc update operation Paolo Bonzini
2 siblings, 0 replies; 4+ messages in thread
From: Liang Li @ 2015-11-05 3:51 UTC (permalink / raw)
To: qemu-devel
Cc: Liang Li, ehabkost, mst, mtosatti, quintela, m.gibula, amit.shah,
pbonzini, rth
This reverts commit de9d61e83d43be9069e6646fa9d57a3f47779d28.
Now 'cpu_clean_all_dirty' is useless, we can revert the related code.
Conflicts:
include/sysemu/kvm.h
Signed-off-by: Liang Li <liang.z.li@intel.com>
---
cpus.c | 9 ---------
include/sysemu/cpus.h | 1 -
include/sysemu/kvm.h | 8 --------
kvm-all.c | 5 -----
4 files changed, 23 deletions(-)
diff --git a/cpus.c b/cpus.c
index d2e9e4f..c6a5d0e 100644
--- a/cpus.c
+++ b/cpus.c
@@ -694,15 +694,6 @@ void cpu_synchronize_all_post_init(void)
}
}
-void cpu_clean_all_dirty(void)
-{
- CPUState *cpu;
-
- CPU_FOREACH(cpu) {
- cpu_clean_state(cpu);
- }
-}
-
static int do_vm_stop(RunState state)
{
int ret = 0;
diff --git a/include/sysemu/cpus.h b/include/sysemu/cpus.h
index 30ddd12..3d1e5ba 100644
--- a/include/sysemu/cpus.h
+++ b/include/sysemu/cpus.h
@@ -11,7 +11,6 @@ void cpu_stop_current(void);
void cpu_synchronize_all_states(void);
void cpu_synchronize_all_post_reset(void);
void cpu_synchronize_all_post_init(void);
-void cpu_clean_all_dirty(void);
void qtest_clock_warp(int64_t dest);
diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
index 461ef65..4ac6176 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -417,7 +417,6 @@ int kvm_physical_memory_addr_from_host(KVMState *s, void *ram_addr,
void kvm_cpu_synchronize_state(CPUState *cpu);
void kvm_cpu_synchronize_post_reset(CPUState *cpu);
void kvm_cpu_synchronize_post_init(CPUState *cpu);
-void kvm_cpu_clean_state(CPUState *cpu);
/* generic hooks - to be moved/refactored once there are more users */
@@ -442,13 +441,6 @@ static inline void cpu_synchronize_post_init(CPUState *cpu)
}
}
-static inline void cpu_clean_state(CPUState *cpu)
-{
- if (kvm_enabled()) {
- kvm_cpu_clean_state(cpu);
- }
-}
-
int kvm_irqchip_add_msi_route(KVMState *s, MSIMessage msg, PCIDevice *dev);
int kvm_irqchip_update_msi_route(KVMState *s, int virq, MSIMessage msg,
PCIDevice *dev);
diff --git a/kvm-all.c b/kvm-all.c
index c442838..1bc1273 100644
--- a/kvm-all.c
+++ b/kvm-all.c
@@ -1766,11 +1766,6 @@ void kvm_cpu_synchronize_post_init(CPUState *cpu)
run_on_cpu(cpu, do_kvm_cpu_synchronize_post_init, cpu);
}
-void kvm_cpu_clean_state(CPUState *cpu)
-{
- cpu->kvm_vcpu_dirty = false;
-}
-
int kvm_cpu_exec(CPUState *cpu)
{
struct kvm_run *run = cpu->kvm_run;
--
1.9.1
^ permalink raw reply related [flat|nested] 4+ messages in thread* Re: [Qemu-devel] [v2 0/2] Optimize the env->tsc update operation
2015-11-05 3:51 [Qemu-devel] [v2 0/2] Optimize the env->tsc update operation Liang Li
2015-11-05 3:51 ` [Qemu-devel] [v2 1/2] kvmclock: add a new function to update env->tsc Liang Li
2015-11-05 3:51 ` [Qemu-devel] [v2 2/2] Revert "Introduce cpu_clean_all_dirty" Liang Li
@ 2015-11-05 10:28 ` Paolo Bonzini
2 siblings, 0 replies; 4+ messages in thread
From: Paolo Bonzini @ 2015-11-05 10:28 UTC (permalink / raw)
To: Liang Li, qemu-devel
Cc: ehabkost, mst, mtosatti, quintela, m.gibula, amit.shah, rth
On 05/11/2015 04:51, Liang Li wrote:
> This patch set aims for reducing the live migration downtime. It
> updates the env->tsc when stopping kvmclock with a new function.
> Rather than using 'cpu_synchronize_all_states()', which updates the
> whole CPU context just for updating the env->tsc.
>
> For a VM with 4 CPUs, this patch set can help to reduce the VM downtime
> about 100us.
>
> Changes in v2:
> * Simplify 'kvm_get_tsc()' function.
> * Introduce a new function 'kvm_synchronize_all_tsc()'
>
> Liang Li (2):
> kvmclock: add a new function to update env->tsc.
> Revert "Introduce cpu_clean_all_dirty"
>
> cpus.c | 9 ---------
> hw/i386/kvm/clock.c | 18 ++----------------
> include/sysemu/cpus.h | 1 -
> include/sysemu/kvm.h | 8 --------
> kvm-all.c | 5 -----
> target-i386/kvm.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
> target-i386/kvm_i386.h | 1 +
> 7 files changed, 48 insertions(+), 39 deletions(-)
>
Nice, I've applied these patches.
Paolo
^ permalink raw reply [flat|nested] 4+ messages in thread