[patch 0/4] fix SMP migration and loadvm/savevm (V2)

[patch 0/4] fix SMP migration and loadvm/savevm (V2)

[Marcelo Tosatti]

Now guarded by KVM_CAP and with increased version_id of cpu_save/load
as requested.

Avi, I prefer not to fold mpstate into kvm_save_registers() as a hidden
register because the MPSTATE is only used during migration, whereas 
save_registers() is not (seems safer).

-- 


-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Register now and save $200. Hurry, offer ends at 11:59 p.m., 
Monday, April 7! Use priority code J8TLD2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

[patch 1/4] QEMU/KVM: properly copy the in-kernel apicbase value

[Marcelo Tosatti]

[-- Attachment #1: qemu-fix-apic --]
[-- Type: text/plain, Size: 2202 bytes --]

The MSR_IA32_APICBASE_ENABLE/MSR_IA32_APICBASE_BSP bits in s->apicbase
are not initialized if in-kernel APIC emulation is used, so save the
actual value passed by cpu_set_apic_base() caller.

Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>

Index: marcelo/git/kvm-userspace.io/qemu/hw/apic.c
===================================================================
--- marcelo.orig/git/kvm-userspace.io/qemu/hw/apic.c
+++ marcelo/git/kvm-userspace.io/qemu/hw/apic.c
@@ -248,8 +248,11 @@ void cpu_set_apic_base(CPUState *env, ui
 #ifdef DEBUG_APIC
     printf("cpu_set_apic_base: %016" PRIx64 "\n", val);
 #endif
-    s->apicbase = (val & 0xfffff000) |
-        (s->apicbase & (MSR_IA32_APICBASE_BSP | MSR_IA32_APICBASE_ENABLE));
+    if (kvm_enabled() && qemu_kvm_irqchip_in_kernel())
+        s->apicbase = val;
+    else
+        s->apicbase = (val & 0xfffff000) |
+            (s->apicbase & (MSR_IA32_APICBASE_BSP | MSR_IA32_APICBASE_ENABLE));
     /* if disabled, cannot be enabled again */
     if (!(val & MSR_IA32_APICBASE_ENABLE)) {
         s->apicbase &= ~MSR_IA32_APICBASE_ENABLE;
Index: marcelo/git/kvm-userspace.io/qemu/qemu-kvm-x86.c
===================================================================
--- marcelo.orig/git/kvm-userspace.io/qemu/qemu-kvm-x86.c
+++ marcelo/git/kvm-userspace.io/qemu/qemu-kvm-x86.c
@@ -248,13 +248,8 @@ void kvm_arch_load_regs(CPUState *env)
     sregs.cr3 = env->cr[3];
     sregs.cr4 = env->cr[4];
 
-    if (kvm_irqchip_in_kernel(kvm_context)) {
-        sregs.cr8 = kvm_get_cr8(kvm_context, env->cpu_index);
-        sregs.apic_base = kvm_get_apic_base(kvm_context, env->cpu_index);
-    } else {
-        sregs.cr8 = cpu_get_apic_tpr(env);
-        sregs.apic_base = cpu_get_apic_base(env);
-    }
+    sregs.cr8 = cpu_get_apic_tpr(env);
+    sregs.apic_base = cpu_get_apic_base(env);
 
     sregs.efer = env->efer;
 

-- 


-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Register now and save $200. Hurry, offer ends at 11:59 p.m., 
Monday, April 7! Use priority code J8TLD2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

[patch 2/4] QEMU/KVM: get/set mpstate libkvm interface

[Marcelo Tosatti]

[-- Attachment #1: fix-mp-migration --]
[-- Type: text/plain, Size: 2190 bytes --]

Add libkvm interface to get/set the mpstate.

Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>

Index: marcelo/git/kvm-userspace.io/libkvm/libkvm.c
===================================================================
--- marcelo.orig/git/kvm-userspace.io/libkvm/libkvm.c
+++ marcelo/git/kvm-userspace.io/libkvm/libkvm.c
@@ -776,6 +776,32 @@ int kvm_set_sregs(kvm_context_t kvm, int
     return ioctl(kvm->vcpu_fd[vcpu], KVM_SET_SREGS, sregs);
 }
 
+int kvm_get_mpstate(kvm_context_t kvm, int vcpu, int *mpstate)
+{
+    int r;
+
+#ifdef KVM_CAP_MP_STATE
+    r = ioctl(kvm->fd, KVM_CHECK_EXTENSION, KVM_CAP_MP_STATE);
+    if (r > 0)
+        return ioctl(kvm->vcpu_fd[vcpu], KVM_GET_MP_STATE, mpstate);
+#endif
+    return -1;
+}
+
+int kvm_set_mpstate(kvm_context_t kvm, int vcpu, int *mpstate)
+{
+    int r;
+
+#ifdef KVM_CAP_MP_STATE
+    r = ioctl(kvm->fd, KVM_CHECK_EXTENSION, KVM_CAP_MP_STATE);
+    if (r > 0)
+        return ioctl(kvm->vcpu_fd[vcpu], KVM_SET_MP_STATE, mpstate);
+#endif
+    return -1;
+}
+
+
+
 static int handle_mmio(kvm_context_t kvm, struct kvm_run *kvm_run)
 {
 	unsigned long addr = kvm_run->mmio.phys_addr;
Index: marcelo/git/kvm-userspace.io/libkvm/libkvm.h
===================================================================
--- marcelo.orig/git/kvm-userspace.io/libkvm/libkvm.h
+++ marcelo/git/kvm-userspace.io/libkvm/libkvm.h
@@ -301,6 +301,18 @@ int kvm_get_sregs(kvm_context_t kvm, int
 int kvm_set_sregs(kvm_context_t kvm, int vcpu, struct kvm_sregs *regs);
 
 /*!
+ *  * \brief Read VCPU MP state
+ *
+ */
+int kvm_get_mpstate(kvm_context_t kvm, int vcpu, int *mpstate);
+
+/*!
+ *  * \brief Write VCPU MP state
+ *
+ */
+int kvm_set_mpstate(kvm_context_t kvm, int vcpu, int *mpstate);
+
+/*!
  * \brief Simulate an external vectored interrupt
  *
  * This allows you to simulate an external vectored interrupt.

-- 


-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Register now and save $200. Hurry, offer ends at 11:59 p.m., 
Monday, April 7! Use priority code J8TLD2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

Re: [patch 2/4] QEMU/KVM: get/set mpstate libkvm interface

[Avi Kivity]

Marcelo Tosatti wrote:
> Add libkvm interface to get/set the mpstate.
>
> Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
>
> Index: marcelo/git/kvm-userspace.io/libkvm/libkvm.c
> ===================================================================
> --- marcelo.orig/git/kvm-userspace.io/libkvm/libkvm.c
> +++ marcelo/git/kvm-userspace.io/libkvm/libkvm.c
> @@ -776,6 +776,32 @@ int kvm_set_sregs(kvm_context_t kvm, int
>      return ioctl(kvm->vcpu_fd[vcpu], KVM_SET_SREGS, sregs);
>  }
>  
> +int kvm_get_mpstate(kvm_context_t kvm, int vcpu, int *mpstate)
> +{
> +    int r;
> +
> +#ifdef KVM_CAP_MP_STATE
> +    r = ioctl(kvm->fd, KVM_CHECK_EXTENSION, KVM_CAP_MP_STATE);
> +    if (r > 0)
> +        return ioctl(kvm->vcpu_fd[vcpu], KVM_GET_MP_STATE, mpstate);
> +#endif
> +    return -1;
> +}
>   

We usually follow the convention of returning -errno in case of error 
instead of -1 in libkvm.

-- 
Any sufficiently difficult bug is indistinguishable from a feature.


-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

[patch 3/4] QEMU/KVM: save and load mp state

[Marcelo Tosatti]

[-- Attachment #1: use-mpstate-ioctls --]
[-- Type: text/plain, Size: 4409 bytes --]

Use the new interface to save and restore MP_STATE for all vcpu's.

Increase version_id for cpu_load/cpu_save.

Fixes SMP migration.

Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>

Index: marcelo/git/kvm-userspace.io/qemu/qemu-kvm-x86.c
===================================================================
--- marcelo.orig/git/kvm-userspace.io/qemu/qemu-kvm-x86.c
+++ marcelo/git/kvm-userspace.io/qemu/qemu-kvm-x86.c
@@ -277,6 +277,24 @@ void kvm_arch_load_regs(CPUState *env)
         perror("kvm_set_msrs FAILED");
 }
 
+void kvm_save_mpstate(CPUState *env)
+{
+    int r;
+
+    r = kvm_get_mpstate(kvm_context, env->cpu_index, &env->mp_state);
+    if (r == -1)
+        env->mp_state = r;
+}
+
+void kvm_load_mpstate(CPUState *env)
+{
+    /*
+     * -1 indicates that the host did not support GET_MP_STATE ioctl,
+     *  so don't touch it.
+     */
+    if (env->mp_state != -1)
+        kvm_set_mpstate(kvm_context, env->cpu_index, &env->mp_state);
+}
 
 void kvm_arch_save_regs(CPUState *env)
 {
Index: marcelo/git/kvm-userspace.io/qemu/qemu-kvm.h
===================================================================
--- marcelo.orig/git/kvm-userspace.io/qemu/qemu-kvm.h
+++ marcelo/git/kvm-userspace.io/qemu/qemu-kvm.h
@@ -18,6 +18,8 @@ int kvm_init_ap(void);
 void kvm_qemu_destroy(void);
 void kvm_load_registers(CPUState *env);
 void kvm_save_registers(CPUState *env);
+void kvm_load_mpstate(CPUState *env);
+void kvm_save_mpstate(CPUState *env);
 int kvm_cpu_exec(CPUState *env);
 int kvm_update_debugger(CPUState *env);
 int kvm_qemu_init_env(CPUState *env);
Index: marcelo/git/kvm-userspace.io/qemu/target-i386/cpu.h
===================================================================
--- marcelo.orig/git/kvm-userspace.io/qemu/target-i386/cpu.h
+++ marcelo/git/kvm-userspace.io/qemu/target-i386/cpu.h
@@ -599,6 +599,7 @@ typedef struct CPUX86State {
     /* in order to simplify APIC support, we leave this pointer to the
        user */
     struct APICState *apic_state;
+    int mp_state;
 } CPUX86State;
 
 CPUX86State *cpu_x86_init(const char *cpu_model);
Index: marcelo/git/kvm-userspace.io/qemu/vl.c
===================================================================
--- marcelo.orig/git/kvm-userspace.io/qemu/vl.c
+++ marcelo/git/kvm-userspace.io/qemu/vl.c
@@ -6655,8 +6655,10 @@ void cpu_save(QEMUFile *f, void *opaque)
     uint32_t hflags;
     int i;
 
-    if (kvm_enabled())
+    if (kvm_enabled()) {
         kvm_save_registers(env);
+        kvm_save_mpstate(env);
+    }
 
     for(i = 0; i < CPU_NB_REGS; i++)
         qemu_put_betls(f, &env->regs[i]);
@@ -6748,6 +6750,7 @@ void cpu_save(QEMUFile *f, void *opaque)
             qemu_put_be32s(f, &env->kvm_interrupt_bitmap[i]);
         }
         qemu_put_be64s(f, &env->tsc);
+        qemu_put_be32s(f, &env->mp_state);
     }
 }
 
@@ -6782,7 +6785,7 @@ int cpu_load(QEMUFile *f, void *opaque, 
     uint32_t hflags;
     uint16_t fpus, fpuc, fptag, fpregs_format;
 
-    if (version_id != 3 && version_id != 4)
+    if (version_id < 3 || version_id > 5)
         return -EINVAL;
     for(i = 0; i < CPU_NB_REGS; i++)
         qemu_get_betls(f, &env->regs[i]);
@@ -6900,6 +6903,10 @@ int cpu_load(QEMUFile *f, void *opaque, 
         }
         qemu_get_be64s(f, &env->tsc);
         kvm_load_registers(env);
+        if (version_id >= 5) {
+            qemu_get_be32s(f, &env->mp_state);
+            kvm_load_mpstate(env);
+        }
     }
     return 0;
 }
Index: marcelo/git/kvm-userspace.io/qemu/hw/pc.c
===================================================================
--- marcelo.orig/git/kvm-userspace.io/qemu/hw/pc.c
+++ marcelo/git/kvm-userspace.io/qemu/hw/pc.c
@@ -750,7 +750,7 @@ CPUState *pc_new_cpu(int cpu, const char
             /* XXX: enable it in all cases */
             env->cpuid_features |= CPUID_APIC;
         }
-        register_savevm("cpu", cpu, 4, cpu_save, cpu_load, env);
+        register_savevm("cpu", cpu, 5, cpu_save, cpu_load, env);
         qemu_register_reset(main_cpu_reset, env);
         if (pci_enabled) {
             apic_init(env);

-- 


-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Register now and save $200. Hurry, offer ends at 11:59 p.m., 
Monday, April 7! Use priority code J8TLD2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

[patch 4/4] QEMU/KVM: ignore SIG_IPI signals in userspace

[Marcelo Tosatti]

[-- Attachment #1: ignore-sigipi --]
[-- Type: text/plain, Size: 1038 bytes --]

Otherwise a signal can be received in userspace and a vcpu goes back
to the kernel while it should stay still.

Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>

Index: marcelo/git/kvm-userspace.io/qemu/qemu-kvm.c
===================================================================
--- marcelo.orig/git/kvm-userspace.io/qemu/qemu-kvm.c
+++ marcelo/git/kvm-userspace.io/qemu/qemu-kvm.c
@@ -350,7 +350,6 @@ static void *ap_main_loop(void *_env)
     vcpu->env = env;
     vcpu->env->thread_id = kvm_get_thread_id();
     sigfillset(&signals);
-    sigdelset(&signals, SIG_IPI);
     sigprocmask(SIG_BLOCK, &signals, NULL);
     kvm_create_vcpu(kvm_context, env->cpu_index);
     kvm_qemu_init_env(env);

-- 


-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Register now and save $200. Hurry, offer ends at 11:59 p.m., 
Monday, April 7! Use priority code J8TLD2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

Re: [patch 0/4] fix SMP migration and loadvm/savevm (V2)

[Avi Kivity]

Marcelo Tosatti wrote:
> Avi, I prefer not to fold mpstate into kvm_save_registers() as a hidden
> register because the MPSTATE is only used during migration, whereas 
> save_registers() is not (seems safer)

But that's the point... what about savevm/loadvm, etc?  They deserve to 
work too.

-- 
Any sufficiently difficult bug is indistinguishable from a feature.


-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Register now and save $200. Hurry, offer ends at 11:59 p.m., 
Monday, April 7! Use priority code J8TLD2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

Re: [patch 0/4] fix SMP migration and loadvm/savevm (V2)

[Marcelo Tosatti]

On Wed, Apr 09, 2008 at 12:59:50AM +0300, Avi Kivity wrote:
> Marcelo Tosatti wrote:
> >Avi, I prefer not to fold mpstate into kvm_save_registers() as a hidden
> >register because the MPSTATE is only used during migration, whereas 
> >save_registers() is not (seems safer)
> 
> But that's the point... what about savevm/loadvm, etc?  They deserve to 
> work too.

savevm/loadvm will work through cpu_save/cpu_load just as live migration
does, so they are covered.


-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

Re: [patch 0/4] fix SMP migration and loadvm/savevm (V2)

[Avi Kivity]

Marcelo Tosatti wrote:
> On Wed, Apr 09, 2008 at 12:59:50AM +0300, Avi Kivity wrote:
>   
>> Marcelo Tosatti wrote:
>>     
>>> Avi, I prefer not to fold mpstate into kvm_save_registers() as a hidden
>>> register because the MPSTATE is only used during migration, whereas 
>>> save_registers() is not (seems safer)
>>>       
>> But that's the point... what about savevm/loadvm, etc?  They deserve to 
>> work too.
>>     
>
> savevm/loadvm will work through cpu_save/cpu_load just as live migration
> does, so they are covered.
>
>   
Right.

-- 
Any sufficiently difficult bug is indistinguishable from a feature.


-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone