[patch 0/5] fix SMP migration and loadvm/savevm (V3)

[patch 0/5] fix SMP migration and loadvm/savevm (V3)

[Marcelo Tosatti]

Address comments from Avi:

- fold mp_state into a structure
- IOW/IOR markers for ioctl definitions
- return -ENOSYS if extension is not supported

-- 


-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

[patch 1/5] QEMU/KVM: properly copy the in-kernel apicbase value

[Marcelo Tosatti]

[-- Attachment #1: apicbase --]
[-- Type: text/plain, Size: 2125 bytes --]

The MSR_IA32_APICBASE_ENABLE/MSR_IA32_APICBASE_BSP bits in s->apicbase
are not initialized if in-kernel APIC emulation is used, so save the
actual value passed by cpu_set_apic_base() caller.

Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>

Index: kvm-userspace.io/qemu/hw/apic.c
===================================================================
--- kvm-userspace.io.orig/qemu/hw/apic.c
+++ kvm-userspace.io/qemu/hw/apic.c
@@ -248,8 +248,11 @@ void cpu_set_apic_base(CPUState *env, ui
 #ifdef DEBUG_APIC
     printf("cpu_set_apic_base: %016" PRIx64 "\n", val);
 #endif
-    s->apicbase = (val & 0xfffff000) |
-        (s->apicbase & (MSR_IA32_APICBASE_BSP | MSR_IA32_APICBASE_ENABLE));
+    if (kvm_enabled() && qemu_kvm_irqchip_in_kernel())
+        s->apicbase = val;
+    else
+        s->apicbase = (val & 0xfffff000) |
+            (s->apicbase & (MSR_IA32_APICBASE_BSP | MSR_IA32_APICBASE_ENABLE));
     /* if disabled, cannot be enabled again */
     if (!(val & MSR_IA32_APICBASE_ENABLE)) {
         s->apicbase &= ~MSR_IA32_APICBASE_ENABLE;
Index: kvm-userspace.io/qemu/qemu-kvm-x86.c
===================================================================
--- kvm-userspace.io.orig/qemu/qemu-kvm-x86.c
+++ kvm-userspace.io/qemu/qemu-kvm-x86.c
@@ -248,13 +248,8 @@ void kvm_arch_load_regs(CPUState *env)
     sregs.cr3 = env->cr[3];
     sregs.cr4 = env->cr[4];
 
-    if (kvm_irqchip_in_kernel(kvm_context)) {
-        sregs.cr8 = kvm_get_cr8(kvm_context, env->cpu_index);
-        sregs.apic_base = kvm_get_apic_base(kvm_context, env->cpu_index);
-    } else {
-        sregs.cr8 = cpu_get_apic_tpr(env);
-        sregs.apic_base = cpu_get_apic_base(env);
-    }
+    sregs.cr8 = cpu_get_apic_tpr(env);
+    sregs.apic_base = cpu_get_apic_base(env);
 
     sregs.efer = env->efer;
 

-- 


-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

[patch 2/5] QEMU/KVM: Add libkvm interface to get/set the mpstate.

[Marcelo Tosatti]

[-- Attachment #1: smpmig-libkvm --]
[-- Type: text/plain, Size: 2126 bytes --]

Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>

Index: kvm-userspace.io/libkvm/libkvm.c
===================================================================
--- kvm-userspace.io.orig/libkvm/libkvm.c
+++ kvm-userspace.io/libkvm/libkvm.c
@@ -776,6 +776,28 @@ int kvm_set_sregs(kvm_context_t kvm, int
     return ioctl(kvm->vcpu_fd[vcpu], KVM_SET_SREGS, sregs);
 }
 
+#ifdef KVM_CAP_MP_STATE
+int kvm_get_mpstate(kvm_context_t kvm, int vcpu, struct kvm_mp_state *mp_state)
+{
+    int r;
+
+    r = ioctl(kvm->fd, KVM_CHECK_EXTENSION, KVM_CAP_MP_STATE);
+    if (r > 0)
+        return ioctl(kvm->vcpu_fd[vcpu], KVM_GET_MP_STATE, mp_state);
+    return -ENOSYS;
+}
+
+int kvm_set_mpstate(kvm_context_t kvm, int vcpu, struct kvm_mp_state *mp_state)
+{
+    int r;
+
+    r = ioctl(kvm->fd, KVM_CHECK_EXTENSION, KVM_CAP_MP_STATE);
+    if (r > 0)
+        return ioctl(kvm->vcpu_fd[vcpu], KVM_SET_MP_STATE, mp_state);
+    return -ENOSYS;
+}
+#endif
+
 static int handle_mmio(kvm_context_t kvm, struct kvm_run *kvm_run)
 {
 	unsigned long addr = kvm_run->mmio.phys_addr;
Index: kvm-userspace.io/libkvm/libkvm.h
===================================================================
--- kvm-userspace.io.orig/libkvm/libkvm.h
+++ kvm-userspace.io/libkvm/libkvm.h
@@ -300,6 +300,22 @@ int kvm_get_sregs(kvm_context_t kvm, int
  */
 int kvm_set_sregs(kvm_context_t kvm, int vcpu, struct kvm_sregs *regs);
 
+#ifdef KVM_CAP_MP_STATE
+/*!
+ *  * \brief Read VCPU MP state
+ *
+ */
+int kvm_get_mpstate(kvm_context_t kvm, int vcpu,
+                    struct kvm_mp_state *mp_state);
+
+/*!
+ *  * \brief Write VCPU MP state
+ *
+ */
+int kvm_set_mpstate(kvm_context_t kvm, int vcpu,
+                    struct kvm_mp_state *mp_state);
+#endif
+
 /*!
  * \brief Simulate an external vectored interrupt
  *

-- 


-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

[patch 3/5] QEMU/KVM: save and load mp state

[Marcelo Tosatti]

[-- Attachment #1: smpmig-qemu --]
[-- Type: text/plain, Size: 4437 bytes --]

Use the new interface to save and restore MP_STATE for all vcpu's.

Increase version_id for cpu_load/cpu_save.

Fixes SMP migration.

Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>

Index: kvm-userspace.io/qemu/hw/pc.c
===================================================================
--- kvm-userspace.io.orig/qemu/hw/pc.c
+++ kvm-userspace.io/qemu/hw/pc.c
@@ -750,7 +750,7 @@ CPUState *pc_new_cpu(int cpu, const char
             /* XXX: enable it in all cases */
             env->cpuid_features |= CPUID_APIC;
         }
-        register_savevm("cpu", cpu, 4, cpu_save, cpu_load, env);
+        register_savevm("cpu", cpu, 5, cpu_save, cpu_load, env);
         qemu_register_reset(main_cpu_reset, env);
         if (pci_enabled) {
             apic_init(env);
Index: kvm-userspace.io/qemu/qemu-kvm-x86.c
===================================================================
--- kvm-userspace.io.orig/qemu/qemu-kvm-x86.c
+++ kvm-userspace.io/qemu/qemu-kvm-x86.c
@@ -277,6 +277,33 @@ void kvm_arch_load_regs(CPUState *env)
         perror("kvm_set_msrs FAILED");
 }
 
+void kvm_save_mpstate(CPUState *env)
+{
+#ifdef KVM_CAP_MP_STATE
+    int r;
+    struct kvm_mp_state mp_state;
+
+    r = kvm_get_mpstate(kvm_context, env->cpu_index, &mp_state);
+    if (r < 0)
+        env->mp_state = -1;
+    else
+        env->mp_state = mp_state.mp_state;
+#endif
+}
+
+void kvm_load_mpstate(CPUState *env)
+{
+#ifdef KVM_CAP_MP_STATE
+    struct kvm_mp_state mp_state = { .mp_state = env->mp_state };
+
+    /*
+     * -1 indicates that the host did not support GET_MP_STATE ioctl,
+     *  so don't touch it.
+     */
+    if (env->mp_state != -1)
+        kvm_set_mpstate(kvm_context, env->cpu_index, &mp_state);
+#endif
+}
 
 void kvm_arch_save_regs(CPUState *env)
 {
Index: kvm-userspace.io/qemu/qemu-kvm.h
===================================================================
--- kvm-userspace.io.orig/qemu/qemu-kvm.h
+++ kvm-userspace.io/qemu/qemu-kvm.h
@@ -18,6 +18,8 @@ int kvm_init_ap(void);
 void kvm_qemu_destroy(void);
 void kvm_load_registers(CPUState *env);
 void kvm_save_registers(CPUState *env);
+void kvm_load_mpstate(CPUState *env);
+void kvm_save_mpstate(CPUState *env);
 int kvm_cpu_exec(CPUState *env);
 int kvm_update_debugger(CPUState *env);
 int kvm_qemu_init_env(CPUState *env);
Index: kvm-userspace.io/qemu/target-i386/cpu.h
===================================================================
--- kvm-userspace.io.orig/qemu/target-i386/cpu.h
+++ kvm-userspace.io/qemu/target-i386/cpu.h
@@ -599,6 +599,7 @@ typedef struct CPUX86State {
     /* in order to simplify APIC support, we leave this pointer to the
        user */
     struct APICState *apic_state;
+    int mp_state;
 } CPUX86State;
 
 CPUX86State *cpu_x86_init(const char *cpu_model);
Index: kvm-userspace.io/qemu/vl.c
===================================================================
--- kvm-userspace.io.orig/qemu/vl.c
+++ kvm-userspace.io/qemu/vl.c
@@ -6655,8 +6655,10 @@ void cpu_save(QEMUFile *f, void *opaque)
     uint32_t hflags;
     int i;
 
-    if (kvm_enabled())
+    if (kvm_enabled()) {
         kvm_save_registers(env);
+        kvm_save_mpstate(env);
+    }
 
     for(i = 0; i < CPU_NB_REGS; i++)
         qemu_put_betls(f, &env->regs[i]);
@@ -6748,6 +6750,7 @@ void cpu_save(QEMUFile *f, void *opaque)
             qemu_put_be32s(f, &env->kvm_interrupt_bitmap[i]);
         }
         qemu_put_be64s(f, &env->tsc);
+        qemu_put_be32s(f, &env->mp_state);
     }
 }
 
@@ -6782,7 +6785,7 @@ int cpu_load(QEMUFile *f, void *opaque, 
     uint32_t hflags;
     uint16_t fpus, fpuc, fptag, fpregs_format;
 
-    if (version_id != 3 && version_id != 4)
+    if (version_id < 3 || version_id > 5)
         return -EINVAL;
     for(i = 0; i < CPU_NB_REGS; i++)
         qemu_get_betls(f, &env->regs[i]);
@@ -6900,6 +6903,10 @@ int cpu_load(QEMUFile *f, void *opaque, 
         }
         qemu_get_be64s(f, &env->tsc);
         kvm_load_registers(env);
+        if (version_id >= 5) {
+            qemu_get_be32s(f, &env->mp_state);
+            kvm_load_mpstate(env);
+        }
     }
     return 0;
 }

-- 


-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

[patch 4/5] QEMU/KVM: ignore SIG_IPI signals in userspace

[Marcelo Tosatti]

[-- Attachment #1: ignore-sigipi --]
[-- Type: text/plain, Size: 997 bytes --]

Otherwise a signal can be received in userspace and a vcpu goes back
to the kernel while it should stay still.

Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>

Index: kvm-userspace.io/qemu/qemu-kvm.c
===================================================================
--- kvm-userspace.io.orig/qemu/qemu-kvm.c
+++ kvm-userspace.io/qemu/qemu-kvm.c
@@ -350,7 +350,6 @@ static void *ap_main_loop(void *_env)
     vcpu->env = env;
     vcpu->env->thread_id = kvm_get_thread_id();
     sigfillset(&signals);
-    sigdelset(&signals, SIG_IPI);
     sigprocmask(SIG_BLOCK, &signals, NULL);
     kvm_create_vcpu(kvm_context, env->cpu_index);
     kvm_qemu_init_env(env);

-- 


-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

[patch 5/5] KVM: add ioctls to save/store mpstate

[Marcelo Tosatti]

[-- Attachment #1: smp-mig-kernel.patch --]
[-- Type: text/plain, Size: 3806 bytes --]

So userspace can save/restore the mpstate during migration.

Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>

Index: kvm/arch/x86/kvm/x86.c
===================================================================
--- kvm.orig/arch/x86/kvm/x86.c
+++ kvm/arch/x86/kvm/x86.c
@@ -814,6 +814,7 @@ int kvm_dev_ioctl_check_extension(long e
 	case KVM_CAP_CLOCKSOURCE:
 	case KVM_CAP_PIT:
 	case KVM_CAP_NOP_IO_DELAY:
+	case KVM_CAP_MP_STATE:
 		r = 1;
 		break;
 	case KVM_CAP_VAPIC:
@@ -3057,6 +3058,24 @@ int kvm_arch_vcpu_ioctl_get_sregs(struct
 	return 0;
 }
 
+int kvm_arch_vcpu_ioctl_get_mpstate(struct kvm_vcpu *vcpu,
+				    struct kvm_mp_state *mp_state)
+{
+	vcpu_load(vcpu);
+	mp_state->mp_state = vcpu->arch.mp_state;
+	vcpu_put(vcpu);
+	return 0;
+}
+
+int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,
+				    struct kvm_mp_state *mp_state)
+{
+	vcpu_load(vcpu);
+	vcpu->arch.mp_state = mp_state->mp_state;
+	vcpu_put(vcpu);
+	return 0;
+}
+
 static void set_segment(struct kvm_vcpu *vcpu,
 			struct kvm_segment *var, int seg)
 {
Index: kvm/include/linux/kvm.h
===================================================================
--- kvm.orig/include/linux/kvm.h
+++ kvm/include/linux/kvm.h
@@ -222,6 +222,11 @@ struct kvm_vapic_addr {
 	__u64 vapic_addr;
 };
 
+/* for KVM_SET_MPSTATE */
+struct kvm_mp_state {
+	__u32 mp_state;
+};
+
 struct kvm_s390_psw {
 	__u64 mask;
 	__u64 addr;
@@ -279,6 +284,7 @@ struct kvm_s390_interrupt {
 #define KVM_CAP_PIT 11
 #define KVM_CAP_NOP_IO_DELAY 12
 #define KVM_CAP_PV_MMU 13
+#define KVM_CAP_MP_STATE 14
 
 /*
  * ioctls for VM fds
@@ -340,5 +346,7 @@ struct kvm_s390_interrupt {
 #define KVM_S390_SET_INITIAL_PSW  _IOW(KVMIO,  0x96, struct kvm_s390_psw)
 /* initial reset for s390 */
 #define KVM_S390_INITIAL_RESET    _IO(KVMIO,  0x97)
+#define KVM_GET_MP_STATE          _IOR(KVMIO,  0x98, struct kvm_mp_state)
+#define KVM_SET_MP_STATE          _IOW(KVMIO,  0x99, struct kvm_mp_state)
 
 #endif
Index: kvm/include/linux/kvm_host.h
===================================================================
--- kvm.orig/include/linux/kvm_host.h
+++ kvm/include/linux/kvm_host.h
@@ -236,6 +236,10 @@ int kvm_arch_vcpu_ioctl_get_sregs(struct
 				  struct kvm_sregs *sregs);
 int kvm_arch_vcpu_ioctl_set_sregs(struct kvm_vcpu *vcpu,
 				  struct kvm_sregs *sregs);
+int kvm_arch_vcpu_ioctl_get_mpstate(struct kvm_vcpu *vcpu,
+				    struct kvm_mp_state *mp_state);
+int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,
+				    struct kvm_mp_state *mp_state);
 int kvm_arch_vcpu_ioctl_debug_guest(struct kvm_vcpu *vcpu,
 				    struct kvm_debug_guest *dbg);
 int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run);
Index: kvm/virt/kvm/kvm_main.c
===================================================================
--- kvm.orig/virt/kvm/kvm_main.c
+++ kvm/virt/kvm/kvm_main.c
@@ -978,6 +978,30 @@ out_free2:
 		r = 0;
 		break;
 	}
+	case KVM_GET_MP_STATE: {
+		struct kvm_mp_state mp_state;
+
+		r = kvm_arch_vcpu_ioctl_get_mpstate(vcpu, &mp_state);
+		if (r)
+			goto out;
+		r = -EFAULT;
+		if (copy_to_user(argp, &mp_state, sizeof mp_state))
+			goto out;
+		r = 0;
+		break;
+	}
+	case KVM_SET_MP_STATE: {
+		struct kvm_mp_state mp_state;
+
+		r = -EFAULT;
+		if (copy_from_user(&mp_state, argp, sizeof mp_state))
+			goto out;
+		r = kvm_arch_vcpu_ioctl_set_mpstate(vcpu, &mp_state);
+		if (r)
+			goto out;
+		r = 0;
+		break;
+	}
 	case KVM_TRANSLATE: {
 		struct kvm_translation tr;
 

-- 


-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

Re: [patch 5/5] KVM: add ioctls to save/store mpstate

[Avi Kivity]

Marcelo Tosatti wrote:
> So userspace can save/restore the mpstate during migration.

Applied, thanks.  I also added public #defines for the possible values.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

Re: [patch 0/5] fix SMP migration and loadvm/savevm (V3)

[Avi Kivity]

Marcelo Tosatti wrote:
> Address comments from Avi:
>
> - fold mp_state into a structure
> - IOW/IOR markers for ioctl definitions
> - return -ENOSYS if extension is not supported
>
>   

Applied all, thanks.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone