[Qemu-devel] [PATCH V5] spapr: Fix stale HTAB during live migration

[Qemu-devel] [PATCH V5] spapr: Fix stale HTAB during live migration

[Samuel Mendoza-Jonas]

If a guest reboots during a running migration, changes to the
hash page table are not necessarily updated on the destination.
Opening a new file descriptor to the HTAB forces the migration
handler to resend the entire table.

Signed-off-by: Samuel Mendoza-Jonas <sam.mj@au1.ibm.com>
---
Changes in v5: Use mutex on when accessing htab_fd_stale
Changes in v4: Readability: need_reset to htab_fd_stale
        Add spapr_check_htab_fd() and use error_report()
Changes in v3: Pointed out by David, htab_save_iterate could
	potentially try to read before htab_fd is open again.
	Leave opening the fd to the functions trying to read.
Changes in v2: Forgot check on kvmppc_get_htab_fd return value
 hw/ppc/spapr.c         | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 include/hw/ppc/spapr.h |  2 ++
 2 files changed, 48 insertions(+)

diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index 3a6d26d..d743308 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -997,6 +997,13 @@ static void spapr_reset_htab(sPAPREnvironment *spapr)
         /* Kernel handles htab, we don't need to allocate one */
         spapr->htab_shift = shift;
         kvmppc_kern_htab = true;
+
+        /* Tell readers to update their file descriptor */
+        pthread_mutex_lock(&spapr->htab_mutex);
+        if (spapr->htab_fd > 0) {
+            spapr->htab_fd_stale = true;
+        }
+        pthread_mutex_unlock(&spapr->htab_mutex);
     } else {
         if (!spapr->htab) {
             /* Allocate an htab if we don't yet have one */
@@ -1014,6 +1021,30 @@ static void spapr_reset_htab(sPAPREnvironment *spapr)
     }
 }
 
+/* A guest reset will cause spapr->htab_fd to become stale if being used.
+ * Reopen the file descriptor to make sure the whole HTAB is properly read.
+ */
+static int spapr_check_htab_fd(sPAPREnvironment *spapr)
+{
+    int rc = 0;
+
+    pthread_mutex_lock(&spapr->htab_mutex);
+
+    if (spapr->htab_fd_stale) {
+        close(spapr->htab_fd);
+        spapr->htab_fd = kvmppc_get_htab_fd(false);
+        if (spapr->htab_fd < 0) {
+            error_report("Unable to open fd for reading hash table from KVM: "
+                    "%s", strerror(errno));
+            rc = -1;
+        }
+        spapr->htab_fd_stale = false;
+    }
+
+    pthread_mutex_unlock(&spapr->htab_mutex);
+    return rc;
+}
+
 static void ppc_spapr_reset(void)
 {
     PowerPCCPU *first_ppc_cpu;
@@ -1156,7 +1187,10 @@ static int htab_save_setup(QEMUFile *f, void *opaque)
     } else {
         assert(kvm_enabled());
 
+        pthread_mutex_lock(&spapr->htab_mutex);
         spapr->htab_fd = kvmppc_get_htab_fd(false);
+        spapr->htab_fd_stale = false;
+        pthread_mutex_unlock(&spapr->htab_mutex);
         if (spapr->htab_fd < 0) {
             fprintf(stderr, "Unable to open fd for reading hash table from KVM: %s\n",
                     strerror(errno));
@@ -1309,6 +1343,11 @@ static int htab_save_iterate(QEMUFile *f, void *opaque)
     if (!spapr->htab) {
         assert(kvm_enabled());
 
+        rc = spapr_check_htab_fd(spapr);
+        if (rc < 0) {
+            return rc;
+        }
+
         rc = kvmppc_save_htab(f, spapr->htab_fd,
                               MAX_KVM_BUF_SIZE, MAX_ITERATION_NS);
         if (rc < 0) {
@@ -1340,6 +1379,11 @@ static int htab_save_complete(QEMUFile *f, void *opaque)
 
         assert(kvm_enabled());
 
+        rc = spapr_check_htab_fd(spapr);
+        if (rc < 0) {
+            return rc;
+        }
+
         rc = kvmppc_save_htab(f, spapr->htab_fd, MAX_KVM_BUF_SIZE, -1);
         if (rc < 0) {
             return rc;
@@ -1525,6 +1569,8 @@ static void ppc_spapr_init(QEMUMachineInitArgs *args)
         spapr->htab_shift++;
     }
 
+    pthread_mutex_init(&spapr->htab_mutex, NULL);
+
     /* Set up Interrupt Controller before we create the VCPUs */
     spapr->icp = xics_system_init(smp_cpus * kvmppc_smt_threads() / smp_threads,
                                   XICS_IRQS);
diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h
index 0c2e3c5..435309a 100644
--- a/include/hw/ppc/spapr.h
+++ b/include/hw/ppc/spapr.h
@@ -71,6 +71,8 @@ typedef struct sPAPREnvironment {
     int htab_save_index;
     bool htab_first_pass;
     int htab_fd;
+    bool htab_fd_stale;
+    pthread_mutex_t htab_mutex;
 
     /* state for Dynamic Reconfiguration Connectors */
     sPAPRDrcEntry drc_table[SPAPR_DRC_TABLE_SIZE];
-- 
1.9.3

Re: [Qemu-devel] [PATCH V5] spapr: Fix stale HTAB during live migration

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 906 bytes --]

On Fri, Aug 22, 2014 at 10:24:10AM +1000, Samuel Mendoza-Jonas wrote:
> If a guest reboots during a running migration, changes to the
> hash page table are not necessarily updated on the destination.
> Opening a new file descriptor to the HTAB forces the migration
> handler to resend the entire table.

Of course, arguably this is a kernel bug - the htab fd should probably
start re-reporting things if the htab reset ioctl() is called.  But I
guess we need a workaround for existing kernels anyway.

However, don't we still have the bug for TCG mode?  spapr_reset_htab()
just memset()s the whole hashtable in that case, which will actually
clear all the dirty bits, whereas we need to set them.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [PATCH V5] spapr: Fix stale HTAB during live migration

[Sam Mendoza-Jonas]

On 26/08/14 13:01, David Gibson wrote:
> On Fri, Aug 22, 2014 at 10:24:10AM +1000, Samuel Mendoza-Jonas wrote:
>> If a guest reboots during a running migration, changes to the
>> hash page table are not necessarily updated on the destination.
>> Opening a new file descriptor to the HTAB forces the migration
>> handler to resend the entire table.
> 
> Of course, arguably this is a kernel bug - the htab fd should probably
> start re-reporting things if the htab reset ioctl() is called.  But I
> guess we need a workaround for existing kernels anyway.

That's right, but as you say it would be nice to cover existing kernels.

> 
> However, don't we still have the bug for TCG mode?  spapr_reset_htab()
> just memset()s the whole hashtable in that case, which will actually
> clear all the dirty bits, whereas we need to set them.
> 

True! I'll add a case for this, it should just be a matter of
checking & setting htab_first_pass.

-- 
Regards,
Sam Mendoza-Jonas
-----------
LTC Ozlabs
IBM