[PATCH for 4.6 0/5] Migration v2 fix

[PATCH for 4.6 0/5] Migration v2 fix

[Wei Liu]

Wei Liu (5):
  libxc: clearer migration v2 debug message
  libxc: migration v2 prefix Memory -> Memory (P2M)
  libxc: fix indentation
  libxc: add assertion to avoid setting same bit more than once
  libxc: de-duplicate gpfns in populate_pfns

 tools/libxc/xc_sr_restore.c        | 20 +++++++++++++++++---
 tools/libxc/xc_sr_restore_x86_pv.c |  4 ++--
 tools/libxc/xc_sr_save.c           |  2 +-
 3 files changed, 20 insertions(+), 6 deletions(-)

-- 
2.1.4

[PATCH for 4.6 1/5] libxc: clearer migration v2 debug message

[Wei Liu]

Previous the message was like:

SAVE:
xc: detail: 32 bits, 3 levels
xc: detail: max_pfn 0xfffff, p2m_frames 1024
xc: detail: max_mfn 0x130000

RESTORE:
xc: detail: max_mfn 0x130000
xc: detail: 32 bits, 3 levels
xc: detail: Expanded p2m from 0 to 0xfffff

It's not immediately clear that the last line in restore messages was
referring to max_pfn. Change the debug message a bit to make that
clearer.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 tools/libxc/xc_sr_restore_x86_pv.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/libxc/xc_sr_restore_x86_pv.c b/tools/libxc/xc_sr_restore_x86_pv.c
index ef26c64..c65a2f1 100644
--- a/tools/libxc/xc_sr_restore_x86_pv.c
+++ b/tools/libxc/xc_sr_restore_x86_pv.c
@@ -66,7 +66,7 @@ static int expand_p2m(struct xc_sr_context *ctx, unsigned long max_pfn)
     for ( i = (old_end_frame ? old_end_frame + 1 : 0); i <= end_frame; ++i )
         ctx->x86_pv.p2m_pfns[i] = INVALID_MFN;
 
-    DPRINTF("Expanded p2m from %#lx to %#lx", old_max, max_pfn);
+    DPRINTF("Changed max_pfn from %#lx to %#lx", old_max, max_pfn);
     return 0;
 }
 
-- 
2.1.4

Re: [PATCH for 4.6 1/5] libxc: clearer migration v2 debug message

[Andrew Cooper]

On 04/09/15 19:32, Wei Liu wrote:
> Previous the message was like:
>
> SAVE:
> xc: detail: 32 bits, 3 levels
> xc: detail: max_pfn 0xfffff, p2m_frames 1024
> xc: detail: max_mfn 0x130000
>
> RESTORE:
> xc: detail: max_mfn 0x130000
> xc: detail: 32 bits, 3 levels
> xc: detail: Expanded p2m from 0 to 0xfffff
>
> It's not immediately clear that the last line in restore messages was
> referring to max_pfn. Change the debug message a bit to make that
> clearer.

I am not sure that this is any clearer; the former means exactly the 
same although I suppose spelling out max_pfn is more consistent with the 
save side.

Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>

[PATCH for 4.6 2/5] libxc: migration v2 prefix Memory -> Memory (P2M)

[Wei Liu]

The prefix "Memory" is confusing because the numbers shown after that
are referring to P2M entries. They have no bearing on how many pages a
domain actually owns or how many actual pages are processed.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 tools/libxc/xc_sr_save.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
index 58667af..bb7ff54 100644
--- a/tools/libxc/xc_sr_save.c
+++ b/tools/libxc/xc_sr_save.c
@@ -659,7 +659,7 @@ static int send_domain_memory_nonlive(struct xc_sr_context *ctx)
     if ( rc )
         goto err;
 
-    xc_set_progress_prefix(xch, "Memory");
+    xc_set_progress_prefix(xch, "Memory (P2M)");
 
     rc = send_all_pages(ctx);
     if ( rc )
-- 
2.1.4

Re: [PATCH for 4.6 2/5] libxc: migration v2 prefix Memory -> Memory (P2M)

[Andrew Cooper]

On 04/09/15 19:32, Wei Liu wrote:
> The prefix "Memory" is confusing because the numbers shown after that
> are referring to P2M entries. They have no bearing on how many pages a
> domain actually owns or how many actual pages are processed.
>
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> ---
>   tools/libxc/xc_sr_save.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
> index 58667af..bb7ff54 100644
> --- a/tools/libxc/xc_sr_save.c
> +++ b/tools/libxc/xc_sr_save.c
> @@ -659,7 +659,7 @@ static int send_domain_memory_nonlive(struct xc_sr_context *ctx)
>       if ( rc )
>           goto err;
>   
> -    xc_set_progress_prefix(xch, "Memory");
> +    xc_set_progress_prefix(xch, "Memory (P2M)");

This is incorrect for autotranslated guests.  I would recommend "Frames" 
instead.

~Andrew

Re: [PATCH for 4.6 2/5] libxc: migration v2 prefix Memory -> Memory (P2M)

[Wei Liu]

On Fri, Sep 04, 2015 at 09:40:55PM +0100, Andrew Cooper wrote:
> On 04/09/15 19:32, Wei Liu wrote:
> >The prefix "Memory" is confusing because the numbers shown after that
> >are referring to P2M entries. They have no bearing on how many pages a
> >domain actually owns or how many actual pages are processed.
> >
> >Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> >---
> >  tools/libxc/xc_sr_save.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> >diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
> >index 58667af..bb7ff54 100644
> >--- a/tools/libxc/xc_sr_save.c
> >+++ b/tools/libxc/xc_sr_save.c
> >@@ -659,7 +659,7 @@ static int send_domain_memory_nonlive(struct xc_sr_context *ctx)
> >      if ( rc )
> >          goto err;
> >-    xc_set_progress_prefix(xch, "Memory");
> >+    xc_set_progress_prefix(xch, "Memory (P2M)");
> 
> This is incorrect for autotranslated guests.  I would recommend "Frames"
> instead.
> 

Fine by me of course.

Wei.

> ~Andrew

[PATCH for 4.6 3/5] libxc: fix indentation

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 tools/libxc/xc_sr_restore_x86_pv.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/libxc/xc_sr_restore_x86_pv.c b/tools/libxc/xc_sr_restore_x86_pv.c
index c65a2f1..bc604b3 100644
--- a/tools/libxc/xc_sr_restore_x86_pv.c
+++ b/tools/libxc/xc_sr_restore_x86_pv.c
@@ -970,7 +970,7 @@ static int x86_pv_localise_page(struct xc_sr_context *ctx,
     }
 
     if ( to_populate && populate_pfns(ctx, to_populate, pfns, NULL) )
-            return -1;
+        return -1;
 
     for ( i = 0; i < (PAGE_SIZE / sizeof(uint64_t)); ++i )
     {
-- 
2.1.4

Re: [PATCH for 4.6 3/5] libxc: fix indentation

[Andrew Cooper]

On 04/09/15 19:32, Wei Liu wrote:
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>

Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>

[PATCH for 4.6 4/5] libxc: add assertion to avoid setting same bit more than once

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 tools/libxc/xc_sr_restore.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index df885b6..adb48da 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -181,6 +181,7 @@ static int pfn_set_populated(struct xc_sr_context *ctx, xen_pfn_t pfn)
         ctx->restore.max_populated_pfn = new_max;
     }
 
+    assert(!test_bit(pfn, ctx->restore.populated_pfns));
     set_bit(pfn, ctx->restore.populated_pfns);
 
     return 0;
-- 
2.1.4

Re: [PATCH for 4.6 4/5] libxc: add assertion to avoid setting same bit more than once

[Andrew Cooper]

On 04/09/15 19:32, Wei Liu wrote:
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>

This surely isn't bisectable with patch 5?  The change is fine (so 
Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>), provided you 
either reverse patches 4 and 5, or fold them together.

~Andrew

Re: [PATCH for 4.6 4/5] libxc: add assertion to avoid setting same bit more than once

[Wei Liu]

On Fri, Sep 04, 2015 at 09:43:16PM +0100, Andrew Cooper wrote:
> On 04/09/15 19:32, Wei Liu wrote:
> >Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> 
> This surely isn't bisectable with patch 5?  The change is fine (so
> Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>), provided you either
> reverse patches 4 and 5, or fold them together.
> 

Oh sure. I will reverse them.

Wei.

> ~Andrew

[PATCH for 4.6 5/5] libxc: de-duplicate gpfns in populate_pfns

[Wei Liu]

The original implementation didn't consider there can be same gpfns
present multiple times in the passed in array. The mechanism to prevent
populating same gpfn multiple times only works if the same gpfn appear
in different batch.

This bug is discovered by save / restore Linux 4.1 32 bit kernel, which
has several PTEs pointing to same gpfn. And gpfn pointed to by those
PTEs are populated in one batch by libxc.  When libxc calls
x86_pv_localise_page, the original implementation failed to detect
duplications in one batch.

The fix is to de-duplicate gpfns in populate_pfns.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 tools/libxc/xc_sr_restore.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)

diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index adb48da..09fe4c0 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -198,7 +198,7 @@ int populate_pfns(struct xc_sr_context *ctx, unsigned count,
     xc_interface *xch = ctx->xch;
     xen_pfn_t *mfns = malloc(count * sizeof(*mfns)),
         *pfns = malloc(count * sizeof(*pfns));
-    unsigned i, nr_pfns = 0;
+    unsigned i, j, nr_pfns = 0;
     int rc = -1;
 
     if ( !mfns || !pfns )
@@ -209,14 +209,27 @@ int populate_pfns(struct xc_sr_context *ctx, unsigned count,
     }
 
     for ( i = 0; i < count; ++i )
+        pfns[i] = mfns[i] = INVALID_MFN;
+
+    for ( i = 0; i < count; ++i )
     {
         if ( (!types || (types &&
                          (types[i] != XEN_DOMCTL_PFINFO_XTAB &&
                           types[i] != XEN_DOMCTL_PFINFO_BROKEN))) &&
              !pfn_is_populated(ctx, original_pfns[i]) )
         {
-            pfns[nr_pfns] = mfns[nr_pfns] = original_pfns[i];
-            ++nr_pfns;
+            bool present = false;
+
+            /* De-duplicate gpfns to avoid populating the same one twice */
+            for ( j = 0; j < nr_pfns; ++j )
+                if ( pfns[j] == original_pfns[i] )
+                    present = true;
+
+            if ( !present )
+            {
+                pfns[nr_pfns] = mfns[nr_pfns] = original_pfns[i];
+                ++nr_pfns;
+            }
         }
     }
 
-- 
2.1.4

Re: [PATCH for 4.6 5/5] libxc: de-duplicate gpfns in populate_pfns

[Andrew Cooper]

     On 04/09/15 19:32, Wei Liu wrote:
> The original implementation didn't consider there can be same gpfns
> present multiple times in the passed in array. The mechanism to prevent
> populating same gpfn multiple times only works if the same gpfn appear
> in different batch.
>
> This bug is discovered by save / restore Linux 4.1 32 bit kernel, which
> has several PTEs pointing to same gpfn. And gpfn pointed to by those
> PTEs are populated in one batch by libxc.  When libxc calls
> x86_pv_localise_page, the original implementation failed to detect
> duplications in one batch.
>
> The fix is to de-duplicate gpfns in populate_pfns.
>
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>

The original version of populate_pfns() synchronously populated pfns 
when they were found referenced in PTEs ahead of the appropriate 
PAGE_DATA record arriving, but this proved to be very inefficient 
depends on the pagetable layout of the guest.

I agree that this is a bug.

> ---
>   tools/libxc/xc_sr_restore.c | 19 ++++++++++++++++---
>   1 file changed, 16 insertions(+), 3 deletions(-)
>
> diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
> index adb48da..09fe4c0 100644
> --- a/tools/libxc/xc_sr_restore.c
> +++ b/tools/libxc/xc_sr_restore.c
> @@ -198,7 +198,7 @@ int populate_pfns(struct xc_sr_context *ctx, unsigned count,
>       xc_interface *xch = ctx->xch;
>       xen_pfn_t *mfns = malloc(count * sizeof(*mfns)),
>           *pfns = malloc(count * sizeof(*pfns));
> -    unsigned i, nr_pfns = 0;
> +    unsigned i, j, nr_pfns = 0;
>       int rc = -1;
>   
>       if ( !mfns || !pfns )
> @@ -209,14 +209,27 @@ int populate_pfns(struct xc_sr_context *ctx, unsigned count,
>       }
>   
>       for ( i = 0; i < count; ++i )
> +        pfns[i] = mfns[i] = INVALID_MFN;
> +
> +    for ( i = 0; i < count; ++i )
>       {
>           if ( (!types || (types &&
>                            (types[i] != XEN_DOMCTL_PFINFO_XTAB &&
>                             types[i] != XEN_DOMCTL_PFINFO_BROKEN))) &&
>                !pfn_is_populated(ctx, original_pfns[i]) )
>           {
> -            pfns[nr_pfns] = mfns[nr_pfns] = original_pfns[i];
> -            ++nr_pfns;
> +            bool present = false;
> +
> +            /* De-duplicate gpfns to avoid populating the same one twice */

Just pfns to match the rest of the code.  (I notice some other memory 
terminology needing cleaning up - I will formulate some patches when I 
am next in a position to do so.)

> +            for ( j = 0; j < nr_pfns; ++j )
> +                if ( pfns[j] == original_pfns[i] )
> +                    present = true;

Adding this nested loop introduces O(N^2) behavior on what is typically 
1024-length inputs.

I recommend moving the pfn_set_populated() call from the subsequent loop 
into this loop, which will cause the pfn_is_populated() call in this 
loop condition to catch repeat populations even in the same batch.

The only way pfn_set_populated() can fail is out of memory, and any 
error anywhere in this function is fatal to migration, so there is no 
chance of proceeding with migration with a pfn marked as populated, but 
set to INVALID_MFN in the p2m.

~Andrew

> +
> +            if ( !present )
> +            {
> +                pfns[nr_pfns] = mfns[nr_pfns] = original_pfns[i];
> +                ++nr_pfns;
> +            }
>           }
>       }
>