[PATCH v2] fix Remus failover regression

[PATCH v2] fix Remus failover regression

[Yang Hongyang]

commit: c2ba706c
tools/libxc: goto correct label on error paths by Andrew Cooper
broke Remus in Xen 4.4 or earlier versions that has this commit
backported.

With Remus, this jump essentially discards the current incomplete
checkpoint received by the backup and restore backup from the
last complete checkpoint.
This is required for Remus to work and this does not break live
migration.
It has been around since Xen 4.0.

CC: Ian Jackson <ian.jackson@eu.citrix.com>
CC: Ian Campbell <ian.campbell@citrix.com>
CC: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Shriram Rajagopalan <rshriram@cs.ubc.ca>
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
 tools/libxc/xc_domain_restore.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/tools/libxc/xc_domain_restore.c b/tools/libxc/xc_domain_restore.c
index e73e0a2..b9a56d5 100644
--- a/tools/libxc/xc_domain_restore.c
+++ b/tools/libxc/xc_domain_restore.c
@@ -1783,20 +1783,29 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
 
     if ( pagebuf_get(xch, ctx, &pagebuf, io_fd, dom) ) {
         PERROR("error when buffering batch, finishing");
-        goto out;
+        /*
+         * Remus: discard the current incomplete checkpoint and restore
+         * backup from the last complete checkpoint.
+         */
+        goto finish;
     }
     memset(&tmptail, 0, sizeof(tmptail));
     tmptail.ishvm = hvm;
     if ( buffer_tail(xch, ctx, &tmptail, io_fd, max_vcpu_id, vcpumap,
                      ext_vcpucontext, vcpuextstate_size) < 0 ) {
         ERROR ("error buffering image tail, finishing");
-        goto out;
+        /*
+         * Remus: discard the current incomplete checkpoint and restore
+         * backup from the last complete checkpoint.
+         */
+        goto finish;
     }
     tailbuf_free(&tailbuf);
     memcpy(&tailbuf, &tmptail, sizeof(tailbuf));
 
     goto loadpages;
 
+  /* With Remus: restore from last complete checkpoint */
   finish:
     if ( hvm )
         goto finish_hvm;
-- 
1.9.1

Re: [PATCH v2] fix Remus failover regression

[Shriram Rajagopalan]

[-- Attachment #1.1: Type: text/plain, Size: 2280 bytes --]

On Jul 28, 2014 12:03 AM, "Yang Hongyang" <yanghy@cn.fujitsu.com> wrote:
>
> commit: c2ba706c
> tools/libxc: goto correct label on error paths by Andrew Cooper
> broke Remus in Xen 4.4 or earlier versions that has this commit
> backported.
>
> With Remus, this jump essentially discards the current incomplete
> checkpoint received by the backup and restore backup from the
> last complete checkpoint.
> This is required for Remus to work and this does not break live
> migration.
> It has been around since Xen 4.0.
>
> CC: Ian Jackson <ian.jackson@eu.citrix.com>
> CC: Ian Campbell <ian.campbell@citrix.com>
> CC: Andrew Cooper <andrew.cooper3@citrix.com>
> CC: Shriram Rajagopalan <rshriram@cs.ubc.ca>
> Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
> ---
>  tools/libxc/xc_domain_restore.c | 13 +++++++++++--
>  1 file changed, 11 insertions(+), 2 deletions(-)
>
> diff --git a/tools/libxc/xc_domain_restore.c
b/tools/libxc/xc_domain_restore.c
> index e73e0a2..b9a56d5 100644
> --- a/tools/libxc/xc_domain_restore.c
> +++ b/tools/libxc/xc_domain_restore.c
> @@ -1783,20 +1783,29 @@ int xc_domain_restore(xc_interface *xch, int
io_fd, uint32_t dom,
>
>      if ( pagebuf_get(xch, ctx, &pagebuf, io_fd, dom) ) {
>          PERROR("error when buffering batch, finishing");
> -        goto out;
> +        /*
> +         * Remus: discard the current incomplete checkpoint and restore
> +         * backup from the last complete checkpoint.
> +         */
> +        goto finish;
>      }
>      memset(&tmptail, 0, sizeof(tmptail));
>      tmptail.ishvm = hvm;
>      if ( buffer_tail(xch, ctx, &tmptail, io_fd, max_vcpu_id, vcpumap,
>                       ext_vcpucontext, vcpuextstate_size) < 0 ) {
>          ERROR ("error buffering image tail, finishing");
> -        goto out;
> +        /*
> +         * Remus: discard the current incomplete checkpoint and restore
> +         * backup from the last complete checkpoint.
> +         */
> +        goto finish;
>      }
>      tailbuf_free(&tailbuf);
>      memcpy(&tailbuf, &tmptail, sizeof(tailbuf));
>
>      goto loadpages;
>
> +  /* With Remus: restore from last complete checkpoint */
>    finish:
>      if ( hvm )
>          goto finish_hvm;
> --
> 1.9.1
>

Acked-by: Shriram Rajagopalan <rshriram@cs.ubc.ca>

[-- Attachment #1.2: Type: text/html, Size: 3322 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH v2] fix Remus failover regression

[Andrew Cooper]

On 28/07/14 05:03, Yang Hongyang wrote:
> commit: c2ba706c
> tools/libxc: goto correct label on error paths by Andrew Cooper
> broke Remus in Xen 4.4 or earlier versions that has this commit
> backported.

My appologies for breaking Remus. (it just goes to show how fragile this
code is).

>
> With Remus, this jump essentially discards the current incomplete
> checkpoint received by the backup and restore backup from the
> last complete checkpoint.
> This is required for Remus to work and this does not break live
> migration.
> It has been around since Xen 4.0.

However, it is a genuine bugfix for regular migration, so simply
reverting it as this patch does is not appropriate.

For regular migration, you absolutely have to goto out; on a failure
otherwise the finish code will run and declare the migration a success
despite only having half a domain restored.

You need something like:

if ( !checkpointed_stream )
    goto err;

/* Remus comment */
goto finish;

to deal with the different error handing requirements of remus and
regular streams.

~Andrew

>
> CC: Ian Jackson <ian.jackson@eu.citrix.com>
> CC: Ian Campbell <ian.campbell@citrix.com>
> CC: Andrew Cooper <andrew.cooper3@citrix.com>
> CC: Shriram Rajagopalan <rshriram@cs.ubc.ca>
> Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
> ---
>  tools/libxc/xc_domain_restore.c | 13 +++++++++++--
>  1 file changed, 11 insertions(+), 2 deletions(-)
>
> diff --git a/tools/libxc/xc_domain_restore.c b/tools/libxc/xc_domain_restore.c
> index e73e0a2..b9a56d5 100644
> --- a/tools/libxc/xc_domain_restore.c
> +++ b/tools/libxc/xc_domain_restore.c
> @@ -1783,20 +1783,29 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
>  
>      if ( pagebuf_get(xch, ctx, &pagebuf, io_fd, dom) ) {
>          PERROR("error when buffering batch, finishing");
> -        goto out;
> +        /*
> +         * Remus: discard the current incomplete checkpoint and restore
> +         * backup from the last complete checkpoint.
> +         */
> +        goto finish;
>      }
>      memset(&tmptail, 0, sizeof(tmptail));
>      tmptail.ishvm = hvm;
>      if ( buffer_tail(xch, ctx, &tmptail, io_fd, max_vcpu_id, vcpumap,
>                       ext_vcpucontext, vcpuextstate_size) < 0 ) {
>          ERROR ("error buffering image tail, finishing");
> -        goto out;
> +        /*
> +         * Remus: discard the current incomplete checkpoint and restore
> +         * backup from the last complete checkpoint.
> +         */
> +        goto finish;
>      }
>      tailbuf_free(&tailbuf);
>      memcpy(&tailbuf, &tmptail, sizeof(tailbuf));
>  
>      goto loadpages;
>  
> +  /* With Remus: restore from last complete checkpoint */
>    finish:
>      if ( hvm )
>          goto finish_hvm;

Re: [PATCH v2] fix Remus failover regression

[Hongyang Yang]

Hi Andrew,

On 07/28/2014 05:24 PM, Andrew Cooper wrote:
> On 28/07/14 05:03, Yang Hongyang wrote:
>> commit: c2ba706c
>> tools/libxc: goto correct label on error paths by Andrew Cooper
>> broke Remus in Xen 4.4 or earlier versions that has this commit
>> backported.
>
> My appologies for breaking Remus. (it just goes to show how fragile this
> code is).
>
>>
>> With Remus, this jump essentially discards the current incomplete
>> checkpoint received by the backup and restore backup from the
>> last complete checkpoint.
>> This is required for Remus to work and this does not break live
>> migration.
>> It has been around since Xen 4.0.
>
> However, it is a genuine bugfix for regular migration, so simply
> reverting it as this patch does is not appropriate.
>
> For regular migration, you absolutely have to goto out; on a failure
> otherwise the finish code will run and declare the migration a success
> despite only having half a domain restored.

I think regular migration shouldn't run into this path (see what I commented
in v1), but I agree that add a check will be better.

>
> You need something like:
>
> if ( !checkpointed_stream )
>      goto err;
>
> /* Remus comment */
> goto finish;
>
> to deal with the different error handing requirements of remus and
> regular streams.
>
> ~Andrew
>
>>
>> CC: Ian Jackson <ian.jackson@eu.citrix.com>
>> CC: Ian Campbell <ian.campbell@citrix.com>
>> CC: Andrew Cooper <andrew.cooper3@citrix.com>
>> CC: Shriram Rajagopalan <rshriram@cs.ubc.ca>
>> Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
>> ---
>>   tools/libxc/xc_domain_restore.c | 13 +++++++++++--
>>   1 file changed, 11 insertions(+), 2 deletions(-)
>>
>> diff --git a/tools/libxc/xc_domain_restore.c b/tools/libxc/xc_domain_restore.c
>> index e73e0a2..b9a56d5 100644
>> --- a/tools/libxc/xc_domain_restore.c
>> +++ b/tools/libxc/xc_domain_restore.c
>> @@ -1783,20 +1783,29 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
>>
>>       if ( pagebuf_get(xch, ctx, &pagebuf, io_fd, dom) ) {
>>           PERROR("error when buffering batch, finishing");
>> -        goto out;
>> +        /*
>> +         * Remus: discard the current incomplete checkpoint and restore
>> +         * backup from the last complete checkpoint.
>> +         */
>> +        goto finish;
>>       }
>>       memset(&tmptail, 0, sizeof(tmptail));
>>       tmptail.ishvm = hvm;
>>       if ( buffer_tail(xch, ctx, &tmptail, io_fd, max_vcpu_id, vcpumap,
>>                        ext_vcpucontext, vcpuextstate_size) < 0 ) {
>>           ERROR ("error buffering image tail, finishing");
>> -        goto out;
>> +        /*
>> +         * Remus: discard the current incomplete checkpoint and restore
>> +         * backup from the last complete checkpoint.
>> +         */
>> +        goto finish;
>>       }
>>       tailbuf_free(&tailbuf);
>>       memcpy(&tailbuf, &tmptail, sizeof(tailbuf));
>>
>>       goto loadpages;
>>
>> +  /* With Remus: restore from last complete checkpoint */
>>     finish:
>>       if ( hvm )
>>           goto finish_hvm;
>
> .
>

-- 
Thanks,
Yang.

Re: [PATCH v2] fix Remus failover regression

[Andrew Cooper]

On 28/07/14 10:29, Hongyang Yang wrote:
> Hi Andrew,
>
> On 07/28/2014 05:24 PM, Andrew Cooper wrote:
>> On 28/07/14 05:03, Yang Hongyang wrote:
>>> commit: c2ba706c
>>> tools/libxc: goto correct label on error paths by Andrew Cooper
>>> broke Remus in Xen 4.4 or earlier versions that has this commit
>>> backported.
>>
>> My appologies for breaking Remus. (it just goes to show how fragile this
>> code is).
>>
>>>
>>> With Remus, this jump essentially discards the current incomplete
>>> checkpoint received by the backup and restore backup from the
>>> last complete checkpoint.
>>> This is required for Remus to work and this does not break live
>>> migration.
>>> It has been around since Xen 4.0.
>>
>> However, it is a genuine bugfix for regular migration, so simply
>> reverting it as this patch does is not appropriate.
>>
>> For regular migration, you absolutely have to goto out; on a failure
>> otherwise the finish code will run and declare the migration a success
>> despite only having half a domain restored.
>
> I think regular migration shouldn't run into this path (see what I
> commented
> in v1), but I agree that add a check will be better.

Hmm - I see what you mean.  I can't spot how a regular migration would
end up at that point.

When I debugged the issue, I was encountering the pagebuf error message
on a regular migrate, although I was debugging a single isolated failure
from logs alone.  With a bit of hindsight now, this probably means that
ctx->last_checkpoint was wrong.

We regularly test migration from before the point that
ctx->last_checkpoint was introduced and broke the migration
backwards-compatibility, but the purpose of checkpointed_stream was to
re-fix this without regressing backwards compatibility.

I have to admit that I somewhat confused as to what actually went on,
but it is also clear that my changes were based on incorrect reasoning
and further rereasoning at this point suggests the changes were wrong.

Therefore, this patch with comments is probably best.

Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>

IanJ: This fix needs backporting to 4.4  (4.3 and older are fine)

~Andrew

Re: [PATCH v2] fix Remus failover regression

[Hongyang Yang]

Ping!

在 07/28/2014 12:03 PM, Yang Hongyang 写道:
> commit: c2ba706c
> tools/libxc: goto correct label on error paths by Andrew Cooper
> broke Remus in Xen 4.4 or earlier versions that has this commit
> backported.
>
> With Remus, this jump essentially discards the current incomplete
> checkpoint received by the backup and restore backup from the
> last complete checkpoint.
> This is required for Remus to work and this does not break live
> migration.
> It has been around since Xen 4.0.
>
> CC: Ian Jackson <ian.jackson@eu.citrix.com>
> CC: Ian Campbell <ian.campbell@citrix.com>
> CC: Andrew Cooper <andrew.cooper3@citrix.com>
> CC: Shriram Rajagopalan <rshriram@cs.ubc.ca>
> Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
> ---
>   tools/libxc/xc_domain_restore.c | 13 +++++++++++--
>   1 file changed, 11 insertions(+), 2 deletions(-)
>
> diff --git a/tools/libxc/xc_domain_restore.c b/tools/libxc/xc_domain_restore.c
> index e73e0a2..b9a56d5 100644
> --- a/tools/libxc/xc_domain_restore.c
> +++ b/tools/libxc/xc_domain_restore.c
> @@ -1783,20 +1783,29 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
>
>       if ( pagebuf_get(xch, ctx, &pagebuf, io_fd, dom) ) {
>           PERROR("error when buffering batch, finishing");
> -        goto out;
> +        /*
> +         * Remus: discard the current incomplete checkpoint and restore
> +         * backup from the last complete checkpoint.
> +         */
> +        goto finish;
>       }
>       memset(&tmptail, 0, sizeof(tmptail));
>       tmptail.ishvm = hvm;
>       if ( buffer_tail(xch, ctx, &tmptail, io_fd, max_vcpu_id, vcpumap,
>                        ext_vcpucontext, vcpuextstate_size) < 0 ) {
>           ERROR ("error buffering image tail, finishing");
> -        goto out;
> +        /*
> +         * Remus: discard the current incomplete checkpoint and restore
> +         * backup from the last complete checkpoint.
> +         */
> +        goto finish;
>       }
>       tailbuf_free(&tailbuf);
>       memcpy(&tailbuf, &tmptail, sizeof(tailbuf));
>
>       goto loadpages;
>
> +  /* With Remus: restore from last complete checkpoint */
>     finish:
>       if ( hvm )
>           goto finish_hvm;
>

-- 
Thanks,
Yang.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH v2] fix Remus failover regression

[Andrew Cooper]

On 07/08/2014 02:16, Hongyang Yang wrote:
> Ping!
>
> 在 07/28/2014 12:03 PM, Yang Hongyang 写道:
>> commit: c2ba706c
>> tools/libxc: goto correct label on error paths by Andrew Cooper
>> broke Remus in Xen 4.4 or earlier versions that has this commit
>> backported.
>>
>> With Remus, this jump essentially discards the current incomplete
>> checkpoint received by the backup and restore backup from the
>> last complete checkpoint.
>> This is required for Remus to work and this does not break live
>> migration.
>> It has been around since Xen 4.0.
>>
>> CC: Ian Jackson <ian.jackson@eu.citrix.com>
>> CC: Ian Campbell <ian.campbell@citrix.com>
>>

It is elsewhere in this thread, but Reviewed-by: Andrew Cooper
<andrew.cooper3@citrix.com>

>> CC: Shriram Rajagopalan <rshriram@cs.ubc.ca>
>> Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
>> ---
>>   tools/libxc/xc_domain_restore.c | 13 +++++++++++--
>>   1 file changed, 11 insertions(+), 2 deletions(-)
>>
>> diff --git a/tools/libxc/xc_domain_restore.c
>> b/tools/libxc/xc_domain_restore.c
>> index e73e0a2..b9a56d5 100644
>> --- a/tools/libxc/xc_domain_restore.c
>> +++ b/tools/libxc/xc_domain_restore.c
>> @@ -1783,20 +1783,29 @@ int xc_domain_restore(xc_interface *xch, int
>> io_fd, uint32_t dom,
>>
>>       if ( pagebuf_get(xch, ctx, &pagebuf, io_fd, dom) ) {
>>           PERROR("error when buffering batch, finishing");
>> -        goto out;
>> +        /*
>> +         * Remus: discard the current incomplete checkpoint and restore
>> +         * backup from the last complete checkpoint.
>> +         */
>> +        goto finish;
>>       }
>>       memset(&tmptail, 0, sizeof(tmptail));
>>       tmptail.ishvm = hvm;
>>       if ( buffer_tail(xch, ctx, &tmptail, io_fd, max_vcpu_id, vcpumap,
>>                        ext_vcpucontext, vcpuextstate_size) < 0 ) {
>>           ERROR ("error buffering image tail, finishing");
>> -        goto out;
>> +        /*
>> +         * Remus: discard the current incomplete checkpoint and restore
>> +         * backup from the last complete checkpoint.
>> +         */
>> +        goto finish;
>>       }
>>       tailbuf_free(&tailbuf);
>>       memcpy(&tailbuf, &tmptail, sizeof(tailbuf));
>>
>>       goto loadpages;
>>
>> +  /* With Remus: restore from last complete checkpoint */
>>     finish:
>>       if ( hvm )
>>           goto finish_hvm;
>>
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH v2] fix Remus failover regression

[Hongyang Yang]

Hi

在 08/07/2014 03:43 PM, Andrew Cooper 写道:
> On 07/08/2014 02:16, Hongyang Yang wrote:
>> Ping!
>>
>> 在 07/28/2014 12:03 PM, Yang Hongyang 写道:
>>> commit: c2ba706c
>>> tools/libxc: goto correct label on error paths by Andrew Cooper
>>> broke Remus in Xen 4.4 or earlier versions that has this commit
>>> backported.
>>>
>>> With Remus, this jump essentially discards the current incomplete
>>> checkpoint received by the backup and restore backup from the
>>> last complete checkpoint.
>>> This is required for Remus to work and this does not break live
>>> migration.
>>> It has been around since Xen 4.0.
>>>
>>> CC: Ian Jackson <ian.jackson@eu.citrix.com>
>>> CC: Ian Campbell <ian.campbell@citrix.com>
>>>
>
> It is elsewhere in this thread, but Reviewed-by: Andrew Cooper
> <andrew.cooper3@citrix.com>

Sorry for the late replay (Just back from a vacation).
Thank you for the review! Seems this patch still not applied.
I will add this to the remus patchset anyway because remus failover
won't work without this fix :(

>
>>> CC: Shriram Rajagopalan <rshriram@cs.ubc.ca>
>>> Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
>>> ---
>>>    tools/libxc/xc_domain_restore.c | 13 +++++++++++--
>>>    1 file changed, 11 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/tools/libxc/xc_domain_restore.c
>>> b/tools/libxc/xc_domain_restore.c
>>> index e73e0a2..b9a56d5 100644
>>> --- a/tools/libxc/xc_domain_restore.c
>>> +++ b/tools/libxc/xc_domain_restore.c
>>> @@ -1783,20 +1783,29 @@ int xc_domain_restore(xc_interface *xch, int
>>> io_fd, uint32_t dom,
>>>
>>>        if ( pagebuf_get(xch, ctx, &pagebuf, io_fd, dom) ) {
>>>            PERROR("error when buffering batch, finishing");
>>> -        goto out;
>>> +        /*
>>> +         * Remus: discard the current incomplete checkpoint and restore
>>> +         * backup from the last complete checkpoint.
>>> +         */
>>> +        goto finish;
>>>        }
>>>        memset(&tmptail, 0, sizeof(tmptail));
>>>        tmptail.ishvm = hvm;
>>>        if ( buffer_tail(xch, ctx, &tmptail, io_fd, max_vcpu_id, vcpumap,
>>>                         ext_vcpucontext, vcpuextstate_size) < 0 ) {
>>>            ERROR ("error buffering image tail, finishing");
>>> -        goto out;
>>> +        /*
>>> +         * Remus: discard the current incomplete checkpoint and restore
>>> +         * backup from the last complete checkpoint.
>>> +         */
>>> +        goto finish;
>>>        }
>>>        tailbuf_free(&tailbuf);
>>>        memcpy(&tailbuf, &tmptail, sizeof(tailbuf));
>>>
>>>        goto loadpages;
>>>
>>> +  /* With Remus: restore from last complete checkpoint */
>>>      finish:
>>>        if ( hvm )
>>>            goto finish_hvm;
>>>
>>
>
> .
>

-- 
Thanks,
Yang.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH v2] fix Remus failover regression

[Ian Campbell]

On Thu, 2014-08-07 at 08:43 +0100, Andrew Cooper wrote:
> On 07/08/2014 02:16, Hongyang Yang wrote:
> > Ping!

> It is elsewhere in this thread, but Reviewed-by: Andrew Cooper
> <andrew.cooper3@citrix.com>

Applied. I added "tools: libxc" to the $SUBJECT and rewrapped since the
first para at least seemed to have gotten mangled.

Re: [PATCH v2] fix Remus failover regression

[Ian Campbell]

On Thu, 2014-08-07 at 08:43 +0100, Andrew Cooper wrote:
> On 07/08/2014 02:16, Hongyang Yang wrote:
> > Ping!

> It is elsewhere in this thread, but Reviewed-by: Andrew Cooper
> <andrew.cooper3@citrix.com>

Applied. I added "tools: libxc" to the $SUBJECT and rewrapped since the
first para at least seemed to have gotten mangled.