Re: [PATCH] migration, xen: Fix block image lock issue on live migration

Re: [PATCH] migration, xen: Fix block image lock issue on live migration

[Anthony PERARD]

Hi Julien,

Can I get a release-ack for this patch?

This fix local live migration of HVM guest when the disk backend is
qdisk.  osstest doesn't report a regression because the kernel or the
glibc is just a bit too old.

Thanks,


On Wed, Nov 22, 2017 at 09:45:03AM +0100, Juan Quintela wrote:
> From: Anthony PERARD <anthony.perard@citrix.com>
> 
> When doing a live migration of a Xen guest with libxl, the images for
> block devices are locked by the original QEMU process, and this prevent
> the QEMU at the destination to take the lock and the migration fail.
> 
> >From QEMU point of view, once the RAM of a domain is migrated, there is
> two QMP commands, "stop" then "xen-save-devices-state", at which point a
> new QEMU is spawned at the destination.
> 
> Release locks in "xen-save-devices-state" so the destination can takes
> them, if it's a live migration.
> 
> This patch add the "live" parameter to "xen-save-devices-state" which
> default to true so older version of libxenlight can work with newer
> version of QEMU.
> 
> Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> Reviewed-by: Juan Quintela <quintela@redhat.com>
> Signed-off-by: Juan Quintela <quintela@redhat.com>
> ---
>  migration/savevm.c  | 23 ++++++++++++++++++++++-
>  qapi/migration.json |  6 +++++-
>  2 files changed, 27 insertions(+), 2 deletions(-)
> 
> diff --git a/migration/savevm.c b/migration/savevm.c
> index 192f2d82cd..b7908f62be 100644
> --- a/migration/savevm.c
> +++ b/migration/savevm.c
> @@ -2242,13 +2242,20 @@ int save_snapshot(const char *name, Error **errp)
>      return ret;
>  }
>  
> -void qmp_xen_save_devices_state(const char *filename, Error **errp)
> +void qmp_xen_save_devices_state(const char *filename, bool has_live, bool live,
> +                                Error **errp)
>  {
>      QEMUFile *f;
>      QIOChannelFile *ioc;
>      int saved_vm_running;
>      int ret;
>  
> +    if (!has_live) {
> +        /* live default to true so old version of Xen tool stack can have a
> +         * successfull live migration */
> +        live = true;
> +    }
> +
>      saved_vm_running = runstate_is_running();
>      vm_stop(RUN_STATE_SAVE_VM);
>      global_state_store_running();
> @@ -2263,6 +2270,20 @@ void qmp_xen_save_devices_state(const char *filename, Error **errp)
>      qemu_fclose(f);
>      if (ret < 0) {
>          error_setg(errp, QERR_IO_ERROR);
> +    } else {
> +        /* libxl calls the QMP command "stop" before calling
> +         * "xen-save-devices-state" and in case of migration failure, libxl
> +         * would call "cont".
> +         * So call bdrv_inactivate_all (release locks) here to let the other
> +         * side of the migration take controle of the images.
> +         */
> +        if (live && !saved_vm_running) {
> +            ret = bdrv_inactivate_all();
> +            if (ret) {
> +                error_setg(errp, "%s: bdrv_inactivate_all() failed (%d)",
> +                           __func__, ret);
> +            }
> +        }
>      }
>  
>   the_end:
> diff --git a/qapi/migration.json b/qapi/migration.json
> index bbc4671ded..03f57c9616 100644
> --- a/qapi/migration.json
> +++ b/qapi/migration.json
> @@ -1075,6 +1075,9 @@
>  # data. See xen-save-devices-state.txt for a description of the binary
>  # format.
>  #
> +# @live: Optional argument to ask QEMU to treat this command as part of a live
> +# migration. Default to true. (since 2.11)
> +#
>  # Returns: Nothing on success
>  #
>  # Since: 1.1
> @@ -1086,7 +1089,8 @@
>  # <- { "return": {} }
>  #
>  ##
> -{ 'command': 'xen-save-devices-state', 'data': {'filename': 'str'} }
> +{ 'command': 'xen-save-devices-state',
> +  'data': {'filename': 'str', '*live':'bool' } }
>  
>  ##
>  # @xen-set-replication:
> -- 
> 2.13.6
> 

-- 
Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] migration, xen: Fix block image lock issue on live migration

[Julien Grall]

+ Stefano

On 11/27/2017 03:00 PM, Anthony PERARD wrote:
> Hi Julien,

Hi Anthony,

> 
> Can I get a release-ack for this patch?
> 
> This fix local live migration of HVM guest when the disk backend is
> qdisk.  osstest doesn't report a regression because the kernel or the
> glibc is just a bit too old.

When does that regression happen? I am considering to release Xen 4.10 
soon and would need more details to decide the inclusion of the patch.

Cheers,

> 
> Thanks,
> 
> 
> On Wed, Nov 22, 2017 at 09:45:03AM +0100, Juan Quintela wrote:
>> From: Anthony PERARD <anthony.perard@citrix.com>
>>
>> When doing a live migration of a Xen guest with libxl, the images for
>> block devices are locked by the original QEMU process, and this prevent
>> the QEMU at the destination to take the lock and the migration fail.
>>
>> >From QEMU point of view, once the RAM of a domain is migrated, there is
>> two QMP commands, "stop" then "xen-save-devices-state", at which point a
>> new QEMU is spawned at the destination.
>>
>> Release locks in "xen-save-devices-state" so the destination can takes
>> them, if it's a live migration.
>>
>> This patch add the "live" parameter to "xen-save-devices-state" which
>> default to true so older version of libxenlight can work with newer
>> version of QEMU.
>>
>> Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
>> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
>> Reviewed-by: Juan Quintela <quintela@redhat.com>
>> Signed-off-by: Juan Quintela <quintela@redhat.com>
>> ---
>>   migration/savevm.c  | 23 ++++++++++++++++++++++-
>>   qapi/migration.json |  6 +++++-
>>   2 files changed, 27 insertions(+), 2 deletions(-)
>>
>> diff --git a/migration/savevm.c b/migration/savevm.c
>> index 192f2d82cd..b7908f62be 100644
>> --- a/migration/savevm.c
>> +++ b/migration/savevm.c
>> @@ -2242,13 +2242,20 @@ int save_snapshot(const char *name, Error **errp)
>>       return ret;
>>   }
>>   
>> -void qmp_xen_save_devices_state(const char *filename, Error **errp)
>> +void qmp_xen_save_devices_state(const char *filename, bool has_live, bool live,
>> +                                Error **errp)
>>   {
>>       QEMUFile *f;
>>       QIOChannelFile *ioc;
>>       int saved_vm_running;
>>       int ret;
>>   
>> +    if (!has_live) {
>> +        /* live default to true so old version of Xen tool stack can have a
>> +         * successfull live migration */
>> +        live = true;
>> +    }
>> +
>>       saved_vm_running = runstate_is_running();
>>       vm_stop(RUN_STATE_SAVE_VM);
>>       global_state_store_running();
>> @@ -2263,6 +2270,20 @@ void qmp_xen_save_devices_state(const char *filename, Error **errp)
>>       qemu_fclose(f);
>>       if (ret < 0) {
>>           error_setg(errp, QERR_IO_ERROR);
>> +    } else {
>> +        /* libxl calls the QMP command "stop" before calling
>> +         * "xen-save-devices-state" and in case of migration failure, libxl
>> +         * would call "cont".
>> +         * So call bdrv_inactivate_all (release locks) here to let the other
>> +         * side of the migration take controle of the images.
>> +         */
>> +        if (live && !saved_vm_running) {
>> +            ret = bdrv_inactivate_all();
>> +            if (ret) {
>> +                error_setg(errp, "%s: bdrv_inactivate_all() failed (%d)",
>> +                           __func__, ret);
>> +            }
>> +        }
>>       }
>>   
>>    the_end:
>> diff --git a/qapi/migration.json b/qapi/migration.json
>> index bbc4671ded..03f57c9616 100644
>> --- a/qapi/migration.json
>> +++ b/qapi/migration.json
>> @@ -1075,6 +1075,9 @@
>>   # data. See xen-save-devices-state.txt for a description of the binary
>>   # format.
>>   #
>> +# @live: Optional argument to ask QEMU to treat this command as part of a live
>> +# migration. Default to true. (since 2.11)
>> +#
>>   # Returns: Nothing on success
>>   #
>>   # Since: 1.1
>> @@ -1086,7 +1089,8 @@
>>   # <- { "return": {} }
>>   #
>>   ##
>> -{ 'command': 'xen-save-devices-state', 'data': {'filename': 'str'} }
>> +{ 'command': 'xen-save-devices-state',
>> +  'data': {'filename': 'str', '*live':'bool' } }
>>   
>>   ##
>>   # @xen-set-replication:
>> -- 
>> 2.13.6
>>
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] migration, xen: Fix block image lock issue on live migration

[Anthony PERARD]

On Wed, Nov 29, 2017 at 12:28:39PM +0000, Julien Grall wrote:
> + Stefano
> 
> On 11/27/2017 03:00 PM, Anthony PERARD wrote:
> > Hi Julien,
> 
> Hi Anthony,
> 
> > 
> > Can I get a release-ack for this patch?
> > 
> > This fix local live migration of HVM guest when the disk backend is
> > qdisk.  osstest doesn't report a regression because the kernel or the
> > glibc is just a bit too old.
> 
> When does that regression happen? I am considering to release Xen 4.10 soon
> and would need more details to decide the inclusion of the patch.

That can happen when an API call Open File Description Locks
(F_OFD_GETLK and F_OFD_SETLK) is available. There are since Linux 3.15
and glibc 2.20.

When this API is available, QEMU is going to set a lock on the file open
for a block device. When we try to live migrate a guest and the lock is
set, the new QEMU is going to fail to open the disk with an error
message that say something like "file locked", and migration fail.

The scenario when the lock is an issue are:
- an HVM guest which us qdisk for disk backend. (Like when the disk
  image is in a format different than raw. e.g. qcow2)
- an HVM guest which only use emulated disk.


BUT I can't reproduce the issue with the QEMU we are going to release...
so there is probably a race involve. I guess the patch can wait.

-- 
Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] migration, xen: Fix block image lock issue on live migration

[Julien Grall]

Hi Anthony,

On 29/11/17 15:06, Anthony PERARD wrote:
> On Wed, Nov 29, 2017 at 12:28:39PM +0000, Julien Grall wrote:
>> + Stefano
>>
>> On 11/27/2017 03:00 PM, Anthony PERARD wrote:
>>> Hi Julien,
>>
>> Hi Anthony,
>>
>>>
>>> Can I get a release-ack for this patch?
>>>
>>> This fix local live migration of HVM guest when the disk backend is
>>> qdisk.  osstest doesn't report a regression because the kernel or the
>>> glibc is just a bit too old.
>>
>> When does that regression happen? I am considering to release Xen 4.10 soon
>> and would need more details to decide the inclusion of the patch.
> 
> That can happen when an API call Open File Description Locks
> (F_OFD_GETLK and F_OFD_SETLK) is available. There are since Linux 3.15
> and glibc 2.20.
> 
> When this API is available, QEMU is going to set a lock on the file open
> for a block device. When we try to live migrate a guest and the lock is
> set, the new QEMU is going to fail to open the disk with an error
> message that say something like "file locked", and migration fail.
> 
> The scenario when the lock is an issue are:
> - an HVM guest which us qdisk for disk backend. (Like when the disk
>    image is in a format different than raw. e.g. qcow2)
> - an HVM guest which only use emulated disk.
> 
> 
> BUT I can't reproduce the issue with the QEMU we are going to release...
> so there is probably a race involve. I guess the patch can wait.

I will defer it to Xen 4.11. We might want to Release notes thought.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH] migration, xen: Fix block image lock issue on live migration

[Anthony PERARD]

When doing a live migration of a Xen guest with libxl, the images for
block devices are locked by the original QEMU process, and this prevent
the QEMU at the destination to take the lock and the migration fail.

From QEMU point of view, once the RAM of a domain is migrated, there is
two QMP commands, "stop" then "xen-save-devices-state", at which point a
new QEMU is spawned at the destination.

Release locks in "xen-save-devices-state" so the destination can takes
them.

Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
---
CCing libxl maintainers:
CC: Ian Jackson <ian.jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
 migration/savevm.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/migration/savevm.c b/migration/savevm.c
index 4a88228614..69d904c179 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -2263,6 +2263,20 @@ void qmp_xen_save_devices_state(const char *filename, Error **errp)
     qemu_fclose(f);
     if (ret < 0) {
         error_setg(errp, QERR_IO_ERROR);
+    } else {
+        /* libxl calls the QMP command "stop" before calling
+         * "xen-save-devices-state" and in case of migration failure, libxl
+         * would call "cont".
+         * So call bdrv_inactivate_all (release locks) here to let the other
+         * side of the migration take controle of the images.
+         */
+        if (!saved_vm_running) {
+            ret = bdrv_inactivate_all();
+            if (ret) {
+                error_setg(errp, "%s: bdrv_inactivate_all() failed (%d)",
+                           __func__, ret);
+            }
+        }
     }
 
  the_end:
-- 
Anthony PERARD


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH] migration, xen: Fix block image lock issue on live migration

[Dr. David Alan Gilbert]

Adding in kwolf;  it looks sane to me; Kevin?
If I'm reading this right, this is just after the device state save.

Dave

* Anthony PERARD (anthony.perard@citrix.com) wrote:
> When doing a live migration of a Xen guest with libxl, the images for
> block devices are locked by the original QEMU process, and this prevent
> the QEMU at the destination to take the lock and the migration fail.
> 
> From QEMU point of view, once the RAM of a domain is migrated, there is
> two QMP commands, "stop" then "xen-save-devices-state", at which point a
> new QEMU is spawned at the destination.
> 
> Release locks in "xen-save-devices-state" so the destination can takes
> them.
> 
> Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
> ---
> CCing libxl maintainers:
> CC: Ian Jackson <ian.jackson@eu.citrix.com>
> CC: Wei Liu <wei.liu2@citrix.com>
> ---
>  migration/savevm.c | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
> 
> diff --git a/migration/savevm.c b/migration/savevm.c
> index 4a88228614..69d904c179 100644
> --- a/migration/savevm.c
> +++ b/migration/savevm.c
> @@ -2263,6 +2263,20 @@ void qmp_xen_save_devices_state(const char *filename, Error **errp)
>      qemu_fclose(f);
>      if (ret < 0) {
>          error_setg(errp, QERR_IO_ERROR);
> +    } else {
> +        /* libxl calls the QMP command "stop" before calling
> +         * "xen-save-devices-state" and in case of migration failure, libxl
> +         * would call "cont".
> +         * So call bdrv_inactivate_all (release locks) here to let the other
> +         * side of the migration take controle of the images.
> +         */
> +        if (!saved_vm_running) {
> +            ret = bdrv_inactivate_all();
> +            if (ret) {
> +                error_setg(errp, "%s: bdrv_inactivate_all() failed (%d)",
> +                           __func__, ret);
> +            }
> +        }
>      }
>  
>   the_end:
> -- 
> Anthony PERARD
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH] migration, xen: Fix block image lock issue on live migration

[Kevin Wolf]

Am 02.10.2017 um 21:18 hat Dr. David Alan Gilbert geschrieben:
> Adding in kwolf;  it looks sane to me; Kevin?
> If I'm reading this right, this is just after the device state save.

Is this actual migration? Because the code looks more like it's copied
and adapted from the snapshot code rather than from the actual migration
code.

If Xen doesn't use the standard mechanisms, I don't know what they need
to do. Snapshots don't need to inactivate images, but migration does.
Compared to the normal migration path, this looks very simplistic, so I
wouldn't be surprised if there was more wrong than just file locking.

This looks like it could work as a hack to the problem at hand. Whether
it is a proper solution, I can't say without investing a lot more time.

Kevin

> * Anthony PERARD (anthony.perard@citrix.com) wrote:
> > When doing a live migration of a Xen guest with libxl, the images for
> > block devices are locked by the original QEMU process, and this prevent
> > the QEMU at the destination to take the lock and the migration fail.
> > 
> > From QEMU point of view, once the RAM of a domain is migrated, there is
> > two QMP commands, "stop" then "xen-save-devices-state", at which point a
> > new QEMU is spawned at the destination.
> > 
> > Release locks in "xen-save-devices-state" so the destination can takes
> > them.
> > 
> > Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
> > ---
> > CCing libxl maintainers:
> > CC: Ian Jackson <ian.jackson@eu.citrix.com>
> > CC: Wei Liu <wei.liu2@citrix.com>
> > ---
> >  migration/savevm.c | 14 ++++++++++++++
> >  1 file changed, 14 insertions(+)
> > 
> > diff --git a/migration/savevm.c b/migration/savevm.c
> > index 4a88228614..69d904c179 100644
> > --- a/migration/savevm.c
> > +++ b/migration/savevm.c
> > @@ -2263,6 +2263,20 @@ void qmp_xen_save_devices_state(const char *filename, Error **errp)
> >      qemu_fclose(f);
> >      if (ret < 0) {
> >          error_setg(errp, QERR_IO_ERROR);
> > +    } else {
> > +        /* libxl calls the QMP command "stop" before calling
> > +         * "xen-save-devices-state" and in case of migration failure, libxl
> > +         * would call "cont".
> > +         * So call bdrv_inactivate_all (release locks) here to let the other
> > +         * side of the migration take controle of the images.
> > +         */
> > +        if (!saved_vm_running) {
> > +            ret = bdrv_inactivate_all();
> > +            if (ret) {
> > +                error_setg(errp, "%s: bdrv_inactivate_all() failed (%d)",
> > +                           __func__, ret);
> > +            }
> > +        }
> >      }
> >  
> >   the_end:
> > -- 
> > Anthony PERARD
> > 
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH] migration, xen: Fix block image lock issue on live migration

[Anthony PERARD]

On Wed, Oct 04, 2017 at 03:03:49PM +0200, Kevin Wolf wrote:
> Am 02.10.2017 um 21:18 hat Dr. David Alan Gilbert geschrieben:
> > Adding in kwolf;  it looks sane to me; Kevin?
> > If I'm reading this right, this is just after the device state save.
> 
> Is this actual migration? Because the code looks more like it's copied
> and adapted from the snapshot code rather than from the actual migration
> code.

Well the Xen tool stack takes care of the migration, we only need to
save the device states from QEMU, I guess similair to a snapshot.

> If Xen doesn't use the standard mechanisms, I don't know what they need
> to do. Snapshots don't need to inactivate images, but migration does.
> Compared to the normal migration path, this looks very simplistic, so I
> wouldn't be surprised if there was more wrong than just file locking.

I realize now that if one would want to take a snapshot of a running
Xen guest, this xen-save-devices-state qmp command will be called as
well.

So I can see a few options to better handle snapshots, we could:
- Add a new parameter to xen-save-devices-state, "live_migration" which
  could default to 'true' so older version of Xen will still works.
- Create a new qmp command that sole purpose is to call
  bdrv_inactivate_all, I don't know what else this command would have to
  do.
- or just take this patch.

Thanks.

> This looks like it could work as a hack to the problem at hand. Whether
> it is a proper solution, I can't say without investing a lot more time.


-- 
Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH] migration, xen: Fix block image lock issue on live migration

[Roger Pau Monné]

On Mon, Oct 02, 2017 at 04:30:58PM +0000, Anthony PERARD wrote:
> When doing a live migration of a Xen guest with libxl, the images for
> block devices are locked by the original QEMU process, and this prevent
> the QEMU at the destination to take the lock and the migration fail.
> 
> From QEMU point of view, once the RAM of a domain is migrated, there is
> two QMP commands, "stop" then "xen-save-devices-state", at which point a
> new QEMU is spawned at the destination.
> 
> Release locks in "xen-save-devices-state" so the destination can takes
> them.

What happens if the migration fails on the destination? Will QEMU pick
the lock again when resuming on the source in this case?

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH] migration, xen: Fix block image lock issue on live migration

[Anthony PERARD]

On Tue, Oct 03, 2017 at 12:33:37PM +0100, Roger Pau Monné wrote:
> On Mon, Oct 02, 2017 at 04:30:58PM +0000, Anthony PERARD wrote:
> > When doing a live migration of a Xen guest with libxl, the images for
> > block devices are locked by the original QEMU process, and this prevent
> > the QEMU at the destination to take the lock and the migration fail.
> > 
> > From QEMU point of view, once the RAM of a domain is migrated, there is
> > two QMP commands, "stop" then "xen-save-devices-state", at which point a
> > new QEMU is spawned at the destination.
> > 
> > Release locks in "xen-save-devices-state" so the destination can takes
> > them.
> 
> What happens if the migration fails on the destination? Will QEMU pick
> the lock again when resuming on the source in this case?

Yes, calling the QMP command "cont" to resume the activity makes QEMU
take the lock again, and libxl would call "cont". (I don't think you can
pick this kind of lock ;-). )

-- 
Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel