Re: [libvirt] limit downtime during life migration from xl/virsh

Re: [libvirt] limit downtime during life migration from xl/virsh

[Daniel P. Berrange]

On Mon, Mar 10, 2014 at 03:36:06PM +0100, Olaf Hering wrote:
> Here is my proposal for a new option for virsh and 2 new options for xl:
> 
> [xl | virsh --live] --max-suspend-time N --timeout N VM host
> 
> --max-suspend-time N: as the name suggests, the VM downtime must not be
> longer than specified. The code doing the migration has to estimate the
> transfer speed. If the VM is about to be suspended, it has to check if
> the remaining dirty pages can be transfered within the required
> timeframe. If not, the migration is aborted, the VM continues to run on
> the src host, the new VM on the dst host is destroyed and an error is
> returned.

Ok, this is already supported by the libvirt virDomainMigrateSetMaxDowntime
API. Strangely you can't set it immediately when invoking 'virsh migrate',
only able to set it once running via 'virsh migrate-setmaxdowntime'.

It makes sense to support it as an arg to 'virsh migrate' itself too,
though I suggest you call it '--maxdowntime' for consistent naming
with the API & existing command.

> --timeout N: if a VM is busy and its workload causes many new dirty
> pages the migrate command would take forever. This option is supposed to
> stop the migration attempt if the number of new dirty pages is too high.
> It would change the semantics of "virsh migrate --timeout n", which
> currently forces a suspend (according to the help text).

The '--timeout' arg isn't anything that's part of the libvirt API,
it is implemented exclusively in virsh client code. That said I
still don't think we can change its semantics in the way you
describe. '--timeout' is a rather poor choice of name for what it
does currently, but we're stuck with it.  So for your proposed
semantics, I think we'll have to introduce a separate '--abort N'
argument to virsh.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [libvirt] limit downtime during life migration from xl/virsh

[Jiri Denemark]

On Mon, Mar 10, 2014 at 15:36:06 +0100, Olaf Hering wrote:
> 
> During live migration of VMs from one host to another the VM is
> suspended for an unpredictable amount of time. The actual downtime
> depends on how many new pages will be dirty and the band width to the
> destination host. Since VM memory size grows faster than transfer rates
> the currently available tuneables will cause troubles for workloads
> within the VM which can not handle large timejumps.
> 
> I have already written code to tweak the inner loop doing the actual
> migration work in libxc. But the patchset exposes the details of the
> loop to the cmdline, as such it is not portable nor is it a friendly UI
> for the hostadmin.
> 
> 
> Here is my proposal for a new option for virsh and 2 new options for xl:
> 
> [xl | virsh --live] --max-suspend-time N --timeout N VM host
> 
> --max-suspend-time N: as the name suggests, the VM downtime must not be
> longer than specified. The code doing the migration has to estimate the
> transfer speed. If the VM is about to be suspended, it has to check if
> the remaining dirty pages can be transfered within the required
> timeframe. If not, the migration is aborted, the VM continues to run on
> the src host, the new VM on the dst host is destroyed and an error is
> returned.

Libvirt already has virDomainMigrateSetMaxDowntime API with this
semantics. However, using virsh, one can set it with virsh
migrate-setmaxdowntime command while migration is happening. Not sure if
exposing it as yet another parameter of already quite complicated
migrate command would buy us much.

> --timeout N: if a VM is busy and its workload causes many new dirty
> pages the migrate command would take forever. This option is supposed to
> stop the migration attempt if the number of new dirty pages is too high.
> It would change the semantics of "virsh migrate --timeout n", which
> currently forces a suspend (according to the help text).

This is not acceptable. If you want an option to automatically cancel
migration after a given timeout, you would need to introduce a new
option instead of changing semantics of an existing option.

Jirka

Re: [libvirt] limit downtime during life migration from xl/virsh

[Daniel P. Berrange]

On Mon, Mar 10, 2014 at 03:47:49PM +0100, Jiri Denemark wrote:
> On Mon, Mar 10, 2014 at 15:36:06 +0100, Olaf Hering wrote:
> > 
> > During live migration of VMs from one host to another the VM is
> > suspended for an unpredictable amount of time. The actual downtime
> > depends on how many new pages will be dirty and the band width to the
> > destination host. Since VM memory size grows faster than transfer rates
> > the currently available tuneables will cause troubles for workloads
> > within the VM which can not handle large timejumps.
> > 
> > I have already written code to tweak the inner loop doing the actual
> > migration work in libxc. But the patchset exposes the details of the
> > loop to the cmdline, as such it is not portable nor is it a friendly UI
> > for the hostadmin.
> > 
> > 
> > Here is my proposal for a new option for virsh and 2 new options for xl:
> > 
> > [xl | virsh --live] --max-suspend-time N --timeout N VM host
> > 
> > --max-suspend-time N: as the name suggests, the VM downtime must not be
> > longer than specified. The code doing the migration has to estimate the
> > transfer speed. If the VM is about to be suspended, it has to check if
> > the remaining dirty pages can be transfered within the required
> > timeframe. If not, the migration is aborted, the VM continues to run on
> > the src host, the new VM on the dst host is destroyed and an error is
> > returned.
> 
> Libvirt already has virDomainMigrateSetMaxDowntime API with this
> semantics. However, using virsh, one can set it with virsh
> migrate-setmaxdowntime command while migration is happening. Not sure if
> exposing it as yet another parameter of already quite complicated
> migrate command would buy us much.

I think it is valuable to have it as a parameter to 'migrate' - since
'migrate' blocks, you'd have to open up a second shell to set the
downtime which is kind of tedious.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [libvirt] limit downtime during life migration from xl/virsh

[Olaf Hering]

On Mon, Mar 10, Jiri Denemark wrote:

> Libvirt already has virDomainMigrateSetMaxDowntime API with this
> semantics. However, using virsh, one can set it with virsh
> migrate-setmaxdowntime command while migration is happening. Not sure if
> exposing it as yet another parameter of already quite complicated
> migrate command would buy us much.

How is the existing code to be used?

  virsh migrate-setmaxdowntime N VM
  virsh migrate --live VM host

In other words, is it some value attached to a VM?

Olaf