[Qemu-devel] migration network requirements

[Qemu-devel] migration network requirements

[Alexey Kardashevskiy]

Hi!

Recent bugreports from our testers at IBM about failed migration across USA 
(for example, from NY to TX) made me wonder - what are the actual basic 
requirements/expectations for migration?

In what network is it normally tested (I use 1Gb/10Gb local network without 
routers)?

Is there any reconnect/restart mechanism if the connection dropped 
(libvirt, qemu)?

Is there any document describing network requirements (libvirt, qemu)?

Any ideas are appreciated :) Thanks!

-- 
Alexey

Re: [Qemu-devel] migration network requirements

[Dr. David Alan Gilbert]

* Alexey Kardashevskiy (aik@ozlabs.ru) wrote:
> Hi!
> 
> Recent bugreports from our testers at IBM about failed migration across USA
> (for example, from NY to TX) made me wonder - what are the actual basic
> requirements/expectations for migration?

Oh that's fun.

> In what network is it normally tested (I use 1Gb/10Gb local network without
> routers)?

That's also my main test, but in the end it's just a TCP stream, so lots of bandwidth,
not that fussy about latency.

> Is there any reconnect/restart mechanism if the connection dropped (libvirt,
> qemu)?

No; one unbroken TCP connection; if you want some reconnect/restart
then tunnel the TCP oversomething that handles that.

> Is there any document describing network requirements (libvirt, qemu)?

Don't think so; libvirt has lots of different modes where it can
tunnel stuff as well, see:
  http://www.libvirt.org/migration.html#transporttunnel

> Any ideas are appreciated :) Thanks!

How is it failing?

Dave

> 
> -- 
> Alexey
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] migration network requirements

[Alexey Kardashevskiy]

On 05/20/2015 07:21 PM, Dr. David Alan Gilbert wrote:
> * Alexey Kardashevskiy (aik@ozlabs.ru) wrote:
>> Hi!
>>
>> Recent bugreports from our testers at IBM about failed migration across USA
>> (for example, from NY to TX) made me wonder - what are the actual basic
>> requirements/expectations for migration?
>
> Oh that's fun.

Was it expected to work? Our intranet is not extremely stable :)


>> In what network is it normally tested (I use 1Gb/10Gb local network without
>> routers)?
>
> That's also my main test, but in the end it's just a TCP stream, so lots of bandwidth,
> not that fussy about latency.
>
>> Is there any reconnect/restart mechanism if the connection dropped (libvirt,
>> qemu)?
>
> No; one unbroken TCP connection; if you want some reconnect/restart
> then tunnel the TCP oversomething that handles that.
>
>> Is there any document describing network requirements (libvirt, qemu)?
>
> Don't think so; libvirt has lots of different modes where it can
> tunnel stuff as well, see:
>    http://www.libvirt.org/migration.html#transporttunnel
>
>> Any ideas are appreciated :) Thanks!
>
> How is it failing?

In libvirt it looks like this (which might be unrelated):
Migration: [  0 %]error: Timed out during operation: cannot acquire state 
change lock

or this (which makes sense):
Migration: [  0 %]error: operation failed: Lost connection to destination host

The host kernel network driver can report "BUG: soft lockup" in the network 
driver. It is from quite old QEMU (2.1-ish) and host kernel though, I just 
wanted to know $subj and if it really makes sense to spend more time on this.



-- 
Alexey

Re: [Qemu-devel] migration network requirements

[Dr. David Alan Gilbert]

* Alexey Kardashevskiy (aik@ozlabs.ru) wrote:
> On 05/20/2015 07:21 PM, Dr. David Alan Gilbert wrote:
> >* Alexey Kardashevskiy (aik@ozlabs.ru) wrote:
> >>Hi!
> >>
> >>Recent bugreports from our testers at IBM about failed migration across USA
> >>(for example, from NY to TX) made me wonder - what are the actual basic
> >>requirements/expectations for migration?
> >
> >Oh that's fun.
> 
> Was it expected to work? Our intranet is not extremely stable :)

It should if your network can survive for the whole migration.

> >>In what network is it normally tested (I use 1Gb/10Gb local network without
> >>routers)?
> >
> >That's also my main test, but in the end it's just a TCP stream, so lots of bandwidth,
> >not that fussy about latency.
> >
> >>Is there any reconnect/restart mechanism if the connection dropped (libvirt,
> >>qemu)?
> >
> >No; one unbroken TCP connection; if you want some reconnect/restart
> >then tunnel the TCP oversomething that handles that.
> >
> >>Is there any document describing network requirements (libvirt, qemu)?
> >
> >Don't think so; libvirt has lots of different modes where it can
> >tunnel stuff as well, see:
> >   http://www.libvirt.org/migration.html#transporttunnel
> >
> >>Any ideas are appreciated :) Thanks!
> >
> >How is it failing?
> 
> In libvirt it looks like this (which might be unrelated):
> Migration: [  0 %]error: Timed out during operation: cannot acquire state
> change lock

Someone was asking about a similar case recently which turned out to be a network
problem; the fact it's failing at 0% makes me think it's failing very early.
Check that any firewalls that are in the way between the two hosts are cooperative.
Check that you can ping and ssh from the source to the destination.
Also note that the migration ports tend to be quite high port numbers, so again
check that's happy.

> or this (which makes sense):
> Migration: [  0 %]error: operation failed: Lost connection to destination host
> 
> The host kernel network driver can report "BUG: soft lockup" in the network
> driver. It is from quite old QEMU (2.1-ish) and host kernel though, I just
> wanted to know $subj and if it really makes sense to spend more time on
> this.

It should work; I'd try a really simple qemu only migration first; just a VM
with no guest loaded or anything and see if that works and then work up.

Dave

> -- 
> Alexey
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK