QEMU P2P migration speed

QEMU P2P migration speed

[Andrey Korolyov]

Hello,

I`ve got strange results during benchmarking migration speed for
different kinds of loads on source/target host: when source host is
'empty', migration takes approx. 30 percent longer than the same for
host already occupied by one VM with CPU overcommit ratio=1.

[src host, three equal vms, each with ability to eat all cores once]
[tgt host, one VM, with same appetite and limitations]

All VMs was put into cgroups with same cpu ceiling and cpu shares values.

Migration time is almost independent of VM RSS(varies by ten percent at
maximum), for situation when VM is active on target host, time is about
85 seconds to migrate 8G between hosts, and when it is turned off,
migration time *increasing* to 120s. For curious ones, frequency
management is completely inactive on both nodes, neither CStates
mechanism. Interconnection is relatively fast (20+Gbit/s by IPoIB).

Anyone have a suggestions on how to possibly explain/fix this?

Re: QEMU P2P migration speed

[Paolo Bonzini]

Il 04/02/2014 18:06, Andrey Korolyov ha scritto:
> Migration time is almost independent of VM RSS(varies by ten percent at
> maximum), for situation when VM is active on target host, time is about
> 85 seconds to migrate 8G between hosts, and when it is turned off,
> migration time *increasing* to 120s. For curious ones, frequency
> management is completely inactive on both nodes, neither CStates
> mechanism. Interconnection is relatively fast (20+Gbit/s by IPoIB).

What version of QEMU?

Paolo

Re: QEMU P2P migration speed

[Andrey Korolyov]

On 02/05/2014 11:27 AM, Paolo Bonzini wrote:
> Il 04/02/2014 18:06, Andrey Korolyov ha scritto:
>> Migration time is almost independent of VM RSS(varies by ten percent at
>> maximum), for situation when VM is active on target host, time is about
>> 85 seconds to migrate 8G between hosts, and when it is turned off,
>> migration time *increasing* to 120s. For curious ones, frequency
>> management is completely inactive on both nodes, neither CStates
>> mechanism. Interconnection is relatively fast (20+Gbit/s by IPoIB).
> 
> What version of QEMU?
> 
> Paolo

Ancie.. ehm, stable - 1.1.2 from wheezy. Should I try 1.6/1.7?

Re: QEMU P2P migration speed

[Paolo Bonzini]

Il 05/02/2014 11:46, Andrey Korolyov ha scritto:
> On 02/05/2014 11:27 AM, Paolo Bonzini wrote:
>> Il 04/02/2014 18:06, Andrey Korolyov ha scritto:
>>> Migration time is almost independent of VM RSS(varies by ten percent at
>>> maximum), for situation when VM is active on target host, time is about
>>> 85 seconds to migrate 8G between hosts, and when it is turned off,
>>> migration time *increasing* to 120s. For curious ones, frequency
>>> management is completely inactive on both nodes, neither CStates
>>> mechanism. Interconnection is relatively fast (20+Gbit/s by IPoIB).
>>
>> What version of QEMU?
>>
>> Paolo
>
> Ancie.. ehm, stable - 1.1.2 from wheezy. Should I try 1.6/1.7?

Yeah, you can checkout the release notes on wiki.qemu.org to find out 
which versions had good improvements.  You can also try compiling 
straight from git, there are more speedups there.

Paolo

Re: QEMU P2P migration speed

[Andrey Korolyov]

On 02/05/2014 07:15 PM, Paolo Bonzini wrote:
> Il 05/02/2014 11:46, Andrey Korolyov ha scritto:
>> On 02/05/2014 11:27 AM, Paolo Bonzini wrote:
>>> Il 04/02/2014 18:06, Andrey Korolyov ha scritto:
>>>> Migration time is almost independent of VM RSS(varies by ten percent at
>>>> maximum), for situation when VM is active on target host, time is about
>>>> 85 seconds to migrate 8G between hosts, and when it is turned off,
>>>> migration time *increasing* to 120s. For curious ones, frequency
>>>> management is completely inactive on both nodes, neither CStates
>>>> mechanism. Interconnection is relatively fast (20+Gbit/s by IPoIB).
>>>
>>> What version of QEMU?
>>>
>>> Paolo
>>
>> Ancie.. ehm, stable - 1.1.2 from wheezy. Should I try 1.6/1.7?
> 
> Yeah, you can checkout the release notes on wiki.qemu.org to find out
> which versions had good improvements.  You can also try compiling
> straight from git, there are more speedups there.
> 
> Paolo
> 

Took and build 1.6.2 and faced a problem - after a couple of bounce
iterations of migration (1->2->1->2) VM is not able to migrate anymore
back in a probabilistic manner with an error 'internal error unexpected
migration status in setup'. Error may disappear over a time, or may not
disappear at all and it may took a lot of tries in a row to succeed.
There are no obvious hints with default logging level in libvirt/qemu
logs and seemingly libvirt is not a cause because "accumulated error
state" preserves over service restarts. Also every VM is affected, not
ones which are experiencing multiple migration actions. Error happens on
3rd-5th second of the migration procedure, if it may help.

What is more interesting, the original counter-intuitive behavior is not
disappeared but increased relative span: 25 vs 70 seconds for fully
commited 8G VM. I am suspecting some mechanism falling back to the idle
and dropping overall performance therefore but can not image one beyond
standard freq/cstates which are definitely turned off.

Re: QEMU P2P migration speed

[Paolo Bonzini]

Il 06/02/2014 14:40, Andrey Korolyov ha scritto:
> Took and build 1.6.2 and faced a problem - after a couple of bounce
> iterations of migration (1->2->1->2) VM is not able to migrate anymore
> back in a probabilistic manner with an error 'internal error unexpected
> migration status in setup'. Error may disappear over a time, or may not
> disappear at all and it may took a lot of tries in a row to succeed.
> There are no obvious hints with default logging level in libvirt/qemu
> logs and seemingly libvirt is not a cause because "accumulated error
> state" preserves over service restarts. Also every VM is affected, not
> ones which are experiencing multiple migration actions. Error happens on
> 3rd-5th second of the migration procedure, if it may help.

You need to update libvirt too.

Paolo

Re: QEMU P2P migration speed

[Andrey Korolyov]

On 02/07/2014 12:14 PM, Paolo Bonzini wrote:
> Il 06/02/2014 14:40, Andrey Korolyov ha scritto:
>> Took and build 1.6.2 and faced a problem - after a couple of bounce
>> iterations of migration (1->2->1->2) VM is not able to migrate anymore
>> back in a probabilistic manner with an error 'internal error unexpected
>> migration status in setup'. Error may disappear over a time, or may not
>> disappear at all and it may took a lot of tries in a row to succeed.
>> There are no obvious hints with default logging level in libvirt/qemu
>> logs and seemingly libvirt is not a cause because "accumulated error
>> state" preserves over service restarts. Also every VM is affected, not
>> ones which are experiencing multiple migration actions. Error happens on
>> 3rd-5th second of the migration procedure, if it may help.
> 
> You need to update libvirt too.
> 
> Paolo

Ok, I will do, but looks like libvirt version(1.0.2) in not relevant -
it meets criteria set by debian packagers and again, 'broken state' is
not relevant to the libvirt state history, it more likely to be qemu/kvm
problem.

Re: QEMU P2P migration speed

[Paolo Bonzini]

Il 07/02/2014 14:07, Andrey Korolyov ha scritto:
> Ok, I will do, but looks like libvirt version(1.0.2) in not relevant -
> it meets criteria set by debian packagers and again, 'broken state' is
> not relevant to the libvirt state history, it more likely to be qemu/kvm
> problem.

It is relevant, qemu introduced a new migration status before "active" 
("setup") and libvirt doesn't recognize it.

Paolo

Re: QEMU P2P migration speed

[Paolo Bonzini]

Il 07/02/2014 14:07, Andrey Korolyov ha scritto:
> Ok, I will do, but looks like libvirt version(1.0.2) in not relevant -
> it meets criteria set by debian packagers

Then Debian's qemu packaging it's wrong, QEMU 1.6 or newer should conflict
with libvirt <1.2.0.

> and again, 'broken state' is
> not relevant to the libvirt state history, it more likely to be qemu/kvm
> problem.

It is relevant, qemu introduced a new migration status before "active" 
("setup") and libvirt doesn't recognize it.  That's why you need at
least 1.2.0.

Paolo

Re: QEMU P2P migration speed

[Michael Tokarev]

07.02.2014 19:32, Paolo Bonzini wrote:
> Il 07/02/2014 14:07, Andrey Korolyov ha scritto:
>> Ok, I will do, but looks like libvirt version(1.0.2) in not relevant -
>> it meets criteria set by debian packagers
> 
> Then Debian's qemu packaging it's wrong, QEMU 1.6 or newer should conflict
> with libvirt <1.2.0.

I've no idea when qemu breaks libvirt.  I found out - just by a chance -
that qemu 1.3+ breaks libvirt <1.0, and I stated this in the deps.
But the thing that 1.6 requires libvirt 1.2 is news for me.

I'll add this requiriment to the debian package.

At any rate, there's no libvirt 1.0 in debian.  Current stable has 0.9,
and current testing has 1.2.1, and this version is also available in
backports for stable.  1.2 was the first version past 0.9 which were
backproted to stable.  There's no other versions of libvirt in debian.

So whomever installed that mess did that on their own, it is definitely
not supported on debian ;)

Thanks,

/mjt

>> and again, 'broken state' is
>> not relevant to the libvirt state history, it more likely to be qemu/kvm
>> problem.
> 
> It is relevant, qemu introduced a new migration status before "active" 
> ("setup") and libvirt doesn't recognize it.  That's why you need at
> least 1.2.0.
> 
> Paolo
> 

Re: QEMU P2P migration speed

[Andrey Korolyov]

On 02/07/2014 07:32 PM, Paolo Bonzini wrote:
> Il 07/02/2014 14:07, Andrey Korolyov ha scritto:
>> Ok, I will do, but looks like libvirt version(1.0.2) in not relevant -
>> it meets criteria set by debian packagers
> 
> Then Debian's qemu packaging it's wrong, QEMU 1.6 or newer should conflict
> with libvirt <1.2.0.
> 
>> and again, 'broken state' is
>> not relevant to the libvirt state history, it more likely to be qemu/kvm
>> problem.
> 
> It is relevant, qemu introduced a new migration status before "active" 
> ("setup") and libvirt doesn't recognize it.  That's why you need at
> least 1.2.0.
> 
> Paolo
> 

Thanks, both issues - with reverted CPU dependency and with migration
itself went away.