[Qemu-devel] slow xbzrle

[Qemu-devel] slow xbzrle

[Stefan Priebe - Profihost AG]

Hello list,

i'm using 1.2 stable and wanted to use xbzrle but xbzrle is extremely slow.

While trying to transfer a simple VM with 4GB memory through a 10GBe nic 
while running a MySQL (with NO LOAD) it takes up to 10 - 15 minutes.

Remaining is often jumping or just lowering pretty slow.

Is this known or is something wrong?

Greets,
Stefan

Re: [Qemu-devel] slow xbzrle

[Orit Wasserman]

On 10/25/2012 10:12 AM, Stefan Priebe - Profihost AG wrote:
> Hello list,
> 
> i'm using 1.2 stable and wanted to use xbzrle but xbzrle is extremely slow.
> 
> While trying to transfer a simple VM with 4GB memory through a 10GBe nic while running a MySQL (with NO LOAD) it takes up to 10 - 15 minutes.
> 
> Remaining is often jumping or just lowering pretty slow.
This happens if the guest is dirtying his memory faster than the speed qemu is sending it.
It can happen with regular migration if the speed is too low or the downtime is too low.
> 
> Is this known or is something wrong?
My guess this workload migrates fine without XBZRLE so it is not the speed or downtime :).
it could be that the cache size is too small resulting with a lot of cache misses which means
XBZRLE makes things worse and just uses CPU to maintain the cache.
Other option is that this workload changes the pages too much which can result with a lot of over flows 
which result in XBZRLE to waste CPU scanning the pages.
As this is idle mysql my guess it the first scenario.

Can you send the cache size, cache miss and overflow (from 'info migrate')?

Thanks,
Orit
> 
> Greets,
> Stefan
> 

Re: [Qemu-devel] slow xbzrle

[Stefan Priebe - Profihost AG]

Am 25.10.2012 11:44, schrieb Orit Wasserman:
>> Is this known or is something wrong?
> My guess this workload migrates fine without XBZRLE so it is not the speed or downtime :).
> it could be that the cache size is too small resulting with a lot of cache misses which means
> XBZRLE makes things worse and just uses CPU to maintain the cache.
> Other option is that this workload changes the pages too much which can result with a lot of over flows
> which result in XBZRLE to waste CPU scanning the pages.
> As this is idle mysql my guess it the first scenario.
>
> Can you send the cache size, cache miss and overflow (from 'info migrate')?

Example got from a log file (it's the info from info migrate just in 
another format):
migration xbzrle
cachesize: 1073741824
transferred 0
pages 0
cachemiss 1315374
overflow 0

Stefan

Re: [Qemu-devel] slow xbzrle

[Orit Wasserman]

On 10/25/2012 12:35 PM, Stefan Priebe - Profihost AG wrote:
> Am 25.10.2012 11:44, schrieb Orit Wasserman:
>>> Is this known or is something wrong?
>> My guess this workload migrates fine without XBZRLE so it is not the speed or downtime :).
>> it could be that the cache size is too small resulting with a lot of cache misses which means
>> XBZRLE makes things worse and just uses CPU to maintain the cache.
>> Other option is that this workload changes the pages too much which can result with a lot of over flows
>> which result in XBZRLE to waste CPU scanning the pages.
>> As this is idle mysql my guess it the first scenario.
>>
>> Can you send the cache size, cache miss and overflow (from 'info migrate')?
> 
> Example got from a log file (it's the info from info migrate just in another format):
> migration xbzrle
> cachesize: 1073741824
> transferred 0
> pages 0
> cachemiss 1315374
Looks like a lot of cache miss, you can try increasing the cache size (migrate_set_cache_size).
But you should remember that for an idle guest XBZRLE is wasteful,
it is useful for workload that changes the same memory pages frequently.

Orit 
> overflow 0
> 
> Stefan

Re: [Qemu-devel] slow xbzrle

[Stefan Priebe - Profihost AG]

Am 25.10.2012 13:39, schrieb Orit Wasserman:
> On 10/25/2012 12:35 PM, Stefan Priebe - Profihost AG wrote:
>> Am 25.10.2012 11:44, schrieb Orit Wasserman:
>>>> Is this known or is something wrong?
>>> My guess this workload migrates fine without XBZRLE so it is not the speed or downtime :).
>>> it could be that the cache size is too small resulting with a lot of cache misses which means
>>> XBZRLE makes things worse and just uses CPU to maintain the cache.
>>> Other option is that this workload changes the pages too much which can result with a lot of over flows
>>> which result in XBZRLE to waste CPU scanning the pages.
>>> As this is idle mysql my guess it the first scenario.
>>>
>>> Can you send the cache size, cache miss and overflow (from 'info migrate')?
>>
>> Example got from a log file (it's the info from info migrate just in another format):
>> migration xbzrle
>> cachesize: 1073741824
>> transferred 0
>> pages 0
>> cachemiss 1315374
> Looks like a lot of cache miss, you can try increasing the cache size (migrate_set_cache_size).
> But you should remember that for an idle guest XBZRLE is wasteful,
> it is useful for workload that changes the same memory pages frequently.

sure here are some values from a loaded VM (20% cpu load with MySQL) - 
with an increased cache size does not look much better:

Oct 25 14:16:39 migration xbzrle cachesize: 536870912 transferred 0 
pages 0 cachemiss 159634 overflow 0
Oct 25 14:16:41 migration xbzrle cachesize: 536870912 transferred 0 
pages 0 cachemiss 173631 overflow 0
Oct 25 14:16:43 migration xbzrle cachesize: 536870912 transferred 0 
pages 0 cachemiss 187627 overflow 0
Oct 25 14:16:45 migration xbzrle cachesize: 536870912 transferred 0 
pages 0 cachemiss 201624 overflow 0

Greets,
Stefan

Re: [Qemu-devel] slow xbzrle

[Orit Wasserman]

On 10/25/2012 02:18 PM, Stefan Priebe - Profihost AG wrote:
> Am 25.10.2012 13:39, schrieb Orit Wasserman:
>> On 10/25/2012 12:35 PM, Stefan Priebe - Profihost AG wrote:
>>> Am 25.10.2012 11:44, schrieb Orit Wasserman:
>>>>> Is this known or is something wrong?
>>>> My guess this workload migrates fine without XBZRLE so it is not the speed or downtime :).
>>>> it could be that the cache size is too small resulting with a lot of cache misses which means
>>>> XBZRLE makes things worse and just uses CPU to maintain the cache.
>>>> Other option is that this workload changes the pages too much which can result with a lot of over flows
>>>> which result in XBZRLE to waste CPU scanning the pages.
>>>> As this is idle mysql my guess it the first scenario.
>>>>
>>>> Can you send the cache size, cache miss and overflow (from 'info migrate')?
>>>
>>> Example got from a log file (it's the info from info migrate just in another format):
>>> migration xbzrle
>>> cachesize: 1073741824
>>> transferred 0
>>> pages 0
>>> cachemiss 1315374
>> Looks like a lot of cache miss, you can try increasing the cache size (migrate_set_cache_size).
>> But you should remember that for an idle guest XBZRLE is wasteful,
>> it is useful for workload that changes the same memory pages frequently.
> 
> sure here are some values from a loaded VM (20% cpu load with MySQL) - with an increased cache size does not look much better:
> 
> Oct 25 14:16:39 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 159634 overflow 0
> Oct 25 14:16:41 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 173631 overflow 0
> Oct 25 14:16:43 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 187627 overflow 0
> Oct 25 14:16:45 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 201624 overflow 0
It looks like the cache got smaller 1073741824 > 536870912 ...
try 2G cache size 
Orit
> 
> Greets,
> Stefan

Re: [Qemu-devel] slow xbzrle

[Stefan Priebe]

Am 25.10.2012 15:15, schrieb Orit Wasserman:
>>> Looks like a lot of cache miss, you can try increasing the cache size (migrate_set_cache_size).
>>> But you should remember that for an idle guest XBZRLE is wasteful,
>>> it is useful for workload that changes the same memory pages frequently.
>>
>> sure here are some values from a loaded VM (20% cpu load with MySQL) - with an increased cache size does not look much better:
>>
>> Oct 25 14:16:39 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 159634 overflow 0
>> Oct 25 14:16:41 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 173631 overflow 0
>> Oct 25 14:16:43 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 187627 overflow 0
>> Oct 25 14:16:45 migration xbzrle cachesize: 536870912 transferred 0 pages 0 cachemiss 201624 overflow 0
> It looks like the cache got smaller 1073741824 > 536870912 ...
> try 2G cache size
> Orit

Here from another VM running into overflow setting cache size to 50% of mem:
Oct 25 22:43:57 migration xbzrle cachesize: 2147483648 transferred 
36294204 pages 520357 cachemiss 392074 overflow 947

Oct 25 22:44:31 migration xbzrle cachesize: 2147483648 transferred 
45438419 pages 653350 cachemiss 392091 overflow 1024

Oct 25 22:44:55 migration xbzrle cachesize: 2147483648 transferred 
51310028 pages 742877 cachemiss 392096 overflow 1054

Greets,
Stefan