[Qemu-devel] [RFC] Migration convergence - a suggestion

[Qemu-devel] [RFC] Migration convergence - a suggestion

[Ronen Hod]

[-- Attachment #1: Type: text/plain, Size: 1899 bytes --]

Well the issue is not new, anyhow, following a conversation with Orit ...

Since we want the migration to finish, I believe that the "migration 
speed" parameter alone cannot do the job.
I suggest using two distinct parameters:
1. Migration speed - will be used to limit the network resources utilization
2. aggressionLevel - A number between 0.0 and 1.0, where low values 
imply minimal interruption to the guest, and 1.0 mean that the guest 
will be completely stalled.

In any case the migration will have to do its work and finish given any 
actual migration-speed, so even low aggressionLevel values will 
sometimes imply that the guest will be throttled substantially.

The algorithm:
The aggressionLevel should determine the targetGuest%CPU (how much CPU 
time we want to allocate to the guest)
With aggressionLevel = 1.0, the guest gets no CPU-resources (stalled).
With aggressionLevel = 0.0, the guest gets minGuest%CPU, such that 
migrationRate == dirtyPagesRate. This minGuest%CPU is continuously 
updated based on the running average of the recent samples (more below).

Note that the targetGuest%CPU allocation is continuously updated due to 
changes guest behavior, network congestion, and alike.

Some more details
- minGuest%CPU (i.e., for dirtyPagesRate == migrationRate) is easy to 
calculate as a running average of
   (migrationRate / dirtyPagesRate * guest%CPU)
- There are several methods to calculate the running average, my 
favorite is IIR, where, roughly speaking,
   newVal = 0.99 * oldVal + 0.01 * newSample
- I would use two measures to ensure that there are more migrated pages 
than "dirty" pages.
   1. The running average (based on recent samples) of the migrated 
pages is larger than that of the new dirty pages
   2. The total number of migrated pages so far is larger than the total 
number of new dirty pages.

And yes, many details are still missing.

Ronen.


[-- Attachment #2: Type: text/html, Size: 2630 bytes --]

Re: [Qemu-devel] [RFC] Migration convergence - a suggestion

[Anthony Liguori]

On 12/20/2011 01:06 AM, Ronen Hod wrote:
> Well the issue is not new, anyhow, following a conversation with Orit ...
>
> Since we want the migration to finish, I believe that the "migration speed"
> parameter alone cannot do the job.
> I suggest using two distinct parameters:
> 1. Migration speed - will be used to limit the network resources utilization
> 2. aggressionLevel - A number between 0.0 and 1.0, where low values imply
> minimal interruption to the guest, and 1.0 mean that the guest will be
> completely stalled.
>
> In any case the migration will have to do its work and finish given any actual
> migration-speed, so even low aggressionLevel values will sometimes imply that
> the guest will be throttled substantially.
>
> The algorithm:
> The aggressionLevel should determine the targetGuest%CPU (how much CPU time we
> want to allocate to the guest)

QEMU has no way to limit the guest CPU time.

Regards,

Anthony Liguori

Re: [Qemu-devel] [RFC] Migration convergence - a suggestion

[Ronen Hod]

On 12/20/2011 03:39 PM, Anthony Liguori wrote:
> On 12/20/2011 01:06 AM, Ronen Hod wrote:
>> Well the issue is not new, anyhow, following a conversation with Orit 
>> ...
>>
>> Since we want the migration to finish, I believe that the "migration 
>> speed"
>> parameter alone cannot do the job.
>> I suggest using two distinct parameters:
>> 1. Migration speed - will be used to limit the network resources 
>> utilization
>> 2. aggressionLevel - A number between 0.0 and 1.0, where low values 
>> imply
>> minimal interruption to the guest, and 1.0 mean that the guest will be
>> completely stalled.
>>
>> In any case the migration will have to do its work and finish given 
>> any actual
>> migration-speed, so even low aggressionLevel values will sometimes 
>> imply that
>> the guest will be throttled substantially.
>>
>> The algorithm:
>> The aggressionLevel should determine the targetGuest%CPU (how much 
>> CPU time we
>> want to allocate to the guest)
>
> QEMU has no way to limit the guest CPU time.

Wouldn't any "yield" (sleep / whatever) limit the guest's CPU time, be 
it in qemu or in KVM.
My intention is to suggest an algorithm that is based on guest 
throttling. Looking at the relevant BZs, I do not see how we can avoid 
it. I certainly have no claims regarding the architecture.
Avi and mst, believe that it is better to continuously control the 
guest's CPU from the outside (libvirt) using cgroups. Although less 
responsive to changes, it should still work.
In the meantime, I also discovered that everybody has a different point 
of view regarding the requirements. Regardless, I believe that the same 
basic mechanics (once decided), can do the work
Some relevant configuration "requirements" are:
1. Max bandwidth
2. Min CPU per guest
3. Max guest stall time
4. Max migration time
These requirements will often conflict, and may imply changes in 
behavior over time.

I would also suggest that the management GUI will let the user select 
the aggression-level (or whatever), and display the implication on all 
the other parameters (total-time, %CPU) based on the current behavior of 
the guest and network.

Regards, Ronen

>
> Regards,
>
> Anthony Liguori

Re: [Qemu-devel] [RFC] Migration convergence - a suggestion

[Anthony Liguori]

On 12/21/2011 03:47 AM, Ronen Hod wrote:
> On 12/20/2011 03:39 PM, Anthony Liguori wrote:
>> On 12/20/2011 01:06 AM, Ronen Hod wrote:
>>> Well the issue is not new, anyhow, following a conversation with Orit ...
>>>
>>> Since we want the migration to finish, I believe that the "migration speed"
>>> parameter alone cannot do the job.
>>> I suggest using two distinct parameters:
>>> 1. Migration speed - will be used to limit the network resources utilization
>>> 2. aggressionLevel - A number between 0.0 and 1.0, where low values imply
>>> minimal interruption to the guest, and 1.0 mean that the guest will be
>>> completely stalled.
>>>
>>> In any case the migration will have to do its work and finish given any actual
>>> migration-speed, so even low aggressionLevel values will sometimes imply that
>>> the guest will be throttled substantially.
>>>
>>> The algorithm:
>>> The aggressionLevel should determine the targetGuest%CPU (how much CPU time we
>>> want to allocate to the guest)
>>
>> QEMU has no way to limit the guest CPU time.
>
> Wouldn't any "yield" (sleep / whatever) limit the guest's CPU time, be it in
> qemu or in KVM.

Practically speaking, not really.

> My intention is to suggest an algorithm that is based on guest throttling.
> Looking at the relevant BZs, I do not see how we can avoid it. I certainly have
> no claims regarding the architecture.
> Avi and mst, believe that it is better to continuously control the guest's CPU
> from the outside (libvirt) using cgroups. Although less responsive to changes,
> it should still work.

Yes, that's really the only way to do it I think.  You'll need to use the CFS 
bandwidth controller.

> In the meantime, I also discovered that everybody has a different point of view
> regarding the requirements. Regardless, I believe that the same basic mechanics
> (once decided), can do the work
> Some relevant configuration "requirements" are:
> 1. Max bandwidth
> 2. Min CPU per guest
> 3. Max guest stall time
> 4. Max migration time
> These requirements will often conflict, and may imply changes in behavior over
> time.
>
> I would also suggest that the management GUI will let the user select the
> aggression-level (or whatever), and display the implication on all the other
> parameters (total-time, %CPU) based on the current behavior of the guest and
> network.

It really depends on your management GUI.  If you have a task oriented GUI, you 
are unlikely to have a "live migration" feature.  Rather, you'll have a 
mechanism to enable load balancing (which may use live migration) or you'll have 
an "evacuate node" feature to allow system maintenance.

Your downtime policy really depends on the feature.  The best thing for QEMU to 
do is make sure we expose all of the necessary hooks to allow a wide variety of 
policies to be implemented.

Regards,

Anthony Liguori

>
> Regards, Ronen
>
>>
>> Regards,
>>
>> Anthony Liguori
>
>