kvm & dyntick

kvm & dyntick

[Avi Kivity]

It occurs to me that kvm could benefit greatly from dyntick:

dyntick-enabled host:
 - generate virtual interrupts at whatever HZ the guest programs its 
timers, be it 100, 250, 1000 or whatever
 - avoid expensive vmexits due to useless timer interrupts

dyntick-enabled guest:
 - reduce the load on the host when the guest is idling
   (currently an idle guest consumes a few percent cpu)

What are the current plans wrt dyntick?  Is it planned for 2.6.21?

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: kvm & dyntick

[Rik van Riel]

Avi Kivity wrote:

> dyntick-enabled guest:
> - reduce the load on the host when the guest is idling
>   (currently an idle guest consumes a few percent cpu)

You do not need dynticks for this actually.  Simple no-tick-on-idle
like Xen has works well enough.

While you're modifying the timer code, you might also want to add
proper accounting for steal time.  Time during which your guest
had a runnable process, but was not actually running itself, should
not be accounted against the currently running process.

I wonder if it would be possible to simply copy some of the timer
code from Xen.  They have the timing quirks worked out very well
and their timer_interrupt() is pretty nice code.

(Now I need to buy myself another VT box so I can help out with KVM :))

http://virt.kernelnewbies.org/ParavirtBenefits has some other features
you may want to have :)))

-- 
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: kvm & dyntick

[Ingo Molnar]

* Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org> wrote:

> It occurs to me that kvm could benefit greatly from dyntick:
> 
> dyntick-enabled host:
> - generate virtual interrupts at whatever HZ the guest programs its 
> timers, be it 100, 250, 1000 or whatever
> - avoid expensive vmexits due to useless timer interrupts
> 
> dyntick-enabled guest:
> - reduce the load on the host when the guest is idling
>   (currently an idle guest consumes a few percent cpu)

yeah. KVM under -rt already works with dynticks enabled on both the host 
and the guest. (but it's more optimal to use a dedicated hypercall to 
set the next guest-interrupt)

> What are the current plans wrt dyntick?  Is it planned for 2.6.21?

yeah, we hope to have it in v2.6.21.

note that s390 (and more recently Xen too) uses a next_timer_interrupt() 
based method to stop the guest tick - which works in terms of reducing 
guest load, but it doesnt stop the host-side interrupt. The highest 
quality approach is to have dynticks on both the host and the guest, and 
this also gives high-resolution timers and a modernized 
time/timer-events subsystem for both the host and the guest.

	Ingo

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: kvm & dyntick

[Ingo Molnar]

* Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org> wrote:

> > dyntick-enabled guest:
> > - reduce the load on the host when the guest is idling
> >   (currently an idle guest consumes a few percent cpu)
> 
> yeah. KVM under -rt already works with dynticks enabled on both the 
> host and the guest. (but it's more optimal to use a dedicated 
> hypercall to set the next guest-interrupt)

using the dynticks code from the -rt kernel makes the overhead of an 
idle guest go down by a factor of 10-15:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2556 mingo     15   0  598m 159m 157m R  1.5  8.0   0:26.20 qemu

( for this to work on my system i have added a 'hyper' clocksource 
  hypercall API for KVM guests to use - this is needed instead of the 
  running-to-slowly TSC. )

	Ingo

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: kvm & dyntick

[Avi Kivity]

Ingo Molnar wrote:
> * Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org> wrote:
>
>   
>>> dyntick-enabled guest:
>>> - reduce the load on the host when the guest is idling
>>>   (currently an idle guest consumes a few percent cpu)
>>>       
>> yeah. KVM under -rt already works with dynticks enabled on both the 
>> host and the guest. (but it's more optimal to use a dedicated 
>> hypercall to set the next guest-interrupt)
>>     
>
> using the dynticks code from the -rt kernel makes the overhead of an 
> idle guest go down by a factor of 10-15:
>
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  2556 mingo     15   0  598m 159m 157m R  1.5  8.0   0:26.20 qemu
>
>   

As usual, great news.

> ( for this to work on my system i have added a 'hyper' clocksource 
>   hypercall API for KVM guests to use - this is needed instead of the 
>   running-to-slowly TSC. )
>   

What's the problem with the TSC?  The only issue I'm aware of is that 
the tsc might go backwards if the vcpu is migrated to another host cpu 
(easily fixed).

A pv clocksource makes sense in any case.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: kvm & dyntick

[Ingo Molnar]

* Avi Kivity <avi@qumranet.com> wrote:

> >( for this to work on my system i have added a 'hyper' clocksource
> >  hypercall API for KVM guests to use - this is needed instead of the 
> >  running-to-slowly TSC. )
> >  
> 
> What's the problem with the TSC?  The only issue I'm aware of is that 
> the tsc might go backwards if the vcpu is migrated to another host cpu 
> (easily fixed).

this is not a problem of KVM - this is a problem of this laptop: its TSC 
stops when going idle. So the TSC is fundamentally unusable for reliable 
timekeeping - and qemu doesnt offer pmtimer emulation to fall back to, 
so the dyntick kernel refused to go high-res under KVM. Once i added the 
hyper clocksource to fall back to, it could utilize the PIT and the 
lapic for clock-events.

	Ingo

RE: [kvm-devel] kvm & dyntick

[Dor Laor]

>* Ingo Molnar <mingo@elte.hu> wrote:
>
>> > dyntick-enabled guest:
>> > - reduce the load on the host when the guest is idling
>> >   (currently an idle guest consumes a few percent cpu)
>>
>> yeah. KVM under -rt already works with dynticks enabled on both the
>> host and the guest. (but it's more optimal to use a dedicated
>> hypercall to set the next guest-interrupt)
>
>using the dynticks code from the -rt kernel makes the overhead of an
>idle guest go down by a factor of 10-15:
>
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 2556 mingo     15   0  598m 159m 157m R  1.5  8.0   0:26.20 qemu
>
>( for this to work on my system i have added a 'hyper' clocksource
>  hypercall API for KVM guests to use - this is needed instead of the
>  running-to-slowly TSC. )
>
>	Ingo

This is great news for PV guests.

Never-the-less we still need to improve our full virtualized guest
support. 
First we need a mechanism (can we use the timeout_granularity?) to
dynamically change the host timer frequency so we can support guests
with 100hz that dynamically change their freq to 1000hz and back.

Afterwards we'll need to compensate the lost alarm signals to the guests
by using one of 
 - hrtimers to inject the lost interrupts for specific guests. The
problem this will increase the overall load.
 - Injecting several virtual irq to the guests one after another (using
interrupt window exit). The question is how the guest will be effected
from this unfair behavior.

Can dyntick help HVMs? Will the answer be the same for guest-dense
hosts? I understood that the main gain of dyn-tick is for idle time.

Re: [kvm-devel] kvm & dyntick

[Thomas Gleixner]

On Fri, 2007-01-12 at 15:25 -0800, Dor Laor wrote:
> This is great news for PV guests.
> 
> Never-the-less we still need to improve our full virtualized guest
> support. 

Full virtualized guests, which have their own dyntick support, are fine
as long as we provide local apic emulation for them.

If a guest does not have that, it will use the periodic mode. There is
no way to circumvent this. We do not know, whether the guest relies on
that periodic interrupt or not.

	tglx

Re: [kvm-devel] kvm & dyntick

[Ingo Molnar]

* Dor Laor <dor.laor@qumranet.com> wrote:

> Afterwards we'll need to compensate the lost alarm signals to the 
> guests by using one of
>  - hrtimers to inject the lost interrupts for specific guests. The 
> problem this will increase the overall load.
>  - Injecting several virtual irq to the guests one after another 
> (using interrupt window exit). The question is how the guest will be 
> effected from this unfair behavior.

well, the most important thing would be to fix qemu to:

 - not use a 1024 /dev/rtc stream of signals as its clock emulation 
   source

i hacked that out of qemu, only to find out that qemu then uses periodic 
itimers. Instead of that it should use one-shot itimers, driven by the 
expiry time of the next clock. I.e. this code in vl.c, in 
host_alarm_handler():

    if (qemu_timer_expired(active_timers[QEMU_TIMER_VIRTUAL],
                           qemu_get_clock(vm_clock)) ||
        qemu_timer_expired(active_timers[QEMU_TIMER_REALTIME],
                           qemu_get_clock(rt_clock))) {

should start an itimer with an expiry time of:

 active_timers[QEMU_TIMER_VIRTUAL]->expire_time - qemu_get_clock(vm_clock)

or:

 active_timers[QEMU_TIMER_REALTIME]->expire_time - qemu_get_clock(rt_clock)

whichever is smaller. Furthermore, whenever timer->expire_time is 
changed in qemu_mod_timer(), this set-the-next-itimer-expiry-time code 
needs to be called. Would anyone like to try that?

this will reduce the host Qemu wakeup rate from 1000-1100/sec to the 
guest's 4-5/sec wakeup rate - resulting in 0.01% CPU overhead from a 
single idle guest. Current unmodified Qemu causes 10-20% CPU overhead 
from a single idle guest.

	Ingo