KVM power usage - high wakeup rates

KVM power usage - high wakeup rates

[Haydn Solomon]

Hi List,

When testing power usage of a windows xp guest using powertop utility,
I'm noticing high wakeup rates. The following is the output of
powertop for an idle windows xp guest.


==============================
===========================
Wakeups-from-idle per second : 2716.6   interval: 10.0s
no ACPI power usage estimate available

Top causes for wakeups:
  52.4% (2434.4)   qemu-system-x86 : sys_timer_settime (posix_timer_fn)
  21.5% (996.7)   qemu-system-x86 : __kvm_migrate_pit_timer (pit_timer_fn)
  15.0% (697.4)       <interrupt> : extra timer interrupt
   4.9% (227.6)      <kernel IPI> : Rescheduling interrupts
   1.6% ( 72.9)   USB device  6-2 : Optical USB Mouse (Logitech)
   1.3% ( 61.7)       <interrupt> : nvidia
   1.3% ( 59.0)       <interrupt> : uhci_hcd:usb4, uhci_hcd:usb6, HDA Intel
===================================================================

I'm getting similar values when running a Fedora 9 guest.

My host is running F9. Details of my /proc/cpuinfo :

processor    : 1
vendor_id    : GenuineIntel
cpu family    : 6
model        : 15
model name    : Intel(R) Core(TM)2 Duo CPU     T7500  @ 2.20GHz
stepping    : 10
cpu MHz        : 2201.000
cache size    : 4096 KB
physical id    : 0
siblings    : 2
core id        : 1
cpu cores    : 2
fpu        : yes
fpu_exception    : yes
cpuid level    : 10
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor
ds_cpl vmx est tm2 ssse3 cx16 xtpr lahf_lm ida
bogomips    : 4387.73
clflush size    : 64
cache_alignment    : 64
address sizes    : 36 bits physical, 48 bits virtual
power management:

The linux image I'm running is Linux localhost.localdomain
2.6.25.4-30.fc9.x86_64 #1 SMP Wed May 21 17:34:18 EDT 2008 x86_64
x86_64 x86_64 GNU/Linux

I'm starting my vm using the following command:

/usr/local/kvm/bin/qemu-system-x86_64 \
-hda windowsxp.img \
-m 650 \
-localtime \
-net user \
-net nic,model=e1000 \
-smb /home/hsolomon \
-std-vga \
-soundhw all \
-usb \
-usbdevice tablet \
-name StandardPC-NAT &

Thanks

Haydn

Re: KVM power usage - high wakeup rates

[Marcelo Tosatti]

On Thu, Jun 19, 2008 at 01:29:39AM -0400, Haydn Solomon wrote:
> Hi List,
> 
> When testing power usage of a windows xp guest using powertop utility,
> I'm noticing high wakeup rates. The following is the output of
> powertop for an idle windows xp guest.
> 
> 
> ==============================
> ===========================
> Wakeups-from-idle per second : 2716.6   interval: 10.0s
> no ACPI power usage estimate available
> 
> Top causes for wakeups:
>   52.4% (2434.4)   qemu-system-x86 : sys_timer_settime (posix_timer_fn)
>   21.5% (996.7)   qemu-system-x86 : __kvm_migrate_pit_timer (pit_timer_fn)
>   15.0% (697.4)       <interrupt> : extra timer interrupt
>    4.9% (227.6)      <kernel IPI> : Rescheduling interrupts
>    1.6% ( 72.9)   USB device  6-2 : Optical USB Mouse (Logitech)
>    1.3% ( 61.7)       <interrupt> : nvidia
>    1.3% ( 59.0)       <interrupt> : uhci_hcd:usb4, uhci_hcd:usb6, HDA Intel
> ===================================================================
> 
> I'm getting similar values when running a Fedora 9 guest.

What difference do you see when closing the VNC client ? 

Re: KVM power usage - high wakeup rates

[Anthony Liguori]

Haydn Solomon wrote:
> Hi List,
>
> When testing power usage of a windows xp guest using powertop utility,
> I'm noticing high wakeup rates. The following is the output of
> powertop for an idle windows xp guest.
>
>
> ==============================
> ===========================
> Wakeups-from-idle per second : 2716.6   interval: 10.0s
> no ACPI power usage estimate available
>
> Top causes for wakeups:
>   52.4% (2434.4)   qemu-system-x86 : sys_timer_settime (posix_timer_fn)
>   21.5% (996.7)   qemu-system-x86 : __kvm_migrate_pit_timer (pit_timer_fn)
>   15.0% (697.4)       <interrupt> : extra timer interrupt
>    4.9% (227.6)      <kernel IPI> : Rescheduling interrupts
>    1.6% ( 72.9)   USB device  6-2 : Optical USB Mouse (Logitech)
>    1.3% ( 61.7)       <interrupt> : nvidia
>    1.3% ( 59.0)       <interrupt> : uhci_hcd:usb4, uhci_hcd:usb6, HDA Intel
> ===================================================================
>
> I'm getting similar values when running a Fedora 9 guest.
>
> My host is running F9. Details of my /proc/cpuinfo :
>
> processor    : 1
> vendor_id    : GenuineIntel
> cpu family    : 6
> model        : 15
> model name    : Intel(R) Core(TM)2 Duo CPU     T7500  @ 2.20GHz
> stepping    : 10
> cpu MHz        : 2201.000
> cache size    : 4096 KB
> physical id    : 0
> siblings    : 2
> core id        : 1
> cpu cores    : 2
> fpu        : yes
> fpu_exception    : yes
> cpuid level    : 10
> wp        : yes
> flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
> syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor
> ds_cpl vmx est tm2 ssse3 cx16 xtpr lahf_lm ida
> bogomips    : 4387.73
> clflush size    : 64
> cache_alignment    : 64
> address sizes    : 36 bits physical, 48 bits virtual
> power management:
>
> The linux image I'm running is Linux localhost.localdomain
> 2.6.25.4-30.fc9.x86_64 #1 SMP Wed May 21 17:34:18 EDT 2008 x86_64
> x86_64 x86_64 GNU/Linux
>
> I'm starting my vm using the following command:
>
> /usr/local/kvm/bin/qemu-system-x86_64 \
> -hda windowsxp.img \
> -m 650 \
> -localtime \
> -net user \
> -net nic,model=e1000 \
> -smb /home/hsolomon \
> -std-vga \
> -soundhw all \
> -usb \
> -usbdevice tablet \
> -name StandardPC-NAT &
>   

USB tablet polls a lot.  You'll probably see far less wake ups if you 
don't use the tablet.

With the default options, my idle winxp guest causes about 550 
wakeups/sec which is pretty close to what I'd expect.  I don't think 
you'll get much better than that without using a dynticks guest.

Regards,

Anthony Liguori

> Thanks
>
> Haydn
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>   

Re: KVM power usage - high wakeup rates

[Haydn Solomon]

I'm not actually using vncserver/client... just SDL.

On Thu, Jun 19, 2008 at 2:26 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
> On Thu, Jun 19, 2008 at 01:29:39AM -0400, Haydn Solomon wrote:
>> Hi List,
>>
>> When testing power usage of a windows xp guest using powertop utility,
>> I'm noticing high wakeup rates. The following is the output of
>> powertop for an idle windows xp guest.
>>
>>
>> ==============================
>> ===========================
>> Wakeups-from-idle per second : 2716.6   interval: 10.0s
>> no ACPI power usage estimate available
>>
>> Top causes for wakeups:
>>   52.4% (2434.4)   qemu-system-x86 : sys_timer_settime (posix_timer_fn)
>>   21.5% (996.7)   qemu-system-x86 : __kvm_migrate_pit_timer (pit_timer_fn)
>>   15.0% (697.4)       <interrupt> : extra timer interrupt
>>    4.9% (227.6)      <kernel IPI> : Rescheduling interrupts
>>    1.6% ( 72.9)   USB device  6-2 : Optical USB Mouse (Logitech)
>>    1.3% ( 61.7)       <interrupt> : nvidia
>>    1.3% ( 59.0)       <interrupt> : uhci_hcd:usb4, uhci_hcd:usb6, HDA Intel
>> ===================================================================
>>
>> I'm getting similar values when running a Fedora 9 guest.
>
> What difference do you see when closing the VNC client ?
>
>

Re: KVM power usage - high wakeup rates

[Haydn Solomon]

If I'm just using SDL, does usb tablet provide any benefit?

On Thu, Jun 19, 2008 at 2:34 PM, Anthony Liguori <anthony@codemonkey.ws> wrote:
> Haydn Solomon wrote:
>>
>> Hi List,
>>
>> When testing power usage of a windows xp guest using powertop utility,
>> I'm noticing high wakeup rates. The following is the output of
>> powertop for an idle windows xp guest.
>>
>>
>> ==============================
>> ===========================
>> Wakeups-from-idle per second : 2716.6   interval: 10.0s
>> no ACPI power usage estimate available
>>
>> Top causes for wakeups:
>>  52.4% (2434.4)   qemu-system-x86 : sys_timer_settime (posix_timer_fn)
>>  21.5% (996.7)   qemu-system-x86 : __kvm_migrate_pit_timer (pit_timer_fn)
>>  15.0% (697.4)       <interrupt> : extra timer interrupt
>>   4.9% (227.6)      <kernel IPI> : Rescheduling interrupts
>>   1.6% ( 72.9)   USB device  6-2 : Optical USB Mouse (Logitech)
>>   1.3% ( 61.7)       <interrupt> : nvidia
>>   1.3% ( 59.0)       <interrupt> : uhci_hcd:usb4, uhci_hcd:usb6, HDA Intel
>> ===================================================================
>>
>> I'm getting similar values when running a Fedora 9 guest.
>>
>> My host is running F9. Details of my /proc/cpuinfo :
>>
>> processor    : 1
>> vendor_id    : GenuineIntel
>> cpu family    : 6
>> model        : 15
>> model name    : Intel(R) Core(TM)2 Duo CPU     T7500  @ 2.20GHz
>> stepping    : 10
>> cpu MHz        : 2201.000
>> cache size    : 4096 KB
>> physical id    : 0
>> siblings    : 2
>> core id        : 1
>> cpu cores    : 2
>> fpu        : yes
>> fpu_exception    : yes
>> cpuid level    : 10
>> wp        : yes
>> flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
>> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
>> syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor
>> ds_cpl vmx est tm2 ssse3 cx16 xtpr lahf_lm ida
>> bogomips    : 4387.73
>> clflush size    : 64
>> cache_alignment    : 64
>> address sizes    : 36 bits physical, 48 bits virtual
>> power management:
>>
>> The linux image I'm running is Linux localhost.localdomain
>> 2.6.25.4-30.fc9.x86_64 #1 SMP Wed May 21 17:34:18 EDT 2008 x86_64
>> x86_64 x86_64 GNU/Linux
>>
>> I'm starting my vm using the following command:
>>
>> /usr/local/kvm/bin/qemu-system-x86_64 \
>> -hda windowsxp.img \
>> -m 650 \
>> -localtime \
>> -net user \
>> -net nic,model=e1000 \
>> -smb /home/hsolomon \
>> -std-vga \
>> -soundhw all \
>> -usb \
>> -usbdevice tablet \
>> -name StandardPC-NAT &
>>
>
> USB tablet polls a lot.  You'll probably see far less wake ups if you don't
> use the tablet.
>
> With the default options, my idle winxp guest causes about 550 wakeups/sec
> which is pretty close to what I'd expect.  I don't think you'll get much
> better than that without using a dynticks guest.
>
> Regards,
>
> Anthony Liguori
>
>> Thanks
>>
>> Haydn
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
>

Re: KVM power usage - high wakeup rates

[Anthony Liguori]

Haydn Solomon wrote:
> If I'm just using SDL, does usb tablet provide any benefit?
>   

You don't have to do explicit key grab.

Regards,

Anthony Liguori

> On Thu, Jun 19, 2008 at 2:34 PM, Anthony Liguori <anthony@codemonkey.ws> wrote:
>   
>> Haydn Solomon wrote:
>>     
>>> Hi List,
>>>
>>> When testing power usage of a windows xp guest using powertop utility,
>>> I'm noticing high wakeup rates. The following is the output of
>>> powertop for an idle windows xp guest.
>>>
>>>
>>> ==============================
>>> ===========================
>>> Wakeups-from-idle per second : 2716.6   interval: 10.0s
>>> no ACPI power usage estimate available
>>>
>>> Top causes for wakeups:
>>>  52.4% (2434.4)   qemu-system-x86 : sys_timer_settime (posix_timer_fn)
>>>  21.5% (996.7)   qemu-system-x86 : __kvm_migrate_pit_timer (pit_timer_fn)
>>>  15.0% (697.4)       <interrupt> : extra timer interrupt
>>>   4.9% (227.6)      <kernel IPI> : Rescheduling interrupts
>>>   1.6% ( 72.9)   USB device  6-2 : Optical USB Mouse (Logitech)
>>>   1.3% ( 61.7)       <interrupt> : nvidia
>>>   1.3% ( 59.0)       <interrupt> : uhci_hcd:usb4, uhci_hcd:usb6, HDA Intel
>>> ===================================================================
>>>
>>> I'm getting similar values when running a Fedora 9 guest.
>>>
>>> My host is running F9. Details of my /proc/cpuinfo :
>>>
>>> processor    : 1
>>> vendor_id    : GenuineIntel
>>> cpu family    : 6
>>> model        : 15
>>> model name    : Intel(R) Core(TM)2 Duo CPU     T7500  @ 2.20GHz
>>> stepping    : 10
>>> cpu MHz        : 2201.000
>>> cache size    : 4096 KB
>>> physical id    : 0
>>> siblings    : 2
>>> core id        : 1
>>> cpu cores    : 2
>>> fpu        : yes
>>> fpu_exception    : yes
>>> cpuid level    : 10
>>> wp        : yes
>>> flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
>>> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
>>> syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor
>>> ds_cpl vmx est tm2 ssse3 cx16 xtpr lahf_lm ida
>>> bogomips    : 4387.73
>>> clflush size    : 64
>>> cache_alignment    : 64
>>> address sizes    : 36 bits physical, 48 bits virtual
>>> power management:
>>>
>>> The linux image I'm running is Linux localhost.localdomain
>>> 2.6.25.4-30.fc9.x86_64 #1 SMP Wed May 21 17:34:18 EDT 2008 x86_64
>>> x86_64 x86_64 GNU/Linux
>>>
>>> I'm starting my vm using the following command:
>>>
>>> /usr/local/kvm/bin/qemu-system-x86_64 \
>>> -hda windowsxp.img \
>>> -m 650 \
>>> -localtime \
>>> -net user \
>>> -net nic,model=e1000 \
>>> -smb /home/hsolomon \
>>> -std-vga \
>>> -soundhw all \
>>> -usb \
>>> -usbdevice tablet \
>>> -name StandardPC-NAT &
>>>
>>>       
>> USB tablet polls a lot.  You'll probably see far less wake ups if you don't
>> use the tablet.
>>
>> With the default options, my idle winxp guest causes about 550 wakeups/sec
>> which is pretty close to what I'd expect.  I don't think you'll get much
>> better than that without using a dynticks guest.
>>
>> Regards,
>>
>> Anthony Liguori
>>
>>     
>>> Thanks
>>>
>>> Haydn
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>       
>>     

Re: KVM power usage - high wakeup rates

[Avi Kivity]

Anthony Liguori wrote:
>
> With the default options, my idle winxp guest causes about 550 
> wakeups/sec which is pretty close to what I'd expect.  I don't think 
> you'll get much better than that without using a dynticks guest.

We keep various timers firing even if the irq lines they are tied into 
are masked.  This generates extra interrupts, and in the case of busy 
guests, extra exits (though the cost is relatively low in that case).

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: KVM power usage - high wakeup rates

[Anders]

Anthony Liguori wrote:
> With the default options, my idle winxp guest causes about 550 
> wakeups/sec which is pretty close to what I'd expect.  I don't think 
> you'll get much better than that without using a dynticks guest.

According to powertop, I had about 15 kvm wakeups/sec with my previous 
setup (nohz, no display). Now I upgraded to kvm-70 and the timer 
interrupt has reappeared so I have this:

Top causes for wakeups:
  25,4% (112,4)      <kernel IPI> : function call interrupts
  22,6% (100,0)   qemu-system-x86 : kvm_arch_vcpu_ioctl_run (pit_timer_fn)
  22,6% (100,0)   qemu-system-x86 : __kvm_migrate_timers (apic_timer_fn)
   8,7% ( 38,5)   qemu-system-x86 : sys_timer_settime (posix_timer_fn)

What could be the cause for this change? The previous kvm was from 
Ubuntu 8.04, there are no changes in the guest.

In general, what are the best host/guest settings for a low number of 
wakeups?


Thanks,
Anders.

Re: KVM power usage - high wakeup rates

[Avi Kivity]

Anders wrote:
> Anthony Liguori wrote:
>> With the default options, my idle winxp guest causes about 550 
>> wakeups/sec which is pretty close to what I'd expect.  I don't think 
>> you'll get much better than that without using a dynticks guest.
>
> According to powertop, I had about 15 kvm wakeups/sec with my previous 
> setup (nohz, no display). Now I upgraded to kvm-70 and the timer 
> interrupt has reappeared so I have this:
>
> Top causes for wakeups:
>  25,4% (112,4)      <kernel IPI> : function call interrupts
>  22,6% (100,0)   qemu-system-x86 : kvm_arch_vcpu_ioctl_run (pit_timer_fn)
>  22,6% (100,0)   qemu-system-x86 : __kvm_migrate_timers (apic_timer_fn)
>   8,7% ( 38,5)   qemu-system-x86 : sys_timer_settime (posix_timer_fn)
>
> What could be the cause for this change? The previous kvm was from 
> Ubuntu 8.04, there are no changes in the guest.
>

Likely, the in-kernel pit.  Try -no-kvm-pit.

> In general, what are the best host/guest settings for a low number of 
> wakeups?

dyntick on both.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: KVM power usage - high wakeup rates

[Anders]

Avi Kivity wrote:
> Anders wrote:
>> According to powertop, I had about 15 kvm wakeups/sec with my 
>> previous setup (nohz, no display). Now I upgraded to kvm-70 and the 
>> timer interrupt has reappeared so I have this:
>>
>> Top causes for wakeups:
>>  25,4% (112,4)      <kernel IPI> : function call interrupts
>>  22,6% (100,0)   qemu-system-x86 : kvm_arch_vcpu_ioctl_run 
>> (pit_timer_fn)
>>  22,6% (100,0)   qemu-system-x86 : __kvm_migrate_timers (apic_timer_fn)
>>   8,7% ( 38,5)   qemu-system-x86 : sys_timer_settime (posix_timer_fn)
>>
>> What could be the cause for this change? The previous kvm was from 
>> Ubuntu 8.04, there are no changes in the guest.
>>
>
> Likely, the in-kernel pit.  Try -no-kvm-pit.

The -no-kvm-pit did change things. However, there is still a periodic tick:

Top causes for wakeups:
  34,7% (102,0)   qemu-system-x86 : sys_timer_settime (posix_timer_fn)
  34,0% (100,0)   qemu-system-x86 : __kvm_migrate_timers (apic_timer_fn)
  13,8% ( 40,7)      <kernel IPI> : function call interrupts

For comparison, this is my normal output. This is with the same kernel 
modules, but the userspace from Ubuntu 8.04:

Top causes for wakeups:
  50,2% ( 66,0)      <kernel IPI> : function call interrupts
  11,3% ( 14,9)               kvm : apic_mmio_write (apic_timer_fn)

>> In general, what are the best host/guest settings for a low number of 
>> wakeups?
>
> dyntick on both.
>

Sure. I guess my question was how to achieve that. Things like the 
-no-kvm-pit advice. Also, I have used "clocksource=tsc acpi=force" on 
the guest to get it to dyntick. But that seems to no longer work. So I 
was wondering whether there was some known configuration that was 
supposed to work.


Anders.