xenpm: opensolaris HVM domU stops getting timer interrupts when C3 used

[Frank Van Der Linden <Frank.Vanderlinden@Sun.COM>]

I've been looking at Xen (3.4.2) power management, and have been trying 
out different settings. One machine I tried was a Toshiba M30 laptop, 
which has an Intel Core 2 Duo CPU:

CPU0: Intel(R) Core(TM)2 Duo CPU     P8400  @ 2.26GHz stepping 06
CPU1: Intel(R) Core(TM)2 Duo CPU     P8400  @ 2.26GHz stepping 06

When dom0 initialized power management, the messages printed are as follows:

Set CPU acpi_id(0) cpuid(0) Px State info:
  _PCT: descriptor=0, length=0, space_id=127, bit_width=0, bit_offset=0, 
reserved=0, address=0
  _PCT: descriptor=0, length=0, space_id=127, bit_width=0, bit_offset=0, 
reserved=0, address=0
  _PSS: state_count=4
  State0: 2261MHz 35000mW 10us 10us 0x927 0x927
  State1: 2260MHz 35000mW 10us 10us 0x4822 0x4822
  State2: 1600MHz 16000mW 10us 10us 0x617 0x617
  State3: 800MHz 14300mW 10us 10us 0x8611 0x8611
  _PSD: num_entries=5 rev=0 domain=0 coord_type=252 num_processors=1
  _PPC: 0
xen_pminfo: @acpi_cpufreq_cpu_init,HARDWARE addr space
CPU 0 initialization completed
Set CPU acpi_id(1) cpuid(1) Px State info:
  _PCT: descriptor=0, length=0, space_id=127, bit_width=0, bit_offset=0, 
reserved=0, address=0
  _PCT: descriptor=0, length=0, space_id=127, bit_width=0, bit_offset=0, 
reserved=0, address=0
  _PSS: state_count=4
  State0: 2261MHz 35000mW 10us 10us 0x927 0x927
  State1: 2260MHz 35000mW 10us 10us 0x4822 0x4822
  State2: 1600MHz 16000mW 10us 10us 0x617 0x617
  State3: 800MHz 14300mW 10us 10us 0x8611 0x8611
  _PSD: num_entries=5 rev=0 domain=1 coord_type=252 num_processors=1
  _PPC: 0
xen_pminfo: @acpi_cpufreq_cpu_init,HARDWARE addr space
CPU 1 initialization completed
cpuid.MWAIT[.eax=40, .ebx=40, .ecx=3, .edx=3122220]
Monitor-Mwait will be used to enter C-1 state
cpuid.MWAIT[.eax=40, .ebx=40, .ecx=3, .edx=3122220]
Monitor-Mwait will be used to enter C-1 state

It seems to mostly work ok, except that an OpenSolaris HVM domU slows 
down to a crawl. When I examined it, I noticed that it wasn't getting 
any timer interrupts. Or, almost none; occasionally the interrupt count 
for the timer jumps, but mostly it stays frozen.

If I limit the maximum C state to C2, things seem ok, but when C3 is 
used, the problems pops up pretty quickly, though not immediately: if I 
initialize the maximum C state to 1, but then change it to C3 when the 
domU is already running, it won't immediately slow down, it might take 
about a minute, during which time the xenpm utility already is showing 
that C3 is being used. So a switch to C3 isn't guaranteed to cause the 
problem, but it eventually will.

I tried other HVM domains, such as Windows, and they don't seem to have 
the problem. But, they might be more "tickless" than OpenSolaris is.

I haven't looked at the PM code much yet, and I haven't tried -unstable, 
but there don't seem to be any changes in -unstable that would fix this.

Is this a known problem? Any hints for diagnosing the problem?

- Frank