Slow dom0 with Xen 4.0 pv_ops kernel from Debian Squeeze

[Mark Cave-Ayland <mark.cave-ayland@siriusit.co.uk>]

Hi everyone,

We're currently looking at upgrading some of our internal machines from 
Debian Lenny to Squeeze and so we're currently testing the new Xen 4.0 
pv_ops dom0 and domU kernel on one of our development servers. 
Information about these packages can be found at 
http://packages.debian.org/testing/kernel/linux-image-2.6.32-5-xen-amd64 
and http://packages.debian.org/testing/kernel/xen-hypervisor-4.0-amd64 
respectively.

While the upgrade initially went without any problems, we are now 
starting to see some performance issues with the new kernel. After about 
24 hours, the dom0 becomes noticeable sluggish and the domUs tend to go 
through frequent spikes of high network latency (between 2.5 - 4.0s in a 
ping test). In particular, one of the most noticeable symptoms we're 
seeting is that the clock in the dom0 doesn't seem to be working 
correctly, e.g. the following command:

ptah:~# date; sleep 1s; date
Mon Jan 17 19:11:17 GMT 2011
Mon Jan 17 19:11:18 GMT 2011

...actually takes somewhere in the region of nearer 10s rather than 1s :(

Since we are running a set of older Intel Xeons, we did find the 
reference in the Xen 4.0 pv_ops wiki related to hpet timer issues, and 
so we've already tried adding GRUB_CMDLINE_XEN="cpuidle=off" to 
/etc/default/grub and re-running update-grub but that doesn't seem to 
have resolved the issue after a reboot.

Can anyone help figure out exactly what is happening? We're happy to 
provide the relevant diagnostic information if anyone can point us in 
the right direction.


Many thanks,

Mark.

-- 
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs