slow linux guest time: off by 7sec/hour

[Michael Tokarev <mjt@tls.msk.ru>]

I'm havin yet another fancy prob with time in guests.
It never happened before so far, but now it is like
this for quite some time.

Here's the result of hourly ntpdate invocation on one
of the guests, other guests shows very similar results:

Nov 25 06:17:09 isrv ntpdate[24713]: step time server 192.168.1.15 offset 7.851828 sec
Nov 25 07:17:08 isrv ntpdate[26221]: step time server 192.168.1.15 offset 6.821011 sec
Nov 25 08:17:08 isrv ntpdate[27823]: step time server 192.168.1.15 offset 6.565726 sec
Nov 25 09:17:08 isrv ntpdate[29999]: step time server 192.168.1.15 offset 6.816590 sec
Nov 25 10:17:08 isrv ntpdate[32048]: step time server 192.168.1.15 offset 5.951779 sec
Nov 25 11:17:08 isrv ntpdate[1981]: step time server 192.168.1.15 offset 7.449376 sec
Nov 25 12:17:08 isrv ntpdate[4204]: step time server 192.168.1.15 offset 6.773106 sec
Nov 25 13:17:08 isrv ntpdate[6075]: step time server 192.168.1.15 offset 6.768047 sec
Nov 25 14:17:08 isrv ntpdate[8752]: step time server 192.168.1.15 offset 6.739191 sec
Nov 25 15:17:08 isrv ntpdate[11450]: step time server 192.168.1.15 offset 6.642222 sec

192.168.1.15 is the host where all guests are running.

All guests are linux 2.6.31.[56], i686 or amd64.  All using
kvm_clock as a clock source.

Host is also 2.6.31.6-amd64.  Kvm is qemu-kvm-0.11.0, the
same version as used since at least September this year
(from my repository at http://www.corpit.ru/debian/tls/kvm/).

The issue started when I upgraded the host kernel the last
time.  Before it was 2.6.31.5, now it's 2.6.31.6.

At the same time I also did a few more - seemengly irrelevant -
changes on the host, and enabling some options in the kernel
config - like EEPROM_LEGACY=m and SECURITY_FILE_CAPABILITIES=y.

One of the host changes was glibc upgrade - from 2.7-19 to
2.10.1-5 (debian lenny=>testing).  In theory it should not
influence the time in guests.

I tried adjtimex on guests, but it does not help -- after
calibration it offers adjustments in range 0.05sec/day
"to agree with CMOS clock".  Obviously it's useless.

I switched one guest from kvm_clock to acpi_pm, to see what
will happen.

Since the whole set of machines are production, I've only a
limited ability to test it where reboots are needed.  But if
anyone have any thoughts about all this, please share ;)

Thank you!

/mjt