RE: pvops domu soft lockup under load (more logs)

[Dan Magenheimer <dan.magenheimer@oracle.com>]

(Resending to list... Our corporate email gateway sadly has problems with
a "+" in an email address so I can't reply to Pim directly... if someone
could forward, I would appreciate it.)

Hi Pim --

I haven't read all of your previous postings, but with what you are
seeing, I'd be suspicious that your TSC's may be getting badly out
of sync, possibly due to power management.

Could you try booting xen on the "bad" machines with the
Xen boot parameter: max_cstate=0

There may also be power management settings in the BIOS that
can be changed.

Hope that helps!
Dan

> > -----Original Message-----
> > From: Pim van Riezen [mailto:pi+lists@panelsix.com]
> > Sent: Friday, April 16, 2010 1:56 AM
> > To: Pim van Riezen
> > Cc: Jeremy Fitzhardinge; xen-devel@lists.xensource.com
> > Subject: Re: [Xen-devel] pvops domu soft lockup under load (more
> logs)
> >
> > Oh,
> >
> > On Apr 16, 2010, at 9:37 , Pim van Riezen wrote:
> >
> > > Another datapoint. This customer has similarly loaded VPS machines
> on
> > a number of different hardware nodes. Not all of them had the lockup
> > problem. I applied the jiffies clocksource to all his machines,
> > regardless of their current problem status. After a day without
> > lockups, the customer complained about time drift (ntp was not
> > activated). The guest that had experienced the soft lockups earlier
> had
> > major clock drift and were way ahead:
> > >
> > > 	16 Apr 09:29:26 ntpdate[11236]: step time server 194.109.22.18
> > offset -7337.731686 sec
> > >
> > > That's over 2 hours accumulated in less than 24 hours of uptime.
> The
> > guests that hadn't been excperiencing the lockup issues berfore
> > switching to the jiffies clocksource hadn't drifted that much after
> the
> > switch and were, at most, 120s behind after the same amount of
> runtime.
> >
> > There's more correlation between the guests that had the lockups and
> > those that didn't: the guests that locked up (and now have a way
> speedy
> > jiffies clock) were all on the same hardware platform, with an older
> > Xeon CPU than on the guests that had no issues. I attached cpuinfo
> for
> > both the broken and the non-broken dom0s. All are on Xen-3.4.1
> > (hypervisor-version doesn't seem to affect this issue) and the latest
> > CentOS 2.6.18 dom0-kernel.
> >
> > Cheers,
> > Pim
> >