stress testing was: Re: blocking Xen 3.X production use: soft lockup bugs

[Steve Traugott <stevegt@TerraLuna.Org>]

On Sat, Aug 05, 2006 at 09:50:29AM +0100, Keir Fraser wrote:
> On 4/8/06 9:21 pm, "Steve Traugott" <stevegt@TerraLuna.Org> wrote:
> > On Thu, Aug 03, 2006 at 09:03:18AM +0100, Keir Fraser wrote:
> >> Also older versions using sedf scheduler (which has now been patched to
> >> avoid this) could end up with domain0 consuming all CPU and starving
> >> other guests, leading to softlockup errors. We haven't seen any such
> >> errors on our own test machines since this was fixed. Of course, that
> >> doesn't mean there aren't problems with other test scenarios!
> > 
> > That is exactly what was happening.  I did more testing yesterday and
> > last night (-testing changeset 9732), and realized that I was only
> > seeing soft lockups on the second of two domU guests, and only when
> > running a heavy load in dom0.  According to 'xm vcpu-list' the second
> > guest was on CPU 0, as was the workload in dom0...  I added more
> > workload processes to consume both CPUs in dom0, and of course when I
> > did that, the first guest ground to a halt and started showing soft
> > lockups as well.
> 
> It's *always* worth trying the tip of 3.0-testing if you're seeing problems
> with a strictly older version. In this case there are about 50 newer
> changesets, many of which are the result of aggressive testing by Suse for
> SLES10. Well worth having.

In between everything else I'm partway through a build of testing tip;
I really need to automate that, including all my local modules, some
stress tests, and so on -- ever since Xen 1.0 we've been using various
combinations of Xen, AFS, DRBD, and soon aoe, in an environment which
is apparently good at finding bugs and regression in all of the
above...  :-}

Because we rely on these things so much it would be in my best
interests to write a "daily build" style stress test harness -- if I
do this, I'd probably do a periodic post similar to Rick's and David's
xm-test results, maybe some web stats and so on.  Does this sound
like something that would be useful to other folks?

Steve
-- 
Stephen G. Traugott  (KG6HDQ)
Managing Partner, TerraLuna LLC
stevegt@TerraLuna.Org -- http://www.t7a.org