Re: slow guest performance with build load, looking for ideas

[Avi Kivity <avi@redhat.com>]

On 06/19/2009 02:07 AM, Erik Jacobson wrote:
> Hello.  I'll top-post since the quoted text is just for reference.
>
> Sorry the follow-up testing took so long.  We're very low on 5500/Nehalem
> resources at the moment and I had to track down lots of stuff before
> getting to the test.
>
> I ran some tests on a 2-socket, 8-core system.  I wasn't pleased with the
> results for a couple reasons.  One, the issue of it being twice as slow
> as the host with no guest was still present.
>
> However, in trying to make use of this system using Fedora 11, I ran in to
> several issues not directly related to virtualization.  So these test runs
> have that grain of salt.  Example issues...
>
>    

<snip>

>   * In some of the timing runs on this system, the "real time" reported by
>     the time command was off by 10 to 11 times.  Issues were found in
>     the messages file that seemed to relate to this including HUGE time
>     adjustments by NTP and kernel hrtimer 'interrupt too slow' messages.
>     This specific problem seems to be intermittent.
>    

This is on the host? It can easily ruin your day.

> So those are the grains of salt.  I've found that, when doing the timing by
> hand instead of using the time command, the build time seems to be around
> 10 to 12 minutes.  I'm not sure how trustworthy the output from the time
> command are in these trials.  In any event, that's still more than double
> for host alone with no guests.
>
> System:
> SGI XE270, 8-core, Xeon X5570 (Nehalem), Hyperthreading turned off
>    

Shoot, was about to blame hyperthreading.

> Test, as before, was simply this for a kernel build.  The .config file has
> plenty of modules configured.
> time (make -j12&&  make -j12 modules)
>
>
>
> host only, no guest, baseline
> -----------------------------
> trial 1:
> real	5m44.823s
> user	28m45.725s
> sys	5m46.633s
>
> trial 2:
> real	5m34.438s
> user	28m14.347s
> sys	5m41.597s
>
>
> guest, 8 vcpu, 4096 mem, virtio, no cache param, disk device supplied in full
> -----------------------------------------------------------------------------
> trial 1:
> real	125m5.995s
> user	31m23.790s
> sys	9m17.602s
>
>
> trial 2 (changed to 7168 mb memory for the guest):
> real	120m48.431s
> user	14m38.967s
> sys	6m12.437s
>
>
> That's real strange...  The 'time' command is showing whacked out results.
>
> I then watched a run by hand and counted it at about 10 minutes.  However,
> this third run had the proper time!  So whatever the weirdness is, it doesn't
> happen every time:
>
> real	9m49.802s
> user	24m46.009s
> sys	8m10.349s
>
> I decided this could be related to ntp running as I saw this in messages:
> Jun 18 16:34:23 localhost ntpd[1916]: time reset -0.229209 s
> Jun 18 16:34:23 localhost ntpd[1916]: kernel time sync status change 0001
> Jun 18 16:40:17 localhost ntpd[1916]: synchronized to 128.162.244.1, stratum 2
>
> and earlier:
>
> Jun 18 16:19:09 localhost ntpd[1916]: synchronized to 128.162.244.1, stratum 2
> Jun 18 16:19:09 localhost ntpd[1916]: time reset +6609.851122 s
> Jun 18 16:23:39 localhost ntpd[1916]: synchronized to 128.162.244.1, stratum 2
> Jun 18 16:24:04 localhost kernel: hrtimer: interrupt too slow, forcing clock min delta to 62725995 ns
>
>
> I then installed all F11 updates in the guest and tried again (host had
> updates all along).  I got these strange results, strange because of the
> timing difference.  I didn't "watch a non-computer clock" for these.
>    

kvm guests should have an accurate clock without ntp in the guest 
(/sys/.../current_clocksource should say 'kvmclock').


Can you post kvm_stat output during the run?

-- 
error compiling committee.c: too many arguments to function