Xen Benchmarking guidelines

["Nick L. Petroni Jr." <npetroni@cs.umd.edu>]

Hi,

I'm trying to perform some simple benchmarks for an HVM domain and I'm 
wondering if there is a set of guidelines available for configuration 
tweaking etc. VMware gives a set of guidelines along the lines of 
"pre-allocate disks, don't use page sharing, etc." and I'm curious if 
there is an analogous document/resource for Xen.

The main problem I am having is high variance in my benchmark runs, 
specifically SPECCPU2006 workloads in an HVM domain. Sometimes I get 
fast runs of a given workload once or twice, but invariably at least one 
of three consecutive runs is much worse than the best overall time.

Below are some details of my specific problem. I'm most interested in test 
repeatability, but maximizing performance would be great too.

Thanks and sorry if I missed this info on this or other forums.

Best,
nick

Details:

When running SPECCPU2006 in HVM domain, experience high variance in test 
times. Here are some specific results to consider (times given for three 
consecutive runs of each workload, in seconds).

Xen First test:
   403.gcc:         2290, 2290, 2290   <-- low variance, poor performance
   456.hmmer:       3530, 3530, 3530   <-- low variance, poor performance
   458.sjeng:       1340, 2640, 2630   <-- High variance
   462.libquantum:  3340, 3820, 3810   <-- High variance, poor perofrmance

Xen Second test:
   403.gcc:         1540, 2780, 2770   <-- High variance
   456.hmmer:       1770, 3580, 3590   <-- High variance
   458.sjeng:       1360, 1360, 2680   <-- High variance
   462.libquantum:  2300, 3130, 3900   <-- High variance

Bare HW test for comparison:
   403.gcc:         767, 775, 784
   456.hmmer:       1660, 1670, 1640
   458.sjeng:       1190, 1220, 1220
   462.libquantum:  2130, 2080, 2030

HW:
  - IBM/Lenovo T60
  - Core Duo T2500 2.00GHz, VT-x and multicore enabled in BIOS
  - 1.5GB physical memory

Xen:
  - Similar results for both 3.0.2-2 and 3.1.0 stable releases

Dom0:
  - Debian Lenny, default 2.6.18-xen kernel build

HVM domain:
  - 1200MB RAM
  - 1 vcpu
  - file disk image
  - Red Hat 7.3 with default 2.4.18-3 (non-smp) kernel
    (yes this is old, but I'm stuck with this distro for other reasons)
  - SPECCPU2006, base run

In general, I'm using standard benchmarking best practices, i.e., 
disabling unnecessary services in host and guest, etc. Perhaps this is
related to VCPU scheduling? Should I try pinning a vcpu? Please let 
me know if I can provide more details. Any help or insight would be much 
appreciated!