Re: A Performance Comparison of Hypervisors

[Anthony Liguori <aliguori@linux.vnet.ibm.com>]

Nicholas Lee wrote:
> Obviously not a very fair comparison [1]. I can't see how this was 
> done well at all.

I wonder why you say this.  I thought the benchmark was done very well.  
What we need is more benchmarking, not less.  Unfortunately, VMware 
makes publishing benchmarks difficult as you have to get their approval.

This benchmark tells us something, the question is what does it tell 
us.  Let's take a look at the benchmarks they choose.  SPECcpu2000 and 
SPECjbb2005 are two favorite benchmarks of virtualization vendors.  They 
are favorites because everyone does well under them :-)  Both aren't 
sensitive to PTE update or context switch latency and don't involve IO 
very much.  Even QEMU wouldn't look so bad against these :-)

I'm not familiar with Passmark, but it looks like it's mostly CPU 
bound.  For all of these virtualization friendly workloads, Xen does 
pretty well compared to VMware.   For some of the Passmark bits, Xen 
actually inches out VMware.  Considering we're Open Source, they really 
have no excuse to ever be slower than we are :-)

The compile workload was, IMHO, the most serious of the benchmarks.  
VMware walloped us on that one.  I suspect that's a some shadow paging 
overhead and perhaps some disk IO overhead.

The Netperf results are a tad silly.  They choose Win2k3 for the guest 
OS.  They installed a paravirtual network driver in their guest 
(vmxnet).  However, since no PV network driver is available for Windows 
for Xen 3.0.3, they used emulated IO[1].  Of course performance is going 
to suck.

I would have rather seen the benchmarks done with a Linux guest using 
the PV drivers that are in the tree.

The only embarrassing part is that they weren't able to boot a Win2k3 
guest with SMP support.  I suspect we need either more QA for HVM or a 
better statement of supported guest confirmations.

Regards,

Anthony Liguori

[1] The PV drivers that come in XenEnterprise are, AFAIK, only for 
XenEnterprise.

> VMWare are a bit silly to release stuff like this, just lowers the 
> whole game.
>
>
> [1] http://www.vmware.com/vmtn/resources/711
> ------------------------------------------------------------------------
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>