From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Nicholas Lee" Subject: A Performance Comparison of Hypervisors Date: Sat, 3 Feb 2007 22:34:55 +1300 Message-ID: <2b6116b30702030134j762eed51ud48dc9567d904e97@mail.gmail.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1737499435==" Return-path: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Mime-version: 1.0 Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Xen development list List-Id: xen-devel@lists.xenproject.org --===============1737499435== Content-Type: multipart/alternative; boundary="----=_Part_40919_25935743.1170495295665" ------=_Part_40919_25935743.1170495295665 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Obviously not a very fair comparison [1]. I can't see how this was done well at all. VMWare are a bit silly to release stuff like this, just lowers the whole game. [1] http://www.vmware.com/vmtn/resources/711 ------=_Part_40919_25935743.1170495295665 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Obviously not a very fair comparison [1]. I can't see how this was done well at all.

VMWare are a bit silly to release stuff like this, just lowers the whole game.


[1] http://www.vmware.com/vmtn/resources/711 ------=_Part_40919_25935743.1170495295665-- --===============1737499435== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --===============1737499435==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Nicholas Lee" Subject: Re: A Performance Comparison of Hypervisors Date: Sun, 4 Feb 2007 23:36:14 +1300 Message-ID: <2b6116b30702040236j651929dau7384131b9dfe2dc3@mail.gmail.com> References: <2b6116b30702030134j762eed51ud48dc9567d904e97@mail.gmail.com> <45C4B500.6000304@linux.vnet.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <45C4B500.6000304@linux.vnet.ibm.com> Content-Disposition: inline List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Anthony Liguori Cc: Xen development list List-Id: xen-devel@lists.xenproject.org On 2/4/07, Anthony Liguori wrote: > > I wonder why you say this. I thought the benchmark was done very well. > What we need is more benchmarking, not less. Unfortunately, VMware > makes publishing benchmarks difficult as you have to get their approval. > ... > [1] The PV drivers that come in XenEnterprise are, AFAIK, only for > XenEnterprise. The person who did the benchmark might have done a brilliant job, but it reads like marketing pamphlet for VMware. A good test show be neutral. Show their best product against Xen's lowest offering is just disrespectful. As you say, the netperf test was silly, a good independent tester wouldn't do something like that. Vmware is very mature. I use both it and Xen and they both work well. Xen certainly works better for some jobs than seems to be presented in that report. Nicholas From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mark Williamson Subject: Re: A Performance Comparison of Hypervisors Date: Mon, 5 Feb 2007 05:08:25 +0000 Message-ID: <200702050508.25728.mark.williamson@cl.cam.ac.uk> References: <2b6116b30702030134j762eed51ud48dc9567d904e97@mail.gmail.com> <45C4B500.6000304@linux.vnet.ibm.com> <2b6116b30702040236j651929dau7384131b9dfe2dc3@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <2b6116b30702040236j651929dau7384131b9dfe2dc3@mail.gmail.com> Content-Disposition: inline List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: xen-devel@lists.xensource.com Cc: Anthony Liguori , Nicholas Lee List-Id: xen-devel@lists.xenproject.org > > I wonder why you say this. I thought the benchmark was done very well. > > What we need is more benchmarking, not less. Unfortunately, VMware > > makes publishing benchmarks difficult as you have to get their approval. > > ... > > > [1] The PV drivers that come in XenEnterprise are, AFAIK, only for > > XenEnterprise. > > The person who did the benchmark might have done a brilliant job, but > it reads like marketing pamphlet for VMware. A good test show be > neutral. Show their best product against Xen's lowest offering is just > disrespectful. I had a quick skim through and I'd agree it has a slightly strange feel to it... They do seem to have picked a reasonably low target to beat with 3.0.3, but OTOH it demonstrates that their comercial product is worth paying for over and above Open Source Xen if you're wanting to run Windows in virtual machines. The numbers don't look ridiculous to me, but nor do the set of tests run give a complete picture for all use cases (e.g. paravirt Linux-only, or mixed use could have a very different balance). Good benchmarks don't demonstrate the nice management aspects of VMware ESX, which is another factor to consider. >As you say, the netperf test was silly, a good independent tester > wouldn't do something like that. > > Vmware is very mature. I use both it and Xen and they both work well. > Xen certainly works better for some jobs than seems to be presented in > that report. It'd be nice to see a comparison of VMware ESX vs Open Source Xen vs XenEnterprise that contrasted the strengths of each, rather than playing exclusively to the strengths of one. This could serve as an effective reference for anyone looking into Enterprise virtualisation - unfortunately it would be a huge task even if permission from all parties was obtained :-( Cheers, Mark -- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Henning Sprang" Subject: Re: A Performance Comparison of Hypervisors Date: Mon, 5 Feb 2007 13:58:31 +0100 Message-ID: <5bb00b3f0702050458l2254493cp4930bc6cda487eb9@mail.gmail.com> References: <2b6116b30702030134j762eed51ud48dc9567d904e97@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <2b6116b30702030134j762eed51ud48dc9567d904e97@mail.gmail.com> Content-Disposition: inline List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Nicholas Lee Cc: Xen development list List-Id: xen-devel@lists.xenproject.org On 2/3/07, Nicholas Lee wrote: > Obviously not a very fair comparison [1]. I can't see how this was done well > at all. > > VMWare are a bit silly to release stuff like this, just lowers the whole > game. Is it still true that they on their side forbid publishing performance benchmarks in their EULA? I heard something like that a while ago. I think, here in Germany they would have not many chances to get this through - a lot of the EULA stuff is not likely to go through a juristic check... > > > [1] http://www.vmware.com/vmtn/resources/711 But why must their benchmark stop at 4 vcpus? Don't they wanna test real high performance systems? How about showing a benchmark of xen with 32 cpus against a vmware server,( which can only use 4 of these 32), and benchmarks that need more than 8GB of RAM :) On the other hand, it's also true that Xen itself does not work 100% so very good as all the advertising can make one expect - when looking closer it's not exactly a lie what some XenSource marketing claims say, but some details are just not told... And some things a user expects are really hard to do... Still I am a free software fan, and this adds enough bonus for the time being... Henning From mboxrd@z Thu Jan 1 00:00:00 1970 From: Anthony Liguori Subject: Re: A Performance Comparison of Hypervisors Date: Sat, 03 Feb 2007 10:14:56 -0600 Message-ID: <45C4B500.6000304@linux.vnet.ibm.com> References: <2b6116b30702030134j762eed51ud48dc9567d904e97@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <2b6116b30702030134j762eed51ud48dc9567d904e97@mail.gmail.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Nicholas Lee Cc: Xen development list List-Id: xen-devel@lists.xenproject.org Nicholas Lee wrote: > Obviously not a very fair comparison [1]. I can't see how this was > done well at all. I wonder why you say this. I thought the benchmark was done very well. What we need is more benchmarking, not less. Unfortunately, VMware makes publishing benchmarks difficult as you have to get their approval. This benchmark tells us something, the question is what does it tell us. Let's take a look at the benchmarks they choose. SPECcpu2000 and SPECjbb2005 are two favorite benchmarks of virtualization vendors. They are favorites because everyone does well under them :-) Both aren't sensitive to PTE update or context switch latency and don't involve IO very much. Even QEMU wouldn't look so bad against these :-) I'm not familiar with Passmark, but it looks like it's mostly CPU bound. For all of these virtualization friendly workloads, Xen does pretty well compared to VMware. For some of the Passmark bits, Xen actually inches out VMware. Considering we're Open Source, they really have no excuse to ever be slower than we are :-) The compile workload was, IMHO, the most serious of the benchmarks. VMware walloped us on that one. I suspect that's a some shadow paging overhead and perhaps some disk IO overhead. The Netperf results are a tad silly. They choose Win2k3 for the guest OS. They installed a paravirtual network driver in their guest (vmxnet). However, since no PV network driver is available for Windows for Xen 3.0.3, they used emulated IO[1]. Of course performance is going to suck. I would have rather seen the benchmarks done with a Linux guest using the PV drivers that are in the tree. The only embarrassing part is that they weren't able to boot a Win2k3 guest with SMP support. I suspect we need either more QA for HVM or a better statement of supported guest confirmations. Regards, Anthony Liguori [1] The PV drivers that come in XenEnterprise are, AFAIK, only for XenEnterprise. > VMWare are a bit silly to release stuff like this, just lowers the > whole game. > > > [1] http://www.vmware.com/vmtn/resources/711 > ------------------------------------------------------------------------ > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Petersson, Mats" Subject: RE: A Performance Comparison of Hypervisors Date: Tue, 6 Feb 2007 11:55:40 +0100 Message-ID: <907625E08839C4409CE5768403633E0B018E1948@sefsexmb1.amd.com> References: <45C4B500.6000304@linux.vnet.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Return-path: Content-class: urn:content-classes:message In-Reply-To: <45C4B500.6000304@linux.vnet.ibm.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Anthony Liguori , Nicholas Lee Cc: Xen development list List-Id: xen-devel@lists.xenproject.org =20 > -----Original Message----- > From: xen-devel-bounces@lists.xensource.com=20 > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of=20 > Anthony Liguori > Sent: 03 February 2007 16:15 > To: Nicholas Lee > Cc: Xen development list > Subject: Re: [Xen-devel] A Performance Comparison of Hypervisors >=20 > Nicholas Lee wrote: > > Obviously not a very fair comparison [1]. I can't see how this was=20 > > done well at all. >=20 > I wonder why you say this. I thought the benchmark was done=20 > very well. =20 > What we need is more benchmarking, not less. Unfortunately, VMware=20 > makes publishing benchmarks difficult as you have to get=20 > their approval. >=20 > This benchmark tells us something, the question is what does it tell=20 > us. Let's take a look at the benchmarks they choose. =20 > SPECcpu2000 and=20 > SPECjbb2005 are two favorite benchmarks of virtualization=20 > vendors. They=20 > are favorites because everyone does well under them :-) Both aren't=20 > sensitive to PTE update or context switch latency and don't=20 > involve IO=20 > very much. Even QEMU wouldn't look so bad against these :-) >=20 > I'm not familiar with Passmark, but it looks like it's mostly CPU=20 > bound. For all of these virtualization friendly workloads, Xen does=20 > pretty well compared to VMware. For some of the Passmark bits, Xen=20 > actually inches out VMware. Considering we're Open Source,=20 > they really=20 > have no excuse to ever be slower than we are :-) >=20 > The compile workload was, IMHO, the most serious of the benchmarks. =20 > VMware walloped us on that one. I suspect that's a some=20 > shadow paging=20 > overhead and perhaps some disk IO overhead. >=20 > The Netperf results are a tad silly. They choose Win2k3 for=20 > the guest=20 > OS. They installed a paravirtual network driver in their guest=20 > (vmxnet). However, since no PV network driver is available=20 > for Windows=20 > for Xen 3.0.3, they used emulated IO[1]. Of course=20 > performance is going=20 > to suck. >=20 > I would have rather seen the benchmarks done with a Linux guest using=20 > the PV drivers that are in the tree. >=20 > The only embarrassing part is that they weren't able to boot a Win2k3=20 > guest with SMP support. I suspect we need either more QA for=20 > HVM or a=20 > better statement of supported guest confirmations. I believe official support for SMP HVM guest wasn't in there until 3.0.4, so not really surprising that it doesn't work right in 3.0.3 ;-) [It was, I think, possible to make SMP HVM guests work, but it involved recompiling the BIOS code, which of course is a bit beyond what you'd expect the average reviewer to do...] -- Mats >=20 > Regards, >=20 > Anthony Liguori >=20 > [1] The PV drivers that come in XenEnterprise are, AFAIK, only for=20 > XenEnterprise. >=20 > > VMWare are a bit silly to release stuff like this, just lowers the=20 > > whole game. > > > > > > [1] http://www.vmware.com/vmtn/resources/711 > >=20 > -------------------------------------------------------------- > ---------- > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > > =20 >=20 >=20 > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >=20 >=20 >=20 From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Nicholas Lee" Subject: Re: A Performance Comparison of Hypervisors Date: Wed, 7 Feb 2007 13:25:44 +1300 Message-ID: <2b6116b30702061625g40e7a7blf672a4de6bc4ea56@mail.gmail.com> References: <2b6116b30702030134j762eed51ud48dc9567d904e97@mail.gmail.com> <45C4B500.6000304@linux.vnet.ibm.com> <2b6116b30702040236j651929dau7384131b9dfe2dc3@mail.gmail.com> <200702050508.25728.mark.williamson@cl.cam.ac.uk> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1736071655==" Return-path: In-Reply-To: <200702050508.25728.mark.williamson@cl.cam.ac.uk> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Mime-version: 1.0 Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Mark Williamson Cc: Anthony Liguori , xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org --===============1736071655== Content-Type: multipart/alternative; boundary="----=_Part_20916_5362476.1170807944559" ------=_Part_20916_5362476.1170807944559 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline On 2/5/07, Mark Williamson wrote: > > It'd be nice to see a comparison of VMware ESX vs Open Source Xen vs > XenEnterprise that contrasted the strengths of each, rather than playing > exclusively to the strengths of one. This could serve as an effective > reference for anyone looking into Enterprise virtualisation - > unfortunately > it would be a huge task even if permission from all parties was obtained > :-( A cook off, is probably the best way to deal with this. Good comparisons are based on an expert for one vs an expert for another competing. As long as the results are repeatable by an independent observer. Nicholas ------=_Part_20916_5362476.1170807944559 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline
On 2/5/07, Mark Williamson <mark.williamson@cl.cam.ac.uk> wrote:
It'd be nice to see a comparison of VMware ESX vs Open Source Xen vs
XenEnterprise that contrasted the strengths of each, rather than playing
exclusively to the strengths of one.  This could serve as an effective
reference for anyone looking into Enterprise virtualisation - unfortunately
it would be a huge task even if permission from all parties was obtained :-(

A cook off, is probably the best way to deal with this.

Good comparisons are based on an expert for one vs an expert for another competing. As long as the results are repeatable by an independent observer.

Nicholas

------=_Part_20916_5362476.1170807944559-- --===============1736071655== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --===============1736071655==--