Xen performance and Dbench

Xen performance and Dbench

[Dan Magenheimer]

[-- Attachment #1: Type: text/plain, Size: 851 bytes --]

I saw the presentation "Virtualization of Linux Servers" at
OLS last month and it had some nice comparisons of Xen
performance vs a lot of other virtualization/container
technologies:

http://ols.fedoraproject.org/OLS/Reprints-2008/camargos-reprint.pdf

As always with benchmarks, there are questions to ask and
points to quibble, but overall Xen looks quite good...
except on Dbench.  Has anybody else run this benchmark
on Xen and gotten better results?  If not, any thoughts
on why Xen (and all virt solutions) would do poorly on this
benchmark?  And whether Xen can/should be fixed?

===================================
Thanks... for the memory
I really could use more / My throughput's on the floor
The balloon is flat / My swap disk's fat / I've OOM's in store
Overcommitted so much
(with apologies to the late great Bob Hope)

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Xen performance and Dbench

[Todd Deshane]

On Thu, Aug 7, 2008 at 1:18 PM, Dan Magenheimer
<dan.magenheimer@oracle.com> wrote:
> I saw the presentation "Virtualization of Linux Servers" at
> OLS last month and it had some nice comparisons of Xen
> performance vs a lot of other virtualization/container
> technologies:
>
> http://ols.fedoraproject.org/OLS/Reprints-2008/camargos-reprint.pdf
>

Thanks for pointing this one out.


> As always with benchmarks, there are questions to ask and
> points to quibble, but overall Xen looks quite good...
> except on Dbench.  Has anybody else run this benchmark
> on Xen and gotten better results?  If not, any thoughts
> on why Xen (and all virt solutions) would do poorly on this
> benchmark?  And whether Xen can/should be fixed?
>

The original Xen paper [1] and our repeated research paper [2]
showed a decent dbench "score" vs. native.

http://www.cl.cam.ac.uk/research/srg/netos/papers/2003-xensosp.pdf
http://web2.clarkson.edu/class/cs644/xen/files/repeatedxen-usenix04.pdf

dbench is known to have a high standard deviation...

some KVM results for dbench:
http://virt.kernelnewbies.org/KVM/Performance

dbench is an interesting test since it is basically an I/O test (Samba test)
http://samba.org/ftp/tridge/dbench/README

I don't know if dbench is the best test to determine if Xen (or other
virts) need
to be fixed.

Cheers,
Todd


> ===================================
> Thanks... for the memory
> I really could use more / My throughput's on the floor
> The balloon is flat / My swap disk's fat / I've OOM's in store
> Overcommitted so much
> (with apologies to the late great Bob Hope)
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
>



-- 
Todd Deshane
http://todddeshane.net
check out our book: http://runningxen.com

Re: Xen performance and Dbench

[Anthony Liguori]

Todd Deshane wrote:
> dbench is an interesting test since it is basically an I/O test (Samba test)
> http://samba.org/ftp/tridge/dbench/README


dbench is actually *not* an I/O test.  It mostly stresses a filesystems 
interaction with the page cache.  It's heavily threaded and tends to 
scale okay but it's rarely impacted heavily by the underlying storage 
systems I/O performance.  It tends to demonstrate shadow page table SMP 
scalability more than I/O performance.

The OLS paper referenced really had bad methodologies.  If you read 
carefully, their host system was a 2-way system.  They ran all of the 
guests UP though.  Since they didn't do a parallel make, it wasn't very 
obvious for their "kernel build" but it became obvious with dbench since 
there were multiple threads.

So native and "VServer" had access to both CPU cores whereas Xen, KVM, 
et al were only running on a single core.  There's no way dbench is 30% 
of native under Xen.  That should have been a big red flag that 
something was wrong.

Regards,

Anthony Liguori

> I don't know if dbench is the best test to determine if Xen (or other
> virts) need
> to be fixed.
> 
> Cheers,
> Todd
> 
> 
>> ===================================
>> Thanks... for the memory
>> I really could use more / My throughput's on the floor
>> The balloon is flat / My swap disk's fat / I've OOM's in store
>> Overcommitted so much
>> (with apologies to the late great Bob Hope)
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
>>
> 
> 
> 

RE: Xen performance and Dbench

[Dan Magenheimer]

Thanks, Anthony, very illuminating.  However one would
expect that the 1-cpu measurement would be about one-half
of the 2-cpu measurement, not closer to one-fourth (27%),
true?  So even if the methodology is bad, the results
may still be indicating something wrong that should
be looked at?

Todd, the references you provide are several years and
several versions of Xen old.  Have you seen any Xen
dbench results on a more recent Xen version?  While
I agree that dbench might not be the best benchmark
in the world, if some customers use and believe it
(or, worse, believe the OLS paper), it would be nice
to be able to set the record straight.

> -----Original Message-----
> From: Anthony Liguori [mailto:anthony@codemonkey.ws]
> Sent: Thursday, August 07, 2008 1:50 PM
> To: deshantm@gmail.com
> Cc: dan.magenheimer@oracle.com; Xen-Devel (E-mail)
> Subject: Re: Xen performance and Dbench
> 
> 
> Todd Deshane wrote:
> > dbench is an interesting test since it is basically an I/O 
> test (Samba test)
> > http://samba.org/ftp/tridge/dbench/README
> 
> 
> dbench is actually *not* an I/O test.  It mostly stresses a 
> filesystems
> interaction with the page cache.  It's heavily threaded and tends to
> scale okay but it's rarely impacted heavily by the underlying storage
> systems I/O performance.  It tends to demonstrate shadow page 
> table SMP
> scalability more than I/O performance.
> 
> The OLS paper referenced really had bad methodologies.  If you read
> carefully, their host system was a 2-way system.  They ran all of the
> guests UP though.  Since they didn't do a parallel make, it 
> wasn't very
> obvious for their "kernel build" but it became obvious with 
> dbench since
> there were multiple threads.
> 
> So native and "VServer" had access to both CPU cores whereas Xen, KVM,
> et al were only running on a single core.  There's no way 
> dbench is 30%
> of native under Xen.  That should have been a big red flag that
> something was wrong.
> 
> Regards,
> 
> Anthony Liguori
> 
> > I don't know if dbench is the best test to determine if Xen 
> (or other
> > virts) need
> > to be fixed.
> >
> > Cheers,
> > Todd
> >
> >
> >> ===================================
> >> Thanks... for the memory
> >> I really could use more / My throughput's on the floor
> >> The balloon is flat / My swap disk's fat / I've OOM's in store
> >> Overcommitted so much
> >> (with apologies to the late great Bob Hope)
> >> _______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@lists.xensource.com
> >> http://lists.xensource.com/xen-devel
> >>
> >>
> >
> >
> >
> 
>

Re: Xen performance and Dbench

[Todd Deshane]

On Thu, Aug 7, 2008 at 6:23 PM, Dan Magenheimer
<dan.magenheimer@oracle.com> wrote:
> Thanks, Anthony, very illuminating.  However one would
> expect that the 1-cpu measurement would be about one-half
> of the 2-cpu measurement, not closer to one-fourth (27%),
> true?  So even if the methodology is bad, the results
> may still be indicating something wrong that should
> be looked at?
>
> Todd, the references you provide are several years and
> several versions of Xen old.  Have you seen any Xen
> dbench results on a more recent Xen version?  While
> I agree that dbench might not be the best benchmark
> in the world, if some customers use and believe it
> (or, worse, believe the OLS paper), it would be nice
> to be able to set the record straight.
>

I don't know of anything on dbench and Xen.

I don't think it would be too hard to try to re-run their Xen
dbench numbers. We could then not make the mistakes
that Anthony pointed out.

More investigation is definitely needed.

Cheers,
Todd