Discordant results between UnixBench and nBench

Discordant results between UnixBench and nBench

[Matthieu Olivier]

All,

I am performing a study about KVM.
(Shame on me, I did not check the version of kvm module. All I know is
that I was using the last Red Hat EL 5.4. Anyway).

 I used a DELL server 2950, with:
- 1 processor Xeon L5240 (3 Ghz, 6 Mo cache)
- 8 GB of RAM
- 4 disks SAS 15k in RAID-1

I created a procedure consist to launch a benchmark on several
virtual machine at the same time. Each session consist in testing X VM,
where X comes from 1 to 8. With nbench, I collected all results and
extracted an average value of results for each VM. Then, I multiplied
these average values by the number of VM in test, getting one unique
and generic value per session of test.
I did nearly the same with unixbench.

On the host side, I launched the same tests, except that instead of
running them in separate VM, I just launched several threads at the
same time thanks to a script.

My main purpose is watching the overhead due to these different
hypervisors, according bechmark results. I am not really in trouble
with KVM itself, but results of my benchmarks:

-> http://img121.imageshack.us/img121/9279/nbench.png
-> http://img704.imageshack.us/img704/953/unixbench.png

Indeed, where I have all results growing from 1 VM tested to 8 VM, we
can see the host growing from 1 to 8 VM. which seems good. But KVM
only increases from 1 to 4 VM, and then stays on the same order of
results, without growing anymore.
The situation is inverse on Unixbench, where KVM seems to
follow host's performances.

So now I wonder which test is reliable. Is unixbench right? It is
possible, I don't know why KVM could be so different from the host
since it can use hardware CPU instructions for virtualization. In the
other hand, nbench is in fact doing 3 tests(Fourrier, Neural Net,
LU decomposition) and unixbench only 1 (whetstone).
This last test could be outdated and therefore nbench in
the truth?

I've got nearly the same problems with integers tests (nbench vs
unixbench) and cache memory (nbench vs cachebench).

Anyone have got an idea?

--
Matthieu Olivier

Re: Discordant results between UnixBench and nBench

[Avi Kivity]

On 12/23/2009 11:43 AM, Matthieu Olivier wrote:
> All,
>
> I am performing a study about KVM.
> (Shame on me, I did not check the version of kvm module. All I know is
> that I was using the last Red Hat EL 5.4. Anyway).
>
>   I used a DELL server 2950, with:
> - 1 processor Xeon L5240 (3 Ghz, 6 Mo cache)
> - 8 GB of RAM
> - 4 disks SAS 15k in RAID-1
>
> I created a procedure consist to launch a benchmark on several
> virtual machine at the same time. Each session consist in testing X VM,
> where X comes from 1 to 8. With nbench, I collected all results and
> extracted an average value of results for each VM. Then, I multiplied
> these average values by the number of VM in test, getting one unique
> and generic value per session of test.
> I did nearly the same with unixbench.
>
> On the host side, I launched the same tests, except that instead of
> running them in separate VM, I just launched several threads at the
> same time thanks to a script.
>
> My main purpose is watching the overhead due to these different
> hypervisors, according bechmark results. I am not really in trouble
> with KVM itself, but results of my benchmarks:
>
> ->  http://img121.imageshack.us/img121/9279/nbench.png
> ->  http://img704.imageshack.us/img704/953/unixbench.png
>
> Indeed, where I have all results growing from 1 VM tested to 8 VM, we
> can see the host growing from 1 to 8 VM. which seems good. But KVM
> only increases from 1 to 4 VM, and then stays on the same order of
> results, without growing anymore.
> The situation is inverse on Unixbench, where KVM seems to
> follow host's performances.
>    

What does 'top' show for nbench with 8 guests?  How about kvm_stat?

Are the results different for the 8 guests, or are they all making the 
same progress?

-- 
error compiling committee.c: too many arguments to function

Re: Discordant results between UnixBench and nBench

[Matthieu Olivier]

On Wed, Dec 23, 2009 at 10:53 AM, Avi Kivity <avi@redhat.com> wrote:
> On 12/23/2009 11:43 AM, Matthieu Olivier wrote:
>>
>> All,
>>
>> I am performing a study about KVM.
>> (Shame on me, I did not check the version of kvm module. All I know is
>> that I was using the last Red Hat EL 5.4. Anyway).
>>
>>  I used a DELL server 2950, with:
>> - 1 processor Xeon L5240 (3 Ghz, 6 Mo cache)
>> - 8 GB of RAM
>> - 4 disks SAS 15k in RAID-1
>>
>> I created a procedure consist to launch a benchmark on several
>> virtual machine at the same time. Each session consist in testing X VM,
>> where X comes from 1 to 8. With nbench, I collected all results and
>> extracted an average value of results for each VM. Then, I multiplied
>> these average values by the number of VM in test, getting one unique
>> and generic value per session of test.
>> I did nearly the same with unixbench.
>>
>> On the host side, I launched the same tests, except that instead of
>> running them in separate VM, I just launched several threads at the
>> same time thanks to a script.
>>
>> My main purpose is watching the overhead due to these different
>> hypervisors, according bechmark results. I am not really in trouble
>> with KVM itself, but results of my benchmarks:
>>
>> ->  http://img121.imageshack.us/img121/9279/nbench.png
>> ->  http://img704.imageshack.us/img704/953/unixbench.png
>>
>> Indeed, where I have all results growing from 1 VM tested to 8 VM, we
>> can see the host growing from 1 to 8 VM. which seems good. But KVM
>> only increases from 1 to 4 VM, and then stays on the same order of
>> results, without growing anymore.
>> The situation is inverse on Unixbench, where KVM seems to
>> follow host's performances.
>>
>
> What does 'top' show for nbench with 8 guests?  How about kvm_stat?
>
> Are the results different for the 8 guests, or are they all making the same
> progress?
>
> --
> error compiling committee.c: too many arguments to function

To tell the truth, I had to test some others hypervisors, so now I
don't have KVM installed anymore.
I used top for following bench progress on 2 or 3 VM, but
unfortunately I did not catch their output in a file, neither I used
kvm_stat (by ignorance of its existence).

But I still have every results of every tests of nbench and unixbench.
From 1 up to 4 VM, each VM have the same results. Indeed, it's because
the Xeon L5240 is a quad core CPU (i forgot to mention that before).
Over 4 tests, scores are still regulars, but decrease slowly as we add
VMs. Here is the result of each VM, when I have tested 7 VM at the
same time with nbench:


-- VM 1:
FOURIER                      15637
NEURAL NET                 30.58
LU DECOMPOSITION     985.18


-- VM 2:
FOURIER                         15512
NEURAL NET                   30.443
LU DECOMPOSITION       1398.1


-- VM 3:
FOURIER                         15374
NEURAL NET                    48.382
LU DECOMPOSITION        1954.6

-- VM 4:
FOURIER                         16159
NEURAL NET                    37.007
LU DECOMPOSITION        1767.1

-- VM 5:
FOURIER                         15196
NEURAL NET                   45.166
LU DECOMPOSITION       1981.1

-- VM 6:
FOURIER                         20948
NEURAL NET                   60.256
LU DECOMPOSITION       1923.1

-- VM 7:
FOURIER                         15227
NEURAL NET                   47.459
LU DECOMPOSITION       1972


According to nbench documentation, i recall that each test is run five
times, and these five scores are
averaged. So the output nbench shows us is already a average value.
Unixbench did the same, i used an option to run each tests 4 times, in
order to reduce exotic values.

I'm not sure i have been clear about how I had my results, so tell me
if you want I re-explain this point.

-- 
Matthieu Olivier

Re: Discordant results between UnixBench and nBench

[Avi Kivity]

On 12/23/2009 12:35 PM, Matthieu Olivier wrote:
>
> To tell the truth, I had to test some others hypervisors, so now I
> don't have KVM installed anymore.
> I used top for following bench progress on 2 or 3 VM, but
> unfortunately I did not catch their output in a file, neither I used
> kvm_stat (by ignorance of its existence).
>    

Well, in order to make progress we need more measurements.

-- 
error compiling committee.c: too many arguments to function

Re: Discordant results between UnixBench and nBench

[Matthieu Olivier]

[-- Attachment #1: Type: text/plain, Size: 715 bytes --]

On Wed, Dec 23, 2009 at 11:37 AM, Avi Kivity <avi@redhat.com> wrote:
> On 12/23/2009 12:35 PM, Matthieu Olivier wrote:
>>
>> To tell the truth, I had to test some others hypervisors, so now I
>> don't have KVM installed anymore.
>> I used top for following bench progress on 2 or 3 VM, but
>> unfortunately I did not catch their output in a file, neither I used
>> kvm_stat (by ignorance of its existence).
>>
>
> Well, in order to make progress we need more measurements.
>
> --
> error compiling committee.c: too many arguments to function
>
>

Ok, there is an excel file which contain some results.
I'm not supposed to give so many informations, but I really need to
understand what's the problem.

-- 
Matthieu

[-- Attachment #2: Analysis.xls --]
[-- Type: application/vnd.ms-excel, Size: 58880 bytes --]

Re: Discordant results between UnixBench and nBench

[Avi Kivity]

On 12/23/2009 02:27 PM, Matthieu Olivier wrote:
>
> Ok, there is an excel file which contain some results.
> I'm not supposed to give so many informations, but I really need to
> understand what's the problem.
>
>    

This is strange.  Is the machine hyperthreaded?

Please supply full /proc/cpuinfo.

-- 
error compiling committee.c: too many arguments to function

Re: Discordant results between UnixBench and nBench

[Matthieu Olivier]

On Wed, Dec 23, 2009 at 1:39 PM, Avi Kivity <avi@redhat.com> wrote:
> On 12/23/2009 02:27 PM, Matthieu Olivier wrote:
>>
>> Ok, there is an excel file which contain some results.
>> I'm not supposed to give so many informations, but I really need to
>> understand what's the problem.
>>
>>
>
> This is strange.  Is the machine hyperthreaded?
>
> Please supply full /proc/cpuinfo.
>
> --
> error compiling committee.c: too many arguments to function
>
>

I can only see 4 cores in /proc/cpuinfo.
According the caract page, there is no hyperthreading.

-> http://ark.intel.com/Product.aspx?id=35130

I also wonder why the host can still get more CPU ressources over 4
threads? I guess the purpose of both benchmarks is to overload the
CPU, so why they can't reach the max?
I suposed that the FPU was able to handle at least 2 operations the
same time, or maybe hyperthreading was included, but I can't check the
first part, and the second isn't true.

Or maybe current x86 processors can handel very well these old
benchmarks. I can't really say :/
-- 
Matthieu

Re: Discordant results between UnixBench and nBench

[Avi Kivity]

On 12/23/2009 02:55 PM, Matthieu Olivier wrote:
>
> I can only see 4 cores in /proc/cpuinfo.
> According the caract page, there is no hyperthreading.
>
> ->  http://ark.intel.com/Product.aspx?id=35130
>
>    

In this case the plateau at 4 guests is perfectly understandable.

> I also wonder why the host can still get more CPU ressources over 4
> threads? I guess the purpose of both benchmarks is to overload the
> CPU, so why they can't reach the max?
>    

Probably a problem with the benchmark itself.

> I suposed that the FPU was able to handle at least 2 operations the
> same time, or maybe hyperthreading was included, but I can't check the
> first part, and the second isn't true.
>
> Or maybe current x86 processors can handel very well these old
> benchmarks. I can't really say :/
>    

No, you should see the same plateau as with kvm.  The processor can't 
run two threads at once.

-- 
error compiling committee.c: too many arguments to function

Re: Discordant results between UnixBench and nBench

[Matthieu Olivier]

On Wed, Dec 23, 2009 at 1:59 PM, Avi Kivity <avi@redhat.com> wrote:
> On 12/23/2009 02:55 PM, Matthieu Olivier wrote:
>>
>> I can only see 4 cores in /proc/cpuinfo.
>> According the caract page, there is no hyperthreading.
>>
>> ->  http://ark.intel.com/Product.aspx?id=35130
>>
>>
>
> In this case the plateau at 4 guests is perfectly understandable.
>
>> I also wonder why the host can still get more CPU ressources over 4
>> threads? I guess the purpose of both benchmarks is to overload the
>> CPU, so why they can't reach the max?
>>
>
> Probably a problem with the benchmark itself.
>
>> I suposed that the FPU was able to handle at least 2 operations the
>> same time, or maybe hyperthreading was included, but I can't check the
>> first part, and the second isn't true.
>>
>> Or maybe current x86 processors can handel very well these old
>> benchmarks. I can't really say :/
>>
>
> No, you should see the same plateau as with kvm.  The processor can't run
> two threads at once.
>
> --
> error compiling committee.c: too many arguments to function
>
>

I have this plateau with integer number tests in unixbench. I have it
too with tests cache in cachebench. I am attempted to discard all
nbench results, but each category (floating, integer and cache) is
tested with 3 or 4 different tests, so the weight is more important.
I'm facing a real dilemn.

I'm going to inquire unixbench and nbench mainteners.
I keep you in touch if I find something revealing.

Thanks for your help Avi.

-- 
Matthieu