Benchmarking on CentOS 5

Benchmarking on CentOS 5

[Cam Macdonell]

Hi,

I wanted to some benchmarking on a CentOS 5 host.  However, the warning 
in dmesg about preempt notifiers indicates I shouldn't benchmark.  Is 
there a workaround (aside from updating the kernel) that handles the issue?

Thanks,
Cam

Re: Benchmarking on CentOS 5

[Andrea Arcangeli]

On Wed, May 28, 2008 at 11:48:21AM -0600, Cam Macdonell wrote:
>
> Hi,
>
> I wanted to some benchmarking on a CentOS 5 host.  However, the warning in 
> dmesg about preempt notifiers indicates I shouldn't benchmark.  Is there a 
> workaround (aside from updating the kernel) that handles the issue?

The problem is that the preempt notifier emulation through hardware
breakpoints is worse at optimizing lightweight vm exists than the real
preempt notifiers supported by recent kernels. If you benchmark
without upgrading the kernel, it has to be clear it is running slower
than it would on a recent optimized host kernel. If you decide to
upgrade and compile the host kernel yourself, you need to configure it
with CONFIG_KVM=m to be sure the preempt notifiers are enabled in the
host kernel and in turn the printk will go away.

Thanks!

Re: Benchmarking on CentOS 5

[Farkas Levente]

Andrea Arcangeli wrote:
> On Wed, May 28, 2008 at 11:48:21AM -0600, Cam Macdonell wrote:
>> Hi,
>>
>> I wanted to some benchmarking on a CentOS 5 host.  However, the warning in 
>> dmesg about preempt notifiers indicates I shouldn't benchmark.  Is there a 
>> workaround (aside from updating the kernel) that handles the issue?
> 
> The problem is that the preempt notifier emulation through hardware
> breakpoints is worse at optimizing lightweight vm exists than the real
> preempt notifiers supported by recent kernels. If you benchmark
> without upgrading the kernel, it has to be clear it is running slower
> than it would on a recent optimized host kernel. If you decide to
> upgrade and compile the host kernel yourself, you need to configure it
> with CONFIG_KVM=m to be sure the preempt notifiers are enabled in the
> host kernel and in turn the printk will go away.

ok. so what is qumranet recommendation?
the just released latest rhel 5.2 kernel is kernel-2.6.18-92.el5 is it 
good enough for kvm host os? or its' better to change some other distro 
eg: fedora 9?
until now we try to use the latest rhel/centos on all of our servers 
while we use fedora (currently 8 but may be upgrade in a few weeks if 9 
become stable) on desktops. but now it seems probably then for a kvm 
host rhel/centos is not enough:-(

-- 
   Levente                               "Si vis pacem para bellum!"

Re: Benchmarking on CentOS 5

[Andrea Arcangeli]

On Wed, May 28, 2008 at 10:27:08PM +0200, Farkas Levente wrote:
> ok. so what is qumranet recommendation?

I'm not aware of a Qumranet reccomandation for this but I can give you
my opinion.

> the just released latest rhel 5.2 kernel is kernel-2.6.18-92.el5 is it good 
> enough for kvm host os? or its' better to change some other distro eg: 
> fedora 9?
> until now we try to use the latest rhel/centos on all of our servers while 
> we use fedora (currently 8 but may be upgrade in a few weeks if 9 become 
> stable) on desktops. but now it seems probably then for a kvm host 
> rhel/centos is not enough:-(

This is a almost the opposite question of the previous one from Cam
;), here you're asking about a production kernel, Cam was asking for a
benchmarking setup.

I think any distro enterprise kernel is better for production systems
than a latest mainline, but that's just me. You'll find others
preferring to run 2.6.25 in production a few days after it is
released.

The slowdown we're talking about here for the preempt notifiers isn't
going to make a big difference in a production system, more important
that you're sure your host kernel is rock solid and well tested
IMHO. But if it was pure benchmarking what you were doing, then using
latest mainline was better to get the best possible score, that is why
such printk is there.

Re: Benchmarking on CentOS 5

[Farkas Levente]

Andrea Arcangeli wrote:
> On Wed, May 28, 2008 at 10:27:08PM +0200, Farkas Levente wrote:
>> ok. so what is qumranet recommendation?
> 
> I'm not aware of a Qumranet reccomandation for this but I can give you
> my opinion.
> 
>> the just released latest rhel 5.2 kernel is kernel-2.6.18-92.el5 is it good 
>> enough for kvm host os? or its' better to change some other distro eg: 
>> fedora 9?
>> until now we try to use the latest rhel/centos on all of our servers while 
>> we use fedora (currently 8 but may be upgrade in a few weeks if 9 become 
>> stable) on desktops. but now it seems probably then for a kvm host 
>> rhel/centos is not enough:-(
> 
> This is a almost the opposite question of the previous one from Cam
> ;), here you're asking about a production kernel, Cam was asking for a
> benchmarking setup.
> 
> I think any distro enterprise kernel is better for production systems
> than a latest mainline, but that's just me. You'll find others
> preferring to run 2.6.25 in production a few days after it is
> released.
> 
> The slowdown we're talking about here for the preempt notifiers isn't
> going to make a big difference in a production system, more important
> that you're sure your host kernel is rock solid and well tested
> IMHO. But if it was pure benchmarking what you were doing, then using
> latest mainline was better to get the best possible score, that is why
> such printk is there.

i want rock solid, but smp guest and both 32 and 64bit guest centos too. 
5-10% performance lost is not important but 50-200% is important. so my 
real question is that kernel-2.6.18-92.el5 has "enough" features for an 
good working kvm if i recompile kvm and kvm-kmod? ehough here means no 
more than 5-10% performance lost.

-- 
   Levente                               "Si vis pacem para bellum!"

Re: Benchmarking on CentOS 5

[Farkas Levente]

this is out production server at the development department (10-15) 
people using it so actually if i tell them that i'll stop the host and 
all guests for max an hour it's acceptable, but more not really. it's 
run it type programs. from my experience in the last 6-12 months is that 
kvm is not production ready. as you can read from this list there are 
far too many change day-by-day which are very core. and this comes from 
the current state of kvm. which indicate that rh can't include in there 
stable server distro such a kvm version which is feature rich and 
stable. i hope rhel 6 will be based on at least kernel-2.6.25 or later. 
and in that case it'll production ready, but even then may be at rhel 
6.1 (since imho rhel 6.0 will be based on a kernel which is released 
nowadays).
imho the biggest problem with the current development of kvm that there 
is not a stable releases which is somewhat related to the current 
release number. eg kvm-0.5.x kvm-0.6.x would be better. but currently 
kvm development is so fast that keep 2-3 parallel branch where there is 
a development and stable release seems to too much work.
so to answer to your question i don't know:-(
i'd like to see a kvm version which is working on rhel/centos 5 host on 
intel and amd, with smp and stable (ie. boot, reboot, save state, a run 
without crash) and the performance is similar (ie 80-90%) to the 
development version. unfortunately it seems currently there is no such 
version:-( may be around 62 one of the version is the best choice.
next week i'll spend a half day to try to find a stable version...

On Fri, May 30, 2008 at 3:38 PM, David S. Ahern <daahern@cisco.com 
<mailto:daahern@cisco.com>> wrote:

    Do you run in a production type environment or lab environment? Do the
    guests run standard IT type programs -- web server, database, etc?

    I ask because something I am working on here might force me to drop back
    to RHEL5, but I still want to use kvm for the virtualization layer. I'm
    trying to get an idea of what to expect. It sounds like it has worked ok
    for you overall, you just have to be picky about specific kvm releases
    you use.

    thanks for the information,

    david


    Farkas Levente wrote:
     > yes as i wrote i always recompile create new rpms for kvm and
    kvm-kmod
     > on centos. the host hasn't any problem (with the last few dozens of
     > kvm), but the guest's are more problematic. we need both 32 and
    64 bit
     > smp guests and it cause a lots of problem. first is the time source,
     > second (as i wrote earlier guest see only 3 of 4 vcpu) and with
    kvm-69
     > we've got a lots of performance lost compared to kvm-62. and see
    random
     > high load on the guests.
     >
     > On Fri, May 30, 2008 at 5:34 AM, David S. Ahern
    <daahern@cisco.com <mailto:daahern@cisco.com>
     > <mailto:daahern@cisco.com <mailto:daahern@cisco.com>>> wrote:
     >
     >     I take it from your posts you are running newer kvm releases
    (e.g.,
     >     kvm-69) on Centos 5 for the host. How's that working out for
    you? Any
     >     major hiccups -- stability issues or runtime issues?
     >
     >     david
     >
     >
     >     Farkas Levente wrote:
     >     > Andrea Arcangeli wrote:
     >     >> On Wed, May 28, 2008 at 10:27:08PM +0200, Farkas Levente
    wrote:
     >     >>> ok. so what is qumranet recommendation?
     >     >>
     >     >> I'm not aware of a Qumranet reccomandation for this but I can
     >     give you
     >     >> my opinion.
     >     >>
     >     >>> the just released latest rhel 5.2 kernel is
    kernel-2.6.18-92.el5 is
     >     >>> it good enough for kvm host os? or its' better to change
    some other
     >     >>> distro eg: fedora 9?
     >     >>> until now we try to use the latest rhel/centos on all of
    our servers
     >     >>> while we use fedora (currently 8 but may be upgrade in a few
     >     weeks if
     >     >>> 9 become stable) on desktops. but now it seems probably
    then for a
     >     >>> kvm host rhel/centos is not enough:-(
     >     >>
     >     >> This is a almost the opposite question of the previous one
    from Cam
     >     >> ;), here you're asking about a production kernel, Cam was
    asking
     >     for a
     >     >> benchmarking setup.
     >     >>
     >     >> I think any distro enterprise kernel is better for
    production systems
     >     >> than a latest mainline, but that's just me. You'll find others
     >     >> preferring to run 2.6.25 in production a few days after it is
     >     >> released.
     >     >>
     >     >> The slowdown we're talking about here for the preempt
    notifiers isn't
     >     >> going to make a big difference in a production system,
    more important
     >     >> that you're sure your host kernel is rock solid and well
    tested
     >     >> IMHO. But if it was pure benchmarking what you were doing,
    then using
     >     >> latest mainline was better to get the best possible score,
    that
     >     is why
     >     >> such printk is there.
     >     >
     >     > i want rock solid, but smp guest and both 32 and 64bit guest
     >     centos too.
     >     > 5-10% performance lost is not important but 50-200% is
    important.
     >     so my
     >     > real question is that kernel-2.6.18-92.el5 has "enough"
    features
     >     for an
     >     > good working kvm if i recompile kvm and kvm-kmod? ehough
    here means no
     >     > more than 5-10% performance lost.
     >     >
     >
     >
     >
     >
     > --
     > Levente "Si vis pacem para bellum!"




-- 
 Levente                               "Si vis pacem para bellum!"

-- 
  Levente                               "Si vis pacem para bellum!"

Re: Benchmarking on CentOS 5

[Andrea Arcangeli]

On Fri, May 30, 2008 at 07:30:41PM +0200, Farkas Levente wrote:
> without crash) and the performance is similar (ie 80-90%) to the 
> development version. unfortunately it seems currently there is no such 

I hope you didn't reach this conclusion about different performance
between enterprise kernel host and mainline kernel host because I
didn't answer promptly to your previous email, sorry!

With regard to your performance question, preempt notifier emulation
really shouldn't be noticeable in real life. It worth to run the
fastest code only if you're doing pure benchmarking, in which case you
want to avoid any unnecessary exception and heavyweight exit (that
should reduces the time to schedule of a couple thousand cycles I
guess, but re-schedule events aren't so frequent and this is a
per-task per-cpu breakpoint with my latest emulation logic taking
advantage of the __switch_to internals).

I can hardly measure any difference here between SLES10 SP2 host and
mainline host with a simple dd from pagecache to /dev/null. The change
between enterprise kernel and current mainline kernel may be more
significant for the final performance than the impact of the preempt
notifier emulation. In any case the difference should be much less
than 5% as far as preempt notifier emulation is concerned. It should
be couple usec lost every couple msec or so.

Re: Benchmarking on CentOS 5

[Amit Shah]

On Friday 30 May 2008 23:00:41 Farkas Levente wrote:
> this is out production server at the development department (10-15)
> people using it so actually if i tell them that i'll stop the host and
> all guests for max an hour it's acceptable, but more not really. it's
> run it type programs. from my experience in the last 6-12 months is that
> kvm is not production ready. as you can read from this list there are
> far too many change day-by-day which are very core. and this comes from
> the current state of kvm. which indicate that rh can't include in there

You'll find the most stable version of kvm in the kernel that your 
distribution ships. Linux-2.6.x (where x > 20) should also be stable. The 
development on kvm will continue to proceed at a fast pace, so you'll see 
several kvm releases and this, as a result, is bound to bring in a few new 
bugs in each iteration.

> imho the biggest problem with the current development of kvm that there
> is not a stable releases which is somewhat related to the current
> release number. eg kvm-0.5.x kvm-0.6.x would be better. but currently

So the short answer is: if you're looking for a stable version of kvm, look at 
a kernel.org kernel or the kvm version provided to you by your distribution.

> kvm development is so fast that keep 2-3 parallel branch where there is
> a development and stable release seems to too much work.
> so to answer to your question i don't know:-(

The "stable" branch of kvm is the one in the most-recently available Linux 
kernel from kernel.org. kvm.git is the development version.

Re: Benchmarking on CentOS 5

[Dor Laor]

On Mon, 2008-06-02 at 14:35 +0530, Amit Shah wrote:
> On Friday 30 May 2008 23:00:41 Farkas Levente wrote:
> > this is out production server at the development department (10-15)
> > people using it so actually if i tell them that i'll stop the host and
> > all guests for max an hour it's acceptable, but more not really. it's
> > run it type programs. from my experience in the last 6-12 months is that
> > kvm is not production ready. as you can read from this list there are
> > far too many change day-by-day which are very core. and this comes from
> > the current state of kvm. which indicate that rh can't include in there
> 
> You'll find the most stable version of kvm in the kernel that your 
> distribution ships. Linux-2.6.x (where x > 20) should also be stable. The 
> development on kvm will continue to proceed at a fast pace, so you'll see 
> several kvm releases and this, as a result, is bound to bring in a few new 
> bugs in each iteration.
> 
> > imho the biggest problem with the current development of kvm that there
> > is not a stable releases which is somewhat related to the current
> > release number. eg kvm-0.5.x kvm-0.6.x would be better. but currently
> 
> So the short answer is: if you're looking for a stable version of kvm, look at 
> a kernel.org kernel or the kvm version provided to you by your distribution.
> 
> > kvm development is so fast that keep 2-3 parallel branch where there is
> > a development and stable release seems to too much work.
> > so to answer to your question i don't know:-(
> 
> The "stable" branch of kvm is the one in the most-recently available Linux 
> kernel from kernel.org. kvm.git is the development version.

In the near future we'll publish a stable branch. There are actually 2
repositories: kernel repo, based on the latest kernel - 2.6.26 and a
userspace repository that will be based on kvm-68.

The idea is to maintain the above repos together and only apply bug
fixes. New features will come with every next kernel release.

We're in the process of creating an automatic test framework for kvm.
It will be open source framework based on autotest and similar to
Anthony's kvmtest. It will help stabilizing both the 'stable' branch and
the master.

> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html