JVM performance on Linux (vs. Solaris/Windows)

JVM performance on Linux (vs. Solaris/Windows)

[K P]

Sun's recently published SPECjbb_2005 numbers on Linux, Windows and
Solaris on their
Opteron system, and the Linux result is the lowest of the three by far:

Linux: http://www.spec.org/jbb2005/results/res2006q1/jbb2005-20060117-00062.html
Solaris: http://www.spec.org/jbb2005/results/res2006q1/jbb2005-20060117-00063.html
Windows: http://www.spec.org/jbb2005/results/res2006q1/jbb2005-20060117-00064.html

It's not evident if Sun spent any time analyzing and tuning the Linux
result. While the
majority of the tuning opportunities for SPECjbb_2005 are likely to be
in the JVM itself, I was
wondering (given the large spread between the OS's) if there were
other typical opportunities
to tune the Linux kernel for JVM performance and SPECjbb_2005.

There are some other results showing excellent scalability with SPECjbb_2005 on
Linux/Itanium (such as SGI's:
http://www.spec.org/jbb2005/results/res2006q2/ ), but it's
not clear if there are other opportunities for tuning unique to Linux
on Opteron, or Linux
in general that should be explored

Comments?

-kp

Re: JVM performance on Linux (vs. Solaris/Windows)

[Martin J. Bligh]

K P wrote:

>Sun's recently published SPECjbb_2005 numbers on Linux, Windows and
>Solaris on their
>Opteron system, and the Linux result is the lowest of the three by far:
>
>Linux: http://www.spec.org/jbb2005/results/res2006q1/jbb2005-20060117-00062.html
>Solaris: http://www.spec.org/jbb2005/results/res2006q1/jbb2005-20060117-00063.html
>Windows: http://www.spec.org/jbb2005/results/res2006q1/jbb2005-20060117-00064.html
>
>It's not evident if Sun spent any time analyzing and tuning the Linux
>result. While the
>majority of the tuning opportunities for SPECjbb_2005 are likely to be
>in the JVM itself, I was
>wondering (given the large spread between the OS's) if there were
>other typical opportunities
>to tune the Linux kernel for JVM performance and SPECjbb_2005.
>
>There are some other results showing excellent scalability with SPECjbb_2005 on
>Linux/Itanium (such as SGI's:
>http://www.spec.org/jbb2005/results/res2006q2/ ), but it's
>not clear if there are other opportunities for tuning unique to Linux
>on Opteron, or Linux
>in general that should be explored
>
>Comments?
>  
>

SpecJBB is a really frigging stupid benchmark. It's *much* more affected
by the stupid crap in Java (like their locking model) than anything in the
OS. 

Best to find another benchmark, oh and preferably somebody vaguely
objective to run it ;-)

M.

Re: JVM performance on Linux (vs. Solaris/Windows)

[Jan Knutar]

On Thursday 13 April 2006 18:56, Martin J. Bligh wrote:

> SpecJBB is a really frigging stupid benchmark. It's *much* more affected
> by the stupid crap in Java (like their locking model) than anything in the
> OS. 

What's even worse is that people actually code by this stupid model...
 

Re: JVM performance on Linux (vs. Solaris/Windows)

[Jan Engelhardt]

>> SpecJBB is a really frigging stupid benchmark. It's *much* more affected
>> by the stupid crap in Java (like their locking model) than anything in the
>> OS. 
>
>What's even worse is that people actually code by this stupid model...
> 
And since it lacks pointers and some other 'fun', things have to be coded 
multiple times and/or differently, which leads to increased code size 
and/or execution time (neglecting the fact it's already a VM).


Jan Engelhardt
-- 

Re: JVM performance on Linux (vs. Solaris/Windows)

[David S. Miller]

From: "Martin J. Bligh" <mbligh@mbligh.org>
Date: Thu, 13 Apr 2006 08:56:49 -0700

> SpecJBB is a really frigging stupid benchmark. It's *much* more affected
> by the stupid crap in Java (like their locking model) than anything in the
> OS. 
> 
> Best to find another benchmark, oh and preferably somebody vaguely
> objective to run it ;-)

This may be true, but on a more serious note someone should check
to make sure Sun turned on hugepage support in the JVM under Linux.

That is one thing that helps this benchmark out enormously, so if
hugepages are disabled you might as well ignore the result.

Re: JVM performance on Linux (vs. Solaris/Windows)

[Jeff V. Merkey]

Martin J. Bligh wrote:

> K P wrote:
>
>> Sun's recently published SPECjbb_2005 numbers on Linux, Windows and
>> Solaris on their
>> Opteron system, and the Linux result is the lowest of the three by far:
>>
>> Linux: 
>> http://www.spec.org/jbb2005/results/res2006q1/jbb2005-20060117-00062.html 
>>
>> Solaris: 
>> http://www.spec.org/jbb2005/results/res2006q1/jbb2005-20060117-00063.html 
>>
>> Windows: 
>> http://www.spec.org/jbb2005/results/res2006q1/jbb2005-20060117-00064.html 
>>
>>
>> It's not evident if Sun spent any time analyzing and tuning the Linux
>> result. While the
>> majority of the tuning opportunities for SPECjbb_2005 are likely to be
>> in the JVM itself, I was
>> wondering (given the large spread between the OS's) if there were
>> other typical opportunities
>> to tune the Linux kernel for JVM performance and SPECjbb_2005.
>>
>> There are some other results showing excellent scalability with 
>> SPECjbb_2005 on
>> Linux/Itanium (such as SGI's:
>> http://www.spec.org/jbb2005/results/res2006q2/ ), but it's
>> not clear if there are other opportunities for tuning unique to Linux
>> on Opteron, or Linux
>> in general that should be explored
>>
>> Comments?
>>  
>>
>
> SpecJBB is a really frigging stupid benchmark. It's *much* more affected
> by the stupid crap in Java (like their locking model) than anything in 
> the
> OS.
> Best to find another benchmark, oh and preferably somebody vaguely
> objective to run it ;-)
>
> M.
>
Note they ran the benchmark on an Opteron 285 instead of a Xeon with 16 
GB of memory.    Opteron peformance currently **SUCKS** with 2.6 series 
kernels under any kind of heavy I/O due to their cloning of the ancient 
82489DX architecture for I/O interrupt access and performance.  Looks 
like the test was stakced against Linux from the start.  Should have 
used a Xeon system. 

AMD needs to get their crappy I/O performance up to snuff.  Looking at 
the test parameteres leads me to believe there was a lot of swapping on 
a system with already poor I/O performance.

Jeff

Re: JVM performance on Linux (vs. Solaris/Windows)

[David Lang]

On Thu, 13 Apr 2006, Jeff V. Merkey wrote:

> Note they ran the benchmark on an Opteron 285 instead of a Xeon with 16 GB of 
> memory.    Opteron peformance currently **SUCKS** with 2.6 series kernels 
> under any kind of heavy I/O due to their cloning of the ancient 82489DX 
> architecture for I/O interrupt access and performance.  Looks like the test 
> was stakced against Linux from the start.  Should have used a Xeon system. 
> AMD needs to get their crappy I/O performance up to snuff.  Looking at the 
> test parameteres leads me to believe there was a lot of swapping on a system 
> with already poor I/O performance.

Jeff, I've seen several reccomendations from databasefolks (postgres and 
mysql) favoring Opterons over Xeons. this doesn't match your statement 
that Opteron performance sucks under any kind of I/O load. I don't 
understand how both can be correct.

David Lang

-- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
  -- C.A.R. Hoare

Re: JVM performance on Linux (vs. Solaris/Windows)

[Jeff V. Merkey]

David Lang wrote:

> On Thu, 13 Apr 2006, Jeff V. Merkey wrote:
>
>> Note they ran the benchmark on an Opteron 285 instead of a Xeon with 
>> 16 GB of memory. Opteron peformance currently **SUCKS** with 2.6 
>> series kernels under any kind of heavy I/O due to their cloning of 
>> the ancient 82489DX architecture for I/O interrupt access and 
>> performance. Looks like the test was stakced against Linux from the 
>> start. Should have used a Xeon system. AMD needs to get their crappy 
>> I/O performance up to snuff. Looking at the test parameteres leads me 
>> to believe there was a lot of swapping on a system with already poor 
>> I/O performance.
>
>
> Jeff, I've seen several reccomendations from databasefolks (postgres 
> and mysql) favoring Opterons over Xeons. this doesn't match your 
> statement that Opteron performance sucks under any kind of I/O load. I 
> don't understand how both can be correct.
>
> David Lang
>
Hi David,

I have tested our Solera products on both Xeon and Opteron Processors. I 
can sustain 500 MB/S capture off the wire on 4 x 1000 Gigabit segments
due to the incredible performance of Xeon based I/O chipsets. My tests 
with Opteron based systems are sick in comparison. An Opteron 200 series 
CPU on a Tyan based motherbord system is discouraging on comparison. The 
Opteron systems will only sustain 150 MB/S with the same software.

The I/O chipset performance for disk and LAN I/O is purtrid compared to 
the 7500 series I/O chipsets.

Jeff

Re: JVM performance on Linux (vs. Solaris/Windows)

[Martin J. Bligh]

> Note they ran the benchmark on an Opteron 285 instead of a Xeon with 
> 16 GB of memory.    Opteron peformance currently **SUCKS** with 2.6 
> series kernels under any kind of heavy I/O due to their cloning of the 
> ancient 82489DX architecture for I/O interrupt access and 
> performance.  Looks like the test was stakced against Linux from the 
> start.  Should have used a Xeon system.
> AMD needs to get their crappy I/O performance up to snuff.  Looking at 
> the test parameteres leads me to believe there was a lot of swapping 
> on a system with already poor I/O performance.
>

Looks to me like it was the same h/w for Linux as Solaris, so I don't
think that's much of an excuse ;-)

M.

Re: JVM performance on Linux (vs. Solaris/Windows)

[K P]

Besides, Specjbb_2005 doesn't do any IO or networking, and these should
not be factors in the result.

So although it's a simplistic benchmark where OS tuning may not provide as
much benefit as JVM tuning, it still doesn't explain why Solaris is ~15% higher
than Linux on the same hardware and JVM (I'm guessing that the tester did
not spend as much time tuning Linux as they did Solaris, hence the question).

-kp


On 4/13/06, Martin J. Bligh <mbligh@mbligh.org> wrote:
>
> > Note they ran the benchmark on an Opteron 285 instead of a Xeon with
> > 16 GB of memory.    Opteron peformance currently **SUCKS** with 2.6
> > series kernels under any kind of heavy I/O due to their cloning of the
> > ancient 82489DX architecture for I/O interrupt access and
> > performance.  Looks like the test was stakced against Linux from the
> > start.  Should have used a Xeon system.
> > AMD needs to get their crappy I/O performance up to snuff.  Looking at
> > the test parameteres leads me to believe there was a lot of swapping
> > on a system with already poor I/O performance.
> >
>
> Looks to me like it was the same h/w for Linux as Solaris, so I don't
> think that's much of an excuse ;-)
>
> M.
>
>

Re: JVM performance on Linux (vs. Solaris/Windows)

[Jan Engelhardt]

>
>So although it's a simplistic benchmark where OS tuning may not provide as
>much benefit as JVM tuning, it still doesn't explain why Solaris is ~15% higher
>than Linux on the same hardware and JVM (I'm guessing that the tester did
>not spend as much time tuning Linux as they did Solaris, hence the question).
>

Have you ever seen a manufacturer not putting his product at the top? 
When Microsoft folks make case studies about Linux+Windows, they come to 
the conclusion that Windows is better.
When Linux folks make case studies about Linux+Windows, they come to the 
conclusion that Linux is better.
Accordingly, a random benchmark picked from the web 
(http://www.sun.com/servers/entry/x2100/) shows Solaris at the top.
And it's not surprisingly in general that a lot of manufacturers from any 
industry claim they're world-leading in fact they are clearly competing 
at the same level with a dozen other ones.


Jan Engelhardt
-- 

Bogus Benchmark (was Re: JVM performance on Linux (vs. Solaris/Windows))

[Linda Walsh]

K P wrote:
> Sun's recently published SPECjbb_2005 numbers on Linux, Windows and
> Solaris on their
> Opteron system, and the Linux result is the lowest of the three by far:
>
> Linux: http://www.spec.org/jbb2005/results/res2006q1/jbb2005-20060117-00062.html
> Solaris: http://www.spec.org/jbb2005/results/res2006q1/jbb2005-20060117-00063.html
> Windows: http://www.spec.org/jbb2005/results/res2006q1/jbb2005-20060117-00064.html
>
> It's not evident if Sun spent any time analyzing and tuning the Linux
> result. While the
> majority of the tuning opportunities for SPECjbb_2005 are likely to be
> in the JVM itself, I was
> wondering (given the large spread between the OS's) if there were
> other typical opportunities
> to tune the Linux kernel for JVM performance and SPECjbb_2005.
----
    I may have missed it, but I didn't see any information on how the
three different binary javaVM's were "built".  What compilers did
they use? What options?  Did they use the OSs' most common compilers
(gcc, MS C, & Sun's C Compiler?). 

    There are many other factors affecting performance.  It's
not clear what background processes were running (affecting memory
cache) or if they ran them in single-user mode.  Do the tests
include graphics?  What windowing/graphics software did they
use?  Where they running a desktop? 

They don't say what kernel version they used (though that might
be implied in the version of RH they used).  They don't
remark on what options were used in mounting the file systems.
They mention that they used "ext3" (which defaults to
"write-through" write operations) vs. NTFS on Windows and Ufs on
Solaris.  NTFS defaults to "write-back" by default.  I'd bet
UFS does as well. 

Were the OS and test databases on the same place on disk?  I.e.
loaded on the same partitions?  Read/Write performance varies,
easily by 100% between the slowest and fastest positions on disk.
Was the spec, _at least_, run in it's own partition?

We're talking a difference of 10-15% between the benchmarks.
Any number of factors could influence a benchmark by more than
that: Compiler options, OS & app disk location, background tasks
loaded,  file system choice and options. 

Speculating that the differences have much of anything to do
with the linux kernel seems awfully premature.

I'd tend to regard this as a fairly worthless benchmark.  In
addition to the questions above, one could still assume they
are using default settings and options.

Anyone concerned with performance is unlikely to be using
defaults.

Linda