Recent kernel SMP scalability Benchmark/White-paper References.

Recent kernel SMP scalability Benchmark/White-paper References.

[Austin Gonyou]

I was looking around on google web, google groups, lkml digests,
Intel.com, RedHat, SuSe, SGI.com, osdl.com, etc for some benchmarks of
recent 2.4.x kernels, say 2.4.x > 16, with references to SMP scalability
problems or successes, etc. Mainly centering around 4-way/8-way x86
testing in terms of memory bandwidth/utilization, threading performance,
etc. 

I've not found much in my search so far, and thought at this point it
might be best to ask on this list to help shorten the search a bit, if
possible. Of the documents I do have, they're more marketing based and
not really *technology* based or touch very heavily as to generic
benchmarking of a standard Linux kernel on SMP. 

I'm hoping to create a white-paper internally, and hopefully externally
at some point, which can be maintained so others don't have to do the
same arduous task of trying to find recent data as it pertains to said
statistics. 

Any help as to recent documentation of this nature would be *overly*
appreciated! 

In addition to this info, I'm trying to gather information as it
pertains to the scalability of Linux kernels on 4/8-way x86 systems
versus Solaris Sparc 4/8-way systems with measurements of the same
statistics. 

I fear I'm searching for a document which does not exist. TIA.

Re: Recent kernel SMP scalability Benchmark/White-paper References.

[Hanna Linder]

Hi Austin,

	Check out http://lse.sourceforge.net
and	http://sourceforge.net/projects/lse

(lse= linux scalability effort)

That might be more information than you were looking for.
Another good resource is the lse mailing list at: 
lse-tech@lists.sourceforge.net

We have bi-weekly conference calls where anyone is welcome
to join and ask questions or report your work or whatever.

As a developer working on SMP scalability on Linux I would 
say it is getting better but we still have work to do. 

Hanna Linder
IBM Linux Technology Center


--On Thursday, May 23, 2002 17:41:55 -0500 Austin Gonyou <austin@digitalroadkill.net> wrote:

> I was looking around on google web, google groups, lkml digests,
> Intel.com, RedHat, SuSe, SGI.com, osdl.com, etc for some benchmarks of
> recent 2.4.x kernels, say 2.4.x > 16, with references to SMP scalability
> problems or successes, etc. Mainly centering around 4-way/8-way x86
> testing in terms of memory bandwidth/utilization, threading performance,
> etc. 
> 
> I've not found much in my search so far, and thought at this point it
> might be best to ask on this list to help shorten the search a bit, if
> possible. Of the documents I do have, they're more marketing based and
> not really *technology* based or touch very heavily as to generic
> benchmarking of a standard Linux kernel on SMP. 
> 
> I'm hoping to create a white-paper internally, and hopefully externally
> at some point, which can be maintained so others don't have to do the
> same arduous task of trying to find recent data as it pertains to said
> statistics. 
> 
> Any help as to recent documentation of this nature would be *overly*
> appreciated! 
> 
> In addition to this info, I'm trying to gather information as it
> pertains to the scalability of Linux kernels on 4/8-way x86 systems
> versus Solaris Sparc 4/8-way systems with measurements of the same
> statistics. 
> 
> I fear I'm searching for a document which does not exist. TIA.
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Recent kernel SMP scalability Benchmark/White-paper References.

[Austin Gonyou]

On Thu, 2002-05-23 at 18:34, Hanna Linder wrote:
> Hi Austin,
> 
> 	Check out http://lse.sourceforge.net
> and	http://sourceforge.net/projects/lse
> 
> (lse= linux scalability effort)
> 
> That might be more information than you were looking for.
> Another good resource is the lse mailing list at: 
> lse-tech@lists.sourceforge.net
> 
> We have bi-weekly conference calls where anyone is welcome
> to join and ask questions or report your work or whatever.
> 

Kewl, I might just see if I can work that in. I saw the LSE, but the
documentation out there hadn't been updated in quite some time as well.
Same with the SGI Linux Scalability Project. It is old, and seems to
center around only MIPS. :(

> As a developer working on SMP scalability on Linux I would 
> say it is getting better but we still have work to do. 
> 
> Hanna Linder
> IBM Linux Technology Center
> 
Would that be the ^^^ one here in tx?


Austin

Re: Recent kernel SMP scalability Benchmark/White-paper References.

[Hanna V Linder]

--On Thursday, May 23, 2002 11:04 PM -0500 Austin Gonyou <austin@digitalroadkill.net> wrote:
>> We have bi-weekly conference calls where anyone is welcome
>> to join and ask questions or report your work or whatever.
>>
>
> Kewl, I might just see if I can work that in. I saw the LSE, but the

Great.

> documentation out there hadn't been updated in quite some time as well.

I have updated some 8-way kernel comparisons running dbench as recently
as a couple weeks ago. Although, I do not know of any specific
Linux(TM) vs Solaris(TM) benchmark results on the lse web site. ;)
What benchmark would reflect your workload?

Speaking of OLS, Is there any interest in an lse-tech BOF or get-
to-gether of some sort in Ottawa this summer? It would be nice
to put some more faces to names from the conference calls.

>>
>> Hanna Linder
>> IBM Linux Technology Center
>>
> Would that be the ^^^ one here in tx?

Same LTC, different state. The LTC engineers live all over the world.
I know some of the guys in Austin...

Hanna

Re: Recent kernel SMP scalability Benchmark/White-paper References.

[Austin Gonyou]

On Fri, 2002-05-24 at 01:15, Hanna V Linder wrote:

> > documentation out there hadn't been updated in quite some time as well.
> 
> I have updated some 8-way kernel comparisons running dbench as recently
> as a couple weeks ago. Although, I do not know of any specific
> Linux(TM) vs Solaris(TM) benchmark results on the lse web site. ;)
> What benchmark would reflect your workload?
> 

Kewl again. I didn't see them, but thanks anyway. I'll check them out. 


> Speaking of OLS, Is there any interest in an lse-tech BOF or get-
> to-gether of some sort in Ottawa this summer? It would be nice
> to put some more faces to names from the conference calls.
> 
I wish. My company doesn't *send* me anywhere. :)

One last question for the list. Is there a recent 2.2.x vs. 2.4.x SMP
kernel comparison available as well? Recent can be as much as 9-10 mo
ago, preferably younger though.

Re: Recent kernel SMP scalability Benchmark/White-paper References.

[Stephane Charette]

On 23 May 2002 17:41:55 -0500, Austin Gonyou wrote:

>I was looking around on google web, google groups, lkml digests,
>Intel.com, RedHat, SuSe, SGI.com, osdl.com, etc for some benchmarks of
>recent 2.4.x kernels, say 2.4.x > 16, with references to SMP scalability
>problems or successes, etc. Mainly centering around 4-way/8-way x86
>testing in terms of memory bandwidth/utilization, threading performance,
>etc. 

All I've found so far is:

http://www.usenix.org/publications/library/proceedings/als2000/full_papers/bryantscale/bryantscale.pdf

which is based on the 2.2.14-SMP -vs- 2.3.99-SMP kernels.

>I'm hoping to create a white-paper internally, and hopefully externally
>at some point, which can be maintained so others don't have to do the
>same arduous task of trying to find recent data as it pertains to said
>statistics.

If you find anything else, or get forwarded any information, please post the relevant information where possible.  There are many of us looking for benchmark information on recent 2.4.x kernels.  Seems like a few companies/projects are currently looking at the costs/benefits/risks of moving up to the new kernel.

Regards,

Stephane Charette

Re: Recent kernel SMP scalability Benchmark/White-paper References.

[Austin Gonyou]

On Thu, 2002-05-23 at 19:23, Stephane Charette wrote:
> On 23 May 2002 17:41:55 -0500, Austin Gonyou wrote:
> 
> >I was looking around on google web, google groups, lkml digests,
> >Intel.com, RedHat, SuSe, SGI.com, osdl.com, etc for some benchmarks of
> >recent 2.4.x kernels, say 2.4.x > 16, with references to SMP scalability
> >problems or successes, etc. Mainly centering around 4-way/8-way x86
> >testing in terms of memory bandwidth/utilization, threading performance,
> >etc. 
> 
> All I've found so far is:
> 
> http://www.usenix.org/publications/library/proceedings/als2000/full_papers/bryantscale/bryantscale.pdf
> 

Yeah..been there, done that. Most stuff I've found is no newer than
2000. :( I really need something recent. 

> which is based on the 2.2.14-SMP -vs- 2.3.99-SMP kernels.
> 
> >I'm hoping to create a white-paper internally, and hopefully externally
> >at some point, which can be maintained so others don't have to do the
> >same arduous task of trying to find recent data as it pertains to said
> >statistics.
> 
> If you find anything else, or get forwarded any information, please post the relevant information where possible.  There are many of us looking for benchmark information on recent 2.4.x kernels.  Seems like a few companies/projects are currently looking at the costs/benefits/risks of moving up to the new kernel.
> 
That is the plan. Like with my recently completed Multi-Realm Kerberos
install docs that I've been working on, everything I collect will get
out for public knowledge.Sharing our experiences is the only way to make
the world a better place. 


> Regards,
> 
> Stephane Charette

Re: Recent kernel SMP scalability Benchmark/White-paper References.

[Austin Gonyou]

On Thu, 2002-05-23 at 21:30, Albert D. Cahalan wrote:
> > problems or successes, etc. Mainly centering around 4-way/8-way x86
> > testing in terms of memory bandwidth/utilization, threading performance,
> 
> Never mind the OS: x86 SMP memory bandwidth is lame.
> AMD does a respectable job with 2-way systems, but
> doesn't produce anything bigger AFAIK.

Right, this is understood, although on the newer Xeons, the underlying
memory architecture is *far* better than it used to be, especially when
we're talking about 400Mhz FSB and up, more pipelined streams, etc.


> Threads are usually a very bad idea, no matter what OS.
> They blow away the cache locality with extra stacks.
> http://www.kegel.com/c10k.html

hrmmm..well..I can't entirely agree with that, mainly in the realm of
the 2.4 kernel with Andrea's patches plus knowing that a threaded app,
when synchronized correctly and designed to perform a specific function
really does perform well. I *can* be more resource intensive, but
usually in disk/network I/O and CPU usage. There can be a memory
bandwidth bottleneck too, but most apps people are going to run which
are threaded and need *a lot* of memory to run are not fast context
switchers/swappers, etc. (e.g. swapping 10 GB RAM or trying to flush
10GB ram to disk for that matter, is nearly impossible in any sane
amount of time. example: oracle or db2)

> > In addition to this info, I'm trying to gather information as it
> > pertains to the scalability of Linux kernels on 4/8-way x86 systems
> > versus Solaris Sparc 4/8-way systems with measurements of the same
> > statistics. 
> 
> This is worthless, because one type of system might not need
> many processors to get your work done. What if the performance
> goes like this:
> 
> CPUs Linux/x86 Solaris/SPARC
>  0        0          0
>  1       10          6
>  2       19         12
>  4       36         24
>  8       60         47
> 16      110         93
> 32      130        182
> 
> Solaris is fastest. Solaris scales almost perfectly.
> Solaris stinks on anything less than 32 CPUs.
> 
> While I pulled these numbers out of my ass, they
> aren't too far from the truth I think. Google has
> about 2200 hits for "Slowaris".

I know about this, we had 3 E10K's at one point with 16P ea to push a
very large set of monolithic DBs. After the tuning was done to that
platform though, we actually were able to get it down to 8CPUs and back
into E4500's. (not to mention actually re-writing most of the Oracle
Packages, reducing the time data stayed in the db so we didn't have to
go through so much, etc)

So, this brings us to where we are now. I've done a crap load of testing
2/4-way configs and comparing them against our E4500's with 8P. Problem
is, I've never been allowed to keep the config around long enough to get
the numbers in a way like you showed above.(political and monetary
issues prevented this a bit I'm afraid) Now we're back into a far better
position to do this type of thing and I just got asked to find
information regarding scalability of x86 arch, versus Sparc in 4 and 8
way scenarios. Our application runs nearly as well on 4 x86 cpus than it
does with 8 Sparc cpus, but equal ram, and nearly equal storage. 

Some seem to believe though that it's just our application running slow
on the Sparc boxes, but I have to say no to this. We've spent years
tuning our application now, with spot checks every couple of weeks to
squeeze out more from our nearly overloaded infrastructure. On top of
that, we've squeezed as much, ATM, as we can out of our disk subsystem
to get better performance there. The only out left is to go to Oracle9i
on Solaris, and then re-tune for *that*. A task not able to be done in a
short amount of time with the sizes of our DBs I assure you.

> You should consider memory. 32-bit hardware gets
> really slow as the amount of physical memory
> gets into the gigabyte range.
> 
> Also consider what happens when a part breaks.
> The Linux/x86 system is easy to repair or replace.
> 
> This is really old, but still fun to look at:
> http://www.cs.uml.edu/~acahalan/linux/benchmark.html

As to memory, I've seen what you're talking about first hand here as
well. 8gb ram on a 250GB oracle db takes a crapload of time to shutdown
if it's been doing any inserts/etc. for any respectable length of time. 
Still, with the 1.6Ghz Xeons and 400Mhz FSB, I would imagine the speed
with which the ram is addressed is quite more impressive than with the
lesser architectures. 

Thanks much for the discussion around this. I truly appreciate it.