Kernel Scalability to 48 cores

Kernel Scalability to 48 cores

[Cyrus Massoumi]

A friend just sent me the following PDF about Linux Scalability to many cores and I thought it might be interesting to the list for further discussion.
It's about scalability problems of the Linux kernel and applications with 48 and more cores.

http://pdos.csail.mit.edu/papers/linux:osdi10.pdf

Re: Kernel Scalability to 48 cores

[Diego Calleja]

On Lunes, 11 de Octubre de 2010 23:30:03 Cyrus Massoumi escribió:
> A friend just sent me the following PDF about Linux Scalability to many cores and I thought it might be interesting to the list for further discussion.
> It's about scalability problems of the Linux kernel and applications with 48 and more cores.
> 
> http://pdos.csail.mit.edu/papers/linux:osdi10.pdf

Some of the contention points mentioned in Figure 1 seem to be the same
things that Nick Piggin is trying to solve with his VFS scalability
patchset.

Re: Kernel Scalability to 48 cores

[Andi Kleen]

Cyrus Massoumi <cyrusm@gmx.net> writes:

> A friend just sent me the following PDF about Linux Scalability to many cores and I thought it might be interesting to the list for further discussion.
> It's about scalability problems of the Linux kernel and applications
> with 48 and more cores.

Note that 48 cores is a small system these days.

> http://pdos.csail.mit.edu/papers/linux:osdi10.pdf

The paper is known, but it seems to be in the usual unproductive 
"publish a paper, but don't care about fixing anything" acedemic mode.

A lot of the issues described there are already being addressed
I believe.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.

Re: Kernel Scalability to 48 cores

[Theodore Tso]

On Oct 12, 2010, at 10:33 AM, Andi Kleen wrote:

> The paper is known, but it seems to be in the usual unproductive 
> "publish a paper, but don't care about fixing anything" acedemic mode.

The thing I found most disappointing about the paper is that it describe techniques that have been in use in industry to provide better scalability to commerical OS's, and which were first applied to Linux something like a decade ago.

Sure, at the time we were going to 4-16 way scalability, but the techniques used were known back then, and in fact the paper is actually pretty naive compared to what people at companies like Sequent had been doing for a very long time.

It's just that a lot of scalability work took a bit of a hiatus, oh, five years ago because at the time, Linux was good enough, and adding more scalability has costs (which is why not all of the changes SGI made for 512 and 1024-way scalability have not necessarily found their way into mainline).   The fact that we've needed to worry about scalability now that 6+ cores/socket are available now, and many more are coming soon, is something we've known for at least the last year or two. 

But since a lot of that work wasn't published in the appropriate hoidy-toidy "tenure-track" approved publication venues, to the academics, it didn't exist.

Oh, well....

-- Ted

Re: Kernel Scalability to 48 cores

[Andi Kleen]

On Tue, Oct 12, 2010 at 12:00:27PM -0400, Theodore Tso wrote:
> 
> On Oct 12, 2010, at 10:33 AM, Andi Kleen wrote:
> 
> > The paper is known, but it seems to be in the usual unproductive 
> > "publish a paper, but don't care about fixing anything" acedemic mode.
> 
> The thing I found most disappointing about the paper is that it describe techniques that have been in use in industry to provide better scalability to commerical OS's, and which were first applied to Linux something like a decade ago.


I haven't gone fully through it yet, maybe there are still some useful
nuggets in there. I would help if they published their patches though.

They used a few new workloads which were useful I guess.

> It's just that a lot of scalability work took a bit of a hiatus, oh, five years ago because at the time, Linux was good enough, and adding more scalability has costs (which is why not all of the changes SGI made for 512 and 1024-way scalability have not necessarily found their way into mainline).   The fact that we've needed to worry about scalability now that 6+ cores/socket are available now, and many more are coming soon, is something we've known for at least the last year or two. 

Already at 8*2 and 12 cores/socket

Fully agreed, but the other aspect is that we also need to worry
now about TBs and more of memory on a single (non extreme) system

More cores come with more memory and now even with 4K pages.

As a simple exercise I configured a 1TB system for less than 50kEUR
at Dell.com some time ago: that's not cheap, but definitely not
an exotic super computer either and prices will go down.

I believe a lot of new challenges also come from that, although
there has been a lot less work on that so far than on core scalability.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.