kernel scalability

kernel scalability

[Anil Gunturu]

Hi,
We are porting linux kernel (2.4.18) for an embdded device that
supports a large number of logical interfaces (upto 200) and routes
(4000). There are only two physical ethernet interfaces, but we are
planning to add multiple ip addresses and routes using the "ip"
utility from iproute2 package. Just wondering if there are any
scalability issues in doing so as far as the kernel is concerned.
Thanks for your help,
-Anil

Kernel Scalability

[Rajib Majumder]

Hi,

I am wondering if 2.4.x/2.6.x kernel is scalable enough to run on
8-CPU hardware. Do we have any scalability comparison data between
2.4/2.6 kernels and beyond 4-CPU?

If yes, is the scalablity is near linear?

Any input is appreciated.

Thanks

Rajib

Re: Kernel Scalability

[Kevin Fox]

There are multiple 512 processor altix's in production. :)

Kevin

On Wed, 2007-05-02 at 20:56 +0800, Rajib Majumder wrote:
> Hi,
> 
> I am wondering if 2.4.x/2.6.x kernel is scalable enough to run on
> 8-CPU hardware. Do we have any scalability comparison data between
> 2.4/2.6 kernels and beyond 4-CPU?
> 
> If yes, is the scalablity is near linear?
> 
> Any input is appreciated.
> 
> Thanks
> 
> Rajib
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Kernel Scalability

[Christoph Lameter]

On Wed, 2 May 2007, Kevin Fox wrote:

> There are multiple 512 processor altix's in production. :)

Ummmm.. There are also 1k configurations. 4k in testing. Expect 16k 
sometime this year.

Re: Kernel Scalability

[Eric W. Biederman]

"Rajib Majumder" <rajibm2005@gmail.com> writes:

> Hi,
>
> I am wondering if 2.4.x/2.6.x kernel is scalable enough to run on
> 8-CPU hardware. Do we have any scalability comparison data between
> 2.4/2.6 kernels and beyond 4-CPU?
>
> If yes, is the scalablity is near linear?

I haven't a clue what the data looks like but it is worth
noting that Linux runs on the SGI Altix and the large Unisys
machines, as well as large RISC machines.  So 128+ cpus seems
to work.

It depends on your application and what it stresses but scalability
isn't just a software issue.  It is also a hardware issue, and the
kernel can't scale linearly unless the hardware does as well.

Eric

Re: Kernel Scalability

[Daniel J Blueman]

On 2 May, 14:00, "Rajib Majumder" <rajibm2005@gmail.com> wrote:
> Hi,
>
> I am wondering if 2.4.x/2.6.x kernel is scalable enough to run on
> 8-CPU hardware. Do we have any scalability comparison data between
> 2.4/2.6 kernels and beyond 4-CPU?
>
> If yes, is the scalablity is near linear?
>
> Any input is appreciated.

There are 128-processor IA64 systems which run recent 2.6 kernels out
there; the per-processor counters, RCU and page-fault scalability work
has been instrumental to the necessary scaling for decent resource
usage on these.

IIRC, there were some patches being developed to improve pagecache
scalability lately too, but I guess it all depends on what kind of
workload you have...

> Thanks
>
> Rajib
-- 
Daniel J Blueman

Re: Kernel Scalability

[Tony Luck]

On 5/2/07, Daniel J Blueman <daniel.blueman@gmail.com> wrote:
>
> There are 128-processor IA64 systems which run recent 2.6 kernels out
> there; the per-processor counters, RCU and page-fault scalability work
> has been instrumental to the necessary scaling for decent resource
> usage on these.

128 cpu is a bit dated ... there are 512 and 1024 cpu systems being
used in production environments.  The highest reported count stands
at 4096 ... but that was only a prototype system, not a production
machine ... a small number of patches were needed to boot (e.g.
with 10 kernel daemons per cpu, a default kernel hits the maximum
number of process ids limit!).

The question of how well it scales is very workload dependent.  Most
of those high cpu count systems are running scientific workloads that
crunch numbers in user mode for 7.5 million years before making a
single syscall to print that the answer is 42. Obviously scaling of
such applications is near linear.

But your original question was about 4-8 cpus with 2.4/2.6 ... you
will almost certainly be a lot happier with a recent 2.6 kernel. A
great deal of work has been done to make it scale well.  But
there are still some dark corners that haven't been cleaned up
so you will have to run your target application and make your
own measurments on scaling.  If you find that it doesn't scale
well, then we'd like to hear about it here.

-Tony

Re: Kernel Scalability

[l.genoni]

On Wed, 2 May 2007, Daniel J Blueman wrote:

> Date: Wed, 2 May 2007 16:43:44 +0100
> From: Daniel J Blueman <daniel.blueman@gmail.com>
> To: Rajib Majumder <rajibm2005@gmail.com>
> Cc: Linux Kernel <linux-kernel@vger.kernel.org>
> Subject: Re: Kernel Scalability
> Resent-Date: Wed, 02 May 2007 17:44:58 +0200
> Resent-From: <l.genoni@sns.it>
> 
> On 2 May, 14:00, "Rajib Majumder" <rajibm2005@gmail.com> wrote:
>> Hi,
>> 
>> I am wondering if 2.4.x/2.6.x kernel is scalable enough to run on
>> 8-CPU hardware. Do we have any scalability comparison data between
>> 2.4/2.6 kernels and beyond 4-CPU?
>> 
>> If yes, is the scalablity is near linear?
>> 
>> Any input is appreciated.
>
> There are 128-processor IA64 systems which run recent 2.6 kernels out
> there; the per-processor counters, RCU and page-fault scalability work
> has been instrumental to the necessary scaling for decent resource
> usage on these.
>
> IIRC, there were some patches being developed to improve pagecache
> scalability lately too, but I guess it all depends on what kind of
> workload you have...
>
To stay on systems probably more familiar to the user who asked this 
question, there are also some 64 core X86_64 bot AMD and Intel out there, 
here the 2.6 kernel is doing 
very well even on those intel CPU with shared L2 cache.
I have some 16 and 32 core Opteron and never had scalability problems.
You have to pay a lot of attention to your kernel configuration  (100 HZ, 
just BKL preemption), and to the 
filesystems you decide to use.

Re: Kernel Scalability

[William Lee Irwin III]

On Wed, 2 May 2007, Daniel J Blueman wrote:
>> There are 128-processor IA64 systems which run recent 2.6 kernels out
>> there; the per-processor counters, RCU and page-fault scalability work
>> has been instrumental to the necessary scaling for decent resource
>> usage on these.
>> IIRC, there were some patches being developed to improve pagecache
>> scalability lately too, but I guess it all depends on what kind of
>> workload you have...

On Wed, May 02, 2007 at 07:39:48PM +0200, l.genoni@oltrelinux.com wrote:
> To stay on systems probably more familiar to the user who asked this 
> question, there are also some 64 core X86_64 bot AMD and Intel out there, 
> here the 2.6 kernel is doing 
> very well even on those intel CPU with shared L2 cache.
> I have some 16 and 32 core Opteron and never had scalability problems.
> You have to pay a lot of attention to your kernel configuration  (100 HZ, 
> just BKL preemption), and to the 
> filesystems you decide to use.

One doesn't typically experience scalability problems per se. One
rather compares slopes of scaling lines and notes relative scaling
across kernels, be they versions of the same kernel or different
codebases altogether. It's still possible to have scalability
problems, of course, where things basically slowdown severely or
appear to stop entirely.

Nonlinear scalability is generally considered to be where the limits
of scaling set in. There is usually linear scaling until some number
of processors, and then the limit is where the linearity breaks down.
Linear scalability is defined as adding resources (e.g. cpus) to the
system resulting in a linear increase in performance.

To characterize a kernel's scalability, one should say three things:
(1) workload/benchmark
(2) linear scaling coefficient
(3) upper limit on linearity

So you would say that when running dbench, Linux scales with a
coefficient of 0.85 up to 78 processors or similar.


-- wli

Re: Kernel Scalability

[Christoph Lameter]

On Wed, 2 May 2007, l.genoni@oltrelinux.com wrote:

> To stay on systems probably more familiar to the user who asked this question,
> there are also some 64 core X86_64 bot AMD and Intel out there, here the 2.6
> kernel is doing very well even on those intel CPU with shared L2 cache.
> I have some 16 and 32 core Opteron and never had scalability problems.
> You have to pay a lot of attention to your kernel configuration  (100 HZ, just
> BKL preemption), and to the filesystems you decide to use.

It becomes a hardware issue at higher cpu counts. Commodity hardware is 
not designed to handle these high processor counts thus workarounds like 
listed above become necessary. Our 512 processor systems have been running 
with 1024 HZ for a long time. Plus you really need a good filesystem 
(XFS).