Re: [Lse-tech] [PATCH] cpusets - big numa cpu and memory placement

[Paul Jackson <pj@sgi.com>]

Martin wrote:
> but they're still close enough, that especially when programming
> them in combination, it seems silly to have 2 separate interfaces. 

The specific attributes needed of CKRM classes are not the same as those
needed for cpusets.

The semantics of the two are distinct -- each has different rules that
have little relevance to the other.

The typical uses of the two have little overlap.  More often than not,
the applications that customers want to run in isolation in cpusets are
not the same as those which customers want to run while sharing compute
resources with a managed balance.

Any merger of two separate mechanisms has its costs - the result risks
being less focused, larger.  These costs must be balanced with what's
called (in Corporate mergers) "synergy".

Programming these two "in combination" simply means using CKRM to manage
resources for some tasks running in a cpuset.  Neither capability or
interface gains in such a use by attempting to merge it with the other.

I can imagine running multiple cpusets, say SetA, SetB, and SetC, using
Nodes 0..31, 32..63, and 64..127, respectively.  Within each, running
the same suite of applications, say a DBMS, web server and credit card
payment handler.  Serving within each three classes of customers: Gold
(corporate), Silver (logged in individuals) and Bronze (anonymous web
surfers).  This is naturally a 3x3 two-dimensional space, not a flat
space that's nine units wide.  The two dimensions are orthogonal here.

No, it is not silly to have 2 separate interfaces.  What's silly is to
presume that everything that seems similar at the 10,000 foot level
should be combined.

The details matter.  Show me the synergy.

Do we have trains that float and ships that roll?

Is there much of a market for a hammer-saw?

It is fitting and proper for kernels to provide independent mechanisms,
and let user space connect them as it will.  Look at the actual hooks
in the kernel code to implement these two facilities.  One hooks the
scheduler and allocator to prohibit running on CPUs outside the cpuset,
and to prohibit allocating memory on Nodes outside the cpuset.  The other
hooks these places, and others, to bias their decisions in order to obtain
the requested balance of resource usage.

These are two distinct sets of hooks, perhaps on the same page of code,
but distinct in placement, logic, means and intention.

Ideally, the kernel provides a single, separate, orthogonal interface to
each mechanism it supports.

If this were a case of proposing two interfaces to the same mechanism,
or what should be the same mechanism, then you'd be 100% right.  We
should merge them.

Perhaps the proper place to resolve this discussion in is a detailed
examination of the kernel hooks required for CKRM and cpusets, the
hooks in the scheduler, allocator and such.

You have both patches available to you.  Examine them.  Especially
examine the hooks in the scheduler and allocator code.  These are not
the same hooks.  I defy you to make them the same and propose such with
a straight face.  If you do so successfully, I will sit up and take
notice.

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373