[RFC] dynsched - different cpu schedulers per cpuset

[RFC] dynsched - different cpu schedulers per cpuset

[Felix Obenhuber]

Hej,

we're a student group witch is working on a research project concerning
the ability to switch the cpu scheduler of the linux kernel at runtime.

We use Peter Williams Plugsched patch [1] to get an interface for the
different scheduler implementations. Some month ago we started to modify
the code to allow different scheduler running on each cpu on an SMP
system. The cpu<->scheduler mapping is controlled via cpusets. Thus you
can switch the scheduler for a cpuset containing multiple cpus and
keep the rest untouched.

The project is hosted on Sourceforge [2] and the current patch applies
against 2.6.18 patched with plugsched.

Threre are still lots of issues - especially the migration of tasks
between cpus with different schedulers is quite buggy.

Switching the scheduler on non smp configured systems works fine (tested
 x86) 

Refer the project instruction site [3] for further information and usage /
patch instructions.

We'd be quite happy, if someone could take a look at what we've done
to gain some feedback/suggestions about the used techniques and
implementation. Some changes are already queued.

The project documentation (description/benchmarks/usage/bugs) is in
progress and will be completed in about 4 weeks.

Thanks a lot.

cheers,

Felix


[1] http://sourceforge.net/projects/cpuse/
[2] http://sourceforge.net/projects/dynsched/
[3] http://dynsched.sourceforge.net
-- 
Felix Obenhuber felixatobenhuber.de
www.obenhuber.de/felix
GPG: F696D489
Sat Nov 18 15:56:31 CET 2006

Re: [RFC] dynsched - different cpu schedulers per cpuset

[Paul Jackson]

Felix wrote:
> The cpu<->scheduler mapping is controlled via cpusets. Thus you
> can switch the scheduler for a cpuset containing multiple cpus and
> keep the rest untouched.

I don't have comments on the main focus of this work - schedulers are
not my expertise.

I just noticed this lkml post because of my interest in cpusets.

You should take a look at the work of Paul Menage (added to the
cc list), who is splitting the cpuset code into:
 1) a generic "container" mechanism,
 2) separate CPU and Memory "controllers", and
 3) various other additional "controllers".

See Paul Menage's most recent patch proposal at:
  http://lkml.org/lkml/2006/11/17/217
  Subject: [PATCH 0/6] Multi-hierarchy Process Containers
  Date:    Fri, 17 Nov 2006 11:11:59 -0800

The container mechanism uses a virtual file system derived from
the cpuset code to provide a file system style (hierarchical names
and classic Unix style file and directory permissions) naming of a
partitioning of the tasks on a system.

By partitioning here, I mean a division of the tasks into several
subsets, aka partition elements, which are non-overlapping and covering.

That is, each task is in one and only one of the partition elements,
these partitions elements are named by the directories in the container
file system, and the regular files in the container file system provide
per-element attributes.

Then kernel facilities that can be considered as providing attributes
for and control of subsets of tasks is represented as a controller,
and attached to such a container.

Your dynamic scheduler mechanisms appear (from what I can tell after a
brief glance) to be a candidate for being such a controller.

The upshot of this is that, if your work should proceed and eventually
be considered for inclusion in the kernel (I have --no-- idea if that
would be a good idea, either for the purposes of your student group,
or for the kernel itself) then it would likely (if Menage's work
is accepted) need to be recast as a "controller" in Menage's terms,
not as an extension to cpusets.

If Menage succeeds, that should not actually be that big of a change,
either semantically, or in coding details.

Good luck.

-- 
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <pj@sgi.com> 1.925.600.0401

Re: [RFC] dynsched - different cpu schedulers per cpuset

[Paul Jackson]

pj wrote:
> See Paul Menage's most recent patch proposal at:
>   http://lkml.org/lkml/2006/11/17/217
>   Subject: [PATCH 0/6] Multi-hierarchy Process Containers
>   Date:    Fri, 17 Nov 2006 11:11:59 -0800

I'm behind the times.  Paul Menage's most recent proposal is at:
    http://lkml.org/lkml/2006/11/23/95
    Subject: [PATCH 0/7] Generic Process Containers (+ ResGroups/BeanCounters)
    Date:    Thu, 23 Nov 2006 04:08:48 -0800

-- 
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <pj@sgi.com> 1.925.600.0401

Re: [RFC] dynsched - different cpu schedulers per cpuset

[Paul Menage]

On 11/29/06, Paul Jackson <pj@sgi.com> wrote:
>
> Your dynamic scheduler mechanisms appear (from what I can tell after a
> brief glance) to be a candidate for being such a controller.

Possibly, if it was some kind of multi-level scheduler - i.e. a
top-level scheduler picks which container to run, and then a
configurable per-container scheduler picks a task from that container.

But (having glanced at the code even less than you) it sounded like it
was intended to be a single level scheduler, configured on a per-cpu
basis. In that case tying it to (exclusive) cpusets sounds like it
might be more reasonable.

Paul.

Re: [RFC] dynsched - different cpu schedulers per cpuset

[Felix Obenhuber]

> Possibly, if it was some kind of multi-level scheduler - i.e. a
> top-level scheduler picks which container to run, and then a
> configurable per-container scheduler picks a task from that container.

What do you think about Plugsched? Peter Williams introduced a simple
scheduler interface for scheduler testing purposes. It should be
possible to integrate into your CPU Controller. The CPU Controller seems
to be quite suited for that purpose, as far as I can appraise after
reading some of your documentation.

Unfortunately, I didn't know about CKRM some months ago, when we decided
to use cpusets. As I wrote, Dynsched is a student project at sunset and
it's future is quite ambiguous. Maybe we'll find some time to take a
look at.

> But (having glanced at the code even less than you) it sounded like it
> was intended to be a single level scheduler, configured on a per-cpu
> basis. In that case tying it to (exclusive) cpusets sounds like it
> might be more reasonable.

You're right. We tie up the scheduler to cpus with some per_cpu
functionality, and chose cpusets for management, cause we didn't want
add another interface (e.g sysfs files for each cpu). Cpusets is quite
adequate for our purpose, but there are lots of issues we're currently
dealing with. Think about (not exclusive) subsets with different
schedulers...

Felix