Re: [RFC] Multiprocessor Control Interfaces

Re: [RFC] Multiprocessor Control Interfaces

[Jason Baietto]

Some people have asked me for more information before they'll attempt
to download and play with this stuff, so here's the --help output from
the latest version of the run(1) command:


Usage: run [OPTIONS] { COMMAND [ARGS] | PROCESS_SPECIFIER }
Set scheduling parameters and CPU bias for a new process or a list
of existing processes.

OPTIONS can be one or more of the following options:

   -b, --bias=LIST        Set the CPU bias to the LIST of CPUs;
                          CPUs are numbered starting from 0
   -s, --policy=POLICY    Set the scheduling policy to POLICY
   -P, --priority=LEVEL   Set the scheduling priority to LEVEL
   -q, --quantum=QUANTUM  Set the SCHED_RR quantum to QUANTUM
   -v, --version          Output version information and exit
   -h, --help             Display this help and exit

PROCESS_SPECIFIER is exactly one of the following options:

   -p, --pid=LIST         Specify LIST of existing PIDs to modify
   -g, --group=LIST       Specify LIST of process groups to modify; all
                          existing processes in the groups will be modified
   -u, --user=LIST        Specify LIST of users to modify; all existing
                          processes owned by the users will be modified
   -n, --name=LIST        Specify LIST of existing process names to modify

Multiple comma separated values can be specified for all LISTs and ranges
are allowed where appropriate (e.g. "run -b 0,1-3 autopilot").

See the run(1) man page for more information.


Take care,
Jason


> Hello All,
> 
> I'm currently working on adding multiprocessor control interfaces
> to Linux.  My current efforts can be found here:
> 
>    http://www.ccur.com/realtime/oss
> 
> These are clean-room implementations of similar tools that have
> been available in our proprietary *nix for quite some time, and
> so the interfaces have a fair amount of mileage under their belts.
> Note that the scope is somewhat wider than just MP.
> 
> There has been some discussion of "chaff" and other interfaces
> recently on this list, so in an effort to hopefully move towards
> a standard more quickly I've gotten permission from my employer
> to GPL the code I've written.  I'm very interested in comments
> and feedback on any or all of this work.
> 
> Here's the README file from the package:
> 
> 
> This package contains:
> 
>    run(1)
>       A multiprocessor control command line tool.
> 
>    mpadvise(3)
>       A multiprocessor control library interface.
> 
> These services rely upon Robert Love's CPU Affinity patch
> (version 2.4.16-1 was used for testing) which is available here:
> 
>    http://www.kernel.org/pub/linux/kernel/people/rml/cpu-affinity/v2.4/
> 
> To build the code, simply unpack it and type "make".  The code has
> been tested on Red Hat 7.1 and 7.2 systems, though it is still
> fairly new and almost certainly contains bugs.
> 
> An attempt was made to abstract the "cpuset" representation of
> the current system in order to have binaries that in theory
> could work on systems with more than 32 cpus.  For this to work,
> the run(1) command would need to be linked against a shared
> mpadvise(3) library (currently only a static library is made).
> 
> This code is being released in the hopes that it will become
> the basis for the Linux multiprocessor control standard interfaces.
> I am very interested in getting feedback on this package,
> so please contact me via email or LKML if you have any.
> 
> This source code is licensed under the GNU GPL Version 2.
> Copyright (C) 2001  Concurrent Computer Corporation
> 
> --
> Jason Baietto
> jason.baietto@ccur.com
> http://www.ccur.com/realtime/oss

Re: [RFC] Multiprocessor Control Interfaces

[Tim Hockin]

Please check, also:

http://www.hockin.org/~thockin/pset.   Not finished, but could be.


> Some people have asked me for more information before they'll attempt
> to download and play with this stuff, so here's the --help output from
> the latest version of the run(1) command:
> 
> 
> Usage: run [OPTIONS] { COMMAND [ARGS] | PROCESS_SPECIFIER }
> Set scheduling parameters and CPU bias for a new process or a list
> of existing processes.
> 
> OPTIONS can be one or more of the following options:
> 
>    -b, --bias=LIST        Set the CPU bias to the LIST of CPUs;
>                           CPUs are numbered starting from 0
>    -s, --policy=POLICY    Set the scheduling policy to POLICY
>    -P, --priority=LEVEL   Set the scheduling priority to LEVEL
>    -q, --quantum=QUANTUM  Set the SCHED_RR quantum to QUANTUM
>    -v, --version          Output version information and exit
>    -h, --help             Display this help and exit
> 
> PROCESS_SPECIFIER is exactly one of the following options:
> 
>    -p, --pid=LIST         Specify LIST of existing PIDs to modify
>    -g, --group=LIST       Specify LIST of process groups to modify; all
>                           existing processes in the groups will be modified
>    -u, --user=LIST        Specify LIST of users to modify; all existing
>                           processes owned by the users will be modified
>    -n, --name=LIST        Specify LIST of existing process names to modify
> 
> Multiple comma separated values can be specified for all LISTs and ranges
> are allowed where appropriate (e.g. "run -b 0,1-3 autopilot").
> 
> See the run(1) man page for more information.
> 
> 
> Take care,
> Jason
> 
> 
> > Hello All,
> > 
> > I'm currently working on adding multiprocessor control interfaces
> > to Linux.  My current efforts can be found here:
> > 
> >    http://www.ccur.com/realtime/oss
> > 
> > These are clean-room implementations of similar tools that have
> > been available in our proprietary *nix for quite some time, and
> > so the interfaces have a fair amount of mileage under their belts.
> > Note that the scope is somewhat wider than just MP.
> > 
> > There has been some discussion of "chaff" and other interfaces
> > recently on this list, so in an effort to hopefully move towards
> > a standard more quickly I've gotten permission from my employer
> > to GPL the code I've written.  I'm very interested in comments
> > and feedback on any or all of this work.
> > 
> > Here's the README file from the package:
> > 
> > 
> > This package contains:
> > 
> >    run(1)
> >       A multiprocessor control command line tool.
> > 
> >    mpadvise(3)
> >       A multiprocessor control library interface.
> > 
> > These services rely upon Robert Love's CPU Affinity patch
> > (version 2.4.16-1 was used for testing) which is available here:
> > 
> >    http://www.kernel.org/pub/linux/kernel/people/rml/cpu-affinity/v2.4/
> > 
> > To build the code, simply unpack it and type "make".  The code has
> > been tested on Red Hat 7.1 and 7.2 systems, though it is still
> > fairly new and almost certainly contains bugs.
> > 
> > An attempt was made to abstract the "cpuset" representation of
> > the current system in order to have binaries that in theory
> > could work on systems with more than 32 cpus.  For this to work,
> > the run(1) command would need to be linked against a shared
> > mpadvise(3) library (currently only a static library is made).
> > 
> > This code is being released in the hopes that it will become
> > the basis for the Linux multiprocessor control standard interfaces.
> > I am very interested in getting feedback on this package,
> > so please contact me via email or LKML if you have any.
> > 
> > This source code is licensed under the GNU GPL Version 2.
> > Copyright (C) 2001  Concurrent Computer Corporation
> > 
> > --
> > Jason Baietto
> > jason.baietto@ccur.com
> > http://www.ccur.com/realtime/oss
> 
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: [RFC] Multiprocessor Control Interfaces

[Jason Baietto]

On Mon, 2001-12-10 at 20:38, Tim Hockin wrote:
>
> http://www.hockin.org/~thockin/pset.   Not finished, but could be.

Yes, many similarities.  I like the idea of having a single system
call that provides so much multiprocessor control and information.
If such a system call was ever standardized upon, I would gladly
support it in my "run" tool, though I fear that it would always be
functionally limited without /proc present.

Re: [RFC] Multiprocessor Control Interfaces

[Tim Hockin]

Jason Baietto wrote:
> 
> On Mon, 2001-12-10 at 20:38, Tim Hockin wrote:
> >
> > http://www.hockin.org/~thockin/pset.   Not finished, but could be.
> 
> Yes, many similarities.  I like the idea of having a single system
> call that provides so much multiprocessor control and information.
> If such a system call was ever standardized upon, I would gladly
> support it in my "run" tool, though I fear that it would always be
> functionally limited without /proc present.

Adding /proc/ interfaces like cpus_allowed or similar is just gravy on top
of pset, which I think is a good interface.

-- 
Tim Hockin
Systems Software Engineer
Sun Microsystems, Cobalt Server Appliances
thockin@sun.com

Re: [RFC] Multiprocessor Control Interfaces

[Jason Baietto]

On Tue, 2001-12-11 at 13:16, Tim Hockin wrote:
> Jason Baietto wrote:
>
> > support it in my "run" tool, though I fear that it would always be
> > functionally limited without /proc present.
> 
> Adding /proc/ interfaces like cpus_allowed or similar is just gravy
> on top of pset, which I think is a good interface.

Sorry for the ambiguity.  My comment was about the functionality of
the "run" tool itself being limited without /proc support.

Although I agree that ultimately "run" is gravy, our customers think
it's pretty tasty gravy :-)  It's very convenient to write a shell
script to start up a complex application without having to reference
pids anywhere and without having to modify any of the programs in
the application to make them CPU affinity aware.  Combined with run's
ability to tune scheduling parameters, you have a very simple but
powerful way to experiment with and tune application performance.

The run tool also allows you to quickly move processes around by
specifying lists of pids, process names, process groups or users,
but /proc is needed for anything more interesting than pids.

Take care,
Jason
--
Jason Baietto
jason.baietto@ccur.com
http://www.ccur.com/realtime/oss

[RFC] Multiprocessor Control Interfaces

[Jason Baietto]

Hello All,

I'm currently working on adding multiprocessor control interfaces
to Linux.  My current efforts can be found here:

   http://www.ccur.com/realtime/oss

These are clean-room implementations of similar tools that have
been available in our proprietary *nix for quite some time, and
so the interfaces have a fair amount of mileage under their belts.
Note that the scope is somewhat wider than just MP.

There has been some discussion of "chaff" and other interfaces
recently on this list, so in an effort to hopefully move towards
a standard more quickly I've gotten permission from my employer
to GPL the code I've written.  I'm very interested in comments
and feedback on any or all of this work.

Here's the README file from the package:


This package contains:

   run(1)
      A multiprocessor control command line tool.

   mpadvise(3)
      A multiprocessor control library interface.

These services rely upon Robert Love's CPU Affinity patch
(version 2.4.16-1 was used for testing) which is available here:

   http://www.kernel.org/pub/linux/kernel/people/rml/cpu-affinity/v2.4/

To build the code, simply unpack it and type "make".  The code has
been tested on Red Hat 7.1 and 7.2 systems, though it is still
fairly new and almost certainly contains bugs.

An attempt was made to abstract the "cpuset" representation of
the current system in order to have binaries that in theory
could work on systems with more than 32 cpus.  For this to work,
the run(1) command would need to be linked against a shared
mpadvise(3) library (currently only a static library is made).

This code is being released in the hopes that it will become
the basis for the Linux multiprocessor control standard interfaces.
I am very interested in getting feedback on this package,
so please contact me via email or LKML if you have any.

This source code is licensed under the GNU GPL Version 2.
Copyright (C) 2001  Concurrent Computer Corporation

--
Jason Baietto
jason.baiett@ccur.com
http://www.ccur.com/realtime/oss

Re: [RFC] Multiprocessor Control Interfaces

[Robert Love]

On Mon, 2001-12-10 at 15:14, Jason Baietto wrote:

> I'm currently working on adding multiprocessor control interfaces
> to Linux.  My current efforts can be found here:
> 
>    http://www.ccur.com/realtime/oss
> 
> These are clean-room implementations of similar tools that have
> been available in our proprietary *nix for quite some time, and
> so the interfaces have a fair amount of mileage under their belts.
> Note that the scope is somewhat wider than just MP.

Ahh, very neat.  This is a useful tool.

One idea would be to allow users to set CPUs processes _can't_ run on. 
On high-end systems sometimes a CPU is affined to a particular IRQ (say,
network interface).  Another situation is where you bind a RT task to a
given CPU.  In these situations, you want everything else to _not_ run
on the CPUs.  I.e., `run --bind=!1' (note its easy to generate the
bitmask here too, by ANDing the inverse of the given CPU against -1).

At any rate, what is needed most is to standardize on an interface for
these scheduling mechanisms.  I guess its just CPU affinity we have to
go ... since not much progress was made of my (proc-based) method vs.
Ingo's (syscall-based) method, at this point either of the two being
merged would make me happy.

> These services rely upon Robert Love's CPU Affinity patch
> (version 2.4.16-1 was used for testing) which is available here:
> 
>    http://www.kernel.org/pub/linux/kernel/people/rml/cpu-affinity/v2.4/

I assume you have no problems with it ... I think I'd like to add the
change that the CPUs reported correspond to the physical CPU number and
not the logical value we derive.  On x86 this won't make a difference,
but its a proper method I suspect.

	Robert Love

Re: [RFC] Multiprocessor Control Interfaces

[Jason Baietto]

On Tue, 2001-12-11 at 01:29, Robert Love wrote:
> 
> One idea would be to allow users to set CPUs processes _can't_
> run on.

A technique that many of our customers currently use with the run
command is to shield a sensitive CPU in rc.sysinit by simply using run
to bias the init process.  For example, on a four CPU system, the
following command:

   run --bias 1-3 --pid 1

would bias init to only run on CPUs 1, 2 or 3.  Since all children
inherit CPU affinity, this effectively makes CPU 0 off limits for any
process that doesn't explicitly bias itself to CPU 0 via "run".

> On high-end systems sometimes a CPU is affined to a particular IRQ
> (say, network interface).  Another situation is where you bind a RT
> task to a given CPU.  In these situations, you want everything else
> to _not_ run on the CPUs.  I.e., `run --bind=!1' (note its easy to
> generate the bitmask here too, by ANDing the inverse of the given
> CPU against -1).

I like it.  I will add the "!" syntax to the next release of run.
However, I suspect that this is a more error prone method of providing
generic process shielding than the init method discussed above as
processes that don't explicitly get biased can still find their way to
the shielded CPU.

> At any rate, what is needed most is to standardize on an interface for
> these scheduling mechanisms.  I guess its just CPU affinity we have to
> go ... since not much progress was made of my (proc-based) method vs.
> Ingo's (syscall-based) method, at this point either of the two being
> merged would make me happy.

I will happily add support to "run" for Ingo's system calls if they
get merged.  However, many of the more powerful features of the "run"
command currently require /proc for other reasons.  For example,
setting the CPU bias for all processes in a specified list of process
groups, or setting the CPU bias for all processes owned by a specified
list of users all require that I walk /proc to find matching processes.

Also, regarding the mpadvise(3) library service that I've proposed,
commands like MPA_CPU_ACTIVE and MPA_PRC_GETRUN currently parse files
in /proc to determine the list of active CPUs on the system and the
CPU that a process is currently running on.  Thus, until I can get all
of the CPU-centric information that I need via system calls, adding
Ingo's system calls doesn't help me too much (though they would allow
"run" to do simple CPU biasing tasks on systems without /proc).

> I assume you have no problems with it ... 

It's exactly what I needed.  I've been using it for a few weeks
without any problems.

> I think I'd like to add the change that the CPUs reported correspond
> to the physical CPU number and not the logical value we derive.  On
> x86 this won't make a difference, but its a proper method I suspect.

I don't have a strong opinion here, though note that in our
proprietary RTOS our policy is to try to never expose physical values
to the user unless they really, really need to know them.  If you
change the interface to physical values, I will probably abstract back
to logical inside my code so the user still deals with logical (though
I could add a --physical option to force the bias to be interpreted as
physical values).

Take care,
Jason