[RFC][PATCH 0/1] MAZE: Mazed processes monitor

[RFC][PATCH 0/1] MAZE: Mazed processes monitor

[Hirofumi Nakagawa]

Hi all.

This patch provides the mazed process monitor,named MAZE.
MAZE's purpose is to notify or terminate registered processes when they are mazed.
The definition of mazed process is a process using excessive CPU cycles,
that is long time keeping TASK_RUNNING state.
MAZE detects mazed processes and sends specified signals to them.
This implements a CGL (Carrier Grade Linux) requirement (AVL.14.0).

Possible uses:
* High-Availability system
* system using many Real-Time processes such as embedded

Please any comments!

Hirofumi Nakagawa

--- Usage
Add monitoring process.

% echo "1234 10000 20000 24 9" > /proc/maze/entries

The numeric values are "pid","soft limit [msec]","hard limit [msec]",
"soft signal" and "hard signal".


Get monitoring process list.

% cat /proc/maze/entries

--- Example
As follows example express the feature of MAZE function.

--
#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[])
{
    int state,i;
    char cmd[128];

    sprintf(cmd, "echo \"%d 10000 20000 24 9\" > /proc/maze/entries", getpid());
    system(cmd);

    if (!fork()) {
        printf("Mazed process start.\n");
        for(;;);
    }

    if (!fork()) {
        printf("Not mazed process start.\n");
        for(i = 0;i < 20; i++)
            sleep(1);
        printf("Not mazed process finish.\n");
        exit(0);
    }

    for (i = 0;i < 2; i++)
         wait(&state);

    return 0;
}
--
---

Re: [RFC][PATCH 0/1] MAZE: Mazed processes monitor

[Andrew Morton]

On Tue, 13 May 2008 20:47:28 +0900 Hirofumi Nakagawa <hnakagawa@miraclelinux.com> wrote:

> Hi all.
> 
> This patch provides the mazed process monitor,named MAZE.
> MAZE's purpose is to notify or terminate registered processes when they are mazed.
> The definition of mazed process is a process using excessive CPU cycles,
> that is long time keeping TASK_RUNNING state.
> MAZE detects mazed processes and sends specified signals to them.
> This implements a CGL (Carrier Grade Linux) requirement (AVL.14.0).
> 
> Possible uses:
> * High-Availability system
> * system using many Real-Time processes such as embedded
> 
> Please any comments!
> 
> Hirofumi Nakagawa
> 
> --- Usage
> Add monitoring process.
> 
> % echo "1234 10000 20000 24 9" > /proc/maze/entries
> 
> The numeric values are "pid","soft limit [msec]","hard limit [msec]",
> "soft signal" and "hard signal".
> 
> 
> Get monitoring process list.
> 
> % cat /proc/maze/entries
> 
> --- Example
> As follows example express the feature of MAZE function.
> 
> --
> #include <sys/types.h>
> #include <sys/wait.h>
> #include <unistd.h>
> 
> #include <stdio.h>
> #include <stdlib.h>
> 
> int main(int argc, char *argv[])
> {
>     int state,i;
>     char cmd[128];
> 
>     sprintf(cmd, "echo \"%d 10000 20000 24 9\" > /proc/maze/entries", getpid());
>     system(cmd);
> 
>     if (!fork()) {
>         printf("Mazed process start.\n");
>         for(;;);
>     }
> 
>     if (!fork()) {
>         printf("Not mazed process start.\n");
>         for(i = 0;i < 20; i++)
>             sleep(1);
>         printf("Not mazed process finish.\n");
>         exit(0);
>     }
> 
>     for (i = 0;i < 2; i++)
>          wait(&state);
> 
>     return 0;
> }

Please include (and maintain) all the above changelog information with
the patch itself rather than separately.  Because if I were to merge
this patch, I'd just have to copy-n-paste it all over anwyay.

Please send the text of AVL.14.0 (or a sufficient part of it for us to
understand the requirement) and maintain that within the changelog
also.

The immediate quesstion is, of course, "why does this need to be done
in-kernel?".  I am unable to answer that from the provided changelog
information, and I should be able to do that.

Thanks.

Re: [RFC][PATCH 0/1] MAZE: Mazed processes monitor

[Andi Kleen]

Hirofumi Nakagawa <hnakagawa@miraclelinux.com> writes:

> This patch provides the mazed process monitor,named MAZE.
> MAZE's purpose is to notify or terminate registered processes when they are mazed.
> The definition of mazed process is a process using excessive CPU cycles,
> that is long time keeping TASK_RUNNING state.

Isn't that very similar to RLIMIT_CPU? The main difference seems to be 
that they're regularly reset and that they can be more fine grained 
than seconds.

How about you implement a way to change the RLIMIT_CPU limit
of a running task (and possibly fix it to use a finer grained unit
if you need <1s resolution).

Then you could run a user space daemon running with lower priority that 
just regularly resets the RLIMIT_CPUs of all running processes.

If some RT process uses so much time that the user daemon cannot
keep up its cpu time limit will be eventually exceeded and it will
be killed.

I think that would be a far cleaner and generic way to implement
this.

-Andi

Re: [RFC][PATCH 0/1] MAZE: Mazed processes monitor

[Hirofumi Nakagawa]

Thank you for your comments and feedbacks.
I got valuable comments.
I'll study your comments closely and post take2.

Thanks.
Hirofumi Nakagawa

Re: [RFC][PATCH 0/1] MAZE: Mazed processes monitor

[Hirofumi Nakagawa]

Hi Andrew,

Thank you for your comment and review.

> The immediate quesstion is, of course, "why does this need to be done
> in-kernel?".  I am unable to answer that from the provided changelog
> information, and I should be able to do that.

It needs to be done in kernel because excessive CPU cycle usage
is directly detected watching kernel internal task_struct.state.
It's hard to get that information outside the kernel.

Additionally, this patch's aim is to implement a
CGL (Carrier Grade Linux) requirement (AVL.14.0).

There are specs that can not be implemented outside the kernel.
These are;
  - Communication between the monitoring process and the kernel
  - Ability to define policy based on process events including
    process/thread creation and exit, and,
  - Ability to set the CPU cycle threshold to a resolution of
    one millisecond.

Thanks.
Hirofumi Nakagawa

Re: [RFC][PATCH 0/1] MAZE: Mazed processes monitor

[Hirofumi Nakagawa]

Hi Andi,

Thank you for your comment and proposal.

> How about you implement a way to change the RLIMIT_CPU limit
> of a running task (and possibly fix it to use a finer grained unit
> if you need <1s resolution).
>
> Then you could run a user space daemon running with lower priority that just regularly resets the RLIMIT_CPUs of all running processes.
>
> If some RT process uses so much time that the user daemon cannot
> keep up its cpu time limit will be eventually exceeded and it will
> be killed.
>
> I think that would be a far cleaner and generic way to implement
> this.
>
> -Andi

I also watch normal processes, which run into infinite loop but
cleanly scheduled out by sched when a time slice expires.
In this case, your deamon still can reset the counter.

> Isn't that very similar to RLIMIT_CPU? The main difference seems to be
> that they're regularly reset and that they can be more fine grained
> than seconds.

The differences with MAZE and rlimit are as follows.
  - MAZE detects excessive CPU cycle usage, but rlimits limits total amount
    of CPU usage.
    MAZE can safely handle CPU intensive but correctly running processes.

  - User processes can add watched processes in MAZE.

  - MAZE allows users to choose a way how to act on the process,
    selecting which signal to send.

Following example shows the differences of features MZE and rlimit

----
void foo(void)
{
     for(;;);
}
----
This code receives a signal shortly,
if registered under either rlimit or MAZE.

----
void foo(void)
{
     for(;;) {
         sleep(1);
     }
}
----
This code receives a signal in case of rlimit.
But, it doesn't receive a signal under MAZE.
There are cases when you need to distinguish these two types.

Thanks.
Hirofumi Nakagawa