Re: [Lse-tech] Re: [PATCH] new CSA patchset for 2.6.8

[John Hesterberg <jh@sgi.com>]

On Fri, Aug 27, 2004 at 07:42:18AM +0200, Guillaume Thouvenin wrote:
> On Thu, Aug 26, 2004 at 10:05:37PM +0200, Tim Schmielau wrote:
> > 
> > It should be easy to combine the data collection enhancements from
> > CSA and ELSA to provide a common superset of information.
> 
> ELSA uses current BSD accounting. The only difference with BSD is that
> accounting is done for a group of processes. I didn't use PAGG and
> rewrite something because I thought (I was wrong) that PAGG project
> wasn't maintained. I continue to maintain ELSA just because there is, 
> until today, no solution for doing job accounting. 
> So, the data collection enhancements from ELSA is not very useful.
> 
> > With the new BSD acct v3 format, it should be possible to do per job
> > accounting entirely from userspace, using pid and ppid information to
> > reconstruct the process tree and some userland database for the
> > pid -> job mapping. It would, however, be greatly simplified if the
> > accounting records provided some kind of job id, and some indicator
> > whether or not this process was the last of a job (group).
> 
> I like this solution.
> In fact what I proposed was to have PAGG and a modified BSD accounting
> that can be used with PAGG as both are already in the -mm tree. But
> manage group of processes from userspace is, IMHO, a better solution as
> modifications in the kernel will be minimal. 

The kernel part of linux-job is a module that uses PAGG, and 
isn't difficult.  We've been running it in production for a
couple years.

I don't think a kernel-based job is a requirement, though,
so I'd like to hear more about how you'd do it otherwise.

The other comments about only one acct record per job vs one
per process might be important, and that might mean the kernel
has to know about the job.

>   Therefore the solution could be to enhance BSD accounting with data
> collection from CSA and provide per job accounting with a userspace
> mechanism. Sounds great to me... 
> 
> Best,
> Guillaume

How does the BSD accounting define jobs?
What determines the job that a process is part of?

An important aspect of linux-job (ie the job part of the pagg/job/csa
stack) is that it is inescapable.  The user doesn't get to determine or
change their job (unlike process groups).  For true accounting, that
determines the real $$$ chargebacks on shared machines, this is
necessary.

Another aspect of jobs that isn't directly related to accounting
is that it gives users and admins a way to query, and kill :-),
all the processes that are part of the job.  The inescapable part
is again important...you can't fork off a process and detach it from
the job to hide it.  In fact, I've heard that some sites use pagg/job
without CSA for this reason.  It might have been an ISP or ASP, and
they liked the containment linux-job provided.

John