Re: Linux scheduler capabilities for batch jobs.

[Pavel Machek <pavel@ucw.cz>]

On Mon 2009-06-01 09:41:07, J Louis wrote:
> Hello All,
> 
> I have been writing user land server code and increasingly I find
> myself writing resource management code which I think would be better
> handled by the scheduler.  The problem can be described as "don't run
> this process if the machine is swapping."  I would think that this
> would be a common enough need that it was already in the kernel.  I'm
> hoping it is and I have simply overlooked it (part of the reason for
> this post,) but I've looked around a good bit and most of the
> scheduler enhancements have to do with real time and latency, not
> batch jobs.
> 
> My problem is analogous to a parallel make.  Say I have an 8 CPU
> machine, and I run "make -j8".  If the total memory of the 8 jobs
> throws the machine into swap, it begins to thrash and runtime is
> awful.  I believe this is aggravated by the scheduler trying to be

This seems to be racy by design.

make -j:
     launch gcc1
     gcc1 does preprocessing (10mb)  
     ok, still not swapping
     launch gcc2
     gcc2 does preprocessing (10mb)  
     ...
     gcc100 does preprocessing (10mb, 1GB total)
now gcc1..gcc100 start optimizing (100mb needed) and boom.

									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html