Re: McVoy's Clusters (was Re: latest linus-2.5 BK broken)

[Jesse Pollard <pollard@tomcat.admin.navo.hpc.mil>]

Nick LeRoy <nleroy@cs.wisc.edu>:
> 
> On Thursday 20 June 2002 12:23 pm, Jesse Pollard wrote:
> <snip>
> > You don't use compute servers much? The problems we are currently running
> > require the cluster (IBM SP) to have 100% uptime for a single job. that
> > job may run for several days. If a detected problem is reported (not yet
> > catastrophic) it is desired/demanded to checkpoint the users process.
> >
> > Currently, we can't - but should be able to by this fall.
> >
> > Having the users job checkpoint midway in it's computations will allow us
> > to remove a node from active service, substitute a different node, and
> > resume the users process without losing many hours of computation (we have
> > a maximum of 300 nodes for computation, another 30 for I/O and front end).
> 
> Have you tried Condor?  Condor is a "high throughput computing" package, 
> specifically targetted at such applications, with the ability to checkpoint & 
> migrate jobs, etc.  Condor is free as in beer, but currently not as in speech 
> (sorry), and is developed by the University of Wisconsin.  
> http://www.condorproject.org is the URL to learn more.  Version 6.4.0 is in 
> the process of being released and should be available within the next couple 
> of days.
> 
> Condor runs on Linux (x86 & Alpha), Solaris, IRIX, HPUX, Digital Unix, and 
> NT, although the NT usually lags the Unix releases.

Condor is designed for a relatively low performance network (10-100Mbit) and
not for things like an IBM SP switch which can carry Gbit data. It needs
availablility on SP-3/4 and Cray SV systems (not that we have problems
with checkpoint there). Also note:

	Cannot use IPC (pipes shared memory), which also leave out PVM/MPI
	job cannot use threads
	cannot use forks

In many of our cases, the jobs are split across many nodes, then spread
across multiple processors in a single node (SP 3 has 4 cpus per node,
SP 4 will have 8-32). The current scientific library uses PVM/MPI to determine
whether it is using shared memory or node/node RPC.

Tightly integrated models wouldn't work well with Condor (disclaimer:
based on a fast look by me, and I don't work on the current jobs).

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.