Re: [RFC/PATCH 0/3] Thinning the git toplevel directory

[Jeff King <peff@peff.net>]

On Wed, Feb 23, 2011 at 02:29:29AM -0600, Jonathan Nieder wrote:

>     If the project grows too much, all those "make" calls start to
>     take time.  It is possible for parallel "make" invocations to
>     trample on each other's work if they are not well coordinated.

Aren't there cases where no amount of coordination will help? E.g.,
let's say I have two subdirs, lib/ and cmds/. From the top-level I do
"make -j". This invokes two recursive sub-makes, one per directory. The
make in cmds/ sees that we haven't built stuff in lib/ yet. So it
invokes a recursive make in lib/. Now we have two parallel makes running
in lib/, stomping on each other. The problem is that no single make was
allowed to see the whole dependency tree.

I know GNU make does have some magic for communicating between recursive
makes. Does it handle this situation?

You can fix this with a rule like "only invoke recursive makes on
directories _below_ you, never above or to the side". But then you can't
run "make" from inside cmds/.

You could have a dependency in the top-level that says recursing into
cmds/ depends on having finished recursing into libs/. But that's not
strictly correct. You want to be working on part of what's in cmds/
(building .o files) in parallel with what's going on in lib/, and then
wait on lib/ only for the linking portion.

Those are all things that are trivial to handle in a single Makefile.

>  B. top-level Makefile slurps in Makefiles from subdirs.  Other
>     Makefiles would

I like this better, but...

>      - keep careful track of what directory "make" was run from; [*]
> [...]
>     [*] is a little hazy and sounds hackish.

Yeah, you have to be careful with paths. I think a more sane way would
be a single top-level Makefile that either contains everything, or
sources tidbits from subdirectories. But in either case, assumes it's
running from the top-level.

A dummy Makefile in each subdir that cd's to the toplevel and runs a
specific target. So from the top-level, "make" would build everything,
"make lib" would build stuff in the lib directory, and the Makefile in
lib/ would just do "cd .. && make lib".

It's not perfect, but it's simple and predictable.

-Peff