[Qemu-devel] Parallel make build fails on fast machine

[Qemu-devel] Parallel make build fails on fast machine

[Mark Cave-Ayland]

Hi all,

Today I tried transferring my QEMU development setup from my laptop onto a faster
desktop machine (Intel i7-6700) and was surprised to find my normal "full" build
script failing:

./configure --target-list='x86_64-softmmu sparc64-softmmu sparc-softmmu ppc-softmmu
arm-softmmu' --prefix=/home/build/rel-qemu-git
make V=1 -j2 install

Things start to go wrong around here:

cc -iquote /home/build/src/qemu/git/qemu/hw/i386/kvm -iquote hw/i386/kvm -iquote
/home/build/src/qemu/git/qemu/tcg -iquote /home/build/src/qemu/git/qemu/tcg/i386
-I/home/build/src/qemu/git/qemu/linux-headers
-I/home/build/src/qemu/git/qemu/linux-headers -iquote . -iquote
/home/build/src/qemu/git/qemu -iquote /home/build/src/qemu/git/qemu/accel/tcg -iquote
/home/build/src/qemu/git/qemu/include -I/usr/include/pixman-1
-I/home/build/src/qemu/git/qemu/dtc/libfdt -Werror  -pthread -I/usr/include/glib-2.0
-I/usr/lib/x86_64-linux-gnu/glib-2.0/include -fPIE -DPIE -m64 -mcx16 -D_GNU_SOURCE
-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls
-Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common
-fwrapv -std=gnu99  -Wendif-labels -Wno-shift-negative-value
-Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security
-Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration
-Wold-style-definition -Wtype-limits -fstack-protector-strong
-I/usr/include/p11-kit-1   -I/usr/include/libpng16
-I/home/build/src/qemu/git/qemu/capstone/include  -I../linux-headers -iquote ..
-iquote /home/build/src/qemu/git/qemu/target/i386 -DNEED_CPU_H -iquote
/home/build/src/qemu/git/qemu/include -MMD -MP -MT hw/i386/kvm/i8259.o -MF
hw/i386/kvm/i8259.d -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -g   -c -o
hw/i386/kvm/i8259.o /home/build/src/qemu/git/qemu/hw/i386/kvm/i8259.c
exec.o: In function `reclaim_ramblock':
/home/build/src/qemu/git/qemu/exec.c:2471: undefined reference to `qemu_ram_munmap'
/home/build/src/qemu/git/qemu/exec.c:2475: undefined reference to `qemu_anon_ram_free'
exec.o: In function `cpu_notify_map_clients_locked':
/home/build/src/qemu/git/qemu/exec.c:3579: undefined reference to `qemu_bh_schedule'
exec.o: In function `mtree_print_phys_entries':
/home/build/src/qemu/git/qemu/exec.c:4169: undefined reference to `qemu_printf'
/home/build/src/qemu/git/qemu/exec.c:4171: undefined reference to `qemu_printf'
/home/build/src/qemu/git/qemu/exec.c:4177: undefined reference to `qemu_printf'
/home/build/src/qemu/git/qemu/exec.c:4175: undefined reference to `qemu_printf'
/home/build/src/qemu/git/qemu/exec.c:4167: undefined reference to `qemu_printf'
exec.o:/home/build/src/qemu/git/qemu/exec.c:4173: more undefined references to
`qemu_printf' follow
exec.o: In function `rcu_read_lock':
/home/build/src/qemu/git/qemu/include/qemu/rcu.h:82: undefined reference to `rcu_gp_ctr'
/home/build/src/qemu/git/qemu/include/qemu/rcu.h:83: undefined reference to `rcu_reader'
exec.o: In function `rcu_read_unlock':
/home/build/src/qemu/git/qemu/include/qemu/rcu.h:93: undefined reference to `rcu_reader'
/home/build/src/qemu/git/qemu/include/qemu/rcu.h:109: undefined reference to
`rcu_gp_event'
exec.o: In function `rcu_read_lock':
/home/build/src/qemu/git/qemu/include/qemu/rcu.h:78: undefined reference to `rcu_reader'
exec.o: In function `cpu_physical_memory_set_dirty_range':
/home/build/src/qemu/git/qemu/include/exec/ram_addr.h:293: undefined reference to
`bitmap_set_atomic'
/home/build/src/qemu/git/qemu/include/exec/ram_addr.h:297: undefined reference to
`bitmap_set_atomic'
/home/build/src/qemu/git/qemu/include/exec/ram_addr.h:301: undefined reference to
`bitmap_set_atomic'
exec.o: In function `qemu_ram_setup_dump':
/home/build/src/qemu/git/qemu/exec.c:2020: undefined reference to `qemu_madvise'
exec.o: In function `cpu_physical_memory_range_includes_clean':
/home/build/src/qemu/git/qemu/include/exec/ram_addr.h:236: undefined reference to
`rcu_reader'
exec.o: In function `cpu_physical_memory_all_dirty':
/home/build/src/qemu/git/qemu/include/exec/ram_addr.h:190: undefined reference to
`find_next_zero_bit'
exec.o: In function `rcu_read_lock':
/home/build/src/qemu/git/qemu/include/qemu/rcu.h:78: undefined reference to `rcu_reader'
exec.o: In function `cpu_physical_memory_all_dirty':
/home/build/src/qemu/git/qemu/include/exec/ram_addr.h:190: undefined reference to
`find_next_zero_bit'
exec.o: In function `rcu_read_lock':
/home/build/src/qemu/git/qemu/include/qemu/rcu.h:78: undefined reference to `rcu_reader'
exec.o: In function `cpu_physical_memory_all_dirty':
/home/build/src/qemu/git/qemu/include/exec/ram_addr.h:190: undefined reference to
`find_next_zero_bit'
exec.o: In function `rcu_read_lock':
/home/build/src/qemu/git/qemu/include/qemu/rcu.h:78: undefined reference to `rcu_reader'
exec.o: In function `rcu_read_unlock':
/home/build/src/qemu/git/qemu/include/qemu/rcu.h:109: undefined reference to
`rcu_gp_event'
exec.o: In function `rcu_read_lock':
/home/build/src/qemu/git/qemu/include/qemu/rcu.h:82: undefined reference to `rcu_gp_ctr'
/home/build/src/qemu/git/qemu/include/qemu/rcu.h:78: undefined reference to `rcu_reader'
/home/build/src/qemu/git/qemu/include/qemu/rcu.h:78: undefined reference to `rcu_reader'
exec.o: In function `cpu_physical_memory_test_and_clear_dirty':
/home/build/src/qemu/git/qemu/exec.c:1378: undefined reference to
`bitmap_test_and_clear_atomic'
exec.o: In function `rcu_read_unlock':
/home/build/src/qemu/git/qemu/include/qemu/rcu.h:93: undefined reference to `rcu_reader'
/home/build/src/qemu/git/qemu/include/qemu/rcu.h:94: undefined reference to `rcu_reader'
/home/build/src/qemu/git/qemu/include/qemu/rcu.h:107: undefined reference to `rcu_reader'
/home/build/src/qemu/git/qemu/include/qemu/rcu.h:108: undefined reference to `rcu_reader'
/home/build/src/qemu/git/qemu/include/qemu/rcu.h:109: undefined reference to
`rcu_gp_event'
/home/build/src/qemu/git/qemu/include/qemu/rcu.h:109: undefined reference to
`qemu_event_set'
exec.o: In function `rcu_read_lock':
/home/build/src/qemu/git/qemu/include/qemu/rcu.h:82: undefined reference to `rcu_gp_ctr'
exec.o: In function `cpu_exec_initfn':
/home/build/src/qemu/git/qemu/exec.c:958: undefined reference to `qemu_get_thread_id'
exec.o: In function `parse_cpu_option':
/home/build/src/qemu/git/qemu/exec.c:1011: undefined reference to `error_fatal'
/home/build/src/qemu/git/qemu/exec.c:998: undefined reference to `error_report'
/home/build/src/qemu/git/qemu/exec.c:1004: undefined reference to `error_report'
exec.o: In function `rcu_read_lock':

For reference I've uploaded the full build output to
https://www.ilande.co.uk/tmp/qemu/qemu-parallel-make-fail.txt. I've also confirmed
that the build completes fine when removing the -j parameter from the make
incantation above.


ATB,

Mark.

Re: [Qemu-devel] Parallel make build fails on fast machine

[Peter Maydell]

On Sat, 6 Jul 2019 at 10:59, Mark Cave-Ayland
<mark.cave-ayland@ilande.co.uk> wrote:
>
> Hi all,
>
> Today I tried transferring my QEMU development setup from my laptop onto a faster
> desktop machine (Intel i7-6700) and was surprised to find my normal "full" build
> script failing:
>
> ./configure --target-list='x86_64-softmmu sparc64-softmmu sparc-softmmu ppc-softmmu
> arm-softmmu' --prefix=/home/build/rel-qemu-git
> make V=1 -j2 install

If you just do 'make' rather than 'make install' does it fail the same way?

thanks
-- PMM

Re: [Qemu-devel] Parallel make build fails on fast machine

[Mark Cave-Ayland]

On 06/07/2019 11:16, Peter Maydell wrote:

> On Sat, 6 Jul 2019 at 10:59, Mark Cave-Ayland
> <mark.cave-ayland@ilande.co.uk> wrote:
>>
>> Hi all,
>>
>> Today I tried transferring my QEMU development setup from my laptop onto a faster
>> desktop machine (Intel i7-6700) and was surprised to find my normal "full" build
>> script failing:
>>
>> ./configure --target-list='x86_64-softmmu sparc64-softmmu sparc-softmmu ppc-softmmu
>> arm-softmmu' --prefix=/home/build/rel-qemu-git
>> make V=1 -j2 install
> 
> If you just do 'make' rather than 'make install' does it fail the same way?

Interesting. A quick test shows that "make V=1 -j2" works fine, it's just "make V=1
-j2 install" which is failing.


ATB,

Mark.

Re: [Qemu-devel] Parallel make build fails on fast machine

[Peter Maydell]

On Sat, 6 Jul 2019 at 11:28, Mark Cave-Ayland
<mark.cave-ayland@ilande.co.uk> wrote:
>
> On 06/07/2019 11:16, Peter Maydell wrote:
> > If you just do 'make' rather than 'make install' does it fail the same way?
>
> Interesting. A quick test shows that "make V=1 -j2" works fine, it's just "make V=1
> -j2 install" which is failing.

Mmm. I suspect most people just do the plain make (with the make install
phase either never or as a second command afterwards), so missing-dependency
bugs between the install targets and the build targets are probably more
likely to lurk around. We should stil fix them, though, assuming we
can track them down...

thanks
-- PMM

Re: [Qemu-devel] Parallel make build fails on fast machine

[Daniel P. Berrangé]

On Sat, Jul 06, 2019 at 09:35:30PM +0100, Peter Maydell wrote:
> On Sat, 6 Jul 2019 at 11:28, Mark Cave-Ayland
> <mark.cave-ayland@ilande.co.uk> wrote:
> >
> > On 06/07/2019 11:16, Peter Maydell wrote:
> > > If you just do 'make' rather than 'make install' does it fail the same way?
> >
> > Interesting. A quick test shows that "make V=1 -j2" works fine, it's just "make V=1
> > -j2 install" which is failing.
> 
> Mmm. I suspect most people just do the plain make (with the make install
> phase either never or as a second command afterwards), so missing-dependency
> bugs between the install targets and the build targets are probably more
> likely to lurk around. We should stil fix them, though, assuming we
> can track them down...

The main 'install' target depends on 'all':

install: all $(if $(BUILD_DOCS),install-doc) install-datadir install-localstatedir \


but I think still allows 'install-doc', 'install-datadir' and
'install-localstatedir'  to parallelize wrt 'all'. I guess the fix is to
make each of those 'install-xxx' targets depend on 'all' instead.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Parallel make build fails on fast machine

[Philippe Mathieu-Daudé]

On 7/8/19 11:41 AM, Daniel P. Berrangé wrote:
> On Sat, Jul 06, 2019 at 09:35:30PM +0100, Peter Maydell wrote:
>> On Sat, 6 Jul 2019 at 11:28, Mark Cave-Ayland
>> <mark.cave-ayland@ilande.co.uk> wrote:
>>>
>>> On 06/07/2019 11:16, Peter Maydell wrote:
>>>> If you just do 'make' rather than 'make install' does it fail the same way?
>>>
>>> Interesting. A quick test shows that "make V=1 -j2" works fine, it's just "make V=1
>>> -j2 install" which is failing.
>>
>> Mmm. I suspect most people just do the plain make (with the make install
>> phase either never or as a second command afterwards), so missing-dependency
>> bugs between the install targets and the build targets are probably more
>> likely to lurk around. We should stil fix them, though, assuming we
>> can track them down...
> 
> The main 'install' target depends on 'all':
> 
> install: all $(if $(BUILD_DOCS),install-doc) install-datadir install-localstatedir \
> 
> 
> but I think still allows 'install-doc', 'install-datadir' and
> 'install-localstatedir'  to parallelize wrt 'all'. I guess the fix is to
> make each of those 'install-xxx' targets depend on 'all' instead.

I'm not sure I follow, are you suggesting this as a kludge for this
release, or you recommend this as a default?

Apparently only 'recurse-install' depends of 'all'.

Re: [Qemu-devel] Parallel make build fails on fast machine

[Daniel P. Berrangé]

On Mon, Jul 08, 2019 at 12:17:12PM +0200, Philippe Mathieu-Daudé wrote:
> On 7/8/19 11:41 AM, Daniel P. Berrangé wrote:
> > On Sat, Jul 06, 2019 at 09:35:30PM +0100, Peter Maydell wrote:
> >> On Sat, 6 Jul 2019 at 11:28, Mark Cave-Ayland
> >> <mark.cave-ayland@ilande.co.uk> wrote:
> >>>
> >>> On 06/07/2019 11:16, Peter Maydell wrote:
> >>>> If you just do 'make' rather than 'make install' does it fail the same way?
> >>>
> >>> Interesting. A quick test shows that "make V=1 -j2" works fine, it's just "make V=1
> >>> -j2 install" which is failing.
> >>
> >> Mmm. I suspect most people just do the plain make (with the make install
> >> phase either never or as a second command afterwards), so missing-dependency
> >> bugs between the install targets and the build targets are probably more
> >> likely to lurk around. We should stil fix them, though, assuming we
> >> can track them down...
> > 
> > The main 'install' target depends on 'all':
> > 
> > install: all $(if $(BUILD_DOCS),install-doc) install-datadir install-localstatedir \
> > 
> > 
> > but I think still allows 'install-doc', 'install-datadir' and
> > 'install-localstatedir'  to parallelize wrt 'all'. I guess the fix is to
> > make each of those 'install-xxx' targets depend on 'all' instead.
> 
> I'm not sure I follow, are you suggesting this as a kludge for this
> release, or you recommend this as a default?

I think it is the right solution in general.

> Apparently only 'recurse-install' depends of 'all'.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Parallel make build fails on fast machine

[Mark Cave-Ayland]

On 08/07/2019 11:19, Daniel P. Berrangé wrote:

> On Mon, Jul 08, 2019 at 12:17:12PM +0200, Philippe Mathieu-Daudé wrote:
>> On 7/8/19 11:41 AM, Daniel P. Berrangé wrote:
>>> On Sat, Jul 06, 2019 at 09:35:30PM +0100, Peter Maydell wrote:
>>>> On Sat, 6 Jul 2019 at 11:28, Mark Cave-Ayland
>>>> <mark.cave-ayland@ilande.co.uk> wrote:
>>>>>
>>>>> On 06/07/2019 11:16, Peter Maydell wrote:
>>>>>> If you just do 'make' rather than 'make install' does it fail the same way?
>>>>>
>>>>> Interesting. A quick test shows that "make V=1 -j2" works fine, it's just "make V=1
>>>>> -j2 install" which is failing.
>>>>
>>>> Mmm. I suspect most people just do the plain make (with the make install
>>>> phase either never or as a second command afterwards), so missing-dependency
>>>> bugs between the install targets and the build targets are probably more
>>>> likely to lurk around. We should stil fix them, though, assuming we
>>>> can track them down...
>>>
>>> The main 'install' target depends on 'all':
>>>
>>> install: all $(if $(BUILD_DOCS),install-doc) install-datadir install-localstatedir \
>>>
>>>
>>> but I think still allows 'install-doc', 'install-datadir' and
>>> 'install-localstatedir'  to parallelize wrt 'all'. I guess the fix is to
>>> make each of those 'install-xxx' targets depend on 'all' instead.
>>
>> I'm not sure I follow, are you suggesting this as a kludge for this
>> release, or you recommend this as a default?
> 
> I think it is the right solution in general.
> 
>> Apparently only 'recurse-install' depends of 'all'.

Something also looks a bit odd with distclean here on a fresh checkout:

build@ezio:~/src/qemu/git/tmp/qemu$ make distclean
  LD      recurse-clean.mo
cc: fatal error: no input files
compilation terminated.
rules.mak:118: recipe for target 'recurse-clean.mo' failed
make: *** [recurse-clean.mo] Error 1

A bit more experimentation shows that it's related to having multiple softmmu targets:

Works:

'./configure' '--target-list=x86_64-softmmu' '--prefix=/home/build/rel-qemu-git'
make V=1 -j4 install

'./configure' '--target-list=x86_64-softmmu arm-softmmu'
'--prefix=/home/build/rel-qemu-git'
make V=1 -j4 install

'./configure' '--target-list=x86_64-softmmu arm-softmmu ppc-softmmu'
'--prefix=/home/build/rel-qemu-git'
make V=1 -j4 install

Fails:

'./configure' '--target-list=x86_64-softmmu arm-softmmu ppc-softmmu sparc-softmmu'
'--prefix=/home/build/rel-qemu-git'
make V=1 -j4 install

'./configure' '--target-list=x86_64-softmmu arm-softmmu ppc-softmmu mips-softmmu'
'--prefix=/home/build/rel-qemu-git'
make V=1 -j4 install

Looking at the output it feels as if we're trying to build something that depends
upon libqemu.a before it's ready, but only with a long list of targets on a fast machine.


ATB,

Mark.

Re: [Qemu-devel] Parallel make build fails on fast machine

[Markus Armbruster]

Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk> writes:

> Something also looks a bit odd with distclean here on a fresh checkout:
>
> build@ezio:~/src/qemu/git/tmp/qemu$ make distclean
>   LD      recurse-clean.mo
> cc: fatal error: no input files
> compilation terminated.
> rules.mak:118: recipe for target 'recurse-clean.mo' failed
> make: *** [recurse-clean.mo] Error 1

This one should be fixed in master (commit 8d358a5ea08).  If it's still
broken for you, let me know.

Re: [Qemu-devel] Parallel make build fails on fast machine

[Mark Cave-Ayland]

On 11/07/2019 15:45, Markus Armbruster wrote:

> Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk> writes:
> 
>> Something also looks a bit odd with distclean here on a fresh checkout:
>>
>> build@ezio:~/src/qemu/git/tmp/qemu$ make distclean
>>   LD      recurse-clean.mo
>> cc: fatal error: no input files
>> compilation terminated.
>> rules.mak:118: recipe for target 'recurse-clean.mo' failed
>> make: *** [recurse-clean.mo] Error 1
> 
> This one should be fixed in master (commit 8d358a5ea08).  If it's still
> broken for you, let me know.

Yes, this is certainly looking much better! There still seems to be something wrong
with the tests/ subdirectory with your "make install" patch applied to git master if
I attempt a "make distclean" after a successful "make V=1 -j2 install" in an
out-of-tree build:

build@ezio:~/src/qemu/git/obj$ ../qemu/configure '--target-list=x86_64-softmmu
sparc64-softmmu sparc-softmmu ppc-softmmu arm-softmmu'
'--prefix=/home/build/rel-qemu-git' '--disable-pie' '--enable-debug'
build@ezio:~/src/qemu/git/obj$ make V=1 -j2 install

(lots of build output cut, but completes successfully)

build@ezio:~/src/qemu/git/obj$ make distclean
Makefile:85: rules.mak: No such file or directory
Makefile:437: tests/Makefile.include: No such file or directory
cat: VERSION: No such file or directory
Makefile:1127: tests/docker/Makefile.include: No such file or directory
Makefile:1128: tests/vm/Makefile.include: No such file or directory
make: *** No rule to make target 'tests/vm/Makefile.include'.  Stop.


ATB,

Mark.

Re: [Qemu-devel] Parallel make build fails on fast machine

[Markus Armbruster]

Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk> writes:

> On 11/07/2019 15:45, Markus Armbruster wrote:
>
>> Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk> writes:
>> 
>>> Something also looks a bit odd with distclean here on a fresh checkout:
>>>
>>> build@ezio:~/src/qemu/git/tmp/qemu$ make distclean
>>>   LD      recurse-clean.mo
>>> cc: fatal error: no input files
>>> compilation terminated.
>>> rules.mak:118: recipe for target 'recurse-clean.mo' failed
>>> make: *** [recurse-clean.mo] Error 1
>> 
>> This one should be fixed in master (commit 8d358a5ea08).  If it's still
>> broken for you, let me know.
>
> Yes, this is certainly looking much better! There still seems to be something wrong
> with the tests/ subdirectory with your "make install" patch applied to git master if
> I attempt a "make distclean" after a successful "make V=1 -j2 install" in an
> out-of-tree build:
>
> build@ezio:~/src/qemu/git/obj$ ../qemu/configure '--target-list=x86_64-softmmu
> sparc64-softmmu sparc-softmmu ppc-softmmu arm-softmmu'
> '--prefix=/home/build/rel-qemu-git' '--disable-pie' '--enable-debug'
> build@ezio:~/src/qemu/git/obj$ make V=1 -j2 install
>
> (lots of build output cut, but completes successfully)
>
> build@ezio:~/src/qemu/git/obj$ make distclean
> Makefile:85: rules.mak: No such file or directory
> Makefile:437: tests/Makefile.include: No such file or directory
> cat: VERSION: No such file or directory
> Makefile:1127: tests/docker/Makefile.include: No such file or directory
> Makefile:1128: tests/vm/Makefile.include: No such file or directory
> make: *** No rule to make target 'tests/vm/Makefile.include'.  Stop.

Double-checking: is this with my '[PATCH for-4.1] Makefile: Fix "make
install" when "make all" needs work' applied?

Re: [Qemu-devel] Parallel make build fails on fast machine

[Markus Armbruster]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Sat, Jul 06, 2019 at 09:35:30PM +0100, Peter Maydell wrote:
>> On Sat, 6 Jul 2019 at 11:28, Mark Cave-Ayland
>> <mark.cave-ayland@ilande.co.uk> wrote:
>> >
>> > On 06/07/2019 11:16, Peter Maydell wrote:
>> > > If you just do 'make' rather than 'make install' does it fail the same way?
>> >
>> > Interesting. A quick test shows that "make V=1 -j2" works fine, it's just "make V=1
>> > -j2 install" which is failing.
>> 
>> Mmm. I suspect most people just do the plain make (with the make install
>> phase either never or as a second command afterwards), so missing-dependency
>> bugs between the install targets and the build targets are probably more
>> likely to lurk around. We should stil fix them, though, assuming we
>> can track them down...
>
> The main 'install' target depends on 'all':
>
> install: all $(if $(BUILD_DOCS),install-doc) install-datadir install-localstatedir \
          $(if $(INSTALL_BLOBS),$(edk2-decompressed)) \
          recurse-install
>
>
> but I think still allows 'install-doc', 'install-datadir' and
> 'install-localstatedir'  to parallelize wrt 'all'. I guess the fix is to
> make each of those 'install-xxx' targets depend on 'all' instead.

Additionally, recurse-install can run in parallel with all, which is
wrong.  I just posted "[PATCH for-4.1] Makefile: Fix "make install" when
"make all" needs work" for that.