Re: [Qemu-devel] [PATCH v6 47/49] tests: add top-level make dependency for docker builds

["Alex Bennée" <alex.bennee@linaro.org>]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> On 06/08/2018 09:33 AM, Alex Bennée wrote:
>> One problem with satisfying your docker dependencies in a sub-make it
>> you might end up trying to satisfy the dependency multiple times. This
>> is especially a problem with debian-sid based cross compilers and CI
>> setups. We solve this by doing a docker build pass at the top level
>> before any sub-makes are called.
>>
>> We still need to satisfy dependencies in the Makefile.target call so
>> people can run tests from individual target directories. We introduce
>> a new Makefile.probe which gets called for each PROBE_TARGET and
>> allows us to build up the list. It does require multiply including
>> config-target.mak which shouldn't cause any issues as it shouldn't
>> define anything that clashes with config-host.mak. However we undefine
>> a few key variables each time around.
>>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> ---
>>  tests/Makefile.include   | 14 +++++++++++++-
>>  tests/tcg/Makefile.probe | 31 +++++++++++++++++++++++++++++++
>>  2 files changed, 44 insertions(+), 1 deletion(-)
>>  create mode 100644 tests/tcg/Makefile.probe
>>
>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>> index ca00247e36..049a387436 100644
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -931,7 +931,19 @@ BUILD_TCG_TARGET_RULES=$(patsubst %,build-tcg-tests-%, $(LINUX_USER_TARGETS))
>>  CLEAN_TCG_TARGET_RULES=$(patsubst %,clean-tcg-tests-%, $(LINUX_USER_TARGETS))
>>  RUN_TCG_TARGET_RULES=$(patsubst %,run-tcg-tests-%, $(LINUX_USER_TARGETS))
>>
>> -build-tcg-tests-%:
>> +# Probe for the Docker Builds we need for later
>> +DOCKER_BUILD_TARGETS:=
>> +$(foreach PROBE_TARGET,$(TARGET_LIST), 				\
>> +	$(eval -include $(SRC_PATH)/tests/tcg/Makefile.probe) 	\
>> +	$(if $(DOCKER_PREREQ), 					\
>> +		$(eval DOCKER_BUILD_TARGETS+=$(DOCKER_PREREQ))))
>> +
>> +$(info DOCKER_BUILD_TARGETS=$(sort $(DOCKER_BUILD_TARGETS)))
>> +
>> +.PHONY: build-docker-prereqs
>> +build-docker-prereqs: $(sort $(DOCKER_BUILD_TARGETS))
>
> It seems now all docker images are built, even if you want to test a
> single arch... Using ~7GB of storage at once.

I've moved build-docker-prereqs to build-tcg and made check-tcg depend
on that. The single arch test case now relies on the sub-make building
things.

>
>> +
>> +build-tcg-tests-%: build-docker-prereqs
>>  	$(call quiet-command,$(MAKE) $(SUBDIR_MAKEFLAGS) -C $* V="$(V)" TARGET_DIR="$*/" guest-tests,)
>>
>>  run-tcg-tests-%: build-tcg-tests-%
>> diff --git a/tests/tcg/Makefile.probe b/tests/tcg/Makefile.probe
>> new file mode 100644
>> index 0000000000..7529e203ad
>> --- /dev/null
>> +++ b/tests/tcg/Makefile.probe
>> @@ -0,0 +1,31 @@
>> +# -*- Mode: makefile -*-
>> +#
>> +# TCG Compiler Probe
>> +#
>> +# This Makefile fragement is included multiple times in the main make
>> +# script to probe for available compilers. This is used to build up a
>> +# selection of required docker targets before we invoke a sub-make for
>> +# each target.
>> +
>> +# First we need the target makefile which tells us the target architecture
>> +-include $(BUILD_DIR)/$(PROBE_TARGET)/config-target.mak
>> +
>> +# Then we load up the target architecture makefiles which tell us
>> +# about the compilers
>> +undefine CROSS_CC_GUEST
>> +undefine DOCKER_IMAGE
>> +DOCKER_PREREQ:=
>> +
>> +-include $(SRC_PATH)/tests/tcg/$(TARGET_BASE_ARCH)/Makefile.include
>> +-include $(SRC_PATH)/tests/tcg/$(TARGET_NAME)/Makefile.include
>> +
>> +ifndef CROSS_CC_GUEST
>> +ifneq ($(DOCKER_IMAGE),)
>> +DOCKER_PREREQ:=docker-image-$(DOCKER_IMAGE)
>> +endif
>> +endif
>> +
>> +# Clean-up
>> +undefine TARGET_NAME
>> +undefine TARGET_BASE_ARCH
>> +undefine TARGET_ABI_DIR
>>


--
Alex Bennée