Re: [PULL 11/14] tests/tcg: reduce the number of plugin tests combinations

["Alex Bennée" <alex.bennee@linaro.org>]

Pierrick Bouvier <pierrick.bouvier@linaro.org> writes:

> On 7/28/25 12:03 PM, Pierrick Bouvier wrote:
>> Hi Alex,
>> On 7/27/25 1:32 AM, Alex Bennée wrote:
>>> As our set of multiarch tests has grown the practice of running every
>>> plugin with every test is becoming unsustainable. If we switch to
>>> ensuring every test gets run with at least one plugin we can speed
>>> things up.
>>>
>>> Some plugins do need to be run with specific tests (for example the
>>> memory instrumentation test). We can handle this by manually adding
>>> them to EXTRA_RUNS. We also need to wrap rules in a CONFIG_PLUGIN test
>>> so we don't enable the runs when plugins are not enabled.
>>>
>>> Reviewed-by: Manos Pitsidianakis <manos.pitsidianakis@linaro.org>
>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>> Message-ID: <20250725154517.3523095-12-alex.bennee@linaro.org>
>>>
>>> diff --git a/tests/tcg/Makefile.target b/tests/tcg/Makefile.target
>>> index a12b15637ea..18afd5be194 100644
>>> --- a/tests/tcg/Makefile.target
>>> +++ b/tests/tcg/Makefile.target
>>> @@ -173,14 +173,25 @@ PLUGINS=$(filter-out $(DISABLE_PLUGINS), \
>>>    # We need to ensure expand the run-plugin-TEST-with-PLUGIN
>>>    # pre-requistes manually here as we can't use stems to handle it. We
>>>    # only expand MULTIARCH_TESTS which are common on most of our targets
>>> -# to avoid an exponential explosion as new tests are added. We also
>>> -# add some special helpers the run-plugin- rules can use below.
>>> +# and rotate the plugins so we don't grow too out of control as new
>>> +# tests are added. Plugins that need to run with a specific test
>>> +# should ensure they add their combination to EXTRA_RUNS.
>>>       ifneq ($(MULTIARCH_TESTS),)
>>> -$(foreach p,$(PLUGINS), \
>>> -	$(foreach t,$(MULTIARCH_TESTS),\
>>> -		$(eval run-plugin-$(t)-with-$(p): $t $p) \
>>> -		$(eval RUN_TESTS+=run-plugin-$(t)-with-$(p))))
>>> +
>>> +NUM_PLUGINS := $(words $(PLUGINS))
>>> +NUM_TESTS := $(words $(MULTIARCH_TESTS))
>>> +
>>> +define mod_plus_one
>>> +  $(shell $(PYTHON) -c "print( ($(1) % $(2)) + 1 )")
>>> +endef
>>> +
>>> +$(foreach _idx, $(shell seq 1 $(NUM_TESTS)), \
>>> +	$(eval _test := $(word $(_idx), $(MULTIARCH_TESTS))) \
>>> +	$(eval _plugin := $(word $(call mod_plus_one, $(_idx), $(NUM_PLUGINS)), $(PLUGINS))) \
>>> +	$(eval run-plugin-$(_test)-with-$(_plugin): $(_test) $(_plugin)) \
>>> +	$(eval RUN_TESTS+=run-plugin-$(_test)-with-$(_plugin)))
>>> +
>>>    endif # MULTIARCH_TESTS
>>>    endif # CONFIG_PLUGIN
>>>    diff --git a/tests/tcg/multiarch/Makefile.target
>>> b/tests/tcg/multiarch/Makefile.target
>>> index bfdf7197a7b..38345ff8805 100644
>>> --- a/tests/tcg/multiarch/Makefile.target
>>> +++ b/tests/tcg/multiarch/Makefile.target
>>> @@ -189,6 +189,10 @@ run-plugin-semiconsole-with-%:
>>>    TESTS += semihosting semiconsole
>>>    endif
>>>    +test-plugin-mem-access: CFLAGS+=-pthread -O0
>>> +test-plugin-mem-access: LDFLAGS+=-pthread -O0
>>> +
>>> +ifeq ($(CONFIG_PLUGIN),y)
>>>    # Test plugin memory access instrumentation
>>>    run-plugin-test-plugin-mem-access-with-libmem.so: \
>>>    	PLUGIN_ARGS=$(COMMA)print-accesses=true
>>> @@ -197,8 +201,8 @@ run-plugin-test-plugin-mem-access-with-libmem.so: \
>>>    	$(SRC_PATH)/tests/tcg/multiarch/check-plugin-output.sh \
>>>    	$(QEMU) $<
>>>    -test-plugin-mem-access: CFLAGS+=-pthread -O0
>>> -test-plugin-mem-access: LDFLAGS+=-pthread -O0
>>> +EXTRA_RUNS += run-plugin-test-plugin-mem-access-with-libmem.so
>>> +endif
>>>       # Update TESTS
>>>    TESTS += $(MULTIARCH_TESTS)
>>> diff --git a/tests/tcg/multiarch/system/Makefile.softmmu-target b/tests/tcg/multiarch/system/Makefile.softmmu-target
>>> index 5acf2700812..4171b4e6aa0 100644
>>> --- a/tests/tcg/multiarch/system/Makefile.softmmu-target
>>> +++ b/tests/tcg/multiarch/system/Makefile.softmmu-target
>>> @@ -71,8 +71,11 @@ endif
>>>    MULTIARCH_RUNS += run-gdbstub-memory run-gdbstub-interrupt \
>>>    	run-gdbstub-untimely-packet run-gdbstub-registers
>>>    +ifeq ($(CONFIG_PLUGIN),y)
>>>    # Test plugin memory access instrumentation
>>> -run-plugin-memory-with-libmem.so: 		\
>>> -	PLUGIN_ARGS=$(COMMA)region-summary=true
>>> -run-plugin-memory-with-libmem.so: 		\
>>> -	CHECK_PLUGIN_OUTPUT_COMMAND=$(MULTIARCH_SYSTEM_SRC)/validate-memory-counts.py $@.out
>>> +run-plugin-memory-with-libmem.so: memory libmem.so
>>> +run-plugin-memory-with-libmem.so: PLUGIN_ARGS=$(COMMA)region-summary=true
>>> +run-plugin-memory-with-libmem.so: CHECK_PLUGIN_OUTPUT_COMMAND=$(MULTIARCH_SYSTEM_SRC)/validate-memory-counts.py $@.out
>>> +
>>> +EXTRA_RUNS += run-plugin-memory-with-libmem.so
>>> +endif
>> I'm not sure how it's related, but check-tcg on aarch64 host now
>> fails
>> [1] since this series was merged, and I suspect it may be related to
>> this patch. I didn't spend time to reproduce and investigate it.
>>
>
> Reverting this patch solved the problem on my side. As well, I'm not
> sure if it's a real problem to run all plugins for all tests. At the
> moment, it takes 5 min on a slow machine with 4 cpus and a sanitized
> build, something we can probably live with for now.

It definitely is - but it really shows up on linux-user builds because
we have an increasing number of multiarch tests and its not sustainable
to increase by $NARCH * $NPLUGIN everytime a new multiarch test is
added.

>
> Unfortunately Alex is out this week, but it would be nice if another
> maintainer could make a PR reverting this patch.
>
> Regards,
> Pierrick
>
>> [1]
>> https://github.com/pbo-linaro/qemu-ci/actions/runs/16575679153/job/46879690693

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro