[Qemu-devel] [RFC PATCH 0/5] Tweak code coverage reporting

[Qemu-devel] [RFC PATCH 0/5] Tweak code coverage reporting

[Alex Bennée]

Hi,

While trying to make sense of some of the gcov reporting in the Travis
tests I came to the conclusion it wasn't really the most helpful
setting. This series pulls the per-test coverage reset and reporting
and adds a new target that uses the gcovr tool to generate nice pretty
html reports.

The travis build is tweaked to summarise the entire testing run at the
end rather than interleave the coverage reports while the tests are
running. Hopefully this makes filtering the noise out a little easier.

Alex Bennée (5):
  build-system: remove per-test GCOV reporting
  .gitignore: add .gcov files
  .travis.yml: add gcovr summary for GCOV build
  build-system: add clean-coverage target
  build-system: add coverage-report target

 .gitignore             |  1 +
 .travis.yml            |  3 +++
 Makefile               | 24 ++++++++++++++++++++++++
 docs/devel/testing.rst | 21 +++++++++++++++------
 tests/Makefile.include | 10 ----------
 5 files changed, 43 insertions(+), 16 deletions(-)

-- 
2.17.1

[Qemu-devel] [RFC PATCH 1/5] build-system: remove per-test GCOV reporting

[Alex Bennée]

I'm not entirely sure who's using this information and certainly in a
CI environment it just washes over as additional noise. Later patches
will provide new reporting options so a user who wants to analyse
individual tests will be able to use that to get the information.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 docs/devel/testing.rst | 11 +++++------
 tests/Makefile.include | 10 ----------
 2 files changed, 5 insertions(+), 16 deletions(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index f33e5a8423..66ef219f69 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -158,12 +158,11 @@ rarely used. See "QEMU iotests" section below for more information.
 GCC gcov support
 ----------------
 
-``gcov`` is a GCC tool to analyze the testing coverage by instrumenting the
-tested code. To use it, configure QEMU with ``--enable-gcov`` option and build.
-Then run ``make check`` as usual. There will be additional ``gcov`` output as
-the testing goes on, showing the test coverage percentage numbers per analyzed
-source file. More detailed reports can be obtained by running ``gcov`` command
-on the output files under ``$build_dir/tests/``, please read the ``gcov``
+``gcov`` is a GCC tool to analyze the testing coverage by
+instrumenting the tested code. To use it, configure QEMU with
+``--enable-gcov`` option and build. Then run ``make check`` as usual.
+Reports can be obtained by running ``gcov`` command on the output
+files under ``$build_dir/tests/``, please read the ``gcov``
 documentation for more information.
 
 QEMU iotests
diff --git a/tests/Makefile.include b/tests/Makefile.include
index ca91da26cb..55d54bd180 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -891,26 +891,16 @@ GCOV_OPTIONS = -n $(if $(V),-f,)
 
 .PHONY: $(patsubst %, check-qtest-%, $(QTEST_TARGETS))
 $(patsubst %, check-qtest-%, $(QTEST_TARGETS)): check-qtest-%: subdir-%-softmmu $(check-qtest-y)
-	$(if $(CONFIG_GCOV),@rm -f *.gcda */*.gcda */*/*.gcda */*/*/*.gcda,)
 	$(call quiet-command,QTEST_QEMU_BINARY=$*-softmmu/qemu-system-$* \
 		QTEST_QEMU_IMG=qemu-img$(EXESUF) \
 		MALLOC_PERTURB_=$${MALLOC_PERTURB_:-$$(( $${RANDOM:-0} % 255 + 1))} \
 		gtester $(GTESTER_OPTIONS) -m=$(SPEED) $(check-qtest-$*-y) $(check-qtest-generic-y),"GTESTER","$@")
-	$(if $(CONFIG_GCOV),@for f in $(gcov-files-$*-y) $(gcov-files-generic-y); do \
-	  echo Gcov report for $$f:;\
-	  $(GCOV) $(GCOV_OPTIONS) $$f -o `dirname $$f`; \
-	done,)
 
 .PHONY: $(patsubst %, check-%, $(check-unit-y) $(check-speed-y))
 $(patsubst %, check-%, $(check-unit-y) $(check-speed-y)): check-%: %
-	$(if $(CONFIG_GCOV),@rm -f *.gcda */*.gcda */*/*.gcda */*/*/*.gcda,)
 	$(call quiet-command, \
 		MALLOC_PERTURB_=$${MALLOC_PERTURB_:-$$(( $${RANDOM:-0} % 255 + 1))} \
 		gtester $(GTESTER_OPTIONS) -m=$(SPEED) $*,"GTESTER","$*")
-	$(if $(CONFIG_GCOV),@for f in $(gcov-files-$(subst tests/,,$*)-y) $(gcov-files-generic-y); do \
-	  echo Gcov report for $$f:;\
-	  $(GCOV) $(GCOV_OPTIONS) $$f -o `dirname $$f`; \
-	done,)
 
 # gtester tests with XML output
 
-- 
2.17.1

[Qemu-devel] [RFC PATCH 2/5] .gitignore: add .gcov files

[Alex Bennée]

These are temporary files generated on gcov runs and shouldn't be
included in the source tree.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 .gitignore | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitignore b/.gitignore
index 9da3b3e626..5668d02782 100644
--- a/.gitignore
+++ b/.gitignore
@@ -155,6 +155,7 @@
 .sdk
 *.gcda
 *.gcno
+*.gcov
 /pc-bios/bios-pq/status
 /pc-bios/vgabios-pq/status
 /pc-bios/optionrom/linuxboot.asm
-- 
2.17.1

[Qemu-devel] [RFC PATCH 3/5] .travis.yml: add gcovr summary for GCOV build

[Alex Bennée]

This gives a more useful summary, sorted by descending % coverage,
after the tests have run. The final numbers will give an idea if our
coverage is getting better or worse.

As quite a lot of lines don't get covered at all we filter out all the
0% lines. If the file doesn't appear it is not being exercised.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 .travis.yml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index fabfe9ec34..83e0577464 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -38,6 +38,7 @@ addons:
       - libvte-2.90-dev
       - sparse
       - uuid-dev
+      - gcovr
 
 # The channel name "irc.oftc.net#qemu" is encrypted against qemu/qemu
 # to prevent IRC notifications from forks. This was created using:
@@ -81,6 +82,8 @@ matrix:
       compiler: clang
     # gprof/gcov are GCC features
     - env: CONFIG="--enable-gprof --enable-gcov --disable-pie --target-list=aarch64-softmmu,arm-softmmu,i386-softmmu,mips-softmmu,mips64-softmmu,ppc64-softmmu,riscv64-softmmu,s390x-softmmu,x86_64-softmmu"
+      after_success:
+        - gcovr -p  | grep -v "0%" | sed s/[0-9]\*[,-]//g
       compiler: gcc
     # We manually include builds which we disable "make check" for
     - env: CONFIG="--enable-debug --enable-tcg-interpreter"
-- 
2.17.1

[Qemu-devel] [RFC PATCH 4/5] build-system: add clean-coverage target

[Alex Bennée]

This can be used to remove any stale coverage data before any
particular test run. This is useful for analysing individual tests.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 Makefile               | 11 +++++++++++
 docs/devel/testing.rst | 11 ++++++++---
 2 files changed, 19 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index e46f2b625a..cb4af8bf80 100644
--- a/Makefile
+++ b/Makefile
@@ -725,6 +725,14 @@ module_block.h: $(SRC_PATH)/scripts/modules/module_block.py config-host.mak
 	$(addprefix $(SRC_PATH)/,$(patsubst %.mo,%.c,$(block-obj-m))), \
 	"GEN","$@")
 
+ifdef CONFIG_GCOV
+.PHONY: clean-coverage
+clean-coverage:
+	$(call quiet-command, \
+		find . \( -name '*.gcda' -o -name '*.gcov' \) -type f -exec rm {} +, \
+		"CLEAN", "coverage files")
+endif
+
 clean:
 # avoid old build problems by removing potentially incorrect old files
 	rm -f config.mak op-i386.h opc-i386.h gen-op-i386.h op-arm.h opc-arm.h gen-op-arm.h
@@ -1075,6 +1083,9 @@ endif
 		echo '')
 	@echo  'Cleaning targets:'
 	@echo  '  clean           - Remove most generated files but keep the config'
+ifdef CONFIG_GCOV
+	@echo  '  clean-coverage  - Remove coverage files'
+endif
 	@echo  '  distclean       - Remove all generated files'
 	@echo  '  dist            - Build a distributable tarball'
 	@echo  ''
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 66ef219f69..a3652aea14 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -161,9 +161,14 @@ GCC gcov support
 ``gcov`` is a GCC tool to analyze the testing coverage by
 instrumenting the tested code. To use it, configure QEMU with
 ``--enable-gcov`` option and build. Then run ``make check`` as usual.
-Reports can be obtained by running ``gcov`` command on the output
-files under ``$build_dir/tests/``, please read the ``gcov``
-documentation for more information.
+
+If you want to gather coverage information on a single test the ``make
+clean-coverage`` target can be used to any existing coverage
+information before running a single test.
+
+Reports can be obtained by running ``gcov`` command
+on the output files under ``$build_dir/tests/``, please read the
+``gcov`` documentation for more information.
 
 QEMU iotests
 ============
-- 
2.17.1

[Qemu-devel] [RFC PATCH 5/5] build-system: add coverage-report target

[Alex Bennée]

This will build a coverage report under the current directory in
reports/coverage. At the users option a report can be generated by
directly invoking something like:

  make foo/bar/coverage-report.html

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 Makefile               | 13 +++++++++++++
 docs/devel/testing.rst | 11 ++++++++---
 2 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index cb4af8bf80..7450e0b7b5 100644
--- a/Makefile
+++ b/Makefile
@@ -988,6 +988,16 @@ docs/interop/qemu-qmp-ref.dvi docs/interop/qemu-qmp-ref.html \
     docs/interop/qemu-qmp-ref.txt docs/interop/qemu-qmp-ref.7: \
 	docs/interop/qemu-qmp-ref.texi docs/interop/qemu-qmp-qapi.texi
 
+# Reports/Analysis
+
+%/coverage-report.html:
+	@mkdir -p $*
+	$(call quiet-command,\
+		gcovr -p --html --html-details -o $@, \
+		"GEN", "coverage-report.html")
+
+.PHONY: coverage-report
+coverage-report: $(CURDIR)/reports/coverage/coverage-report.html
 
 ifdef CONFIG_WIN32
 
@@ -1097,6 +1107,9 @@ endif
 	@echo  'Documentation targets:'
 	@echo  '  html info pdf txt'
 	@echo  '                  - Build documentation in specified format'
+ifdef CONFIG_GCOV
+	@echo  '  coverage-report - Create code coverage report'
+endif
 	@echo  ''
 ifdef CONFIG_WIN32
 	@echo  'Windows targets:'
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index a3652aea14..9dcdd19260 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -166,9 +166,14 @@ If you want to gather coverage information on a single test the ``make
 clean-coverage`` target can be used to any existing coverage
 information before running a single test.
 
-Reports can be obtained by running ``gcov`` command
-on the output files under ``$build_dir/tests/``, please read the
-``gcov`` documentation for more information.
+You can generate a HTML coverage report by executing ``make
+coverage-report`` which will generate into
+./reports/coverage/coverage-report.html. If you want to generate it
+elsewhere simply execute ``make /foo/bar/baz/coverage-report.html``.
+
+Further analysis can be conducted by running the ``gcov`` command
+directly on the various .gcda output files. Please read the ``gcov``
+documentation for more information.
 
 QEMU iotests
 ============
-- 
2.17.1

Re: [Qemu-devel] [RFC PATCH 1/5] build-system: remove per-test GCOV reporting

[Philippe Mathieu-Daudé]

On 06/20/2018 10:20 AM, Alex Bennée wrote:
> I'm not entirely sure who's using this information and certainly in a
> CI environment it just washes over as additional noise. Later patches
> will provide new reporting options so a user who wants to analyse
> individual tests will be able to use that to get the information.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> ---
>  docs/devel/testing.rst | 11 +++++------
>  tests/Makefile.include | 10 ----------
>  2 files changed, 5 insertions(+), 16 deletions(-)
> 
> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> index f33e5a8423..66ef219f69 100644
> --- a/docs/devel/testing.rst
> +++ b/docs/devel/testing.rst
> @@ -158,12 +158,11 @@ rarely used. See "QEMU iotests" section below for more information.
>  GCC gcov support
>  ----------------
>  
> -``gcov`` is a GCC tool to analyze the testing coverage by instrumenting the
> -tested code. To use it, configure QEMU with ``--enable-gcov`` option and build.
> -Then run ``make check`` as usual. There will be additional ``gcov`` output as
> -the testing goes on, showing the test coverage percentage numbers per analyzed
> -source file. More detailed reports can be obtained by running ``gcov`` command
> -on the output files under ``$build_dir/tests/``, please read the ``gcov``
> +``gcov`` is a GCC tool to analyze the testing coverage by
> +instrumenting the tested code. To use it, configure QEMU with
> +``--enable-gcov`` option and build. Then run ``make check`` as usual.
> +Reports can be obtained by running ``gcov`` command on the output
> +files under ``$build_dir/tests/``, please read the ``gcov``
>  documentation for more information.
>  
>  QEMU iotests
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index ca91da26cb..55d54bd180 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -891,26 +891,16 @@ GCOV_OPTIONS = -n $(if $(V),-f,)
>  
>  .PHONY: $(patsubst %, check-qtest-%, $(QTEST_TARGETS))
>  $(patsubst %, check-qtest-%, $(QTEST_TARGETS)): check-qtest-%: subdir-%-softmmu $(check-qtest-y)
> -	$(if $(CONFIG_GCOV),@rm -f *.gcda */*.gcda */*/*.gcda */*/*/*.gcda,)
>  	$(call quiet-command,QTEST_QEMU_BINARY=$*-softmmu/qemu-system-$* \
>  		QTEST_QEMU_IMG=qemu-img$(EXESUF) \
>  		MALLOC_PERTURB_=$${MALLOC_PERTURB_:-$$(( $${RANDOM:-0} % 255 + 1))} \
>  		gtester $(GTESTER_OPTIONS) -m=$(SPEED) $(check-qtest-$*-y) $(check-qtest-generic-y),"GTESTER","$@")
> -	$(if $(CONFIG_GCOV),@for f in $(gcov-files-$*-y) $(gcov-files-generic-y); do \
> -	  echo Gcov report for $$f:;\
> -	  $(GCOV) $(GCOV_OPTIONS) $$f -o `dirname $$f`; \
> -	done,)
>  
>  .PHONY: $(patsubst %, check-%, $(check-unit-y) $(check-speed-y))
>  $(patsubst %, check-%, $(check-unit-y) $(check-speed-y)): check-%: %
> -	$(if $(CONFIG_GCOV),@rm -f *.gcda */*.gcda */*/*.gcda */*/*/*.gcda,)
>  	$(call quiet-command, \
>  		MALLOC_PERTURB_=$${MALLOC_PERTURB_:-$$(( $${RANDOM:-0} % 255 + 1))} \
>  		gtester $(GTESTER_OPTIONS) -m=$(SPEED) $*,"GTESTER","$*")
> -	$(if $(CONFIG_GCOV),@for f in $(gcov-files-$(subst tests/,,$*)-y) $(gcov-files-generic-y); do \
> -	  echo Gcov report for $$f:;\
> -	  $(GCOV) $(GCOV_OPTIONS) $$f -o `dirname $$f`; \
> -	done,)
>  
>  # gtester tests with XML output
>  
> 

Re: [Qemu-devel] [RFC PATCH 2/5] .gitignore: add .gcov files

[Philippe Mathieu-Daudé]

On 06/20/2018 10:20 AM, Alex Bennée wrote:
> These are temporary files generated on gcov runs and shouldn't be
> included in the source tree.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> ---
>  .gitignore | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/.gitignore b/.gitignore
> index 9da3b3e626..5668d02782 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -155,6 +155,7 @@
>  .sdk
>  *.gcda
>  *.gcno
> +*.gcov
>  /pc-bios/bios-pq/status
>  /pc-bios/vgabios-pq/status
>  /pc-bios/optionrom/linuxboot.asm
> 

Re: [Qemu-devel] [RFC PATCH 3/5] .travis.yml: add gcovr summary for GCOV build

[Philippe Mathieu-Daudé]

On 06/20/2018 10:20 AM, Alex Bennée wrote:
> This gives a more useful summary, sorted by descending % coverage,
> after the tests have run. The final numbers will give an idea if our
> coverage is getting better or worse.
> 
> As quite a lot of lines don't get covered at all we filter out all the
> 0% lines. If the file doesn't appear it is not being exercised.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  .travis.yml | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/.travis.yml b/.travis.yml
> index fabfe9ec34..83e0577464 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -38,6 +38,7 @@ addons:
>        - libvte-2.90-dev
>        - sparse
>        - uuid-dev
> +      - gcovr
>  
>  # The channel name "irc.oftc.net#qemu" is encrypted against qemu/qemu
>  # to prevent IRC notifications from forks. This was created using:
> @@ -81,6 +82,8 @@ matrix:
>        compiler: clang
>      # gprof/gcov are GCC features
>      - env: CONFIG="--enable-gprof --enable-gcov --disable-pie --target-list=aarch64-softmmu,arm-softmmu,i386-softmmu,mips-softmmu,mips64-softmmu,ppc64-softmmu,riscv64-softmmu,s390x-softmmu,x86_64-softmmu"
> +      after_success:
> +        - gcovr -p  | grep -v "0%" | sed s/[0-9]\*[,-]//g

Can we use 'fgrep -B 1 -v "0%"' (--before-context=1) to also remove the
filepath line?

Any idea why I get this? "(WARNING) Unrecognized GCOV output: '====='"

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

>        compiler: gcc
>      # We manually include builds which we disable "make check" for
>      - env: CONFIG="--enable-debug --enable-tcg-interpreter"
> 

Re: [Qemu-devel] [RFC PATCH 4/5] build-system: add clean-coverage target

[Philippe Mathieu-Daudé]

Hi Alex,

On 06/20/2018 10:20 AM, Alex Bennée wrote:
> This can be used to remove any stale coverage data before any
> particular test run. This is useful for analysing individual tests.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  Makefile               | 11 +++++++++++
>  docs/devel/testing.rst | 11 ++++++++---
>  2 files changed, 19 insertions(+), 3 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index e46f2b625a..cb4af8bf80 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -725,6 +725,14 @@ module_block.h: $(SRC_PATH)/scripts/modules/module_block.py config-host.mak
>  	$(addprefix $(SRC_PATH)/,$(patsubst %.mo,%.c,$(block-obj-m))), \
>  	"GEN","$@")
>  
> +ifdef CONFIG_GCOV
> +.PHONY: clean-coverage
> +clean-coverage:
> +	$(call quiet-command, \
> +		find . \( -name '*.gcda' -o -name '*.gcov' \) -type f -exec rm {} +, \
> +		"CLEAN", "coverage files")

I also see ".gcno" files.
>From GCC man page:

      -ftest-coverage
           Produce a notes file that the gcov code-coverage
           utility can use to show program coverage.  Each
           source file's note file is called auxname.gcno.

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Adding gcno:
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> +endif
> +
>  clean:
>  # avoid old build problems by removing potentially incorrect old files
>  	rm -f config.mak op-i386.h opc-i386.h gen-op-i386.h op-arm.h opc-arm.h gen-op-arm.h
> @@ -1075,6 +1083,9 @@ endif
>  		echo '')
>  	@echo  'Cleaning targets:'
>  	@echo  '  clean           - Remove most generated files but keep the config'
> +ifdef CONFIG_GCOV
> +	@echo  '  clean-coverage  - Remove coverage files'
> +endif
>  	@echo  '  distclean       - Remove all generated files'
>  	@echo  '  dist            - Build a distributable tarball'
>  	@echo  ''
> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> index 66ef219f69..a3652aea14 100644
> --- a/docs/devel/testing.rst
> +++ b/docs/devel/testing.rst
> @@ -161,9 +161,14 @@ GCC gcov support
>  ``gcov`` is a GCC tool to analyze the testing coverage by
>  instrumenting the tested code. To use it, configure QEMU with
>  ``--enable-gcov`` option and build. Then run ``make check`` as usual.
> -Reports can be obtained by running ``gcov`` command on the output
> -files under ``$build_dir/tests/``, please read the ``gcov``
> -documentation for more information.
> +
> +If you want to gather coverage information on a single test the ``make
> +clean-coverage`` target can be used to any existing coverage
> +information before running a single test.
> +
> +Reports can be obtained by running ``gcov`` command
> +on the output files under ``$build_dir/tests/``, please read the
> +``gcov`` documentation for more information.
>  
>  QEMU iotests
>  ============
> 

Re: [Qemu-devel] [RFC PATCH 3/5] .travis.yml: add gcovr summary for GCOV build

[Philippe Mathieu-Daudé]

On 06/20/2018 10:20 AM, Alex Bennée wrote:
> This gives a more useful summary, sorted by descending % coverage,
> after the tests have run. The final numbers will give an idea if our
> coverage is getting better or worse.
> 
> As quite a lot of lines don't get covered at all we filter out all the
> 0% lines. If the file doesn't appear it is not being exercised.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  .travis.yml | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/.travis.yml b/.travis.yml
> index fabfe9ec34..83e0577464 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -38,6 +38,7 @@ addons:
>        - libvte-2.90-dev
>        - sparse
>        - uuid-dev
> +      - gcovr
>  
>  # The channel name "irc.oftc.net#qemu" is encrypted against qemu/qemu
>  # to prevent IRC notifications from forks. This was created using:
> @@ -81,6 +82,8 @@ matrix:
>        compiler: clang
>      # gprof/gcov are GCC features
>      - env: CONFIG="--enable-gprof --enable-gcov --disable-pie --target-list=aarch64-softmmu,arm-softmmu,i386-softmmu,mips-softmmu,mips64-softmmu,ppc64-softmmu,riscv64-softmmu,s390x-softmmu,x86_64-softmmu"

I just noticed the linux-user tests are not covered.

I'd duplicate this entry and use --disable-system --disable-bsd-user.

> +      after_success:
> +        - gcovr -p  | grep -v "0%" | sed s/[0-9]\*[,-]//g
>        compiler: gcc
>      # We manually include builds which we disable "make check" for
>      - env: CONFIG="--enable-debug --enable-tcg-interpreter"
> 

Re: [Qemu-devel] [RFC PATCH 5/5] build-system: add coverage-report target

[Philippe Mathieu-Daudé]

Hi Alex,

On 06/20/2018 10:20 AM, Alex Bennée wrote:
> This will build a coverage report under the current directory in
> reports/coverage. At the users option a report can be generated by
> directly invoking something like:
> 
>   make foo/bar/coverage-report.html
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  Makefile               | 13 +++++++++++++
>  docs/devel/testing.rst | 11 ++++++++---
>  2 files changed, 21 insertions(+), 3 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index cb4af8bf80..7450e0b7b5 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -988,6 +988,16 @@ docs/interop/qemu-qmp-ref.dvi docs/interop/qemu-qmp-ref.html \
>      docs/interop/qemu-qmp-ref.txt docs/interop/qemu-qmp-ref.7: \
>  	docs/interop/qemu-qmp-ref.texi docs/interop/qemu-qmp-qapi.texi
>  
> +# Reports/Analysis
> +
> +%/coverage-report.html:

What about the files in the root directory?

> +	@mkdir -p $*
> +	$(call quiet-command,\
> +		gcovr -p --html --html-details -o $@, \
> +		"GEN", "coverage-report.html")

I think this also needs "-r $(SRC_PATH)" for out-of-tree builds.

>From GCOVR(1):

       -r ROOT, --root=ROOT
              Defines the root directory for source files.

> +
> +.PHONY: coverage-report
> +coverage-report: $(CURDIR)/reports/coverage/coverage-report.html
>  
>  ifdef CONFIG_WIN32
>  
> @@ -1097,6 +1107,9 @@ endif
>  	@echo  'Documentation targets:'
>  	@echo  '  html info pdf txt'
>  	@echo  '                  - Build documentation in specified format'
> +ifdef CONFIG_GCOV
> +	@echo  '  coverage-report - Create code coverage report'
> +endif
>  	@echo  ''
>  ifdef CONFIG_WIN32
>  	@echo  'Windows targets:'
> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> index a3652aea14..9dcdd19260 100644
> --- a/docs/devel/testing.rst
> +++ b/docs/devel/testing.rst
> @@ -166,9 +166,14 @@ If you want to gather coverage information on a single test the ``make
>  clean-coverage`` target can be used to any existing coverage
>  information before running a single test.
>  
> -Reports can be obtained by running ``gcov`` command
> -on the output files under ``$build_dir/tests/``, please read the
> -``gcov`` documentation for more information.
> +You can generate a HTML coverage report by executing ``make
> +coverage-report`` which will generate into
> +./reports/coverage/coverage-report.html. If you want to generate it
> +elsewhere simply execute ``make /foo/bar/baz/coverage-report.html``.

$ make coverage-report
  GEN     coverage-report.html
/bin/sh: 1: gcovr: not found
make: *** [Makefile:995: reports/coverage/coverage-report.html] Error 127

Can you add a line about this prerequisite? (I don't think it's worth a
check in ./configure).

> +
> +Further analysis can be conducted by running the ``gcov`` command
> +directly on the various .gcda output files. Please read the ``gcov``
> +documentation for more information.
>  
>  QEMU iotests
>  ============
> 

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Re: [Qemu-devel] [RFC PATCH 3/5] .travis.yml: add gcovr summary for GCOV build

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> On 06/20/2018 10:20 AM, Alex Bennée wrote:
>> This gives a more useful summary, sorted by descending % coverage,
>> after the tests have run. The final numbers will give an idea if our
>> coverage is getting better or worse.
>>
>> As quite a lot of lines don't get covered at all we filter out all the
>> 0% lines. If the file doesn't appear it is not being exercised.
>>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> ---
>>  .travis.yml | 3 +++
>>  1 file changed, 3 insertions(+)
>>
>> diff --git a/.travis.yml b/.travis.yml
>> index fabfe9ec34..83e0577464 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -38,6 +38,7 @@ addons:
>>        - libvte-2.90-dev
>>        - sparse
>>        - uuid-dev
>> +      - gcovr
>>
>>  # The channel name "irc.oftc.net#qemu" is encrypted against qemu/qemu
>>  # to prevent IRC notifications from forks. This was created using:
>> @@ -81,6 +82,8 @@ matrix:
>>        compiler: clang
>>      # gprof/gcov are GCC features
>>      - env: CONFIG="--enable-gprof --enable-gcov --disable-pie --target-list=aarch64-softmmu,arm-softmmu,i386-softmmu,mips-softmmu,mips64-softmmu,ppc64-softmmu,riscv64-softmmu,s390x-softmmu,x86_64-softmmu"
>
> I just noticed the linux-user tests are not covered.

I did try and calculate coverage of a risu run through SVE and didn't
get any gcda files so I think there is something else that needs adding
first.

>
> I'd duplicate this entry and use --disable-system --disable-bsd-user.
>
>> +      after_success:
>> +        - gcovr -p  | grep -v "0%" | sed s/[0-9]\*[,-]//g
>>        compiler: gcc
>>      # We manually include builds which we disable "make check" for
>>      - env: CONFIG="--enable-debug --enable-tcg-interpreter"
>>


--
Alex Bennée

Re: [Qemu-devel] [RFC PATCH 4/5] build-system: add clean-coverage target

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Hi Alex,
>
> On 06/20/2018 10:20 AM, Alex Bennée wrote:
>> This can be used to remove any stale coverage data before any
>> particular test run. This is useful for analysing individual tests.
>>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> ---
>>  Makefile               | 11 +++++++++++
>>  docs/devel/testing.rst | 11 ++++++++---
>>  2 files changed, 19 insertions(+), 3 deletions(-)
>>
>> diff --git a/Makefile b/Makefile
>> index e46f2b625a..cb4af8bf80 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -725,6 +725,14 @@ module_block.h: $(SRC_PATH)/scripts/modules/module_block.py config-host.mak
>>  	$(addprefix $(SRC_PATH)/,$(patsubst %.mo,%.c,$(block-obj-m))), \
>>  	"GEN","$@")
>>
>> +ifdef CONFIG_GCOV
>> +.PHONY: clean-coverage
>> +clean-coverage:
>> +	$(call quiet-command, \
>> +		find . \( -name '*.gcda' -o -name '*.gcov' \) -type f -exec rm {} +, \
>> +		"CLEAN", "coverage files")
>
> I also see ".gcno" files.
> From GCC man page:
>
>       -ftest-coverage
>            Produce a notes file that the gcov code-coverage
>            utility can use to show program coverage.  Each
>            source file's note file is called auxname.gcno.

I explicitly left that out - the gcno file is regenerated by the build.
There is no reason to wipe it between coverage runs. A full clean should
remove them however.

>
> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> Adding gcno:
> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>
>> +endif
>> +
>>  clean:
>>  # avoid old build problems by removing potentially incorrect old files
>>  	rm -f config.mak op-i386.h opc-i386.h gen-op-i386.h op-arm.h opc-arm.h gen-op-arm.h
>> @@ -1075,6 +1083,9 @@ endif
>>  		echo '')
>>  	@echo  'Cleaning targets:'
>>  	@echo  '  clean           - Remove most generated files but keep the config'
>> +ifdef CONFIG_GCOV
>> +	@echo  '  clean-coverage  - Remove coverage files'
>> +endif
>>  	@echo  '  distclean       - Remove all generated files'
>>  	@echo  '  dist            - Build a distributable tarball'
>>  	@echo  ''
>> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
>> index 66ef219f69..a3652aea14 100644
>> --- a/docs/devel/testing.rst
>> +++ b/docs/devel/testing.rst
>> @@ -161,9 +161,14 @@ GCC gcov support
>>  ``gcov`` is a GCC tool to analyze the testing coverage by
>>  instrumenting the tested code. To use it, configure QEMU with
>>  ``--enable-gcov`` option and build. Then run ``make check`` as usual.
>> -Reports can be obtained by running ``gcov`` command on the output
>> -files under ``$build_dir/tests/``, please read the ``gcov``
>> -documentation for more information.
>> +
>> +If you want to gather coverage information on a single test the ``make
>> +clean-coverage`` target can be used to any existing coverage
>> +information before running a single test.
>> +
>> +Reports can be obtained by running ``gcov`` command
>> +on the output files under ``$build_dir/tests/``, please read the
>> +``gcov`` documentation for more information.
>>
>>  QEMU iotests
>>  ============
>>


--
Alex Bennée

Re: [Qemu-devel] [RFC PATCH 4/5] build-system: add clean-coverage target

[Philippe Mathieu-Daudé]

On 06/20/2018 06:06 PM, Alex Bennée wrote:
> 
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> 
>> Hi Alex,
>>
>> On 06/20/2018 10:20 AM, Alex Bennée wrote:
>>> This can be used to remove any stale coverage data before any
>>> particular test run. This is useful for analysing individual tests.
>>>
>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>> ---
>>>  Makefile               | 11 +++++++++++
>>>  docs/devel/testing.rst | 11 ++++++++---
>>>  2 files changed, 19 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/Makefile b/Makefile
>>> index e46f2b625a..cb4af8bf80 100644
>>> --- a/Makefile
>>> +++ b/Makefile
>>> @@ -725,6 +725,14 @@ module_block.h: $(SRC_PATH)/scripts/modules/module_block.py config-host.mak
>>>  	$(addprefix $(SRC_PATH)/,$(patsubst %.mo,%.c,$(block-obj-m))), \
>>>  	"GEN","$@")
>>>
>>> +ifdef CONFIG_GCOV
>>> +.PHONY: clean-coverage
>>> +clean-coverage:
>>> +	$(call quiet-command, \
>>> +		find . \( -name '*.gcda' -o -name '*.gcov' \) -type f -exec rm {} +, \
>>> +		"CLEAN", "coverage files")
>>
>> I also see ".gcno" files.
>> From GCC man page:
>>
>>       -ftest-coverage
>>            Produce a notes file that the gcov code-coverage
>>            utility can use to show program coverage.  Each
>>            source file's note file is called auxname.gcno.
> 
> I explicitly left that out - the gcno file is regenerated by the build.
> There is no reason to wipe it between coverage runs. A full clean should
> remove them however.

Oh OK, fine then.

> 
>>
>> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> Adding gcno:
>> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>
>>> +endif
>>> +
>>>  clean:
>>>  # avoid old build problems by removing potentially incorrect old files
>>>  	rm -f config.mak op-i386.h opc-i386.h gen-op-i386.h op-arm.h opc-arm.h gen-op-arm.h
>>> @@ -1075,6 +1083,9 @@ endif
>>>  		echo '')
>>>  	@echo  'Cleaning targets:'
>>>  	@echo  '  clean           - Remove most generated files but keep the config'
>>> +ifdef CONFIG_GCOV
>>> +	@echo  '  clean-coverage  - Remove coverage files'
>>> +endif
>>>  	@echo  '  distclean       - Remove all generated files'
>>>  	@echo  '  dist            - Build a distributable tarball'
>>>  	@echo  ''
>>> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
>>> index 66ef219f69..a3652aea14 100644
>>> --- a/docs/devel/testing.rst
>>> +++ b/docs/devel/testing.rst
>>> @@ -161,9 +161,14 @@ GCC gcov support
>>>  ``gcov`` is a GCC tool to analyze the testing coverage by
>>>  instrumenting the tested code. To use it, configure QEMU with
>>>  ``--enable-gcov`` option and build. Then run ``make check`` as usual.
>>> -Reports can be obtained by running ``gcov`` command on the output
>>> -files under ``$build_dir/tests/``, please read the ``gcov``
>>> -documentation for more information.
>>> +
>>> +If you want to gather coverage information on a single test the ``make
>>> +clean-coverage`` target can be used to any existing coverage
>>> +information before running a single test.
>>> +
>>> +Reports can be obtained by running ``gcov`` command
>>> +on the output files under ``$build_dir/tests/``, please read the
>>> +``gcov`` documentation for more information.
>>>
>>>  QEMU iotests
>>>  ============
>>>
> 
> 
> --
> Alex Bennée
>