[RFC PATCH 0/1] perf test cs-etm: Add end-to-end tests for CoreSight decoding

[RFC PATCH 0/1] perf test cs-etm: Add end-to-end tests for CoreSight decoding

[German Gomez]

Hi

While discussing running more tests for CoreSight, we thought it might
be a good idea to upstream some EndToEnd tests for CoreSight decoding in
order to lock down the behaviour. I am sending this as RFC to get some
feedback from the community first.

The test relies on pre-geneated perf.data files that are downloaded
during the test. I'm not sure it's a good idea to commit those files to
the Linux repository, so they would have to live in an external source
and be downloaded during the test.

For this RFC, the files are stored in a Github repository [1]. As an
idea, I think we could store them in a new repo in the ARM-software
namespace. Any hosting suggestions are very welcome.

Thanks,
German

[1] https://github.com/ARM-software/data/tree/984cde8fb0bb22591e284826a80b338bb79c3655/perf/coresight

German Gomez (1):
  perf test cs-etm: Add end-to-end tests for CoreSight decoding

 tools/perf/tests/shell/lib/arm_auxtrace.sh    | 21 +++++++
 .../tests/shell/test_arm_coresight_decoder.sh | 57 +++++++++++++++++++
 2 files changed, 78 insertions(+)
 create mode 100644 tools/perf/tests/shell/lib/arm_auxtrace.sh
 create mode 100755 tools/perf/tests/shell/test_arm_coresight_decoder.sh

-- 
2.25.1

[RFC PATCH 1/1] perf test cs-etm: Add end-to-end tests for CoreSight decoding

[German Gomez]

Add a shell script to test for regressions in the decoding of CoreSight
samples. The test uses supporting files which are not committed to the
linux repository. Instead they are downloaded from an external source:

[1] perf.data
[2] perf.data.inject
[3] perf.data.tar.bz2

File [1] is an example recording of a CoreSight trace. File [2] is the
output of running perf-inject on file [1] (i.e. the expected samples).
Lastly file [3] are the contents of the "~/.debug" directory.

Signed-off-by: German Gomez <german.gomez@arm.com>
---
 tools/perf/tests/shell/lib/arm_auxtrace.sh    | 21 +++++++
 .../tests/shell/test_arm_coresight_decoder.sh | 57 +++++++++++++++++++
 2 files changed, 78 insertions(+)
 create mode 100644 tools/perf/tests/shell/lib/arm_auxtrace.sh
 create mode 100755 tools/perf/tests/shell/test_arm_coresight_decoder.sh

diff --git a/tools/perf/tests/shell/lib/arm_auxtrace.sh b/tools/perf/tests/shell/lib/arm_auxtrace.sh
new file mode 100644
index 0000000000000..5e117c33a4c98
--- /dev/null
+++ b/tools/perf/tests/shell/lib/arm_auxtrace.sh
@@ -0,0 +1,21 @@
+# TODO(german):
+#	This is a palceholder location, where the test data will/would reside.
+#	We need to find a suitable location to host the perf.data files used for testing
+GITHUB_REPO=https://github.com/ARM-software/data/raw/984cde8fb0bb22591e284826a80b338bb79c3655/perf/
+
+# download test files to the current working directory:
+# 	perf.data         (perf.data that contains auxtrace test data)
+# 	perf.data.inject  (output of perf inject -i perf.data)
+# 	perf.data.tar.bz2 (output of perf archive)
+arm_download_auxtrace_test_files() {
+	local data="$GITHUB_REPO/$1/perf.data"
+	local data_inject="$GITHUB_REPO/$1/perf.data.inject"
+	local data_debug="$GITHUB_REPO/$1/perf.data.tar.bz2"
+
+	# skip the test if the download fails for whetever reason
+	timeout 30 curl -LSs --fail "$data"		> perf.data		|| exit 2
+	timeout 30 curl -LSs --fail "$data_inject"	> perf.data.inject	|| exit 2
+	timeout 30 curl -LSs --fail "$data_debug"	> perf.data.tar.bz2	|| exit 2
+}
+
+export arm_download_auxtrace_test_files
diff --git a/tools/perf/tests/shell/test_arm_coresight_decoder.sh b/tools/perf/tests/shell/test_arm_coresight_decoder.sh
new file mode 100755
index 0000000000000..342a6bc70428f
--- /dev/null
+++ b/tools/perf/tests/shell/test_arm_coresight_decoder.sh
@@ -0,0 +1,57 @@
+#!/bin/bash
+# e2e test Arm CoreSight decoding
+
+# SPDX-License-Identifier: GPL-2.0
+# German Gomez <german.gomez@arm.com>, 2022
+
+set -e
+source $(dirname $0)/lib/arm_auxtrace.sh
+TEMP_DIR=$(mktemp -d)
+
+cleanup_files() {
+	rm -rf ${TEMP_DIR}
+}
+
+trap cleanup_files exit term int
+
+# This test compares the perf-script outputs of the files perf.data and perf.data.inject.
+# The former contains AUXTRACE events. The later contains the expected SAMPLE events.
+test_decoding() {
+	echo "Testing $1..."
+
+	cd $TEMP_DIR
+
+	arm_download_auxtrace_test_files "$1"
+
+	# unpack debug directory
+	rm -rf ~/.debug
+	mkdir ~/.debug
+	tar xf perf.data.tar.bz2 -C ~/.debug
+
+	perf script -i perf.data	$3 $2	| tr -s " " > perf.data.script
+	perf script -i perf.data.inject	$3	| tr -s " " > perf.data.inject.script
+
+	# Fail the test if there are any differences in the generated samples.
+	diff perf.data.script perf.data.inject.script > /dev/null
+}
+
+test_coresight() { 
+	# test only if perf has OpenCSD support
+	if ! ldd perf | grep -q "opencsd"
+	then
+		echo "[Skipped: missing OpenCSD support]"
+		return
+	fi
+
+	# the "--itrace" params must match the ones used to generate perf.data.inject files
+
+	test_decoding "coresight/sort_single_thread"		\
+		      "--itrace=i10ib"				\
+		      "--fields hw:cpu,pid,tid,ip,dso,addr,comm,event"
+
+	test_decoding "coresight/sort_multi_thread"		\
+		      "--itrace=i100ib"				\
+		      "--fields hw:cpu,pid,tid,ip,dso,addr,comm,event"
+}
+
+test_coresight
-- 
2.25.1

Re: [RFC PATCH 0/1] perf test cs-etm: Add end-to-end tests for CoreSight decoding

[Arnaldo Carvalho de Melo]

Em Mon, May 23, 2022 at 03:49:51PM +0100, German Gomez escreveu:
> Hi
> 
> While discussing running more tests for CoreSight, we thought it might
> be a good idea to upstream some EndToEnd tests for CoreSight decoding in
> order to lock down the behaviour. I am sending this as RFC to get some
> feedback from the community first.
> 
> The test relies on pre-geneated perf.data files that are downloaded
> during the test. I'm not sure it's a good idea to commit those files to
> the Linux repository, so they would have to live in an external source
> and be downloaded during the test.

That is ok, but please cache it locally, so that from the second 'perf
test' run onwards one doesn´t have to incur in the download delay and
also be able to run the tests while not connected.

- Arnaldo
 
> For this RFC, the files are stored in a Github repository [1]. As an
> idea, I think we could store them in a new repo in the ARM-software
> namespace. Any hosting suggestions are very welcome.
> 
> Thanks,
> German
> 
> [1] https://github.com/ARM-software/data/tree/984cde8fb0bb22591e284826a80b338bb79c3655/perf/coresight
> 
> German Gomez (1):
>   perf test cs-etm: Add end-to-end tests for CoreSight decoding
> 
>  tools/perf/tests/shell/lib/arm_auxtrace.sh    | 21 +++++++
>  .../tests/shell/test_arm_coresight_decoder.sh | 57 +++++++++++++++++++
>  2 files changed, 78 insertions(+)
>  create mode 100644 tools/perf/tests/shell/lib/arm_auxtrace.sh
>  create mode 100755 tools/perf/tests/shell/test_arm_coresight_decoder.sh
> 
> -- 
> 2.25.1

-- 

- Arnaldo

Re: [RFC PATCH 0/1] perf test cs-etm: Add end-to-end tests for CoreSight decoding

[Ian Rogers]

On Mon, May 23, 2022 at 1:05 PM Arnaldo Carvalho de Melo
<acme@kernel.org> wrote:
>
> Em Mon, May 23, 2022 at 03:49:51PM +0100, German Gomez escreveu:
> > Hi
> >
> > While discussing running more tests for CoreSight, we thought it might
> > be a good idea to upstream some EndToEnd tests for CoreSight decoding in
> > order to lock down the behaviour. I am sending this as RFC to get some
> > feedback from the community first.
> >
> > The test relies on pre-geneated perf.data files that are downloaded
> > during the test. I'm not sure it's a good idea to commit those files to
> > the Linux repository, so they would have to live in an external source
> > and be downloaded during the test.
>
> That is ok, but please cache it locally, so that from the second 'perf
> test' run onwards one doesn´t have to incur in the download delay and
> also be able to run the tests while not connected.

I have some continuous tests running that don't have internet access.
Could we have an environment variable to give a path to an already
downloaded version? I may be able to fake having a cached downloaded
version. Where would such a download live?

Thanks,
Ian

> - Arnaldo
>
> > For this RFC, the files are stored in a Github repository [1]. As an
> > idea, I think we could store them in a new repo in the ARM-software
> > namespace. Any hosting suggestions are very welcome.
> >
> > Thanks,
> > German
> >
> > [1] https://github.com/ARM-software/data/tree/984cde8fb0bb22591e284826a80b338bb79c3655/perf/coresight
> >
> > German Gomez (1):
> >   perf test cs-etm: Add end-to-end tests for CoreSight decoding
> >
> >  tools/perf/tests/shell/lib/arm_auxtrace.sh    | 21 +++++++
> >  .../tests/shell/test_arm_coresight_decoder.sh | 57 +++++++++++++++++++
> >  2 files changed, 78 insertions(+)
> >  create mode 100644 tools/perf/tests/shell/lib/arm_auxtrace.sh
> >  create mode 100755 tools/perf/tests/shell/test_arm_coresight_decoder.sh
> >
> > --
> > 2.25.1
>
> --
>
> - Arnaldo

Re: [RFC PATCH 0/1] perf test cs-etm: Add end-to-end tests for CoreSight decoding

[German Gomez]

Hi Ian, Arnaldo, thanks a lot for your feedback

On 23/05/2022 22:29, Ian Rogers wrote:
> On Mon, May 23, 2022 at 1:05 PM Arnaldo Carvalho de Melo
> <acme@kernel.org> wrote:
>> Em Mon, May 23, 2022 at 03:49:51PM +0100, German Gomez escreveu:
>>> Hi
>>>
>>> While discussing running more tests for CoreSight, we thought it might
>>> be a good idea to upstream some EndToEnd tests for CoreSight decoding in
>>> order to lock down the behaviour. I am sending this as RFC to get some
>>> feedback from the community first.
>>>
>>> The test relies on pre-geneated perf.data files that are downloaded
>>> during the test. I'm not sure it's a good idea to commit those files to
>>> the Linux repository, so they would have to live in an external source
>>> and be downloaded during the test.
>> That is ok, but please cache it locally, so that from the second 'perf
>> test' run onwards one doesn´t have to incur in the download delay and
>> also be able to run the tests while not connected.

Agreed, I will keep this in mind.

> I have some continuous tests running that don't have internet access.
> Could we have an environment variable to give a path to an already
> downloaded version? I may be able to fake having a cached downloaded
> version. Where would such a download live?

Yes sure, download location could be (in order of priority) $PERF_TEST_AUXTRACE_DECODER > $XDG_CACHE_HOME > ~/.cache. I can add a mechanism to override the download step if needed.

Also I think it would make sense for it to be generic/extensible for SPE and PT as well.

Imho best location for Arm files at the moment is github/ARM-software, since we already kind of use it for perf work (the pmu json files, for example). We also have gitlab.arm.com but I'm not very familiar with that one, or if it can be used at all.

Thanks,
German

>
> Thanks,
> Ian
>
>> - Arnaldo
>>
>>> For this RFC, the files are stored in a Github repository [1]. As an
>>> idea, I think we could store them in a new repo in the ARM-software
>>> namespace. Any hosting suggestions are very welcome.
>>>
>>> Thanks,
>>> German
>>>
>>> [1] https://github.com/ARM-software/data/tree/984cde8fb0bb22591e284826a80b338bb79c3655/perf/coresight
>>>
>>> German Gomez (1):
>>>   perf test cs-etm: Add end-to-end tests for CoreSight decoding
>>>
>>>  tools/perf/tests/shell/lib/arm_auxtrace.sh    | 21 +++++++
>>>  .../tests/shell/test_arm_coresight_decoder.sh | 57 +++++++++++++++++++
>>>  2 files changed, 78 insertions(+)
>>>  create mode 100644 tools/perf/tests/shell/lib/arm_auxtrace.sh
>>>  create mode 100755 tools/perf/tests/shell/test_arm_coresight_decoder.sh
>>>
>>> --
>>> 2.25.1
>> --
>>
>> - Arnaldo