From: Arnaldo Carvalho de Melo <acme@kernel.org>
To: Michael Petlan <mpetlan@redhat.com>
Cc: linux-perf-users@vger.kernel.org, acme@redhat.com,
qzhao@redhat.com, cjense@google.com,
Ian Rogers <irogers@google.com>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
Namhyung Kim <namhyung@kernel.org>, Jiri Olsa <jolsa@kernel.org>,
Adrian Hunter <adrian.hunter@intel.com>
Subject: Re: [PATCH 2/2] perf test: Fix JSON format linter test checks
Date: Fri, 27 Jan 2023 09:26:03 -0300 [thread overview]
Message-ID: <Y9PC2+Yqh3W8wyNU@kernel.org> (raw)
In-Reply-To: <20230120134039.17788-3-mpetlan@redhat.com>
Em Fri, Jan 20, 2023 at 02:40:39PM +0100, Michael Petlan escreveu:
> The test fails on CPUs with topdown metrics, where it is common to print
> two metrics per line. Since these are included in default event set for
> `perf stat -- something`, it causes the "no args" subtest to fail due to
> unexpected member count. We need to accept 7 or 9 members in that case.
>
> Coming to that, counting commas in the JSON line and consider it a number
> of elements is incorrect and misleading. There should be +1 element than
> the count of commas, while also, commas can potentially appear in the
> event names. Count " : " substrings rather, since these better fit to
> what should be actually counted.
Before I apply this first patch I can run, as root, the 'perf test' JSON
lint 100 times without problems:
[root@quaco ~]# for a in $(seq 100) ; do echo -n $a ; perf test 98 ; done
1 98: perf stat JSON output linter : Ok
2 98: perf stat JSON output linter : Ok
3 98: perf stat JSON output linter : Ok
4 98: perf stat JSON output linter : Ok
<SNIP>
96 98: perf stat JSON output linter : Ok
97 98: perf stat JSON output linter : Ok
98 98: perf stat JSON output linter : Ok
99 98: perf stat JSON output linter : Ok
100 98: perf stat JSON output linter : Ok
[root@quaco ~]#
After applying it it fails seemingly randomly, I'll remove both patches
from my tmp.perf/core branch and investigate.
[acme@quaco perf]$ git log --oneline -1
fdafd42aac3629db (HEAD) perf stat: Fix JSON metric printout for multiple metrics per line
[acme@quaco perf]$ perf -v
perf version 6.2.rc5.gfdafd42aac36
[acme@quaco perf]$
[root@quaco ~]# perf -v
perf version 6.2.rc5.gfdafd42aac36
[root@quaco ~]# for a in $(seq 100) ; do echo -n $a ; perf test 98 ; done
[root@quaco ~]# perf -v
perf version 6.2.rc5.gfdafd42aac36
[root@quaco ~]# for a in $(seq 100) ; do echo -n $a ; perf test 98 ; done
1 98: perf stat JSON output linter : Ok
2 98: perf stat JSON output linter : Ok
3 98: perf stat JSON output linter : FAILED!
4 98: perf stat JSON output linter : Ok
5 98: perf stat JSON output linter : Ok
6 98: perf stat JSON output linter : Ok
7 98: perf stat JSON output linter : FAILED!
8 98: perf stat JSON output linter : Ok
9 98: perf stat JSON output linter : Ok
10 98: perf stat JSON output linter : Ok
11 98: perf stat JSON output linter : Ok
12 98: perf stat JSON output linter : Ok
13 98: perf stat JSON output linter : Ok
14 98: perf stat JSON output linter : Ok
15 98: perf stat JSON output linter : Ok
16 98: perf stat JSON output linter : FAILED!
17 98: perf stat JSON output linter : Ok
18 98: perf stat JSON output linter : Ok
19 98: perf stat JSON output linter : Ok
20 98: perf stat JSON output linter : FAILED!
21 98: perf stat JSON output linter : Ok
22 98: perf stat JSON output linter : Ok
23 98: perf stat JSON output linter : FAILED!
24 98: perf stat JSON output linter : Ok
25 98: perf stat JSON output linter : Ok
26 98: perf stat JSON output linter : Ok
27 98: perf stat JSON output linter : Ok
28 98: perf stat JSON output linter : FAILED!
29 98: perf stat JSON output linter : Ok
30 98: perf stat JSON output linter : Ok
31 98: perf stat JSON output linter : FAILED!
32 98: perf stat JSON output linter : FAILED!
33 98: perf stat JSON output linter : FAILED!
34 98: perf stat JSON output linter : Ok
35 98: perf stat JSON output linter : Ok
36 98: perf stat JSON output linter : Ok
37 98: perf stat JSON output linter : Ok
38 98: perf stat JSON output linter : FAILED!
39 98: perf stat JSON output linter : FAILED!
40 98: perf stat JSON output linter : FAILED!
41 98: perf stat JSON output linter : Ok
42 98: perf stat JSON output linter : Ok
43 98: perf stat JSON output linter : Ok
44 98: perf stat JSON output linter : Ok
45 98: perf stat JSON output linter : Ok
46 98: perf stat JSON output linter : FAILED!
47 98: perf stat JSON output linter : Ok
48 98: perf stat JSON output linter : FAILED!
49 98: perf stat JSON output linter : FAILED!
50 98: perf stat JSON output linter : FAILED!
51 98: perf stat JSON output linter : FAILED!
52 98: perf stat JSON output linter : Ok
53 98: perf stat JSON output linter : FAILED!
54 98: perf stat JSON output linter : Ok
55 98: perf stat JSON output linter : Ok
56 98: perf stat JSON output linter : FAILED!
57 98: perf stat JSON output linter : Ok
58 98: perf stat JSON output linter : Ok
59 98: perf stat JSON output linter : FAILED!
60 98: perf stat JSON output linter : Ok
61 98: perf stat JSON output linter : FAILED!
62 98: perf stat JSON output linter : Ok
63 98: perf stat JSON output linter : Ok
64 98: perf stat JSON output linter : Ok
65 98: perf stat JSON output linter : Ok
66 98: perf stat JSON output linter : Ok
67 98: perf stat JSON output linter : Ok
68 98: perf stat JSON output linter : Ok
69 98: perf stat JSON output linter : Ok
70 98: perf stat JSON output linter : Ok
71 98: perf stat JSON output linter : FAILED!
72 98: perf stat JSON output linter : FAILED!
73 98: perf stat JSON output linter : Ok
74 98: perf stat JSON output linter : FAILED!
75 98: perf stat JSON output linter : Ok
76 98: perf stat JSON output linter : FAILED!
77 98: perf stat JSON output linter : Ok
78 98: perf stat JSON output linter : FAILED!
79 98: perf stat JSON output linter : FAILED!
80 98: perf stat JSON output linter : FAILED!
81 98: perf stat JSON output linter : Ok
82 98: perf stat JSON output linter : FAILED!
83 98: perf stat JSON output linter : FAILED!
84 98: perf stat JSON output linter : Ok
85 98: perf stat JSON output linter : Ok
86 98: perf stat JSON output linter : Ok
87 98: perf stat JSON output linter : FAILED!
88 98: perf stat JSON output linter : FAILED!
89 98: perf stat JSON output linter : Ok
90 98: perf stat JSON output linter : FAILED!
91 98: perf stat JSON output linter : FAILED!
92 98: perf stat JSON output linter : Ok
93 98: perf stat JSON output linter : FAILED!
94 98: perf stat JSON output linter : FAILED!
95 98: perf stat JSON output linter : FAILED!
96 98: perf stat JSON output linter : FAILED!
97 98: perf stat JSON output linter : FAILED!
98 98: perf stat JSON output linter : Ok
99 98: perf stat JSON output linter : Ok
100 98: perf stat JSON output linter : FAILED!
[root@quaco ~]#
> Fixes: 0c343af2a2f8 ("perf test: JSON format checking")
>
> Signed-off-by: Michael Petlan <mpetlan@redhat.com>
> ---
> .../tests/shell/lib/perf_json_output_lint.py | 16 ++++++++--------
> 1 file changed, 8 insertions(+), 8 deletions(-)
>
> diff --git a/tools/perf/tests/shell/lib/perf_json_output_lint.py b/tools/perf/tests/shell/lib/perf_json_output_lint.py
> index d90f8d102eb9..4f1bbb3f07ec 100644
> --- a/tools/perf/tests/shell/lib/perf_json_output_lint.py
> +++ b/tools/perf/tests/shell/lib/perf_json_output_lint.py
> @@ -40,17 +40,17 @@ def is_counter_value(num):
> return isfloat(num) or num == '<not counted>' or num == '<not supported>'
>
> def check_json_output(expected_items):
> - if expected_items != -1:
> + if expected_items:
> for line in Lines:
> if 'failed' not in line:
> count = 0
> - count = line.count(',')
> - if count != expected_items and count >= 1 and count <= 3 and 'metric-value' in line:
> + count = line.count(' : ')
> + if count not in expected_items and count >= 2 and count <= 4 and 'metric-value' in line:
> # Events that generate >1 metric may have isolated metric
> # values and possibly other prefixes like interval, core and
> # aggregate-number.
> continue
> - if count != expected_items:
> + if count not in expected_items:
> raise RuntimeError(f'wrong number of fields. counted {count} expected {expected_items}'
> f' in \'{line}\'')
> checks = {
> @@ -82,14 +82,14 @@ def check_json_output(expected_items):
>
> try:
> if args.no_args or args.system_wide or args.event:
> - expected_items = 6
> + expected_items = [7, 9]
> elif args.interval or args.per_thread or args.system_wide_no_aggr:
> - expected_items = 7
> + expected_items = [8, 10]
> elif args.per_core or args.per_socket or args.per_node or args.per_die:
> - expected_items = 8
> + expected_items = [9, 11]
> else:
> # If no option is specified, don't check the number of items.
> - expected_items = -1
> + expected_items = []
> check_json_output(expected_items)
> except:
> print('Test failed for input:\n' + '\n'.join(Lines))
> --
> 2.18.4
>
--
- Arnaldo
next prev parent reply other threads:[~2023-01-27 12:27 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-01-20 13:40 [PATCH 0/2] perf test: Fix JSON linter Michael Petlan
2023-01-20 13:40 ` [PATCH 1/2] perf stat: Fix JSON metric printout for multiple metrics per line Michael Petlan
2023-01-23 6:31 ` Namhyung Kim
2023-05-22 12:11 ` Michael Petlan
2023-06-06 11:16 ` Michael Petlan
2023-01-20 13:40 ` [PATCH 2/2] perf test: Fix JSON format linter test checks Michael Petlan
2023-01-23 6:36 ` Namhyung Kim
2023-01-24 16:49 ` Michael Petlan
2023-01-24 17:26 ` Namhyung Kim
2023-01-27 12:26 ` Arnaldo Carvalho de Melo [this message]
2023-01-27 12:30 ` Arnaldo Carvalho de Melo
2023-01-31 17:14 ` Michael Petlan
2023-02-02 1:18 ` Arnaldo Carvalho de Melo
2023-01-20 17:37 ` [PATCH 0/2] perf test: Fix JSON linter Ian Rogers
2023-01-23 6:48 ` Namhyung Kim
2023-01-23 13:38 ` Arnaldo Carvalho de Melo
2023-01-24 17:39 ` Michael Petlan
2023-01-25 0:37 ` Ian Rogers
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Y9PC2+Yqh3W8wyNU@kernel.org \
--to=acme@kernel.org \
--cc=acme@redhat.com \
--cc=adrian.hunter@intel.com \
--cc=cjense@google.com \
--cc=irogers@google.com \
--cc=jolsa@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-perf-users@vger.kernel.org \
--cc=mpetlan@redhat.com \
--cc=namhyung@kernel.org \
--cc=qzhao@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.