From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2FE6FC433EF for ; Thu, 7 Jul 2022 20:12:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236462AbiGGUMc (ORCPT ); Thu, 7 Jul 2022 16:12:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50802 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236708AbiGGUMY (ORCPT ); Thu, 7 Jul 2022 16:12:24 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 168A160520 for ; Thu, 7 Jul 2022 13:12:23 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id z9-20020a258689000000b0066e38ab7122so9398923ybk.9 for ; Thu, 07 Jul 2022 13:12:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=tFW8VFrPqiDt4A42R2rdLJY8CJp1Wy+H1xcOhCNPxog=; b=Rn6JrCUtfkakxQ60+6FAQ4STRa7Key1gCigctNzJ02Njz7n6jwVl15rlIFGk+YQ7Bg frVrlmPOAm4Jpts3omH1b93EKvfePIXPbIglc9usLMJZ4bmFD4dL4RTJYpL6rqeWRIrG ylSuPxlOs3xFhf88wOkfHRJUjOflTaGKZtoOdLWd1HLrUASvDxI/jXi3bpATzqAbjDwH mgKb/bcr+3xlnYT2THb2YecIRcVLhKp8CRO1RT0Fj48EIYeDMG7N4LYBr3xvOCyssk6O ByckgYq4XKY+AzAzn0VJf1BO4pu9YtGXpjFZ00Eu/gp/j1i7C3zvXrCwHmZcxrzeRNXi a81A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=tFW8VFrPqiDt4A42R2rdLJY8CJp1Wy+H1xcOhCNPxog=; b=6nbnHaxp1rma28hD78zdcwglTmK74uGThxPJH4paFG2OMys2VlXIQWg3EgtqcFWFNE G1769C+vRwSjRsf/2ygDLYRiFJcfeUrBFY5xxvUznKEKM0ES8zxynKwdRkz04RFjXnQQ pi6MNGk7W9STfiDxfs/9rGf8cBrDJx3PiA7XD/1YZYrembmEccoEZhE72FUTLPqL4M59 Ghgv6FjK6DY0misWrWUH0xTMf/k3rSky/mLgR2tkudNnR4FouDqZEYK533L+DiWLceoZ MCmtcewoPUPKiIRPk6AfunozaaX+P3GDJzc2sqqOcd9Nd7MiMZExxACPRIlDQ34Y7grs wiRg== X-Gm-Message-State: AJIora8IysjglfvRC+affYn3vQLevs99yB/W3P10jEn4GMmwXzlvTWjB un7L1231G+t9wZGlo7+4R8NokvRlMTcX X-Google-Smtp-Source: AGRyM1vYTeg8ZWnMrebSTLUXApWydRdUJGbbTc4ql7usQE9POHEV2jrTwgfVslyVSzDAY8hTCBBbmznwaojz X-Received: from irogers.svl.corp.google.com ([2620:15c:2d4:203:bf75:f79:d509:a8d1]) (user=irogers job=sendgmr) by 2002:a81:1355:0:b0:31c:9441:ec1b with SMTP id 82-20020a811355000000b0031c9441ec1bmr26860818ywt.101.1657224742397; Thu, 07 Jul 2022 13:12:22 -0700 (PDT) Date: Thu, 7 Jul 2022 13:12:13 -0700 In-Reply-To: <20220707201213.331663-1-irogers@google.com> Message-Id: <20220707201213.331663-3-irogers@google.com> Mime-Version: 1.0 References: <20220707201213.331663-1-irogers@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [PATCH v6 2/2] perf test: Json format checking From: Ian Rogers To: Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , Mark Rutland , Alexander Shishkin , Jiri Olsa , Namhyung Kim , Kan Liang , Zhengjun Xing , Sandipan Das , Claire Jensen , Alyssa Ross , Like Xu , James Clark , Florian Fischer , linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, Claire Jensen Cc: Stephane Eranian , Ian Rogers Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-perf-users@vger.kernel.org From: Claire Jensen Add field checking tests for perf stat JSON output. Sanity checks the expected number of fields are present, that the expected keys are present and they have the correct values. Signed-off-by: Claire Jensen Signed-off-by: Ian Rogers --- .../tests/shell/lib/perf_json_output_lint.py | 95 +++++++++++ tools/perf/tests/shell/stat+json_output.sh | 147 ++++++++++++++++++ 2 files changed, 242 insertions(+) create mode 100644 tools/perf/tests/shell/lib/perf_json_output_lint.py create mode 100755 tools/perf/tests/shell/stat+json_output.sh diff --git a/tools/perf/tests/shell/lib/perf_json_output_lint.py b/tools/perf/tests/shell/lib/perf_json_output_lint.py new file mode 100644 index 000000000000..aaa4a8677b6c --- /dev/null +++ b/tools/perf/tests/shell/lib/perf_json_output_lint.py @@ -0,0 +1,95 @@ +#!/usr/bin/python +# SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) +# Basic sanity check of perf JSON output as specified in the man page. + +import argparse +import sys +import json + +ap = argparse.ArgumentParser() +ap.add_argument('--no-args', action='store_true') +ap.add_argument('--interval', action='store_true') +ap.add_argument('--system-wide-no-aggr', action='store_true') +ap.add_argument('--system-wide', action='store_true') +ap.add_argument('--event', action='store_true') +ap.add_argument('--per-core', action='store_true') +ap.add_argument('--per-thread', action='store_true') +ap.add_argument('--per-die', action='store_true') +ap.add_argument('--per-node', action='store_true') +ap.add_argument('--per-socket', action='store_true') +args = ap.parse_args() + +Lines = sys.stdin.readlines() + +def isfloat(num): + try: + float(num) + return True + except ValueError: + return False + + +def isint(num): + try: + int(num) + return True + except ValueError: + return False + +def is_counter_value(num): + return isfloat(num) or num == '' or num == '' + +def check_json_output(expected_items): + if expected_items != -1: + for line in Lines: + if 'failed' not in line: + count = 0 + count = line.count(',') + if count != expected_items and (count == 1 or count == 2) and 'metric-value' in line: + # Events that generate >1 metric may have isolated metric + # values and possibly have an interval prefix. + continue + if count != expected_items: + raise RuntimeError(f'wrong number of fields. counted {count} expected {expected_items}' + f' in \'{line}\'') + checks = { + 'aggregate-number': lambda x: isfloat(x), + 'core': lambda x: True, + 'counter-value': lambda x: is_counter_value(x), + 'cgroup': lambda x: True, + 'cpu': lambda x: isint(x), + 'die': lambda x: True, + 'event': lambda x: True, + 'event-runtime': lambda x: isfloat(x), + 'interval': lambda x: isfloat(x), + 'metric-unit': lambda x: True, + 'metric-value': lambda x: isfloat(x), + 'node': lambda x: True, + 'pcnt-running': lambda x: isfloat(x), + 'socket': lambda x: True, + 'thread': lambda x: True, + 'unit': lambda x: True, + } + input = '[\n' + ','.join(Lines) + '\n]' + for item in json.loads(input): + for key, value in item.items(): + if key not in checks: + raise RuntimeError(f'Unexpected key: key={key} value={value}') + if not checks[key](value): + raise RuntimeError(f'Check failed for: key={key} value={value}') + + +try: + if args.no_args or args.system_wide or args.event: + expected_items = 6 + elif args.interval or args.per_thread or args.system_wide_no_aggr: + expected_items = 7 + elif args.per_core or args.per_socket or args.per_node or args.per_die: + expected_items = 8 + else: + # If no option is specified, don't check the number of items. + expected_items = -1 + check_json_output(expected_items) +except: + print('Test failed for input:\n' + '\n'.join(Lines)) + raise diff --git a/tools/perf/tests/shell/stat+json_output.sh b/tools/perf/tests/shell/stat+json_output.sh new file mode 100755 index 000000000000..ea8714a36051 --- /dev/null +++ b/tools/perf/tests/shell/stat+json_output.sh @@ -0,0 +1,147 @@ +#!/bin/bash +# perf stat JSON output linter +# SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) +# Checks various perf stat JSON output commands for the +# correct number of fields. + +set -e + +pythonchecker=$(dirname $0)/lib/perf_json_output_lint.py +if [ "x$PYTHON" == "x" ] +then + if which python3 > /dev/null + then + PYTHON=python3 + elif which python > /dev/null + then + PYTHON=python + else + echo Skipping test, python not detected please set environment variable PYTHON. + exit 2 + fi +fi + +# Return true if perf_event_paranoid is > $1 and not running as root. +function ParanoidAndNotRoot() +{ + [ $(id -u) != 0 ] && [ $(cat /proc/sys/kernel/perf_event_paranoid) -gt $1 ] +} + +check_no_args() +{ + echo -n "Checking json output: no args " + perf stat -j true 2>&1 | $PYTHON $pythonchecker --no-args + echo "[Success]" +} + +check_system_wide() +{ + echo -n "Checking json output: system wide " + if ParanoidAndNotRoot 0 + then + echo "[Skip] paranoia and not root" + return + fi + perf stat -j -a true 2>&1 | $PYTHON $pythonchecker --system-wide + echo "[Success]" +} + +check_system_wide_no_aggr() +{ + echo -n "Checking json output: system wide " + if ParanoidAndNotRoot 0 + then + echo "[Skip] paranoia and not root" + return + fi + echo -n "Checking json output: system wide no aggregation " + perf stat -j -A -a --no-merge true 2>&1 | $PYTHON $pythonchecker --system-wide-no-aggr + echo "[Success]" +} + +check_interval() +{ + echo -n "Checking json output: interval " + perf stat -j -I 1000 true 2>&1 | $PYTHON $pythonchecker --interval + echo "[Success]" +} + + +check_event() +{ + echo -n "Checking json output: event " + perf stat -j -e cpu-clock true 2>&1 | $PYTHON $pythonchecker --event + echo "[Success]" +} + +check_per_core() +{ + echo -n "Checking json output: per core " + if ParanoidAndNotRoot 0 + then + echo "[Skip] paranoia and not root" + return + fi + perf stat -j --per-core -a true 2>&1 | $PYTHON $pythonchecker --per-core + echo "[Success]" +} + +check_per_thread() +{ + echo -n "Checking json output: per thread " + if ParanoidAndNotRoot 0 + then + echo "[Skip] paranoia and not root" + return + fi + perf stat -j --per-thread -a true 2>&1 | $PYTHON $pythonchecker --per-thread + echo "[Success]" +} + +check_per_die() +{ + echo -n "Checking json output: per die " + if ParanoidAndNotRoot 0 + then + echo "[Skip] paranoia and not root" + return + fi + perf stat -j --per-die -a true 2>&1 | $PYTHON $pythonchecker --per-die + echo "[Success]" +} + +check_per_node() +{ + echo -n "Checking json output: per node " + if ParanoidAndNotRoot 0 + then + echo "[Skip] paranoia and not root" + return + fi + perf stat -j --per-node -a true 2>&1 | $PYTHON $pythonchecker --per-node + echo "[Success]" +} + +check_per_socket() +{ + echo -n "Checking json output: per socket " + if ParanoidAndNotRoot 0 + then + echo "[Skip] paranoia and not root" + return + fi + perf stat -j --per-socket -a true 2>&1 | $PYTHON $pythonchecker --per-socket + echo "[Success]" +} + +check_no_args +check_system_wide +check_system_wide_no_aggr +check_interval +check_event +check_per_core +check_per_thread +check_per_die +check_per_node +check_per_socket +exit 0 -- 2.37.0.rc0.161.g10f37bed90-goog