* [PATCH v5 0/3] Add metric value validation test
@ 2023-06-20 17:00 Weilin Wang
2023-06-20 17:00 ` [PATCH v5 1/3] perf test: " Weilin Wang
` (3 more replies)
0 siblings, 4 replies; 9+ messages in thread
From: Weilin Wang @ 2023-06-20 17:00 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Jiri Olsa,
Namhyung Kim, Adrian Hunter, Ian Rogers, linux-perf-users,
linux-kernel
Cc: Weilin Wang, Kan Liang, Samantha Alt, Perry Taylor, Caleb Biggers,
ravi.bangoria
This is the fifth version of metric value validation tests.
We made the following changes from v4 to v5:
- Update "()" to "{}" to avoid creating sub shell and successfully skip test on non-Intel
platform. [Ravi]
v4: https://lore.kernel.org/lkml/20230618172820.751560-1-weilin.wang@intel.com/
Weilin Wang (3):
perf test: Add metric value validation test
perf test: Add skip list for metrics known would fail
perf test: Rerun failed metrics with longer workload
.../tests/shell/lib/perf_metric_validation.py | 574 ++++++++++++++++++
.../lib/perf_metric_validation_rules.json | 398 ++++++++++++
tools/perf/tests/shell/stat_metrics_values.sh | 30 +
3 files changed, 1002 insertions(+)
create mode 100644 tools/perf/tests/shell/lib/perf_metric_validation.py
create mode 100644 tools/perf/tests/shell/lib/perf_metric_validation_rules.json
create mode 100755 tools/perf/tests/shell/stat_metrics_values.sh
base-commit: 7cdda6998ee55140e64894e25048df7157344fc9
--
2.39.1
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v5 1/3] perf test: Add metric value validation test 2023-06-20 17:00 [PATCH v5 0/3] Add metric value validation test Weilin Wang @ 2023-06-20 17:00 ` Weilin Wang 2023-06-20 17:00 ` [PATCH v5 2/3] perf test: Add skip list for metrics known would fail Weilin Wang ` (2 subsequent siblings) 3 siblings, 0 replies; 9+ messages in thread From: Weilin Wang @ 2023-06-20 17:00 UTC (permalink / raw) To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Jiri Olsa, Namhyung Kim, Adrian Hunter, Ian Rogers, linux-perf-users, linux-kernel Cc: Weilin Wang, Kan Liang, Samantha Alt, Perry Taylor, Caleb Biggers, ravi.bangoria Add metric value validation test to check if metric values are with in correct value ranges. There are three types of tests included: 1) positive-value test checks if all the metrics collected are non-negative; 2) single-value test checks if the list of metrics have values in given value ranges; 3) relationship test checks if multiple metrics follow a given relationship, e.g. memory_bandwidth_read + memory_bandwidth_write = memory_bandwidth_total. Signed-off-by: Weilin Wang <weilin.wang@intel.com> --- .../tests/shell/lib/perf_metric_validation.py | 514 ++++++++++++++++++ .../lib/perf_metric_validation_rules.json | 387 +++++++++++++ tools/perf/tests/shell/stat_metrics_values.sh | 30 + 3 files changed, 931 insertions(+) create mode 100644 tools/perf/tests/shell/lib/perf_metric_validation.py create mode 100644 tools/perf/tests/shell/lib/perf_metric_validation_rules.json create mode 100755 tools/perf/tests/shell/stat_metrics_values.sh diff --git a/tools/perf/tests/shell/lib/perf_metric_validation.py b/tools/perf/tests/shell/lib/perf_metric_validation.py new file mode 100644 index 000000000000..81bd2bf38b67 --- /dev/null +++ b/tools/perf/tests/shell/lib/perf_metric_validation.py @@ -0,0 +1,514 @@ +#SPDX-License-Identifier: GPL-2.0 +import re +import csv +import json +import argparse +from pathlib import Path +import subprocess + +class Validator: + def __init__(self, rulefname, reportfname='', t=5, debug=False, datafname='', fullrulefname='', workload='true', metrics=''): + self.rulefname = rulefname + self.reportfname = reportfname + self.rules = None + self.collectlist=metrics + self.metrics = set() + self.tolerance = t + + self.workloads = [x for x in workload.split(",") if x] + self.wlidx = 0 # idx of current workloads + self.allresults = dict() # metric results of all workload + self.allignoremetrics = dict() # metrics with no results or negative results + self.allfailtests = dict() + self.alltotalcnt = dict() + self.allpassedcnt = dict() + self.allerrlist = dict() + + self.results = dict() # metric results of current workload + # vars for test pass/failure statistics + self.ignoremetrics= set() # metrics with no results or negative results, neg result counts as a failed test + self.failtests = dict() + self.totalcnt = 0 + self.passedcnt = 0 + # vars for errors + self.errlist = list() + + # vars for Rule Generator + self.pctgmetrics = set() # Percentage rule + + # vars for debug + self.datafname = datafname + self.debug = debug + self.fullrulefname = fullrulefname + + def read_json(self, filename: str) -> dict: + try: + with open(Path(filename).resolve(), "r") as f: + data = json.loads(f.read()) + except OSError as e: + print(f"Error when reading file {e}") + sys.exit() + + return data + + def json_dump(self, data, output_file): + parent = Path(output_file).parent + if not parent.exists(): + parent.mkdir(parents=True) + + with open(output_file, "w+") as output_file: + json.dump(data, + output_file, + ensure_ascii=True, + indent=4) + + def get_results(self, idx:int = 0): + return self.results[idx] + + def get_bounds(self, lb, ub, error, alias={}, ridx:int = 0) -> list: + """ + Get bounds and tolerance from lb, ub, and error. + If missing lb, use 0.0; missing ub, use float('inf); missing error, use self.tolerance. + + @param lb: str/float, lower bound + @param ub: str/float, upper bound + @param error: float/str, error tolerance + @returns: lower bound, return inf if the lower bound is a metric value and is not collected + upper bound, return -1 if the upper bound is a metric value and is not collected + tolerance, denormalized base on upper bound value + """ + # init ubv and lbv to invalid values + def get_bound_value (bound, initval, ridx): + val = initval + if isinstance(bound, int) or isinstance(bound, float): + val = bound + elif isinstance(bound, str): + if bound == '': + val = float("inf") + elif bound in alias: + vall = self.get_value(alias[ub], ridx) + if vall: + val = vall[0] + elif bound.replace('.', '1').isdigit(): + val = float(bound) + else: + print("Wrong bound: {0}".format(bound)) + else: + print("Wrong bound: {0}".format(bound)) + return val + + ubv = get_bound_value(ub, -1, ridx) + lbv = get_bound_value(lb, float('inf'), ridx) + t = get_bound_value(error, self.tolerance, ridx) + + # denormalize error threshold + denormerr = t * ubv / 100 if ubv != 100 and ubv > 0 else t + + return lbv, ubv, denormerr + + def get_value(self, name:str, ridx:int = 0) -> list: + """ + Get value of the metric from self.results. + If result of this metric is not provided, the metric name will be added into self.ignoremetics and self.errlist. + All future test(s) on this metric will fail. + + @param name: name of the metric + @returns: list with value found in self.results; list is empty when not value found. + """ + results = [] + data = self.results[ridx] if ridx in self.results else self.results[0] + if name not in self.ignoremetrics: + if name in data: + results.append(data[name]) + elif name.replace('.', '1').isdigit(): + results.append(float(name)) + else: + self.errlist.append("Metric '%s' is not collected or the value format is incorrect"%(name)) + self.ignoremetrics.add(name) + return results + + def check_bound(self, val, lb, ub, err): + return True if val <= ub + err and val >= lb - err else False + + # Positive Value Sanity check + def pos_val_test(self): + """ + Check if metrics value are non-negative. + One metric is counted as one test. + Failure: when metric value is negative or not provided. + Metrics with negative value will be added into the self.failtests['PositiveValueTest'] and self.ignoremetrics. + """ + negmetric = set() + missmetric = set() + pcnt = 0 + tcnt = 0 + for name, val in self.get_results().items(): + if val is None or val == '': + missmetric.add(name) + self.errlist.append("Metric '%s' is not collected"%(name)) + elif val < 0: + negmetric.add("{0}(={1:.4f})".format(name, val)) + else: + pcnt += 1 + tcnt += 1 + + self.failtests['PositiveValueTest']['Total Tests'] = tcnt + self.failtests['PositiveValueTest']['Passed Tests'] = pcnt + if len(negmetric) or len(missmetric)> 0: + self.ignoremetrics.update(negmetric) + self.ignoremetrics.update(missmetric) + self.failtests['PositiveValueTest']['Failed Tests'].append({'NegativeValue':list(negmetric), 'MissingValue':list(missmetric)}) + + return + + def evaluate_formula(self, formula:str, alias:dict, ridx:int = 0): + """ + Evaluate the value of formula. + + @param formula: the formula to be evaluated + @param alias: the dict has alias to metric name mapping + @returns: value of the formula is success; -1 if the one or more metric value not provided + """ + stack = [] + b = 0 + errs = [] + sign = "+" + f = str() + + #TODO: support parenthesis? + for i in range(len(formula)): + if i+1 == len(formula) or formula[i] in ('+', '-', '*', '/'): + s = alias[formula[b:i]] if i+1 < len(formula) else alias[formula[b:]] + v = self.get_value(s, ridx) + if not v: + errs.append(s) + else: + f = f + "{0}(={1:.4f})".format(s, v[0]) + if sign == "*": + stack[-1] = stack[-1] * v + elif sign == "/": + stack[-1] = stack[-1] / v + elif sign == '-': + stack.append(-v[0]) + else: + stack.append(v[0]) + if i + 1 < len(formula): + sign = formula[i] + f += sign + b = i + 1 + + if len(errs) > 0: + return -1, "Metric value missing: "+','.join(errs) + + val = sum(stack) + return val, f + + # Relationships Tests + def relationship_test(self, rule: dict): + """ + Validate if the metrics follow the required relationship in the rule. + eg. lower_bound <= eval(formula)<= upper_bound + One rule is counted as ont test. + Failure: when one or more metric result(s) not provided, or when formula evaluated outside of upper/lower bounds. + + @param rule: dict with metric name(+alias), formula, and required upper and lower bounds. + """ + alias = dict() + for m in rule['Metrics']: + alias[m['Alias']] = m['Name'] + lbv, ubv, t = self.get_bounds(rule['RangeLower'], rule['RangeUpper'], rule['ErrorThreshold'], alias, ridx=rule['RuleIndex']) + val, f = self.evaluate_formula(rule['Formula'], alias, ridx=rule['RuleIndex']) + if val == -1: + self.failtests['RelationshipTest']['Failed Tests'].append({'RuleIndex': rule['RuleIndex'], 'Description':f}) + elif not self.check_bound(val, lbv, ubv, t): + lb = rule['RangeLower'] + ub = rule['RangeUpper'] + if isinstance(lb, str): + if lb in alias: + lb = alias[lb] + if isinstance(ub, str): + if ub in alias: + ub = alias[ub] + self.failtests['RelationshipTest']['Failed Tests'].append({'RuleIndex': rule['RuleIndex'], 'Formula':f, + 'RangeLower': lb, 'LowerBoundValue': self.get_value(lb), + 'RangeUpper': ub, 'UpperBoundValue':self.get_value(ub), + 'ErrorThreshold': t, 'CollectedValue': val}) + else: + self.passedcnt += 1 + self.failtests['RelationshipTest']['Passed Tests'] += 1 + self.totalcnt += 1 + self.failtests['RelationshipTest']['Total Tests'] += 1 + + return + + + # Single Metric Test + def single_test(self, rule:dict): + """ + Validate if the metrics are in the required value range. + eg. lower_bound <= metrics_value <= upper_bound + One metric is counted as one test in this type of test. + One rule may include one or more metrics. + Failure: when the metric value not provided or the value is outside the bounds. + This test updates self.total_cnt and records failed tests in self.failtest['SingleMetricTest']. + + @param rule: dict with metrics to validate and the value range requirement + """ + lbv, ubv, t = self.get_bounds(rule['RangeLower'], rule['RangeUpper'], rule['ErrorThreshold']) + metrics = rule['Metrics'] + passcnt = 0 + totalcnt = 0 + faillist = [] + for m in metrics: + totalcnt += 1 + result = self.get_value(m['Name']) + if len(result) > 0 and self.check_bound(result[0], lbv, ubv, t): + passcnt += 1 + else: + faillist.append({'MetricName':m['Name'], 'CollectedValue':result}) + + self.totalcnt += totalcnt + self.passedcnt += passcnt + self.failtests['SingleMetricTest']['Total Tests'] += totalcnt + self.failtests['SingleMetricTest']['Passed Tests'] += passcnt + if len(faillist) != 0: + self.failtests['SingleMetricTest']['Failed Tests'].append({'RuleIndex':rule['RuleIndex'], + 'RangeLower': rule['RangeLower'], + 'RangeUpper': rule['RangeUpper'], + 'ErrorThreshold':rule['ErrorThreshold'], + 'Failure':faillist}) + + return + + def create_report(self): + """ + Create final report and write into a JSON file. + """ + alldata = list() + for i in range(0, len(self.workloads)): + reportstas = {"Total Rule Count": self.alltotalcnt[i], "Passed Rule Count": self.allpassedcnt[i]} + data = {"Metric Validation Statistics": reportstas, "Tests in Category": self.allfailtests[i], + "Errors":self.allerrlist[i]} + alldata.append({"Workload": self.workloads[i], "Report": data}) + + json_str = json.dumps(alldata, indent=4) + print("Test validation finished. Final report: ") + print(json_str) + + if self.debug: + allres = [{"Workload": self.workloads[i], "Results": self.allresults[i]} for i in range(0, len(self.workloads))] + self.json_dump(allres, self.datafname) + + def check_rule(self, testtype, metric_list): + """ + Check if the rule uses metric(s) that not exist in current platform. + + @param metric_list: list of metrics from the rule. + @return: False when find one metric out in Metric file. (This rule should not skipped.) + True when all metrics used in the rule are found in Metric file. + """ + if testtype == "RelationshipTest": + for m in metric_list: + if m['Name'] not in self.metrics: + return False + return True + + # Start of Collector and Converter + def convert(self, data: list, idx: int): + """ + Convert collected metric data from the -j output to dict of {metric_name:value}. + """ + for json_string in data: + try: + result =json.loads(json_string) + if "metric-unit" in result and result["metric-unit"] != "(null)" and result["metric-unit"] != "": + name = result["metric-unit"].split(" ")[1] if len(result["metric-unit"].split(" ")) > 1 \ + else result["metric-unit"] + if idx not in self.results: self.results[idx] = dict() + self.results[idx][name.lower()] = float(result["metric-value"]) + except ValueError as error: + continue + return + + def collect_perf(self, data_file: str, workload: str): + """ + Collect metric data with "perf stat -M" on given workload with -a and -j. + """ + self.results = dict() + tool = 'perf' + print(f"Starting perf collection") + print(f"Workload: {workload}") + collectlist = dict() + if self.collectlist != "": + collectlist[0] = {x for x in self.collectlist.split(",")} + else: + collectlist[0] = set(list(self.metrics)) + # Create metric set for relationship rules + for rule in self.rules: + if rule["TestType"] == "RelationshipTest": + metrics = [m["Name"] for m in rule["Metrics"]] + if not any(m not in collectlist[0] for m in metrics): + collectlist[rule["RuleIndex"]] = set(metrics) + + for idx, metrics in collectlist.items(): + if idx == 0: wl = "sleep 0.5".split() + else: wl = workload.split() + for metric in metrics: + command = [tool, 'stat', '-j', '-M', f"{metric}", "-a"] + command.extend(wl) + cmd = subprocess.run(command, stderr=subprocess.PIPE, encoding='utf-8') + data = [x+'}' for x in cmd.stderr.split('}\n') if x] + self.convert(data, idx) + # End of Collector and Converter + + # Start of Rule Generator + def parse_perf_metrics(self): + """ + Read and parse perf metric file: + 1) find metrics with '1%' or '100%' as ScaleUnit for Percent check + 2) create metric name list + """ + command = ['perf', 'list', '-j', '--details', 'metrics'] + cmd = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, encoding='utf-8') + try: + data = json.loads(cmd.stdout) + for m in data: + if 'MetricName' not in m: + print("Warning: no metric name") + continue + name = m['MetricName'] + self.metrics.add(name) + if 'ScaleUnit' in m and (m['ScaleUnit'] == '1%' or m['ScaleUnit'] == '100%'): + self.pctgmetrics.add(name.lower()) + except ValueError as error: + print(f"Error when parsing metric data") + sys.exit() + + return + + def create_rules(self): + """ + Create full rules which includes: + 1) All the rules from the "relationshi_rules" file + 2) SingleMetric rule for all the 'percent' metrics + + Reindex all the rules to avoid repeated RuleIndex + """ + self.rules = self.read_json(self.rulefname)['RelationshipRules'] + pctgrule = {'RuleIndex':0, + 'TestType':'SingleMetricTest', + 'RangeLower':'0', + 'RangeUpper': '100', + 'ErrorThreshold': self.tolerance, + 'Description':'Metrics in percent unit have value with in [0, 100]', + 'Metrics': [{'Name': m} for m in self.pctgmetrics]} + self.rules.append(pctgrule) + + # Re-index all rules to avoid repeated RuleIndex + idx = 1 + for r in self.rules: + r['RuleIndex'] = idx + idx += 1 + + if self.debug: + #TODO: need to test and generate file name correctly + data = {'RelationshipRules':self.rules, 'SupportedMetrics': [{"MetricName": name} for name in self.metrics]} + self.json_dump(data, self.fullrulefname) + + return + # End of Rule Generator + + def _storewldata(self, key): + ''' + Store all the data of one workload into the corresponding data structure for all workloads. + @param key: key to the dictionaries (index of self.workloads). + ''' + self.allresults[key] = self.results + self.allignoremetrics[key] = self.ignoremetrics + self.allfailtests[key] = self.failtests + self.alltotalcnt[key] = self.totalcnt + self.allpassedcnt[key] = self.passedcnt + self.allerrlist[key] = self.errlist + + #Initialize data structures before data validation of each workload + def _init_data(self): + + testtypes = ['PositiveValueTest', 'RelationshipTest', 'SingleMetricTest'] + self.results = dict() + self.ignoremetrics= set() + self.errlist = list() + self.failtests = {k:{'Total Tests':0, 'Passed Tests':0, 'Failed Tests':[]} for k in testtypes} + self.totalcnt = 0 + self.passedcnt = 0 + + def test(self): + ''' + The real entry point of the test framework. + This function loads the validation rule JSON file and Standard Metric file to create rules for + testing and namemap dictionaries. + It also reads in result JSON file for testing. + + In the test process, it passes through each rule and launch correct test function bases on the + 'TestType' field of the rule. + + The final report is written into a JSON file. + ''' + self.parse_perf_metrics() + self.create_rules() + for i in range(0, len(self.workloads)): + self._init_data() + self.collect_perf(self.datafname, self.workloads[i]) + # Run positive value test + self.pos_val_test() + for r in self.rules: + # skip rules that uses metrics not exist in this platform + testtype = r['TestType'] + if not self.check_rule(testtype, r['Metrics']): + continue + if testtype == 'RelationshipTest': + self.relationship_test(r) + elif testtype == 'SingleMetricTest': + self.single_test(r) + else: + print("Unsupported Test Type: ", testtype) + self.errlist.append("Unsupported Test Type from rule: " + r['RuleIndex']) + self._storewldata(i) + print("Workload: ", self.workloads[i]) + print("Total metrics collected: ", self.failtests['PositiveValueTest']['Total Tests']) + print("Non-negative metric count: ", self.failtests['PositiveValueTest']['Passed Tests']) + print("Total Test Count: ", self.totalcnt) + print("Passed Test Count: ", self.passedcnt) + + self.create_report() + return sum(self.alltotalcnt.values()) != sum(self.allpassedcnt.values()) +# End of Class Validator + + +def main() -> None: + parser = argparse.ArgumentParser(description="Launch metric value validation") + + parser.add_argument("-rule", help="Base validation rule file", required=True) + parser.add_argument("-output_dir", help="Path for validator output file, report file", required=True) + parser.add_argument("-debug", help="Debug run, save intermediate data to files", action="store_true", default=False) + parser.add_argument("-wl", help="Workload to run while data collection", default="true") + parser.add_argument("-m", help="Metric list to validate", default="") + args = parser.parse_args() + outpath = Path(args.output_dir) + reportf = Path.joinpath(outpath, 'perf_report.json') + fullrule = Path.joinpath(outpath, 'full_rule.json') + datafile = Path.joinpath(outpath, 'perf_data.json') + + validator = Validator(args.rule, reportf, debug=args.debug, + datafname=datafile, fullrulefname=fullrule, workload=args.wl, + metrics=args.m) + ret = validator.test() + + return ret + + +if __name__ == "__main__": + import sys + sys.exit(main()) + + + diff --git a/tools/perf/tests/shell/lib/perf_metric_validation_rules.json b/tools/perf/tests/shell/lib/perf_metric_validation_rules.json new file mode 100644 index 000000000000..debaa910da9f --- /dev/null +++ b/tools/perf/tests/shell/lib/perf_metric_validation_rules.json @@ -0,0 +1,387 @@ +{ + "RelationshipRules": [ + { + "RuleIndex": 1, + "Formula": "a+b", + "TestType": "RelationshipTest", + "RangeLower": "c", + "RangeUpper": "c", + "ErrorThreshold": 5.0, + "Description": "Intel(R) Optane(TM) Persistent Memory(PMEM) bandwidth total includes Intel(R) Optane(TM) Persistent Memory(PMEM) read bandwidth and Intel(R) Optane(TM) Persistent Memory(PMEM) write bandwidth", + "Metrics": [ + { + "Name": "pmem_memory_bandwidth_read", + "Alias": "a" + }, + { + "Name": "pmem_memory_bandwidth_write", + "Alias": "b" + }, + { + "Name": "pmem_memory_bandwidth_total", + "Alias": "c" + } + ] + }, + { + "RuleIndex": 2, + "Formula": "a+b", + "TestType": "RelationshipTest", + "RangeLower": "c", + "RangeUpper": "c", + "ErrorThreshold": 5.0, + "Description": "DDR memory bandwidth total includes DDR memory read bandwidth and DDR memory write bandwidth", + "Metrics": [ + { + "Name": "memory_bandwidth_read", + "Alias": "a" + }, + { + "Name": "memory_bandwidth_write", + "Alias": "b" + }, + { + "Name": "memory_bandwidth_total", + "Alias": "c" + } + ] + }, + { + "RuleIndex": 3, + "Formula": "a+b", + "TestType": "RelationshipTest", + "RangeLower": "100", + "RangeUpper": "100", + "ErrorThreshold": 5.0, + "Description": "Total memory read accesses includes memory reads from last level cache (LLC) addressed to local DRAM and memory reads from the last level cache (LLC) addressed to remote DRAM.", + "Metrics": [ + { + "Name": "numa_reads_addressed_to_local_dram", + "Alias": "a" + }, + { + "Name": "numa_reads_addressed_to_remote_dram", + "Alias": "b" + } + ] + }, + { + "RuleIndex": 4, + "Formula": "a", + "TestType": "SingleMetricTest", + "RangeLower": "0.125", + "RangeUpper": "", + "ErrorThreshold": "", + "Description": "", + "Metrics": [ + { + "Name": "cpi", + "Alias": "a" + } + ] + }, + { + "RuleIndex": 5, + "Formula": "", + "TestType": "SingleMetricTest", + "RangeLower": "0", + "RangeUpper": "1", + "ErrorThreshold": 5.0, + "Description": "Ratio values should be within value range [0,1)", + "Metrics": [ + { + "Name": "loads_per_instr", + "Alias": "" + }, + { + "Name": "stores_per_instr", + "Alias": "" + }, + { + "Name": "l1d_mpi", + "Alias": "" + }, + { + "Name": "l1d_demand_data_read_hits_per_instr", + "Alias": "" + }, + { + "Name": "l1_i_code_read_misses_with_prefetches_per_instr", + "Alias": "" + }, + { + "Name": "l2_demand_data_read_hits_per_instr", + "Alias": "" + }, + { + "Name": "l2_mpi", + "Alias": "" + }, + { + "Name": "l2_demand_data_read_mpi", + "Alias": "" + }, + { + "Name": "l2_demand_code_mpi", + "Alias": "" + } + ] + }, + { + "RuleIndex": 6, + "Formula": "a+b+c+d", + "TestType": "RelationshipTest", + "RangeLower": "100", + "RangeUpper": "100", + "ErrorThreshold": 5.0, + "Description": "Sum of TMA level 1 metrics should be 100%", + "Metrics": [ + { + "Name": "tma_frontend_bound", + "Alias": "a" + }, + { + "Name": "tma_bad_speculation", + "Alias": "b" + }, + { + "Name": "tma_backend_bound", + "Alias": "c" + }, + { + "Name": "tma_retiring", + "Alias": "d" + } + ] + }, + { + "RuleIndex": 7, + "Formula": "a+b", + "TestType": "RelationshipTest", + "RangeLower": "c", + "RangeUpper": "c", + "ErrorThreshold": 5.0, + "Description": "Sum of the level 2 children should equal level 1 parent", + "Metrics": [ + { + "Name": "tma_fetch_latency", + "Alias": "a" + }, + { + "Name": "tma_fetch_bandwidth", + "Alias": "b" + }, + { + "Name": "tma_frontend_bound", + "Alias": "c" + } + ] + }, + { + "RuleIndex": 8, + "Formula": "a+b", + "TestType": "RelationshipTest", + "RangeLower": "c", + "RangeUpper": "c", + "ErrorThreshold": 5.0, + "Description": "Sum of the level 2 children should equal level 1 parent", + "Metrics": [ + { + "Name": "tma_branch_mispredicts", + "Alias": "a" + }, + { + "Name": "tma_machine_clears", + "Alias": "b" + }, + { + "Name": "tma_bad_speculation", + "Alias": "c" + } + ] + }, + { + "RuleIndex": 9, + "Formula": "a+b", + "TestType": "RelationshipTest", + "RangeLower": "c", + "RangeUpper": "c", + "ErrorThreshold": 5.0, + "Description": "Sum of the level 2 children should equal level 1 parent", + "Metrics": [ + { + "Name": "tma_memory_bound", + "Alias": "a" + }, + { + "Name": "tma_core_bound", + "Alias": "b" + }, + { + "Name": "tma_backend_bound", + "Alias": "c" + } + ] + }, + { + "RuleIndex": 10, + "Formula": "a+b", + "TestType": "RelationshipTest", + "RangeLower": "c", + "RangeUpper": "c", + "ErrorThreshold": 5.0, + "Description": "Sum of the level 2 children should equal level 1 parent", + "Metrics": [ + { + "Name": "tma_light_operations", + "Alias": "a" + }, + { + "Name": "tma_heavy_operations", + "Alias": "b" + }, + { + "Name": "tma_retiring", + "Alias": "c" + } + ] + }, + { + "RuleIndex": 11, + "Formula": "a+b+c", + "TestType": "RelationshipTest", + "RangeLower": "100", + "RangeUpper": "100", + "ErrorThreshold": 5.0, + "Description": "The all_requests includes the memory_page_empty, memory_page_misses, and memory_page_hits equals.", + "Metrics": [ + { + "Name": "memory_page_empty_vs_all_requests", + "Alias": "a" + }, + { + "Name": "memory_page_misses_vs_all_requests", + "Alias": "b" + }, + { + "Name": "memory_page_hits_vs_all_requests", + "Alias": "c" + } + ] + }, + { + "RuleIndex": 12, + "Formula": "a-b", + "TestType": "RelationshipTest", + "RangeLower": "0", + "RangeUpper": "", + "ErrorThreshold": 5.0, + "Description": "CPU utilization in kernel mode should always be <= cpu utilization", + "Metrics": [ + { + "Name": "cpu_utilization", + "Alias": "a" + }, + { + "Name": "cpu_utilization_in_kernel_mode", + "Alias": "b" + } + ] + }, + { + "RuleIndex": 13, + "Formula": "a-b", + "TestType": "RelationshipTest", + "RangeLower": "0", + "RangeUpper": "", + "ErrorThreshold": 5.0, + "Description": "Total L2 misses per instruction should be >= L2 demand data read misses per instruction", + "Metrics": [ + { + "Name": "l2_mpi", + "Alias": "a" + }, + { + "Name": "l2_demand_data_read_mpi", + "Alias": "b" + } + ] + }, + { + "RuleIndex": 14, + "Formula": "a-b", + "TestType": "RelationshipTest", + "RangeLower": "0", + "RangeUpper": "", + "ErrorThreshold": 5.0, + "Description": "Total L2 misses per instruction should be >= L2 demand code misses per instruction", + "Metrics": [ + { + "Name": "l2_mpi", + "Alias": "a" + }, + { + "Name": "l2_demand_code_mpi", + "Alias": "b" + } + ] + }, + { + "RuleIndex": 15, + "Formula": "b+c+d", + "TestType": "RelationshipTest", + "RangeLower": "a", + "RangeUpper": "a", + "ErrorThreshold": 5.0, + "Description": "L3 data read, rfo, code misses per instruction equals total L3 misses per instruction.", + "Metrics": [ + { + "Name": "llc_mpi", + "Alias": "a" + }, + { + "Name": "llc_data_read_mpi_demand_plus_prefetch", + "Alias": "b" + }, + { + "Name": "llc_rfo_read_mpi_demand_plus_prefetch", + "Alias": "c" + }, + { + "Name": "llc_code_read_mpi_demand_plus_prefetch", + "Alias": "d" + } + ] + }, + { + "RuleIndex": 16, + "Formula": "a", + "TestType": "SingleMetricTest", + "RangeLower": "0", + "RangeUpper": "8", + "ErrorThreshold": 0.0, + "Description": "Setting generous range for allowable frequencies", + "Metrics": [ + { + "Name": "uncore_freq", + "Alias": "a" + } + ] + }, + { + "RuleIndex": 17, + "Formula": "a", + "TestType": "SingleMetricTest", + "RangeLower": "0", + "RangeUpper": "8", + "ErrorThreshold": 0.0, + "Description": "Setting generous range for allowable frequencies", + "Metrics": [ + { + "Name": "cpu_operating_frequency", + "Alias": "a" + } + ] + } + ] +} \ No newline at end of file diff --git a/tools/perf/tests/shell/stat_metrics_values.sh b/tools/perf/tests/shell/stat_metrics_values.sh new file mode 100755 index 000000000000..ad94c936de7e --- /dev/null +++ b/tools/perf/tests/shell/stat_metrics_values.sh @@ -0,0 +1,30 @@ +#!/bin/bash +# perf metrics value validation +# SPDX-License-Identifier: GPL-2.0 +if [ "x$PYTHON" == "x" ] +then + if which python3 > /dev/null + then + PYTHON=python3 + else + echo Skipping test, python3 not detected please set environment variable PYTHON. + exit 2 + fi +fi + +grep -q GenuineIntel /proc/cpuinfo || { echo Skipping non-Intel; exit 2; } + +pythonvalidator=$(dirname $0)/lib/perf_metric_validation.py +rulefile=$(dirname $0)/lib/perf_metric_validation_rules.json +tmpdir=$(mktemp -d /tmp/__perf_test.program.XXXXX) +workload="perf bench futex hash -r 2 -s" + +# Add -debug, save data file and full rule file +echo "Launch python validation script $pythonvalidator" +echo "Output will be stored in: $tmpdir" +$PYTHON $pythonvalidator -rule $rulefile -output_dir $tmpdir -wl "${workload}" +ret=$? +rm -rf $tmpdir + +exit $ret + -- 2.39.1 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v5 2/3] perf test: Add skip list for metrics known would fail 2023-06-20 17:00 [PATCH v5 0/3] Add metric value validation test Weilin Wang 2023-06-20 17:00 ` [PATCH v5 1/3] perf test: " Weilin Wang @ 2023-06-20 17:00 ` Weilin Wang 2023-06-20 17:00 ` [PATCH v5 3/3] perf test: Rerun failed metrics with longer workload Weilin Wang 2023-06-22 0:08 ` [PATCH v5 0/3] Add metric value validation test Namhyung Kim 3 siblings, 0 replies; 9+ messages in thread From: Weilin Wang @ 2023-06-20 17:00 UTC (permalink / raw) To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Jiri Olsa, Namhyung Kim, Adrian Hunter, Ian Rogers, linux-perf-users, linux-kernel Cc: Weilin Wang, Kan Liang, Samantha Alt, Perry Taylor, Caleb Biggers, ravi.bangoria Add skip list for metrics known would fail because some of the metrics are very likely to fail due to multiplexing or other errors. So add all of the flaky tests into the skip list. Signed-off-by: Weilin Wang <weilin.wang@intel.com> --- .../tests/shell/lib/perf_metric_validation.py | 31 ++++++++++++++++--- .../lib/perf_metric_validation_rules.json | 11 +++++++ 2 files changed, 38 insertions(+), 4 deletions(-) diff --git a/tools/perf/tests/shell/lib/perf_metric_validation.py b/tools/perf/tests/shell/lib/perf_metric_validation.py index 81bd2bf38b67..3c3a9b4f8b82 100644 --- a/tools/perf/tests/shell/lib/perf_metric_validation.py +++ b/tools/perf/tests/shell/lib/perf_metric_validation.py @@ -12,7 +12,7 @@ class Validator: self.reportfname = reportfname self.rules = None self.collectlist=metrics - self.metrics = set() + self.metrics = set(metrics) self.tolerance = t self.workloads = [x for x in workload.split(",") if x] @@ -148,6 +148,7 @@ class Validator: self.errlist.append("Metric '%s' is not collected"%(name)) elif val < 0: negmetric.add("{0}(={1:.4f})".format(name, val)) + self.collectlist[0].append(name) else: pcnt += 1 tcnt += 1 @@ -266,6 +267,7 @@ class Validator: passcnt += 1 else: faillist.append({'MetricName':m['Name'], 'CollectedValue':result}) + self.collectlist[0].append(m['Name']) self.totalcnt += totalcnt self.passedcnt += passcnt @@ -348,7 +350,7 @@ class Validator: if rule["TestType"] == "RelationshipTest": metrics = [m["Name"] for m in rule["Metrics"]] if not any(m not in collectlist[0] for m in metrics): - collectlist[rule["RuleIndex"]] = set(metrics) + collectlist[rule["RuleIndex"]] = [",".join(list(set(metrics)))] for idx, metrics in collectlist.items(): if idx == 0: wl = "sleep 0.5".split() @@ -356,9 +358,12 @@ class Validator: for metric in metrics: command = [tool, 'stat', '-j', '-M', f"{metric}", "-a"] command.extend(wl) + print(" ".join(command)) cmd = subprocess.run(command, stderr=subprocess.PIPE, encoding='utf-8') data = [x+'}' for x in cmd.stderr.split('}\n') if x] self.convert(data, idx) + self.collectlist = dict() + self.collectlist[0] = list() # End of Collector and Converter # Start of Rule Generator @@ -386,6 +391,20 @@ class Validator: return + def remove_unsupported_rules(self, rules, skiplist: set = None): + for m in skiplist: + self.metrics.discard(m) + new_rules = [] + for rule in rules: + add_rule = True + for m in rule["Metrics"]: + if m["Name"] not in self.metrics: + add_rule = False + break + if add_rule: + new_rules.append(rule) + return new_rules + def create_rules(self): """ Create full rules which includes: @@ -394,7 +413,10 @@ class Validator: Reindex all the rules to avoid repeated RuleIndex """ - self.rules = self.read_json(self.rulefname)['RelationshipRules'] + data = self.read_json(self.rulefname) + rules = data['RelationshipRules'] + skiplist = set(data['SkipList']) + self.rules = self.remove_unsupported_rules(rules, skiplist) pctgrule = {'RuleIndex':0, 'TestType':'SingleMetricTest', 'RangeLower':'0', @@ -453,7 +475,8 @@ class Validator: The final report is written into a JSON file. ''' - self.parse_perf_metrics() + if not self.collectlist: + self.parse_perf_metrics() self.create_rules() for i in range(0, len(self.workloads)): self._init_data() diff --git a/tools/perf/tests/shell/lib/perf_metric_validation_rules.json b/tools/perf/tests/shell/lib/perf_metric_validation_rules.json index debaa910da9f..eb6f59e018b7 100644 --- a/tools/perf/tests/shell/lib/perf_metric_validation_rules.json +++ b/tools/perf/tests/shell/lib/perf_metric_validation_rules.json @@ -1,4 +1,15 @@ { + "SkipList": [ + "tsx_aborted_cycles", + "tsx_transactional_cycles", + "C2_Pkg_Residency", + "C6_Pkg_Residency", + "C1_Core_Residency", + "C6_Core_Residency", + "tma_false_sharing", + "tma_remote_cache", + "tma_contested_accesses" + ], "RelationshipRules": [ { "RuleIndex": 1, -- 2.39.1 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v5 3/3] perf test: Rerun failed metrics with longer workload 2023-06-20 17:00 [PATCH v5 0/3] Add metric value validation test Weilin Wang 2023-06-20 17:00 ` [PATCH v5 1/3] perf test: " Weilin Wang 2023-06-20 17:00 ` [PATCH v5 2/3] perf test: Add skip list for metrics known would fail Weilin Wang @ 2023-06-20 17:00 ` Weilin Wang 2023-06-22 0:08 ` [PATCH v5 0/3] Add metric value validation test Namhyung Kim 3 siblings, 0 replies; 9+ messages in thread From: Weilin Wang @ 2023-06-20 17:00 UTC (permalink / raw) To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Jiri Olsa, Namhyung Kim, Adrian Hunter, Ian Rogers, linux-perf-users, linux-kernel Cc: Weilin Wang, Kan Liang, Samantha Alt, Perry Taylor, Caleb Biggers, ravi.bangoria Rerun failed metrics with longer workload to avoid false failure because sometimes metric value test fails when running in very short amount of time. Skip rerun if equal to or more than 20 metrics fail. Signed-off-by: Weilin Wang <weilin.wang@intel.com> --- .../tests/shell/lib/perf_metric_validation.py | 129 +++++++++++------- 1 file changed, 83 insertions(+), 46 deletions(-) diff --git a/tools/perf/tests/shell/lib/perf_metric_validation.py b/tools/perf/tests/shell/lib/perf_metric_validation.py index 3c3a9b4f8b82..50a34a9cc040 100644 --- a/tools/perf/tests/shell/lib/perf_metric_validation.py +++ b/tools/perf/tests/shell/lib/perf_metric_validation.py @@ -11,8 +11,9 @@ class Validator: self.rulefname = rulefname self.reportfname = reportfname self.rules = None - self.collectlist=metrics - self.metrics = set(metrics) + self.collectlist:str = metrics + self.metrics = self.__set_metrics(metrics) + self.skiplist = set() self.tolerance = t self.workloads = [x for x in workload.split(",") if x] @@ -41,6 +42,12 @@ class Validator: self.debug = debug self.fullrulefname = fullrulefname + def __set_metrics(self, metrics=''): + if metrics != '': + return set(metrics.split(",")) + else: + return set() + def read_json(self, filename: str) -> dict: try: with open(Path(filename).resolve(), "r") as f: @@ -113,7 +120,7 @@ class Validator: All future test(s) on this metric will fail. @param name: name of the metric - @returns: list with value found in self.results; list is empty when not value found. + @returns: list with value found in self.results; list is empty when value is not found. """ results = [] data = self.results[ridx] if ridx in self.results else self.results[0] @@ -123,7 +130,6 @@ class Validator: elif name.replace('.', '1').isdigit(): results.append(float(name)) else: - self.errlist.append("Metric '%s' is not collected or the value format is incorrect"%(name)) self.ignoremetrics.add(name) return results @@ -138,27 +144,32 @@ class Validator: Failure: when metric value is negative or not provided. Metrics with negative value will be added into the self.failtests['PositiveValueTest'] and self.ignoremetrics. """ - negmetric = set() - missmetric = set() + negmetric = dict() pcnt = 0 tcnt = 0 + rerun = list() for name, val in self.get_results().items(): - if val is None or val == '': - missmetric.add(name) - self.errlist.append("Metric '%s' is not collected"%(name)) - elif val < 0: - negmetric.add("{0}(={1:.4f})".format(name, val)) - self.collectlist[0].append(name) + if val < 0: + negmetric[name] = val + rerun.append(name) else: pcnt += 1 tcnt += 1 + if len(rerun) > 0 and len(rerun) < 20: + second_results = dict() + self.second_test(rerun, second_results) + for name, val in second_results.items(): + if name not in negmetric: continue + if val >= 0: + del negmetric[name] + pcnt += 1 self.failtests['PositiveValueTest']['Total Tests'] = tcnt self.failtests['PositiveValueTest']['Passed Tests'] = pcnt - if len(negmetric) or len(missmetric)> 0: - self.ignoremetrics.update(negmetric) - self.ignoremetrics.update(missmetric) - self.failtests['PositiveValueTest']['Failed Tests'].append({'NegativeValue':list(negmetric), 'MissingValue':list(missmetric)}) + if len(negmetric.keys()): + self.ignoremetrics.update(negmetric.keys()) + negmessage = ["{0}(={1:.4f})".format(name, val) for name, val in negmetric.items()] + self.failtests['PositiveValueTest']['Failed Tests'].append({'NegativeValue': negmessage}) return @@ -259,21 +270,36 @@ class Validator: metrics = rule['Metrics'] passcnt = 0 totalcnt = 0 - faillist = [] + faillist = list() + failures = dict() + rerun = list() for m in metrics: totalcnt += 1 result = self.get_value(m['Name']) - if len(result) > 0 and self.check_bound(result[0], lbv, ubv, t): + if len(result) > 0 and self.check_bound(result[0], lbv, ubv, t) or m['Name'] in self.skiplist: passcnt += 1 else: - faillist.append({'MetricName':m['Name'], 'CollectedValue':result}) - self.collectlist[0].append(m['Name']) + failures[m['Name']] = result + rerun.append(m['Name']) + + if len(rerun) > 0 and len(rerun) < 20: + second_results = dict() + self.second_test(rerun, second_results) + for name, val in second_results.items(): + if name not in failures: continue + if self.check_bound(val, lbv, ubv, t): + passcnt += 1 + del failures[name] + else: + failures[name] = val + self.results[0][name] = val self.totalcnt += totalcnt self.passedcnt += passcnt self.failtests['SingleMetricTest']['Total Tests'] += totalcnt self.failtests['SingleMetricTest']['Passed Tests'] += passcnt - if len(faillist) != 0: + if len(failures.keys()) != 0: + faillist = [{'MetricName':name, 'CollectedValue':val} for name, val in failures.items()] self.failtests['SingleMetricTest']['Failed Tests'].append({'RuleIndex':rule['RuleIndex'], 'RangeLower': rule['RangeLower'], 'RangeUpper': rule['RangeUpper'], @@ -316,7 +342,7 @@ class Validator: return True # Start of Collector and Converter - def convert(self, data: list, idx: int): + def convert(self, data: list, metricvalues:dict): """ Convert collected metric data from the -j output to dict of {metric_name:value}. """ @@ -326,20 +352,29 @@ class Validator: if "metric-unit" in result and result["metric-unit"] != "(null)" and result["metric-unit"] != "": name = result["metric-unit"].split(" ")[1] if len(result["metric-unit"].split(" ")) > 1 \ else result["metric-unit"] - if idx not in self.results: self.results[idx] = dict() - self.results[idx][name.lower()] = float(result["metric-value"]) + metricvalues[name.lower()] = float(result["metric-value"]) except ValueError as error: continue return - def collect_perf(self, data_file: str, workload: str): + def _run_perf(self, metric, workload: str): + tool = 'perf' + command = [tool, 'stat', '-j', '-M', f"{metric}", "-a"] + wl = workload.split() + command.extend(wl) + print(" ".join(command)) + cmd = subprocess.run(command, stderr=subprocess.PIPE, encoding='utf-8') + data = [x+'}' for x in cmd.stderr.split('}\n') if x] + return data + + + def collect_perf(self, workload: str): """ Collect metric data with "perf stat -M" on given workload with -a and -j. """ self.results = dict() - tool = 'perf' print(f"Starting perf collection") - print(f"Workload: {workload}") + print(f"Long workload: {workload}") collectlist = dict() if self.collectlist != "": collectlist[0] = {x for x in self.collectlist.split(",")} @@ -353,17 +388,20 @@ class Validator: collectlist[rule["RuleIndex"]] = [",".join(list(set(metrics)))] for idx, metrics in collectlist.items(): - if idx == 0: wl = "sleep 0.5".split() - else: wl = workload.split() + if idx == 0: wl = "true" + else: wl = workload for metric in metrics: - command = [tool, 'stat', '-j', '-M', f"{metric}", "-a"] - command.extend(wl) - print(" ".join(command)) - cmd = subprocess.run(command, stderr=subprocess.PIPE, encoding='utf-8') - data = [x+'}' for x in cmd.stderr.split('}\n') if x] - self.convert(data, idx) - self.collectlist = dict() - self.collectlist[0] = list() + data = self._run_perf(metric, wl) + if idx not in self.results: self.results[idx] = dict() + self.convert(data, self.results[idx]) + return + + def second_test(self, collectlist, second_results): + workload = self.workloads[self.wlidx] + for metric in collectlist: + data = self._run_perf(metric, workload) + self.convert(data, second_results) + # End of Collector and Converter # Start of Rule Generator @@ -381,7 +419,7 @@ class Validator: if 'MetricName' not in m: print("Warning: no metric name") continue - name = m['MetricName'] + name = m['MetricName'].lower() self.metrics.add(name) if 'ScaleUnit' in m and (m['ScaleUnit'] == '1%' or m['ScaleUnit'] == '100%'): self.pctgmetrics.add(name.lower()) @@ -391,14 +429,12 @@ class Validator: return - def remove_unsupported_rules(self, rules, skiplist: set = None): - for m in skiplist: - self.metrics.discard(m) + def remove_unsupported_rules(self, rules): new_rules = [] for rule in rules: add_rule = True for m in rule["Metrics"]: - if m["Name"] not in self.metrics: + if m["Name"] in self.skiplist or m["Name"] not in self.metrics: add_rule = False break if add_rule: @@ -415,15 +451,15 @@ class Validator: """ data = self.read_json(self.rulefname) rules = data['RelationshipRules'] - skiplist = set(data['SkipList']) - self.rules = self.remove_unsupported_rules(rules, skiplist) + self.skiplist = set([name.lower() for name in data['SkipList']]) + self.rules = self.remove_unsupported_rules(rules) pctgrule = {'RuleIndex':0, 'TestType':'SingleMetricTest', 'RangeLower':'0', 'RangeUpper': '100', 'ErrorThreshold': self.tolerance, 'Description':'Metrics in percent unit have value with in [0, 100]', - 'Metrics': [{'Name': m} for m in self.pctgmetrics]} + 'Metrics': [{'Name': m.lower()} for m in self.pctgmetrics]} self.rules.append(pctgrule) # Re-index all rules to avoid repeated RuleIndex @@ -479,8 +515,9 @@ class Validator: self.parse_perf_metrics() self.create_rules() for i in range(0, len(self.workloads)): + self.wlidx = i self._init_data() - self.collect_perf(self.datafname, self.workloads[i]) + self.collect_perf(self.workloads[i]) # Run positive value test self.pos_val_test() for r in self.rules: -- 2.39.1 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH v5 0/3] Add metric value validation test 2023-06-20 17:00 [PATCH v5 0/3] Add metric value validation test Weilin Wang ` (2 preceding siblings ...) 2023-06-20 17:00 ` [PATCH v5 3/3] perf test: Rerun failed metrics with longer workload Weilin Wang @ 2023-06-22 0:08 ` Namhyung Kim 2023-06-22 21:35 ` Namhyung Kim 3 siblings, 1 reply; 9+ messages in thread From: Namhyung Kim @ 2023-06-22 0:08 UTC (permalink / raw) To: Weilin Wang Cc: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Jiri Olsa, Adrian Hunter, Ian Rogers, linux-perf-users, linux-kernel, Kan Liang, Samantha Alt, Perry Taylor, Caleb Biggers, ravi.bangoria Hello, On Tue, Jun 20, 2023 at 10:00 AM Weilin Wang <weilin.wang@intel.com> wrote: > > This is the fifth version of metric value validation tests. > > We made the following changes from v4 to v5: > - Update "()" to "{}" to avoid creating sub shell and successfully skip test on non-Intel > platform. [Ravi] > > v4: https://lore.kernel.org/lkml/20230618172820.751560-1-weilin.wang@intel.com/ > > Weilin Wang (3): > perf test: Add metric value validation test > perf test: Add skip list for metrics known would fail > perf test: Rerun failed metrics with longer workload Tested-by: Namhyung Kim <namhyung@kernel.org> Thanks, Namhyung $ ./perf test -v validation 107: perf metrics value validation : --- start --- test child forked, pid 1900992 Launch python validation script ./tests/shell/lib/perf_metric_validation.py Output will be stored in: /tmp/__perf_test.program.Mm9Rw Starting perf collection ... Workload: perf bench futex hash -r 2 -s Total metrics collected: 200 Non-negative metric count: 200 Total Test Count: 100 Passed Test Count: 100 Test validation finished. Final report: [ { "Workload": "perf bench futex hash -r 2 -s", "Report": { "Metric Validation Statistics": { "Total Rule Count": 100, "Passed Rule Count": 100 }, "Tests in Category": { "PositiveValueTest": { "Total Tests": 200, "Passed Tests": 200, "Failed Tests": [] }, "RelationshipTest": { "Total Tests": 5, "Passed Tests": 5, "Failed Tests": [] }, "SingleMetricTest": { "Total Tests": 95, "Passed Tests": 95, "Failed Tests": [] } }, "Errors": [] } } ] test child finished with 0 ---- end ---- perf metrics value validation: Ok ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v5 0/3] Add metric value validation test 2023-06-22 0:08 ` [PATCH v5 0/3] Add metric value validation test Namhyung Kim @ 2023-06-22 21:35 ` Namhyung Kim 2023-09-05 17:30 ` Namhyung Kim 0 siblings, 1 reply; 9+ messages in thread From: Namhyung Kim @ 2023-06-22 21:35 UTC (permalink / raw) To: Weilin Wang Cc: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Jiri Olsa, Adrian Hunter, Ian Rogers, linux-perf-users, linux-kernel, Kan Liang, Samantha Alt, Perry Taylor, Caleb Biggers, ravi.bangoria On Wed, Jun 21, 2023 at 5:08 PM Namhyung Kim <namhyung@kernel.org> wrote: > > Hello, > > On Tue, Jun 20, 2023 at 10:00 AM Weilin Wang <weilin.wang@intel.com> wrote: > > > > This is the fifth version of metric value validation tests. > > > > We made the following changes from v4 to v5: > > - Update "()" to "{}" to avoid creating sub shell and successfully skip test on non-Intel > > platform. [Ravi] > > > > v4: https://lore.kernel.org/lkml/20230618172820.751560-1-weilin.wang@intel.com/ > > > > Weilin Wang (3): > > perf test: Add metric value validation test > > perf test: Add skip list for metrics known would fail > > perf test: Rerun failed metrics with longer workload > > Tested-by: Namhyung Kim <namhyung@kernel.org> Applied to perf-tools-next, thanks! ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v5 0/3] Add metric value validation test 2023-06-22 21:35 ` Namhyung Kim @ 2023-09-05 17:30 ` Namhyung Kim 2023-09-07 4:58 ` Wang, Weilin 0 siblings, 1 reply; 9+ messages in thread From: Namhyung Kim @ 2023-09-05 17:30 UTC (permalink / raw) To: Weilin Wang Cc: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Jiri Olsa, Adrian Hunter, Ian Rogers, linux-perf-users, linux-kernel, Kan Liang, Samantha Alt, Perry Taylor, Caleb Biggers, ravi.bangoria Hello Weilin, Now I'm seeing occasional failures in this test. Can you please take a look? Thanks, Namhyung --8<-- $ sudo ./perf test -v 105 105: perf metrics value validation : --- start --- test child forked, pid 78217 Launch python validation script ./tests/shell/lib/perf_metric_validation.py Output will be stored in: /tmp/__perf_test.program.ymbCm Starting perf collection Long workload: perf bench futex hash -r 2 -s perf stat -j -M tma_branch_mispredicts -a true perf stat -j -M tma_info_memory_core_l2_cache_fill_bw -a true perf stat -j -M tma_retiring -a true perf stat -j -M tma_info_bad_spec_branch_misprediction_cost -a true perf stat -j -M tma_port_0 -a true perf stat -j -M tma_itlb_misses -a true perf stat -j -M tma_other_light_ops -a true perf stat -j -M smi_cycles -a true perf stat -j -M c7_core_residency -a true perf stat -j -M tma_split_stores -a true perf stat -j -M tma_info_frontend_dsb_coverage -a true perf stat -j -M tma_info_frontend_icache_miss_latency -a true perf stat -j -M tma_store_stlb_hit -a true perf stat -j -M tma_dram_bound -a true perf stat -j -M tma_info_system_dram_bw_use -a true perf stat -j -M tma_info_bad_spec_ipmisp_cond_taken -a true perf stat -j -M tma_info_botlnk_l0_core_bound_likely -a true perf stat -j -M tma_machine_clears -a true perf stat -j -M tma_info_system_kernel_cpi -a true perf stat -j -M tma_decoder0_alone -a true perf stat -j -M tma_info_system_cpu_utilization -a true perf stat -j -M tma_clears_resteers -a true perf stat -j -M tma_fp_vector_512b -a true perf stat -j -M tma_dtlb_store -a true perf stat -j -M tma_info_branches_cond_nt -a true perf stat -j -M tma_info_memory_core_l3_cache_access_bw -a true perf stat -j -M tma_info_memory_oro_load_l2_mlp -a true perf stat -j -M c8_pkg_residency -a true perf stat -j -M tma_mem_bandwidth -a true perf stat -j -M tma_fp_vector_256b -a true perf stat -j -M tma_branch_resteers -a true perf stat -j -M tma_false_sharing -a true perf stat -j -M tma_info_core_ilp -a true perf stat -j -M tma_bad_speculation -a true perf stat -j -M tma_info_system_power_license1_utilization -a true perf stat -j -M tma_load_stlb_miss -a true perf stat -j -M tma_info_system_average_frequency -a true perf stat -j -M tma_info_frontend_fetch_upc -a true perf stat -j -M tma_info_branches_cond_tk -a true perf stat -j -M tma_info_bad_spec_ipmispredict -a true perf stat -j -M tma_info_branches_callret -a true perf stat -j -M tma_info_inst_mix_iparith_avx512 -a true perf stat -j -M tma_info_frontend_l2mpki_code_all -a true perf stat -j -M tma_divider -a true perf stat -j -M tma_info_system_mem_parallel_reads -a true perf stat -j -M tma_info_branches_jump -a true perf stat -j -M tma_mixing_vectors -a true perf stat -j -M tma_cisc -a true perf stat -j -M tma_info_thread_ipc -a true perf stat -j -M tma_lcp -a true perf stat -j -M tma_assists -a true perf stat -j -M tma_dtlb_load -a true perf stat -j -M tma_info_memory_load_miss_real_latency -a true perf stat -j -M tma_fetch_latency -a true perf stat -j -M tma_info_bad_spec_ipmisp_ret -a true perf stat -j -M tma_sq_full -a true perf stat -j -M tma_info_pipeline_retire -a true perf stat -j -M tma_info_memory_tlb_page_walks_utilization -a true perf stat -j -M tma_store_latency -a true perf stat -j -M tma_port_5 -a true perf stat -j -M tma_info_memory_thread_l2_cache_fill_bw_1t -a true perf stat -j -M tma_load_op_utilization -a true perf stat -j -M tma_info_frontend_lsd_coverage -a true perf stat -j -M tma_info_memory_thread_l3_cache_access_bw_1t -a true perf stat -j -M tma_l3_hit_latency -a true perf stat -j -M tma_info_system_kernel_utilization -a true perf stat -j -M tma_backend_bound -a true perf stat -j -M c7_pkg_residency -a true perf stat -j -M tma_info_memory_fb_hpki -a true perf stat -j -M tma_info_botlnk_l2_ic_misses -a true perf stat -j -M tma_info_memory_l3mpki -a true perf stat -j -M tma_l1_bound -a true perf stat -j -M c6_pkg_residency -a true perf stat -j -M tma_nop_instructions -a true perf stat -j -M tma_load_stlb_hit -a true perf stat -j -M tma_microcode_sequencer -a true perf stat -j -M tma_mem_latency -a true perf stat -j -M tma_data_sharing -a true perf stat -j -M tma_core_bound -a true perf stat -j -M tma_store_bound -a true perf stat -j -M tma_store_op_utilization -a true perf stat -j -M tsx_aborted_cycles -a true perf stat -j -M tma_mispredicts_resteers -a true perf stat -j -M tma_memory_operations -a true perf stat -j -M tma_info_memory_oro_data_l2_mlp -a true perf stat -j -M tma_4k_aliasing -a true perf stat -j -M c10_pkg_residency -a true perf stat -j -M tma_info_memory_mlp -a true perf stat -j -M tma_fp_vector -a true perf stat -j -M tma_info_bottleneck_big_code -a true perf stat -j -M tma_info_core_flopc -a true perf stat -j -M tma_memory_bound -a true perf stat -j -M tma_info_memory_core_l3_cache_fill_bw -a true perf stat -j -M tma_alu_op_utilization -a true perf stat -j -M c9_pkg_residency -a true perf stat -j -M tma_info_system_gflops -a true perf stat -j -M tma_split_loads -a true perf stat -j -M c2_pkg_residency -a true perf stat -j -M tsx_cycles_per_transaction -a true perf stat -j -M tma_branch_instructions -a true perf stat -j -M tma_ms_switches -a true perf stat -j -M c6_core_residency -a true perf stat -j -M tma_streaming_stores -a true perf stat -j -M tma_l3_bound -a true perf stat -j -M tma_info_memory_l2mpki_load -a true perf stat -j -M tma_slow_pause -a true perf stat -j -M tma_info_frontend_ipunknown_branch -a true perf stat -j -M tma_contested_accesses -a true perf stat -j -M tma_info_bottleneck_branching_overhead -a true perf stat -j -M tma_x87_use -a true perf stat -j -M tma_frontend_bound -a true perf stat -j -M tma_store_fwd_blk -a true perf stat -j -M tma_info_system_mem_request_latency -a true perf stat -j -M tma_info_bad_spec_ipmisp_indirect -a true perf stat -j -M tma_info_system_turbo_utilization -a true perf stat -j -M tma_heavy_operations -a true perf stat -j -M tma_light_operations -a true perf stat -j -M tma_ports_utilization -a true perf stat -j -M tma_info_memory_l2hpki_load -a true perf stat -j -M tma_dsb_switches -a true perf stat -j -M tma_info_memory_l1mpki -a true perf stat -j -M tma_info_memory_core_l1d_cache_fill_bw -a true perf stat -j -M tma_info_core_core_clks -a true perf stat -j -M tma_info_memory_tlb_code_stlb_mpki -a true perf stat -j -M tma_info_memory_tlb_load_stlb_mpki -a true perf stat -j -M tma_info_system_mem_read_latency -a true perf stat -j -M tma_ports_utilized_2 -a true perf stat -j -M tma_info_inst_mix_iparith -a true perf stat -j -M tma_ports_utilized_3m -a true perf stat -j -M tma_info_system_smt_2t_utilization -a true perf stat -j -M tma_info_thread_slots -a true perf stat -j -M tma_info_thread_clks -a true perf stat -j -M tma_info_core_coreipc -a true perf stat -j -M tma_info_memory_oro_load_l3_miss_latency -a true perf stat -j -M tma_info_inst_mix_iparith_scalar_sp -a true perf stat -j -M tma_lock_latency -a true perf stat -j -M tma_info_inst_mix_ipflop -a true perf stat -j -M tma_unknown_branches -a true perf stat -j -M tma_info_bottleneck_memory_latency -a true perf stat -j -M tma_serializing_operation -a true perf stat -j -M tma_lsd -a true perf stat -j -M tma_info_system_ipfarbranch -a true perf stat -j -M tma_info_inst_mix_iptb -a true perf stat -j -M tma_info_botlnk_l2_dsb_misses -a true perf stat -j -M tma_mite -a true perf stat -j -M tma_icache_misses -a true perf stat -j -M tma_info_memory_l1mpki_load -a true perf stat -j -M tma_fp_arith -a true perf stat -j -M tma_info_memory_l2hpki_all -a true perf stat -j -M tma_port_1 -a true perf stat -j -M tma_info_system_power_license0_utilization -a true perf stat -j -M tma_mite_4wide -a true perf stat -j -M tma_info_frontend_dsb_switch_cost -a true perf stat -j -M tma_info_thread_execute_per_issue -a true perf stat -j -M tma_info_bad_spec_ipmisp_cond_ntaken -a true perf stat -j -M tma_fp_scalar -a true perf stat -j -M tma_info_inst_mix_ipswpf -a true perf stat -j -M tma_info_inst_mix_bptkbranch -a true perf stat -j -M tma_info_memory_l2mpki_all -a true perf stat -j -M tma_info_inst_mix_instructions -a true perf stat -j -M tma_info_inst_mix_ipload -a true perf stat -j -M tma_info_thread_slots_utilization -a true perf stat -j -M tma_info_thread_cpi -a true perf stat -j -M tma_info_bottleneck_instruction_fetch_bw -a true perf stat -j -M tma_info_inst_mix_ipbranch -a true perf stat -j -M tma_ports_utilized_1 -a true perf stat -j -M tma_info_inst_mix_iparith_avx256 -a true perf stat -j -M tma_store_stlb_miss -a true perf stat -j -M tma_dsb -a true perf stat -j -M smi_num -a true perf stat -j -M tsx_transactional_cycles -a true perf stat -j -M tma_fp_vector_128b -a true perf stat -j -M tma_fetch_bandwidth -a true perf stat -j -M tma_fb_full -a true perf stat -j -M tma_info_bottleneck_mispredictions -a true perf stat -j -M tma_port_6 -a true perf stat -j -M tma_info_frontend_l2mpki_code -a true perf stat -j -M tma_info_core_fp_arith_utilization -a true perf stat -j -M tma_info_inst_mix_iparith_avx128 -a true perf stat -j -M tma_info_memory_l2mpki -a true perf stat -j -M tma_info_bottleneck_memory_bandwidth -a true perf stat -j -M tma_info_memory_thread_l1d_cache_fill_bw_1t -a true perf stat -j -M tma_info_memory_oro_load_l2_miss_latency -a true perf stat -j -M c3_pkg_residency -a true perf stat -j -M tma_few_uops_instructions -a true perf stat -j -M tma_info_pipeline_execute -a true perf stat -j -M tma_info_frontend_ipdsb_miss_ret -a true perf stat -j -M tma_info_memory_thread_l3_cache_fill_bw_1t -a true perf stat -j -M tma_info_thread_uptb -a true perf stat -j -M tma_info_inst_mix_ipcall -a true perf stat -j -M tma_l2_bound -a true perf stat -j -M tma_ports_utilized_0 -a true perf stat -j -M tma_info_bottleneck_memory_data_tlbs -a true perf stat -j -M tma_info_inst_mix_iparith_scalar_dp -a true perf stat -j -M tma_info_inst_mix_ipstore -a true perf stat -j -M tma_info_branches_other_branches -a true perf stat -j -M tma_info_thread_uoppi -a true perf stat -j -M tma_info_system_power_license2_utilization -a true perf stat -j -M tsx_cycles_per_elision -a true perf stat -j -M tma_info_memory_tlb_store_stlb_mpki -a true perf stat -j -M tma_backend_bound,tma_bad_speculation,tma_retiring,tma_frontend_bound -a perf bench futex hash -r 2 -s # Running 'futex/hash' benchmark: Run summary [PID 78627]: 8 threads, each operating on 1024 [private] futexes for 2 secs. Averaged 1625344 operations/sec (+- 0.78%), total secs = 2 perf stat -j -M tma_fetch_bandwidth,tma_frontend_bound,tma_fetch_latency -a perf bench futex hash -r 2 -s # Running 'futex/hash' benchmark: Run summary [PID 78637]: 8 threads, each operating on 1024 [private] futexes for 2 secs. Averaged 1622144 operations/sec (+- 0.25%), total secs = 2 perf stat -j -M tma_machine_clears,tma_branch_mispredicts,tma_bad_speculation -a perf bench futex hash -r 2 -s # Running 'futex/hash' benchmark: Run summary [PID 78647]: 8 threads, each operating on 1024 [private] futexes for 2 secs. Averaged 1503103 operations/sec (+- 1.58%), total secs = 2 perf stat -j -M tma_core_bound,tma_memory_bound,tma_backend_bound -a perf bench futex hash -r 2 -s # Running 'futex/hash' benchmark: Run summary [PID 78680]: 8 threads, each operating on 1024 [private] futexes for 2 secs. Averaged 1415935 operations/sec (+- 2.26%), total secs = 2 perf stat -j -M tma_heavy_operations,tma_light_operations,tma_retiring -a perf bench futex hash -r 2 -s # Running 'futex/hash' benchmark: Run summary [PID 78690]: 8 threads, each operating on 1024 [private] futexes for 2 secs. Averaged 1519872 operations/sec (+- 0.38%), total secs = 2 perf stat -j -M tma_other_light_ops -a perf bench futex hash -r 2 -s # Running 'futex/hash' benchmark: Run summary [PID 78700]: 8 threads, each operating on 1024 [private] futexes for 2 secs. Averaged 1490816 operations/sec (+- 0.21%), total secs = 2 perf stat -j -M tma_split_stores -a perf bench futex hash -r 2 -s # Running 'futex/hash' benchmark: Run summary [PID 78710]: 8 threads, each operating on 1024 [private] futexes for 2 secs. Averaged 1474816 operations/sec (+- 0.25%), total secs = 2 perf stat -j -M tma_store_stlb_hit -a perf bench futex hash -r 2 -s # Running 'futex/hash' benchmark: Run summary [PID 78721]: 8 threads, each operating on 1024 [private] futexes for 2 secs. Averaged 1460032 operations/sec (+- 0.35%), total secs = 2 perf stat -j -M tma_dram_bound -a perf bench futex hash -r 2 -s # Running 'futex/hash' benchmark: Run summary [PID 78732]: 8 threads, each operating on 1024 [private] futexes for 2 secs. Averaged 1421632 operations/sec (+- 0.56%), total secs = 2 perf stat -j -M tma_dtlb_store -a perf bench futex hash -r 2 -s # Running 'futex/hash' benchmark: Run summary [PID 78742]: 8 threads, each operating on 1024 [private] futexes for 2 secs. Averaged 1431424 operations/sec (+- 0.24%), total secs = 2 perf stat -j -M tma_mem_bandwidth -a perf bench futex hash -r 2 -s # Running 'futex/hash' benchmark: Run summary [PID 78752]: 8 threads, each operating on 1024 [private] futexes for 2 secs. Averaged 1344768 operations/sec (+- 1.51%), total secs = 2 perf stat -j -M tma_l3_hit_latency -a perf bench futex hash -r 2 -s # Running 'futex/hash' benchmark: Run summary [PID 78765]: 8 threads, each operating on 1024 [private] futexes for 2 secs. Averaged 1369728 operations/sec (+- 1.18%), total secs = 2 perf stat -j -M tma_mem_latency -a perf bench futex hash -r 2 -s # Running 'futex/hash' benchmark: Run summary [PID 78776]: 8 threads, each operating on 1024 [private] futexes for 2 secs. Averaged 1361536 operations/sec (+- 0.79%), total secs = 2 perf stat -j -M tma_data_sharing -a perf bench futex hash -r 2 -s # Running 'futex/hash' benchmark: Run summary [PID 78786]: 8 threads, each operating on 1024 [private] futexes for 2 secs. Averaged 1302656 operations/sec (+- 1.90%), total secs = 2 perf stat -j -M tma_memory_operations -a perf bench futex hash -r 2 -s # Running 'futex/hash' benchmark: Run summary [PID 78797]: 8 threads, each operating on 1024 [private] futexes for 2 secs. Averaged 1333184 operations/sec (+- 1.08%), total secs = 2 perf stat -j -M tma_4k_aliasing -a perf bench futex hash -r 2 -s # Running 'futex/hash' benchmark: Run summary [PID 78807]: 8 threads, each operating on 1024 [private] futexes for 2 secs. Averaged 1268352 operations/sec (+- 2.13%), total secs = 2 perf stat -j -M tma_l3_bound -a perf bench futex hash -r 2 -s # Running 'futex/hash' benchmark: Run summary [PID 78827]: 8 threads, each operating on 1024 [private] futexes for 2 secs. Averaged 1289408 operations/sec (+- 2.65%), total secs = 2 perf stat -j -M tma_store_fwd_blk -a perf bench futex hash -r 2 -s # Running 'futex/hash' benchmark: Run summary [PID 78847]: 8 threads, each operating on 1024 [private] futexes for 2 secs. Averaged 1176256 operations/sec (+- 1.57%), total secs = 2 perf stat -j -M tma_lock_latency -a perf bench futex hash -r 2 -s # Running 'futex/hash' benchmark: Run summary [PID 78857]: 8 threads, each operating on 1024 [private] futexes for 2 secs. Averaged 1050112 operations/sec (+- 2.42%), total secs = 2 perf stat -j -M tma_store_stlb_miss -a perf bench futex hash -r 2 -s # Running 'futex/hash' benchmark: Run summary [PID 78867]: 8 threads, each operating on 1024 [private] futexes for 2 secs. Averaged 1117248 operations/sec (+- 1.96%), total secs = 2 perf stat -j -M tma_l2_bound -a perf bench futex hash -r 2 -s # Running 'futex/hash' benchmark: Run summary [PID 78877]: 8 threads, each operating on 1024 [private] futexes for 2 secs. Averaged 1174847 operations/sec (+- 0.74%), total secs = 2 Workload: perf bench futex hash -r 2 -s Total metrics collected: 197 Non-negative metric count: 197 Total Test Count: 100 Passed Test Count: 96 Test validation finished. Final report: [ { "Workload": "perf bench futex hash -r 2 -s", "Report": { "Metric Validation Statistics": { "Total Rule Count": 100, "Passed Rule Count": 96 }, "Tests in Category": { "PositiveValueTest": { "Total Tests": 197, "Passed Tests": 197, "Failed Tests": [] }, "RelationshipTest": { "Total Tests": 5, "Passed Tests": 5, "Failed Tests": [] }, "SingleMetricTest": { "Total Tests": 95, "Passed Tests": 91, "Failed Tests": [ { "RuleIndex": 6, "RangeLower": "0", "RangeUpper": "100", "ErrorThreshold": 5, "Failure": [ { "MetricName": "tma_split_stores", "CollectedValue": [] }, { "MetricName": "tma_4k_aliasing", "CollectedValue": [] }, { "MetricName": "tma_l3_bound", "CollectedValue": [] }, { "MetricName": "tma_store_fwd_blk", "CollectedValue": [] } ] } ] } }, "Errors": [] } } ] test child finished with -1 ---- end ---- perf metrics value validation: FAILED! On Thu, Jun 22, 2023 at 2:35 PM Namhyung Kim <namhyung@kernel.org> wrote: > > On Wed, Jun 21, 2023 at 5:08 PM Namhyung Kim <namhyung@kernel.org> wrote: > > > > Hello, > > > > On Tue, Jun 20, 2023 at 10:00 AM Weilin Wang <weilin.wang@intel.com> wrote: > > > > > > This is the fifth version of metric value validation tests. > > > > > > We made the following changes from v4 to v5: > > > - Update "()" to "{}" to avoid creating sub shell and successfully skip test on non-Intel > > > platform. [Ravi] > > > > > > v4: https://lore.kernel.org/lkml/20230618172820.751560-1-weilin.wang@intel.com/ > > > > > > Weilin Wang (3): > > > perf test: Add metric value validation test > > > perf test: Add skip list for metrics known would fail > > > perf test: Rerun failed metrics with longer workload > > > > Tested-by: Namhyung Kim <namhyung@kernel.org> > > Applied to perf-tools-next, thanks! ^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: [PATCH v5 0/3] Add metric value validation test 2023-09-05 17:30 ` Namhyung Kim @ 2023-09-07 4:58 ` Wang, Weilin 2023-09-07 16:33 ` Namhyung Kim 0 siblings, 1 reply; 9+ messages in thread From: Wang, Weilin @ 2023-09-07 4:58 UTC (permalink / raw) To: Namhyung Kim Cc: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Jiri Olsa, Hunter, Adrian, Ian Rogers, linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org, Kan Liang, Alt, Samantha, Taylor, Perry, Biggers, Caleb, ravi.bangoria@amd.com Hi Namhyung, > -----Original Message----- > From: Namhyung Kim <namhyung@kernel.org> > Sent: Tuesday, September 5, 2023 10:31 AM > To: Wang, Weilin <weilin.wang@intel.com> > Cc: Peter Zijlstra <peterz@infradead.org>; Ingo Molnar <mingo@redhat.com>; > Arnaldo Carvalho de Melo <acme@kernel.org>; Jiri Olsa <jolsa@kernel.org>; > Hunter, Adrian <adrian.hunter@intel.com>; Ian Rogers > <irogers@google.com>; linux-perf-users@vger.kernel.org; linux- > kernel@vger.kernel.org; Kan Liang <kan.liang@linux.intel.com>; Alt, Samantha > <samantha.alt@intel.com>; Taylor, Perry <perry.taylor@intel.com>; Biggers, > Caleb <caleb.biggers@intel.com>; ravi.bangoria@amd.com > Subject: Re: [PATCH v5 0/3] Add metric value validation test > > Hello Weilin, > > Now I'm seeing occasional failures in this test. Can you please take a look? > > Thanks, > Namhyung > > > --8<-- > > $ sudo ./perf test -v 105 > 105: perf metrics value validation : > --- start --- > test child forked, pid 78217 > Launch python validation script ./tests/shell/lib/perf_metric_validation.py > Output will be stored in: /tmp/__perf_test.program.ymbCm Starting perf > collection Long workload: perf bench futex hash -r 2 -s perf stat -j -M > tma_branch_mispredicts -a true perf stat -j -M > tma_info_memory_core_l2_cache_fill_bw -a true perf stat -j -M tma_retiring - > a true perf stat -j -M tma_info_bad_spec_branch_misprediction_cost -a true > perf stat -j -M tma_port_0 -a true perf stat -j -M tma_itlb_misses -a true perf > stat -j -M tma_other_light_ops -a true perf stat -j -M smi_cycles -a true perf > stat -j -M c7_core_residency -a true perf stat -j -M tma_split_stores -a true > perf stat -j -M tma_info_frontend_dsb_coverage -a true perf stat -j -M > tma_info_frontend_icache_miss_latency -a true perf stat -j -M > tma_store_stlb_hit -a true perf stat -j -M tma_dram_bound -a true perf stat -j > -M tma_info_system_dram_bw_use -a true perf stat -j -M > tma_info_bad_spec_ipmisp_cond_taken -a true perf stat -j -M > tma_info_botlnk_l0_core_bound_likely -a true perf stat -j -M > tma_machine_clears -a true perf stat -j -M tma_info_system_kernel_cpi -a true > perf stat -j -M tma_decoder0_alone -a true perf stat -j -M > tma_info_system_cpu_utilization -a true perf stat -j -M tma_clears_resteers -a > true perf stat -j -M tma_fp_vector_512b -a true perf stat -j -M tma_dtlb_store > -a true perf stat -j -M tma_info_branches_cond_nt -a true perf stat -j -M > tma_info_memory_core_l3_cache_access_bw -a true perf stat -j -M > tma_info_memory_oro_load_l2_mlp -a true perf stat -j -M c8_pkg_residency - > a true perf stat -j -M tma_mem_bandwidth -a true perf stat -j -M > tma_fp_vector_256b -a true perf stat -j -M tma_branch_resteers -a true perf > stat -j -M tma_false_sharing -a true perf stat -j -M tma_info_core_ilp -a true > perf stat -j -M tma_bad_speculation -a true perf stat -j -M > tma_info_system_power_license1_utilization -a true perf stat -j -M > tma_load_stlb_miss -a true perf stat -j -M > tma_info_system_average_frequency -a true perf stat -j -M > tma_info_frontend_fetch_upc -a true perf stat -j -M > tma_info_branches_cond_tk -a true perf stat -j -M > tma_info_bad_spec_ipmispredict -a true perf stat -j -M > tma_info_branches_callret -a true perf stat -j -M > tma_info_inst_mix_iparith_avx512 -a true perf stat -j -M > tma_info_frontend_l2mpki_code_all -a true perf stat -j -M tma_divider -a true > perf stat -j -M tma_info_system_mem_parallel_reads -a true perf stat -j -M > tma_info_branches_jump -a true perf stat -j -M tma_mixing_vectors -a true > perf stat -j -M tma_cisc -a true perf stat -j -M tma_info_thread_ipc -a true perf > stat -j -M tma_lcp -a true perf stat -j -M tma_assists -a true perf stat -j -M > tma_dtlb_load -a true perf stat -j -M > tma_info_memory_load_miss_real_latency -a true perf stat -j -M > tma_fetch_latency -a true perf stat -j -M tma_info_bad_spec_ipmisp_ret -a > true perf stat -j -M tma_sq_full -a true perf stat -j -M tma_info_pipeline_retire > -a true perf stat -j -M tma_info_memory_tlb_page_walks_utilization -a true > perf stat -j -M tma_store_latency -a true perf stat -j -M tma_port_5 -a true > perf stat -j -M tma_info_memory_thread_l2_cache_fill_bw_1t -a true perf stat > -j -M tma_load_op_utilization -a true perf stat -j -M > tma_info_frontend_lsd_coverage -a true perf stat -j -M > tma_info_memory_thread_l3_cache_access_bw_1t -a true perf stat -j -M > tma_l3_hit_latency -a true perf stat -j -M tma_info_system_kernel_utilization - > a true perf stat -j -M tma_backend_bound -a true perf stat -j -M > c7_pkg_residency -a true perf stat -j -M tma_info_memory_fb_hpki -a true > perf stat -j -M tma_info_botlnk_l2_ic_misses -a true perf stat -j -M > tma_info_memory_l3mpki -a true perf stat -j -M tma_l1_bound -a true perf > stat -j -M c6_pkg_residency -a true perf stat -j -M tma_nop_instructions -a > true perf stat -j -M tma_load_stlb_hit -a true perf stat -j -M > tma_microcode_sequencer -a true perf stat -j -M tma_mem_latency -a true > perf stat -j -M tma_data_sharing -a true perf stat -j -M tma_core_bound -a > true perf stat -j -M tma_store_bound -a true perf stat -j -M > tma_store_op_utilization -a true perf stat -j -M tsx_aborted_cycles -a true perf > stat -j -M tma_mispredicts_resteers -a true perf stat -j -M > tma_memory_operations -a true perf stat -j -M > tma_info_memory_oro_data_l2_mlp -a true perf stat -j -M tma_4k_aliasing -a > true perf stat -j -M c10_pkg_residency -a true perf stat -j -M > tma_info_memory_mlp -a true perf stat -j -M tma_fp_vector -a true perf stat - > j -M tma_info_bottleneck_big_code -a true perf stat -j -M > tma_info_core_flopc -a true perf stat -j -M tma_memory_bound -a true perf > stat -j -M tma_info_memory_core_l3_cache_fill_bw -a true perf stat -j -M > tma_alu_op_utilization -a true perf stat -j -M c9_pkg_residency -a true perf > stat -j -M tma_info_system_gflops -a true perf stat -j -M tma_split_loads -a > true perf stat -j -M c2_pkg_residency -a true perf stat -j -M > tsx_cycles_per_transaction -a true perf stat -j -M tma_branch_instructions -a > true perf stat -j -M tma_ms_switches -a true perf stat -j -M c6_core_residency > -a true perf stat -j -M tma_streaming_stores -a true perf stat -j -M > tma_l3_bound -a true perf stat -j -M tma_info_memory_l2mpki_load -a true > perf stat -j -M tma_slow_pause -a true perf stat -j -M > tma_info_frontend_ipunknown_branch -a true perf stat -j -M > tma_contested_accesses -a true perf stat -j -M > tma_info_bottleneck_branching_overhead -a true perf stat -j -M tma_x87_use > -a true perf stat -j -M tma_frontend_bound -a true perf stat -j -M > tma_store_fwd_blk -a true perf stat -j -M > tma_info_system_mem_request_latency -a true perf stat -j -M > tma_info_bad_spec_ipmisp_indirect -a true perf stat -j -M > tma_info_system_turbo_utilization -a true perf stat -j -M > tma_heavy_operations -a true perf stat -j -M tma_light_operations -a true > perf stat -j -M tma_ports_utilization -a true perf stat -j -M > tma_info_memory_l2hpki_load -a true perf stat -j -M tma_dsb_switches -a > true perf stat -j -M tma_info_memory_l1mpki -a true perf stat -j -M > tma_info_memory_core_l1d_cache_fill_bw -a true perf stat -j -M > tma_info_core_core_clks -a true perf stat -j -M > tma_info_memory_tlb_code_stlb_mpki -a true perf stat -j -M > tma_info_memory_tlb_load_stlb_mpki -a true perf stat -j -M > tma_info_system_mem_read_latency -a true perf stat -j -M > tma_ports_utilized_2 -a true perf stat -j -M tma_info_inst_mix_iparith -a true > perf stat -j -M tma_ports_utilized_3m -a true perf stat -j -M > tma_info_system_smt_2t_utilization -a true perf stat -j -M > tma_info_thread_slots -a true perf stat -j -M tma_info_thread_clks -a true perf > stat -j -M tma_info_core_coreipc -a true perf stat -j -M > tma_info_memory_oro_load_l3_miss_latency -a true perf stat -j -M > tma_info_inst_mix_iparith_scalar_sp -a true perf stat -j -M tma_lock_latency - > a true perf stat -j -M tma_info_inst_mix_ipflop -a true perf stat -j -M > tma_unknown_branches -a true perf stat -j -M > tma_info_bottleneck_memory_latency -a true perf stat -j -M > tma_serializing_operation -a true perf stat -j -M tma_lsd -a true perf stat -j -M > tma_info_system_ipfarbranch -a true perf stat -j -M tma_info_inst_mix_iptb -a > true perf stat -j -M tma_info_botlnk_l2_dsb_misses -a true perf stat -j -M > tma_mite -a true perf stat -j -M tma_icache_misses -a true perf stat -j -M > tma_info_memory_l1mpki_load -a true perf stat -j -M tma_fp_arith -a true > perf stat -j -M tma_info_memory_l2hpki_all -a true perf stat -j -M tma_port_1 > -a true perf stat -j -M tma_info_system_power_license0_utilization -a true > perf stat -j -M tma_mite_4wide -a true perf stat -j -M > tma_info_frontend_dsb_switch_cost -a true perf stat -j -M > tma_info_thread_execute_per_issue -a true perf stat -j -M > tma_info_bad_spec_ipmisp_cond_ntaken -a true perf stat -j -M tma_fp_scalar > -a true perf stat -j -M tma_info_inst_mix_ipswpf -a true perf stat -j -M > tma_info_inst_mix_bptkbranch -a true perf stat -j -M > tma_info_memory_l2mpki_all -a true perf stat -j -M > tma_info_inst_mix_instructions -a true perf stat -j -M > tma_info_inst_mix_ipload -a true perf stat -j -M > tma_info_thread_slots_utilization -a true perf stat -j -M tma_info_thread_cpi - > a true perf stat -j -M tma_info_bottleneck_instruction_fetch_bw -a true perf > stat -j -M tma_info_inst_mix_ipbranch -a true perf stat -j -M > tma_ports_utilized_1 -a true perf stat -j -M tma_info_inst_mix_iparith_avx256 > -a true perf stat -j -M tma_store_stlb_miss -a true perf stat -j -M tma_dsb -a > true perf stat -j -M smi_num -a true perf stat -j -M tsx_transactional_cycles -a > true perf stat -j -M tma_fp_vector_128b -a true perf stat -j -M > tma_fetch_bandwidth -a true perf stat -j -M tma_fb_full -a true perf stat -j -M > tma_info_bottleneck_mispredictions -a true perf stat -j -M tma_port_6 -a true > perf stat -j -M tma_info_frontend_l2mpki_code -a true perf stat -j -M > tma_info_core_fp_arith_utilization -a true perf stat -j -M > tma_info_inst_mix_iparith_avx128 -a true perf stat -j -M > tma_info_memory_l2mpki -a true perf stat -j -M > tma_info_bottleneck_memory_bandwidth -a true perf stat -j -M > tma_info_memory_thread_l1d_cache_fill_bw_1t -a true perf stat -j -M > tma_info_memory_oro_load_l2_miss_latency -a true perf stat -j -M > c3_pkg_residency -a true perf stat -j -M tma_few_uops_instructions -a true > perf stat -j -M tma_info_pipeline_execute -a true perf stat -j -M > tma_info_frontend_ipdsb_miss_ret -a true perf stat -j -M > tma_info_memory_thread_l3_cache_fill_bw_1t -a true perf stat -j -M > tma_info_thread_uptb -a true perf stat -j -M tma_info_inst_mix_ipcall -a true > perf stat -j -M tma_l2_bound -a true perf stat -j -M tma_ports_utilized_0 -a > true perf stat -j -M tma_info_bottleneck_memory_data_tlbs -a true perf stat -j > -M tma_info_inst_mix_iparith_scalar_dp -a true perf stat -j -M > tma_info_inst_mix_ipstore -a true perf stat -j -M > tma_info_branches_other_branches -a true perf stat -j -M > tma_info_thread_uoppi -a true perf stat -j -M > tma_info_system_power_license2_utilization -a true perf stat -j -M > tsx_cycles_per_elision -a true perf stat -j -M > tma_info_memory_tlb_store_stlb_mpki -a true perf stat -j -M > tma_backend_bound,tma_bad_speculation,tma_retiring,tma_frontend_boun > d > -a perf bench futex hash -r 2 -s > # Running 'futex/hash' benchmark: > Run summary [PID 78627]: 8 threads, each operating on 1024 [private] > futexes for 2 secs. > > Averaged 1625344 operations/sec (+- 0.78%), total secs = 2 perf stat -j -M > tma_fetch_bandwidth,tma_frontend_bound,tma_fetch_latency -a perf bench > futex hash -r 2 -s # Running 'futex/hash' benchmark: > Run summary [PID 78637]: 8 threads, each operating on 1024 [private] > futexes for 2 secs. > > Averaged 1622144 operations/sec (+- 0.25%), total secs = 2 perf stat -j -M > tma_machine_clears,tma_branch_mispredicts,tma_bad_speculation -a perf > bench futex hash -r 2 -s # Running 'futex/hash' benchmark: > Run summary [PID 78647]: 8 threads, each operating on 1024 [private] > futexes for 2 secs. > > Averaged 1503103 operations/sec (+- 1.58%), total secs = 2 perf stat -j -M > tma_core_bound,tma_memory_bound,tma_backend_bound -a perf bench > futex hash -r 2 -s # Running 'futex/hash' benchmark: > Run summary [PID 78680]: 8 threads, each operating on 1024 [private] > futexes for 2 secs. > > Averaged 1415935 operations/sec (+- 2.26%), total secs = 2 perf stat -j -M > tma_heavy_operations,tma_light_operations,tma_retiring > -a perf bench futex hash -r 2 -s > # Running 'futex/hash' benchmark: > Run summary [PID 78690]: 8 threads, each operating on 1024 [private] > futexes for 2 secs. > > Averaged 1519872 operations/sec (+- 0.38%), total secs = 2 perf stat -j -M > tma_other_light_ops -a perf bench futex hash -r 2 -s # Running 'futex/hash' > benchmark: > Run summary [PID 78700]: 8 threads, each operating on 1024 [private] > futexes for 2 secs. > > Averaged 1490816 operations/sec (+- 0.21%), total secs = 2 perf stat -j -M > tma_split_stores -a perf bench futex hash -r 2 -s # Running 'futex/hash' > benchmark: > Run summary [PID 78710]: 8 threads, each operating on 1024 [private] > futexes for 2 secs. > > Averaged 1474816 operations/sec (+- 0.25%), total secs = 2 perf stat -j -M > tma_store_stlb_hit -a perf bench futex hash -r 2 -s # Running 'futex/hash' > benchmark: > Run summary [PID 78721]: 8 threads, each operating on 1024 [private] > futexes for 2 secs. > > Averaged 1460032 operations/sec (+- 0.35%), total secs = 2 perf stat -j -M > tma_dram_bound -a perf bench futex hash -r 2 -s # Running 'futex/hash' > benchmark: > Run summary [PID 78732]: 8 threads, each operating on 1024 [private] > futexes for 2 secs. > > Averaged 1421632 operations/sec (+- 0.56%), total secs = 2 perf stat -j -M > tma_dtlb_store -a perf bench futex hash -r 2 -s # Running 'futex/hash' > benchmark: > Run summary [PID 78742]: 8 threads, each operating on 1024 [private] > futexes for 2 secs. > > Averaged 1431424 operations/sec (+- 0.24%), total secs = 2 perf stat -j -M > tma_mem_bandwidth -a perf bench futex hash -r 2 -s # Running 'futex/hash' > benchmark: > Run summary [PID 78752]: 8 threads, each operating on 1024 [private] > futexes for 2 secs. > > Averaged 1344768 operations/sec (+- 1.51%), total secs = 2 perf stat -j -M > tma_l3_hit_latency -a perf bench futex hash -r 2 -s # Running 'futex/hash' > benchmark: > Run summary [PID 78765]: 8 threads, each operating on 1024 [private] > futexes for 2 secs. > > Averaged 1369728 operations/sec (+- 1.18%), total secs = 2 perf stat -j -M > tma_mem_latency -a perf bench futex hash -r 2 -s # Running 'futex/hash' > benchmark: > Run summary [PID 78776]: 8 threads, each operating on 1024 [private] > futexes for 2 secs. > > Averaged 1361536 operations/sec (+- 0.79%), total secs = 2 perf stat -j -M > tma_data_sharing -a perf bench futex hash -r 2 -s # Running 'futex/hash' > benchmark: > Run summary [PID 78786]: 8 threads, each operating on 1024 [private] > futexes for 2 secs. > > Averaged 1302656 operations/sec (+- 1.90%), total secs = 2 perf stat -j -M > tma_memory_operations -a perf bench futex hash -r 2 -s # Running > 'futex/hash' benchmark: > Run summary [PID 78797]: 8 threads, each operating on 1024 [private] > futexes for 2 secs. > > Averaged 1333184 operations/sec (+- 1.08%), total secs = 2 perf stat -j -M > tma_4k_aliasing -a perf bench futex hash -r 2 -s # Running 'futex/hash' > benchmark: > Run summary [PID 78807]: 8 threads, each operating on 1024 [private] > futexes for 2 secs. > > Averaged 1268352 operations/sec (+- 2.13%), total secs = 2 perf stat -j -M > tma_l3_bound -a perf bench futex hash -r 2 -s # Running 'futex/hash' > benchmark: > Run summary [PID 78827]: 8 threads, each operating on 1024 [private] > futexes for 2 secs. > > Averaged 1289408 operations/sec (+- 2.65%), total secs = 2 perf stat -j -M > tma_store_fwd_blk -a perf bench futex hash -r 2 -s # Running 'futex/hash' > benchmark: > Run summary [PID 78847]: 8 threads, each operating on 1024 [private] > futexes for 2 secs. > > Averaged 1176256 operations/sec (+- 1.57%), total secs = 2 perf stat -j -M > tma_lock_latency -a perf bench futex hash -r 2 -s # Running 'futex/hash' > benchmark: > Run summary [PID 78857]: 8 threads, each operating on 1024 [private] > futexes for 2 secs. > > Averaged 1050112 operations/sec (+- 2.42%), total secs = 2 perf stat -j -M > tma_store_stlb_miss -a perf bench futex hash -r 2 -s # Running 'futex/hash' > benchmark: > Run summary [PID 78867]: 8 threads, each operating on 1024 [private] > futexes for 2 secs. > > Averaged 1117248 operations/sec (+- 1.96%), total secs = 2 perf stat -j -M > tma_l2_bound -a perf bench futex hash -r 2 -s # Running 'futex/hash' > benchmark: > Run summary [PID 78877]: 8 threads, each operating on 1024 [private] > futexes for 2 secs. > > Averaged 1174847 operations/sec (+- 0.74%), total secs = 2 > Workload: perf bench futex hash -r 2 -s Total metrics collected: 197 Non- > negative metric count: 197 Total Test Count: 100 Passed Test Count: 96 Test > validation finished. Final report: > [ > { > "Workload": "perf bench futex hash -r 2 -s", > "Report": { > "Metric Validation Statistics": { > "Total Rule Count": 100, > "Passed Rule Count": 96 > }, > "Tests in Category": { > "PositiveValueTest": { > "Total Tests": 197, > "Passed Tests": 197, > "Failed Tests": [] > }, > "RelationshipTest": { > "Total Tests": 5, > "Passed Tests": 5, > "Failed Tests": [] > }, > "SingleMetricTest": { > "Total Tests": 95, > "Passed Tests": 91, > "Failed Tests": [ > { > "RuleIndex": 6, > "RangeLower": "0", > "RangeUpper": "100", > "ErrorThreshold": 5, > "Failure": [ > { > "MetricName": "tma_split_stores", > "CollectedValue": [] > }, > { > "MetricName": "tma_4k_aliasing", > "CollectedValue": [] > }, > { > "MetricName": "tma_l3_bound", > "CollectedValue": [] > }, > { > "MetricName": "tma_store_fwd_blk", > "CollectedValue": [] > } > ] > } > ] > } > }, > "Errors": [] > } > } > ] > test child finished with -1 > ---- end ---- > perf metrics value validation: FAILED! > > The report shows there are four metrics return with no values. Do you know which processor this was tested on? I can look into the test and see if I could improve the final report to make it easier to debug when fails. Thanks, Weilin > > On Thu, Jun 22, 2023 at 2:35 PM Namhyung Kim <namhyung@kernel.org> > wrote: > > > > On Wed, Jun 21, 2023 at 5:08 PM Namhyung Kim <namhyung@kernel.org> > wrote: > > > > > > Hello, > > > > > > On Tue, Jun 20, 2023 at 10:00 AM Weilin Wang <weilin.wang@intel.com> > wrote: > > > > > > > > This is the fifth version of metric value validation tests. > > > > > > > > We made the following changes from v4 to v5: > > > > - Update "()" to "{}" to avoid creating sub shell and > > > > successfully skip test on non-Intel platform. [Ravi] > > > > > > > > v4: > > > > https://lore.kernel.org/lkml/20230618172820.751560-1- > weilin.wang@i > > > > ntel.com/ > > > > > > > > Weilin Wang (3): > > > > perf test: Add metric value validation test > > > > perf test: Add skip list for metrics known would fail > > > > perf test: Rerun failed metrics with longer workload > > > > > > Tested-by: Namhyung Kim <namhyung@kernel.org> > > > > Applied to perf-tools-next, thanks! ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v5 0/3] Add metric value validation test 2023-09-07 4:58 ` Wang, Weilin @ 2023-09-07 16:33 ` Namhyung Kim 0 siblings, 0 replies; 9+ messages in thread From: Namhyung Kim @ 2023-09-07 16:33 UTC (permalink / raw) To: Wang, Weilin Cc: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Jiri Olsa, Hunter, Adrian, Ian Rogers, linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org, Kan Liang, Alt, Samantha, Taylor, Perry, Biggers, Caleb, ravi.bangoria@amd.com On Wed, Sep 6, 2023 at 9:58 PM Wang, Weilin <weilin.wang@intel.com> wrote: > > Hi Namhyung, > > The report shows there are four metrics return with no values. Do you know which processor this was tested on? It's a laptop with 11th gen CPUs (tigerlake). $ head /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 140 model name : 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz stepping : 1 microcode : 0xac cpu MHz : 3899.531 cache size : 12288 KB physical id : 0 > > I can look into the test and see if I could improve the final report to make it easier to debug when fails. Thanks, Namhyung ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2023-09-07 16:41 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2023-06-20 17:00 [PATCH v5 0/3] Add metric value validation test Weilin Wang 2023-06-20 17:00 ` [PATCH v5 1/3] perf test: " Weilin Wang 2023-06-20 17:00 ` [PATCH v5 2/3] perf test: Add skip list for metrics known would fail Weilin Wang 2023-06-20 17:00 ` [PATCH v5 3/3] perf test: Rerun failed metrics with longer workload Weilin Wang 2023-06-22 0:08 ` [PATCH v5 0/3] Add metric value validation test Namhyung Kim 2023-06-22 21:35 ` Namhyung Kim 2023-09-05 17:30 ` Namhyung Kim 2023-09-07 4:58 ` Wang, Weilin 2023-09-07 16:33 ` Namhyung Kim
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).