[PATCH net-next v3 0/4] selftests: net: add performance metric reporting

[PATCH net-next v3 0/4] selftests: net: add performance metric reporting

[Stanislav Fomichev]

Let Python selftests report numeric measurements, so that CI can track
the performance of drivers over time, and add two performance tests.

ksft_metric() reports a measurement of the current test case, which is
printed as JSON right before the test result:

  # ktap-metric-json: {"goal":"higher","name":"throughput","value":389.127}

The goal tells a performance metric, which is better higher or lower,
from an error metric, which must be zero. Everything else, such as
units, history and regression thresholds, is up to the consumer.

The tests are in a new drivers/net/perf target, apart from the
functional tests: they need real hardware, take a while, and make no
sense on debug kernels. iperf.py measures single-stream TCP throughput
with iperf3, devmem.py the throughput of udmabuf-backed devmem TCP with
kperf [1].

System monitoring stays in NIPA, which runs each case of the perf tests
in a process of its own and attaches CPU and NIC metrics to its results.
The NIPA changes are at [2]; git clone & ./nipa local view starts an
http server on :44109 with two sample runs.

[1] https://github.com/facebookexperimental/kperf
[2] https://github.com/fomichev/nipa/tree/nipa-local-hw-metrics

v3:
- drop the timestamp of metrics (Jakub)
- drop the note on nested output from the docs (Jakub)
- report plain numbers, drop kind/type (Jakub)
- replace ksft_metric_policy() with a goal per metric: higher, lower
  or zero (Jakub)
- move the tests to drivers/net/perf, with a lib/py of their own (Jakub)
- fix the format of the Makefile lists (Jakub)
- report metrics from the tests rather than the runners (Jakub)
- drop the kperf section of the README (Jakub)
- stop reporting TCP retransmits
- drop the metric labels, report the throughput of the receiver only

v2:
- move system monitoring to NIPA (Jakub)
- move performance tests to drivers/net/hw/performance (Jakub)
- rename metric shape to type and document supported types (Paolo);
  kind= instead of type= (type shadows a builtin)
- remove direction from the metric example (Paolo)
- document policy processing order (Paolo)
- support producer-selected summary operations, including arbitrary pXX
- label throughput metrics in Gbps
- do not mark the iperf benchmark disruptive
- use 2 GiB udmabuf buffers for kperf devmem
- drop the ncdevmem devmem probe
- use a random kperf control port
- report kperf server cleanup failures

Stanislav Fomichev (4):
  selftests: net: py: add metric output
  selftests: drv-net: add an iperf performance test
  selftests: drv-net: add a kperf runner
  selftests: drv-net: measure devmem performance with kperf

 tools/testing/selftests/Makefile              |   2 +-
 .../testing/selftests/drivers/net/README.rst  |  17 ++
 .../selftests/drivers/net/lib/py/__init__.py  |   4 +-
 .../selftests/drivers/net/lib/py/kperf.py     | 181 ++++++++++++++++++
 .../selftests/drivers/net/lib/py/load.py      |  20 +-
 .../selftests/drivers/net/perf/Makefile       |  12 ++
 .../testing/selftests/drivers/net/perf/config |   4 +
 .../selftests/drivers/net/perf/devmem.py      |  96 ++++++++++
 .../selftests/drivers/net/perf/iperf.py       |  23 +++
 .../drivers/net/perf/lib/py/__init__.py       |  59 ++++++
 .../selftests/drivers/net/perf/settings       |   1 +
 .../testing/selftests/net/lib/py/__init__.py  |   9 +-
 tools/testing/selftests/net/lib/py/ksft.py    |  28 +++
 13 files changed, 447 insertions(+), 9 deletions(-)
 create mode 100644 tools/testing/selftests/drivers/net/lib/py/kperf.py
 create mode 100644 tools/testing/selftests/drivers/net/perf/Makefile
 create mode 100644 tools/testing/selftests/drivers/net/perf/config
 create mode 100755 tools/testing/selftests/drivers/net/perf/devmem.py
 create mode 100755 tools/testing/selftests/drivers/net/perf/iperf.py
 create mode 100644 tools/testing/selftests/drivers/net/perf/lib/py/__init__.py
 create mode 100644 tools/testing/selftests/drivers/net/perf/settings

-- 
2.53.0-Meta

[PATCH net-next v3 1/4] selftests: net: py: add metric output

[Stanislav Fomichev]

Add ksft_metric() to report a numeric measurement of the current test case
as JSON, right before the test result. Each metric states its goal: higher
or lower for a performance metric, depending on which way is better, or
zero for an error metric, so a consumer can tell wins from regressions.

Values are plain numbers without a timestamp. A helper for other kinds of
values, or a timestamp for values sampled over time, can be added when a
test needs one.

Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
---
v2: https://lore.kernel.org/netdev/20261005201115.2690830-1-sdf@fomichev.me/
v1: https://lore.kernel.org/netdev/20260916190409.1222272-1-sdf@fomichev.me/
---
 .../testing/selftests/drivers/net/README.rst  | 17 +++++++++++
 .../testing/selftests/net/lib/py/__init__.py  |  9 +++---
 tools/testing/selftests/net/lib/py/ksft.py    | 28 +++++++++++++++++++
 3 files changed, 50 insertions(+), 4 deletions(-)

diff --git a/tools/testing/selftests/drivers/net/README.rst b/tools/testing/selftests/drivers/net/README.rst
index 3fe49bce4f3a..ab61bbc322fd 100644
--- a/tools/testing/selftests/drivers/net/README.rst
+++ b/tools/testing/selftests/drivers/net/README.rst
@@ -259,6 +259,23 @@ ksft_pr()
 
 Use ``ksft_pr()`` instead of ``print()`` to avoid breaking TAP format.
 
+ksft_metric()
+~~~~~~~~~~~~~
+
+Use ``ksft_metric()`` to report a numeric measurement of the current test
+case. Metrics are printed as compact JSON right before the test result. For
+example::
+
+  ksft_metric("throughput", 100.5, goal="higher")
+
+prints::
+
+  # ktap-metric-json: {"goal":"higher","name":"throughput","value":100.5}
+
+``goal`` tells a performance metric, which is better ``higher`` or ``lower``,
+from an error metric, which must be ``zero``. Metrics do not affect the
+result of the test.
+
 ksft_disruptive
 ~~~~~~~~~~~~~~~
 
diff --git a/tools/testing/selftests/net/lib/py/__init__.py b/tools/testing/selftests/net/lib/py/__init__.py
index e269a260d58e..b1696f432c7c 100644
--- a/tools/testing/selftests/net/lib/py/__init__.py
+++ b/tools/testing/selftests/net/lib/py/__init__.py
@@ -8,8 +8,8 @@ from .consts import KSRC
 from .ksft import KsftFailEx, KsftSkipEx, KsftXfailEx, ksft_pr, ksft_eq, \
     ksft_ne, ksft_true, ksft_not_none, ksft_in, ksft_not_in, ksft_is, \
     ksft_ge, ksft_gt, ksft_lt, ksft_raises, ksft_busy_wait, \
-    ktap_result, ksft_disruptive, ksft_setup, ksft_run, ksft_exit, \
-    ksft_variants, KsftNamedVariant
+    ktap_result, ksft_disruptive, ksft_metric, ksft_setup, ksft_run, \
+    ksft_exit, ksft_variants, KsftNamedVariant
 from .netns import NetNS, NetNSEnter, UserNetNS
 from .nsim import NetdevSim, NetdevSimDev
 from .utils import CmdExitFailure, fd_read_timeout, cmd, bkg, defer, \
@@ -24,8 +24,9 @@ __all__ = ["KSRC",
            "KsftFailEx", "KsftSkipEx", "KsftXfailEx", "ksft_pr", "ksft_eq",
            "ksft_ne", "ksft_true", "ksft_not_none", "ksft_in", "ksft_not_in",
            "ksft_is", "ksft_ge", "ksft_gt", "ksft_lt", "ksft_raises",
-           "ksft_busy_wait", "ktap_result", "ksft_disruptive", "ksft_setup",
-           "ksft_run", "ksft_exit", "ksft_variants", "KsftNamedVariant",
+           "ksft_busy_wait", "ktap_result", "ksft_disruptive", "ksft_metric",
+           "ksft_setup", "ksft_run", "ksft_exit", "ksft_variants",
+           "KsftNamedVariant",
            "NetNS", "NetNSEnter", "UserNetNS",
            "CmdExitFailure", "fd_read_timeout", "cmd", "bkg", "defer",
            "bpftool", "ip", "ethtool", "bpftrace", "rand_port", "rand_ports",
diff --git a/tools/testing/selftests/net/lib/py/ksft.py b/tools/testing/selftests/net/lib/py/ksft.py
index 81287c2daff0..3b97bdff4cbb 100644
--- a/tools/testing/selftests/net/lib/py/ksft.py
+++ b/tools/testing/selftests/net/lib/py/ksft.py
@@ -4,6 +4,7 @@ import fnmatch
 import functools
 import getopt
 import inspect
+import json
 import os
 import signal
 import sys
@@ -16,6 +17,7 @@ from . import utils
 KSFT_RESULT = None
 KSFT_RESULT_ALL = True
 KSFT_DISRUPTIVE = True
+KSFT_METRICS = []
 
 
 class KsftFailEx(Exception):
@@ -93,6 +95,31 @@ KSFT_DISRUPTIVE = True
     print(pfx, prefixed, **kwargs)
 
 
+def ksft_metric(name, value, *, goal):
+    """
+    Report a numeric measurement of the current test case.
+
+    Metrics are printed right before the result of the test case. They do
+    not affect the result.
+
+    Args:
+      name - name of the metric, e.g. "throughput"
+      value - the measured number
+      goal - "higher" or "lower" for a performance metric, depending on
+             which way is better, "zero" for an error metric, which must
+             be zero
+    """
+    KSFT_METRICS.append({"name": name, "value": value, "goal": goal})
+
+
+def _ksft_flush_metrics():
+    for metric in KSFT_METRICS:
+        payload = json.dumps(metric, allow_nan=False, separators=(",", ":"),
+                             sort_keys=True)
+        ksft_pr(f"ktap-metric-json: {payload}")
+    KSFT_METRICS.clear()
+
+
 def _fail(*args):
     global KSFT_RESULT
     KSFT_RESULT = False
@@ -450,6 +477,7 @@ KsftCaseFunction = namedtuple("KsftCaseFunction",
         if not cnt_key:
             cnt_key = 'pass' if KSFT_RESULT else 'fail'
 
+        _ksft_flush_metrics()
         ktap_result(KSFT_RESULT, cnt, name, comment=comment)
         totals[cnt_key] += 1
 
-- 
2.53.0-Meta

[PATCH net-next v3 2/4] selftests: drv-net: add an iperf performance test

[Stanislav Fomichev]

Add the drivers/net/perf target, with a lib/py of its own like the one of
drivers/net/hw. Add Iperf3Runner.measure(), which returns an Iperf3Result
that other results can join later, and report its throughput from the test.

Single-stream iperf3 throughput is flaky and can change considerably
between back-to-back runs on the same hosts.

Tested on two mlx5 hosts. Captured metric output:

  # # ktap-metric-json: {"goal":"higher","name":"throughput","value":49.92071602072054}

Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
---
 tools/testing/selftests/Makefile              |  2 +-
 .../selftests/drivers/net/lib/py/load.py      | 20 ++++++-
 .../selftests/drivers/net/perf/Makefile       | 11 ++++
 .../testing/selftests/drivers/net/perf/config |  1 +
 .../selftests/drivers/net/perf/iperf.py       | 23 ++++++++
 .../drivers/net/perf/lib/py/__init__.py       | 57 +++++++++++++++++++
 .../selftests/drivers/net/perf/settings       |  1 +
 7 files changed, 111 insertions(+), 4 deletions(-)
 create mode 100644 tools/testing/selftests/drivers/net/perf/Makefile
 create mode 100644 tools/testing/selftests/drivers/net/perf/config
 create mode 100755 tools/testing/selftests/drivers/net/perf/iperf.py
 create mode 100644 tools/testing/selftests/drivers/net/perf/lib/py/__init__.py
 create mode 100644 tools/testing/selftests/drivers/net/perf/settings

diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
index 07c3719f8b04..c55d7792b27c 100644
--- a/tools/testing/selftests/Makefile
+++ b/tools/testing/selftests/Makefile
@@ -159,7 +159,7 @@ TARGETS_HOTPLUG = cpu-hotplug
 TARGETS_HOTPLUG += memory-hotplug
 
 # Networking tests want the net/lib target, include it automatically
-ifneq ($(filter net drivers/net drivers/net/hw,$(TARGETS)),)
+ifneq ($(filter net drivers/net drivers/net/hw drivers/net/perf,$(TARGETS)),)
 ifeq ($(filter net/lib,$(TARGETS)),)
 	INSTALL_DEP_TARGETS := net/lib
 endif
diff --git a/tools/testing/selftests/drivers/net/lib/py/load.py b/tools/testing/selftests/drivers/net/lib/py/load.py
index e24660e5c27f..f337b576c4b4 100644
--- a/tools/testing/selftests/drivers/net/lib/py/load.py
+++ b/tools/testing/selftests/drivers/net/lib/py/load.py
@@ -3,10 +3,18 @@
 import re
 import time
 import json
+from dataclasses import dataclass
 
 from lib.py import ksft_pr, cmd, ip, rand_port, wait_port_listen
 
 
+@dataclass
+class Iperf3Result:
+    """Average throughput (Gbps) of an iperf3 run."""
+
+    throughput: float
+
+
 class Iperf3Runner:
     """
     Sets up and runs iperf3 traffic.
@@ -53,9 +61,9 @@ from lib.py import ksft_pr, cmd, ip, rand_port, wait_port_listen
             kwargs["timeout"] = duration + 5
         return cmd(cmdline, **kwargs)
 
-    def measure_bandwidth(self, reverse=False):
+    def measure(self, reverse=False):
         """
-        Runs an iperf3 measurement and returns the average bandwidth (Gbps).
+        Runs an iperf3 measurement and returns an Iperf3Result.
         Discards the first and last few reporting intervals and uses only the
         middle part of the run where throughput is typically stable.
         """
@@ -78,7 +86,13 @@ from lib.py import ksft_pr, cmd, ip, rand_port, wait_port_listen
 
         avg = sum(stable) / len(stable)
 
-        return avg
+        return Iperf3Result(avg)
+
+    def measure_bandwidth(self, reverse=False):
+        """
+        Runs an iperf3 measurement and returns the average bandwidth (Gbps).
+        """
+        return self.measure(reverse=reverse).throughput
 
 
 class GenerateTraffic:
diff --git a/tools/testing/selftests/drivers/net/perf/Makefile b/tools/testing/selftests/drivers/net/perf/Makefile
new file mode 100644
index 000000000000..31b0b60de4f6
--- /dev/null
+++ b/tools/testing/selftests/drivers/net/perf/Makefile
@@ -0,0 +1,11 @@
+# SPDX-License-Identifier: GPL-2.0
+
+TEST_PROGS := \
+	iperf.py \
+# end of TEST_PROGS
+
+TEST_INCLUDES := \
+	$(wildcard lib/py/*.py ../lib/py/*.py) \
+# end of TEST_INCLUDES
+
+include ../../../lib.mk
diff --git a/tools/testing/selftests/drivers/net/perf/config b/tools/testing/selftests/drivers/net/perf/config
new file mode 100644
index 000000000000..6450f34933e6
--- /dev/null
+++ b/tools/testing/selftests/drivers/net/perf/config
@@ -0,0 +1 @@
+CONFIG_IPV6=y
diff --git a/tools/testing/selftests/drivers/net/perf/iperf.py b/tools/testing/selftests/drivers/net/perf/iperf.py
new file mode 100755
index 000000000000..5b6f0bce2a03
--- /dev/null
+++ b/tools/testing/selftests/drivers/net/perf/iperf.py
@@ -0,0 +1,23 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+"""Measure TCP throughput with iperf3."""
+
+from lib.py import Iperf3Runner, NetDrvEpEnv, ksft_exit, ksft_metric, ksft_run
+
+
+def tcp(cfg):
+    """Measure TCP throughput."""
+    result = Iperf3Runner(cfg).measure()
+    ksft_metric("throughput", result.throughput, goal="higher")
+
+
+def main():
+    """Run the iperf performance test."""
+    with NetDrvEpEnv(__file__, nsim_test=False) as cfg:
+        ksft_run([tcp], args=(cfg,))
+    ksft_exit()
+
+
+if __name__ == "__main__":
+    main()
diff --git a/tools/testing/selftests/drivers/net/perf/lib/py/__init__.py b/tools/testing/selftests/drivers/net/perf/lib/py/__init__.py
new file mode 100644
index 000000000000..2bb2cd08ff7d
--- /dev/null
+++ b/tools/testing/selftests/drivers/net/perf/lib/py/__init__.py
@@ -0,0 +1,57 @@
+# SPDX-License-Identifier: GPL-2.0
+
+"""
+Driver test environment (performance tests).
+NetDrvEnv and NetDrvEpEnv are the main environment classes.
+NetDrvContEnv extends NetDrvEpEnv with netkit container support.
+Former is for local host only tests, latter creates / connects
+to a remote endpoint. See NIPA wiki for more information about
+running and writing driver tests.
+"""
+
+import sys
+from pathlib import Path
+
+KSFT_DIR = (Path(__file__).parent / "../../../../..").resolve()
+
+try:
+    sys.path.append(KSFT_DIR.as_posix())
+
+    # Import one by one to avoid pylint false positives
+    from net.lib.py import NetNS, NetNSEnter, NetdevSimDev, UserNetNS
+    from net.lib.py import EthtoolFamily, NetdevFamily, NetshaperFamily, \
+        NlError, RtnlFamily, DevlinkFamily, PSPFamily, Netlink
+    from net.lib.py import CmdExitFailure
+    from net.lib.py import bkg, cmd, bpftool, bpftrace, defer, ethtool, \
+        fd_read_timeout, ip, rand_port, rand_ports, wait_port_listen, \
+        wait_file, ctl_file_write, tool
+    from net.lib.py import bpf_map_set, bpf_map_dump, bpf_prog_map_ids
+    from net.lib.py import KsftSkipEx, KsftFailEx, KsftXfailEx
+    from net.lib.py import ksft_disruptive, ksft_exit, ksft_metric, ksft_pr, \
+        ksft_run, ksft_setup, ksft_variants, KsftNamedVariant
+    from net.lib.py import ksft_eq, ksft_ge, ksft_in, ksft_is, ksft_lt, \
+        ksft_ne, ksft_not_in, ksft_raises, ksft_true, ksft_gt, ksft_not_none
+    from drivers.net.lib.py import set_ethtool_feat
+    from drivers.net.lib.py import GenerateTraffic, Remote, Iperf3Runner
+    from drivers.net.lib.py import NetDrvEnv, NetDrvEpEnv, NetDrvContEnv
+
+    __all__ = ["NetNS", "NetNSEnter", "NetdevSimDev", "UserNetNS",
+               "EthtoolFamily", "NetdevFamily", "NetshaperFamily",
+               "NlError", "RtnlFamily", "DevlinkFamily", "PSPFamily", "Netlink",
+               "CmdExitFailure",
+               "bkg", "cmd", "bpftool", "bpftrace", "defer", "ethtool",
+               "fd_read_timeout", "ip", "rand_port", "rand_ports",
+               "wait_port_listen", "wait_file", "ctl_file_write", "tool",
+               "bpf_map_set", "bpf_map_dump", "bpf_prog_map_ids",
+               "KsftSkipEx", "KsftFailEx", "KsftXfailEx",
+               "ksft_disruptive", "ksft_exit", "ksft_metric", "ksft_pr",
+               "ksft_run", "ksft_setup", "ksft_variants", "KsftNamedVariant",
+               "ksft_eq", "ksft_ge", "ksft_in", "ksft_is", "ksft_lt",
+               "ksft_ne", "ksft_not_in", "ksft_raises", "ksft_true", "ksft_gt",
+               "ksft_not_none",
+               "NetDrvEnv", "NetDrvEpEnv", "NetDrvContEnv", "GenerateTraffic",
+               "Remote", "Iperf3Runner", "set_ethtool_feat"]
+except ModuleNotFoundError as e:
+    print("Failed importing `net` library from kernel sources")
+    print(str(e))
+    sys.exit(4)
diff --git a/tools/testing/selftests/drivers/net/perf/settings b/tools/testing/selftests/drivers/net/perf/settings
new file mode 100644
index 000000000000..e7b9417537fb
--- /dev/null
+++ b/tools/testing/selftests/drivers/net/perf/settings
@@ -0,0 +1 @@
+timeout=0
-- 
2.53.0-Meta

[PATCH net-next v3 3/4] selftests: drv-net: add a kperf runner

[Stanislav Fomichev]

Add a helper to manage kperf servers, run the client, and return the
aggregate throughput of both ends. Use a random control port and a unique
PID file per server, as the hosts may share /tmp, and report server cleanup
failures. Stopping the servers waits for their sessions to exit, as they
may still be releasing devmem and RX steering, which the next test needs. A
server which can log does so to a file next to its PID file, and the end of
both logs is printed when the client fails, as the client can only tell
that a server went away.

Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
---
 .../selftests/drivers/net/lib/py/__init__.py  |   4 +-
 .../selftests/drivers/net/lib/py/kperf.py     | 181 ++++++++++++++++++
 .../drivers/net/perf/lib/py/__init__.py       |   6 +-
 3 files changed, 188 insertions(+), 3 deletions(-)
 create mode 100644 tools/testing/selftests/drivers/net/lib/py/kperf.py

diff --git a/tools/testing/selftests/drivers/net/lib/py/__init__.py b/tools/testing/selftests/drivers/net/lib/py/__init__.py
index 40ccdcf0c378..7a32f6516a12 100644
--- a/tools/testing/selftests/drivers/net/lib/py/__init__.py
+++ b/tools/testing/selftests/drivers/net/lib/py/__init__.py
@@ -49,11 +49,13 @@ KSFT_DIR = (Path(__file__).parent / "../../../..").resolve()
 
     from .env import NetDrvEnv, NetDrvEpEnv, NetDrvContEnv
     from .feat import set_ethtool_feat
+    from .kperf import KperfResult, KperfRunner
     from .load import GenerateTraffic, Iperf3Runner
     from .remote import Remote
 
     __all__ += ["NetDrvEnv", "NetDrvEpEnv", "NetDrvContEnv", "GenerateTraffic",
-                "Remote", "Iperf3Runner", "set_ethtool_feat"]
+                "Remote", "Iperf3Runner", "KperfResult", "KperfRunner",
+                "set_ethtool_feat"]
 except ModuleNotFoundError as e:
     print("Failed importing `net` library from kernel sources")
     print(str(e))
diff --git a/tools/testing/selftests/drivers/net/lib/py/kperf.py b/tools/testing/selftests/drivers/net/lib/py/kperf.py
new file mode 100644
index 000000000000..53870d0caf14
--- /dev/null
+++ b/tools/testing/selftests/drivers/net/lib/py/kperf.py
@@ -0,0 +1,181 @@
+# SPDX-License-Identifier: GPL-2.0
+
+"""Run and parse kperf network performance measurements."""
+
+import re
+import shlex
+import subprocess
+import time
+import uuid
+from dataclasses import dataclass
+
+from lib.py import cmd, ksft_pr, rand_port, wait_port_listen
+
+
+_SECTION_RE = re.compile(r"==\s+(Source|Target)\b")
+_RATE_RE = re.compile(r"\b(Tx|Rx)\s*([0-9]+(?:\.[0-9]+)?)\s+Gbps\b")
+
+
+@dataclass
+class KperfResult:
+    """Aggregate throughput (Gbps) of both endpoints."""
+
+    source_throughput: float
+    target_throughput: float
+
+
+class KperfRunner:
+    """Manage kperf servers and run measurements between test endpoints."""
+
+    def __init__(self, env):
+        self.env = env
+        self._pid_prefix = f"/tmp/ksft-kperf-{uuid.uuid4().hex[:8]}"
+        self._port = rand_port()
+        self._started = []
+        env.require_cmd("kperf-client", local=True)
+        env.require_cmd("kperf-server", local=True, remote=True)
+
+    def _pid_file(self, remote):
+        # The hosts may share /tmp (netns), and kperf-server kills the
+        # process named in an existing PID file.
+        return f"{self._pid_prefix}-{'remote' if remote else 'local'}.pid"
+
+    def _log_file(self, remote):
+        return f"{self._pid_prefix}-{'remote' if remote else 'local'}.log"
+
+    def _server(self, remote, kill=False):
+        host = self.env.remote if remote else None
+        address = self.env.remote_addr if remote else self.env.addr
+        args = ["kperf-server", "--pid-file", self._pid_file(remote)]
+        if kill:
+            args.append("--kill")
+        else:
+            args += ["--addr", address, "--port", str(self._port)]
+            # The output of a daemon is lost, unless it can log
+            usage = cmd("kperf-server --help", host=host, fail=False)
+            if "--log-file" in usage.stdout + usage.stderr:
+                args += ["--log-file", self._log_file(remote)]
+        return cmd(shlex.join(args) if remote else args, host=host,
+                   fail=not kill)
+
+    def _print_logs(self):
+        """Print the end of the logs of the servers, to tell why the client
+        failed.
+        """
+        for remote in self._started:
+            host = self.env.remote if remote else None
+            args = ["tail", "-n", "30", self._log_file(remote)]
+            log = cmd(shlex.join(args) if remote else args, host=host,
+                      fail=False)
+            if log.stdout.strip():
+                name = "remote" if remote else "local"
+                ksft_pr(f"kperf-server log ({name}):\n{log.stdout.strip()}")
+
+    def __enter__(self):
+        try:
+            for remote in (False, True):
+                self._server(remote)
+                self._started.append(remote)
+            wait_port_listen(self._port)
+            wait_port_listen(self._port, host=self.env.remote)
+        except Exception:
+            self.close()
+            raise
+        return self
+
+    def _wait_exit(self, remote, timeout=60):
+        """Wait for the sessions of a server to exit.
+
+        Killing the server leaves its sessions, which exit once they have
+        released their devmem and RX steering, which the next test needs.
+        Being forked, they run with the server's command line.  The
+        bracket keeps pgrep from matching the shell which runs it.
+        """
+        host = self.env.remote if remote else None
+        args = ["pgrep", "-f",
+                f"[k]perf-server --pid-file {self._pid_file(remote)}"]
+        deadline = time.monotonic() + timeout
+        while True:
+            pgrep = cmd(shlex.join(args) if remote else args, host=host,
+                        fail=False)
+            if pgrep.ret:
+                return None
+            if time.monotonic() > deadline:
+                return pgrep
+            time.sleep(0.1)
+
+    def close(self):
+        """Stop the started kperf servers, and wait for their sessions."""
+        failures = []
+        while self._started:
+            remote = self._started.pop()
+            command = self._server(remote, kill=True)
+            if not command.ret:
+                command = self._wait_exit(remote)
+            if command:
+                failures.append(command)
+            host = self.env.remote if remote else None
+            args = ["rm", "-f", self._log_file(remote)]
+            cmd(shlex.join(args) if remote else args, host=host, fail=False)
+        if failures:
+            details = "\n".join(str(command) for command in failures)
+            raise RuntimeError(f"failed to stop kperf server(s)\n{details}")
+
+    def __exit__(self, _exc_type, _exc_value, _exc_tb):
+        self.close()
+
+    def run(self, client_args=()):
+        """Run kperf and return its KperfResult."""
+        argv = ["kperf-client", "--src", self.env.remote_addr,
+                "--dst", self.env.addr, "--src-svc", str(self._port),
+                "--dst-svc", str(self._port)]
+        argv += [str(arg) for arg in client_args]
+        command = cmd(argv, background=True)
+        try:
+            command.process(terminate=False, fail=False, timeout=300)
+        except subprocess.TimeoutExpired as error:
+            command.proc.kill()
+            command.process(terminate=False, fail=False)
+            self._print_logs()
+            raise RuntimeError(f"kperf client timed out\n{command!r}") \
+                from error
+        if command.ret:
+            self._print_logs()
+            raise RuntimeError(
+                f"kperf client exited with status {command.ret}\n{command!r}")
+
+        try:
+            result = self.parse(command.stdout + "\n" + command.stderr)
+        except ValueError:
+            ksft_pr(command)
+            self._print_logs()
+            raise
+
+        return result
+
+    @staticmethod
+    def parse(output):
+        """Parse human-readable kperf client output."""
+        throughput = {"Source": [], "Target": []}
+        section = None
+
+        for line in output.splitlines():
+            match = _SECTION_RE.search(line)
+            if match:
+                section = match.group(1)
+                continue
+            if section is None:
+                continue
+
+            match = _RATE_RE.search(line)
+            if match:
+                direction, value = match.groups()
+                if (section, direction) in (("Source", "Tx"),
+                                             ("Target", "Rx")):
+                    throughput[section].append(float(value))
+
+        if not all(throughput.values()):
+            raise ValueError("incomplete kperf result")
+
+        return KperfResult(sum(throughput["Source"]),
+                           sum(throughput["Target"]))
diff --git a/tools/testing/selftests/drivers/net/perf/lib/py/__init__.py b/tools/testing/selftests/drivers/net/perf/lib/py/__init__.py
index 2bb2cd08ff7d..c07f1ac5e80a 100644
--- a/tools/testing/selftests/drivers/net/perf/lib/py/__init__.py
+++ b/tools/testing/selftests/drivers/net/perf/lib/py/__init__.py
@@ -32,7 +32,8 @@ KSFT_DIR = (Path(__file__).parent / "../../../../..").resolve()
     from net.lib.py import ksft_eq, ksft_ge, ksft_in, ksft_is, ksft_lt, \
         ksft_ne, ksft_not_in, ksft_raises, ksft_true, ksft_gt, ksft_not_none
     from drivers.net.lib.py import set_ethtool_feat
-    from drivers.net.lib.py import GenerateTraffic, Remote, Iperf3Runner
+    from drivers.net.lib.py import GenerateTraffic, Remote, Iperf3Runner, \
+        KperfResult, KperfRunner
     from drivers.net.lib.py import NetDrvEnv, NetDrvEpEnv, NetDrvContEnv
 
     __all__ = ["NetNS", "NetNSEnter", "NetdevSimDev", "UserNetNS",
@@ -50,7 +51,8 @@ KSFT_DIR = (Path(__file__).parent / "../../../../..").resolve()
                "ksft_ne", "ksft_not_in", "ksft_raises", "ksft_true", "ksft_gt",
                "ksft_not_none",
                "NetDrvEnv", "NetDrvEpEnv", "NetDrvContEnv", "GenerateTraffic",
-               "Remote", "Iperf3Runner", "set_ethtool_feat"]
+               "Remote", "Iperf3Runner", "KperfResult", "KperfRunner",
+               "set_ethtool_feat"]
 except ModuleNotFoundError as e:
     print("Failed importing `net` library from kernel sources")
     print(str(e))
-- 
2.53.0-Meta

[PATCH net-next v3 4/4] selftests: drv-net: measure devmem performance with kperf

[Stanislav Fomichev]

Add a separate udmabuf-backed devmem performance test using KperfRunner.
Keep correctness cases in drivers/net/hw. Skip the test when kperf or
/dev/udmabuf is missing, and fail it when the kperf run fails.

Example metric output:

  # # ktap-metric-json: {"goal":"higher","name":"throughput","value":389.127}

Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
---
 .../selftests/drivers/net/perf/Makefile       |  1 +
 .../testing/selftests/drivers/net/perf/config |  3 +
 .../selftests/drivers/net/perf/devmem.py      | 96 +++++++++++++++++++
 3 files changed, 100 insertions(+)
 create mode 100755 tools/testing/selftests/drivers/net/perf/devmem.py

diff --git a/tools/testing/selftests/drivers/net/perf/Makefile b/tools/testing/selftests/drivers/net/perf/Makefile
index 31b0b60de4f6..1a90537d2726 100644
--- a/tools/testing/selftests/drivers/net/perf/Makefile
+++ b/tools/testing/selftests/drivers/net/perf/Makefile
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0
 
 TEST_PROGS := \
+	devmem.py \
 	iperf.py \
 # end of TEST_PROGS
 
diff --git a/tools/testing/selftests/drivers/net/perf/config b/tools/testing/selftests/drivers/net/perf/config
index 6450f34933e6..111e9b58610d 100644
--- a/tools/testing/selftests/drivers/net/perf/config
+++ b/tools/testing/selftests/drivers/net/perf/config
@@ -1 +1,4 @@
 CONFIG_IPV6=y
+CONFIG_NET_DEVMEM=y
+CONFIG_SYNC_FILE=y
+CONFIG_UDMABUF=y
diff --git a/tools/testing/selftests/drivers/net/perf/devmem.py b/tools/testing/selftests/drivers/net/perf/devmem.py
new file mode 100755
index 000000000000..4c1fb9b51fff
--- /dev/null
+++ b/tools/testing/selftests/drivers/net/perf/devmem.py
@@ -0,0 +1,96 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+"""Measure udmabuf-backed devmem TCP throughput with kperf."""
+
+from lib.py import cmd, defer, ethtool, EthtoolFamily, KperfRunner
+from lib.py import ksft_disruptive, ksft_exit, ksft_metric, ksft_run
+from lib.py import KsftSkipEx, NetDrvEpEnv
+
+
+UDMABUF_LIMIT = "/sys/module/udmabuf/parameters/size_limit_mb"
+UDMABUF_SIZE_MB = 2048
+
+
+def _set_udmabuf_limit(host) -> None:
+    current = cmd(f"cat {UDMABUF_LIMIT}", host=host).stdout.strip()
+    if int(current) >= UDMABUF_SIZE_MB:
+        return
+
+    cmd(f"echo {UDMABUF_SIZE_MB} > {UDMABUF_LIMIT}", host=host, shell=True)
+    defer(cmd, f"echo {current} > {UDMABUF_LIMIT}", host=host, shell=True)
+
+
+def _enable_hw_gro(cfg) -> None:
+    features = ethtool(f"-k {cfg.ifname}", json=True)[0]
+    hw_gro = features["rx-gro-hw"]
+    if not hw_gro["active"]:
+        if hw_gro["fixed"]:
+            raise KsftSkipEx("Test requires rx-gro-hw support")
+        ethtool(f"-K {cfg.ifname} rx-gro-hw on")
+        defer(ethtool, f"-K {cfg.ifname} rx-gro-hw off")
+
+
+@ksft_disruptive
+def kperf(cfg) -> None:
+    """Measure udmabuf-backed devmem performance with kperf."""
+    runner = KperfRunner(cfg)
+
+    for name, host in (("local", None), ("remote", cfg.remote)):
+        if cmd("test -c /dev/udmabuf", host=host, fail=False).ret:
+            raise KsftSkipEx(f"Test requires /dev/udmabuf on the {name} host")
+        _set_udmabuf_limit(host)
+
+    # kperf reprograms the RSS indirection table for its queues.
+    defer(ethtool, f"-X {cfg.ifname} default")
+
+    _enable_hw_gro(cfg)
+
+    ethnl = EthtoolFamily()
+    rings = ethnl.rings_get({"header": {"dev-index": cfg.ifindex}})
+    restore = {
+        "header": {"dev-index": cfg.ifindex},
+        "tcp-data-split": rings.get("tcp-data-split", "unknown"),
+    }
+    config = {
+        "header": {"dev-index": cfg.ifindex},
+        "tcp-data-split": "enabled",
+    }
+    # Memory providers require a zero threshold where the driver has one.
+    if "hds-thresh" in rings:
+        restore["hds-thresh"] = rings["hds-thresh"]
+        config["hds-thresh"] = 0
+    defer(ethnl.rings_set, restore)
+    ethnl.rings_set(config)
+
+    devmem_args = [
+        "--time", "15",
+        "--num-connections", "4",
+        "--read-size", str(64 << 20),
+        "--write-size", str(64 << 20),
+        "--pin-off", "4",
+        "--num-rx-queues", "4",
+        "--dmabuf-rx-size-mb", str(UDMABUF_SIZE_MB),
+        "--dmabuf-tx-size-mb", str(UDMABUF_SIZE_MB),
+        "--validate", "no",
+        "--devmem-rx",
+        "--devmem-tx",
+        "--devmem-rx-memory", "host",
+        "--devmem-tx-memory", "host",
+    ]
+
+    with runner:
+        result = runner.run(client_args=devmem_args)
+
+    ksft_metric("throughput", result.target_throughput, goal="higher")
+
+
+def main() -> None:
+    """Run the devmem performance test."""
+    with NetDrvEpEnv(__file__, nsim_test=False) as cfg:
+        ksft_run([kperf], args=(cfg,))
+    ksft_exit()
+
+
+if __name__ == "__main__":
+    main()
-- 
2.53.0-Meta

Re: [PATCH net-next v3 1/4] selftests: net: py: add metric output

[Björn Töpel]

Stanislav Fomichev <sdf.kernel@gmail.com> writes:

> Add ksft_metric() to report a numeric measurement of the current test case
> as JSON, right before the test result. Each metric states its goal: higher
> or lower for a performance metric, depending on which way is better, or
> zero for an error metric, so a consumer can tell wins from regressions.
>
> Values are plain numbers without a timestamp. A helper for other kinds of
> values, or a timestamp for values sampled over time, can be added when a
> test needs one.
>
> Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>

Reviewed-by: Björn Töpel <bjorn@kernel.org>

Re: [PATCH net-next v3 2/4] selftests: drv-net: add an iperf performance test

[Björn Töpel]

Stanislav Fomichev <sdf.kernel@gmail.com> writes:

> Add the drivers/net/perf target, with a lib/py of its own like the one of
> drivers/net/hw. Add Iperf3Runner.measure(), which returns an Iperf3Result
> that other results can join later, and report its throughput from the test.
>
> Single-stream iperf3 throughput is flaky and can change considerably
> between back-to-back runs on the same hosts.
>
> Tested on two mlx5 hosts. Captured metric output:
>
>   # # ktap-metric-json: {"goal":"higher","name":"throughput","value":49.92071602072054}
>
> Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
> ---
>  tools/testing/selftests/Makefile              |  2 +-
>  .../selftests/drivers/net/lib/py/load.py      | 20 ++++++-
>  .../selftests/drivers/net/perf/Makefile       | 11 ++++
>  .../testing/selftests/drivers/net/perf/config |  1 +
>  .../selftests/drivers/net/perf/iperf.py       | 23 ++++++++
>  .../drivers/net/perf/lib/py/__init__.py       | 57 +++++++++++++++++++
>  .../selftests/drivers/net/perf/settings       |  1 +
>  7 files changed, 111 insertions(+), 4 deletions(-)
>  create mode 100644 tools/testing/selftests/drivers/net/perf/Makefile
>  create mode 100644 tools/testing/selftests/drivers/net/perf/config
>  create mode 100755 tools/testing/selftests/drivers/net/perf/iperf.py
>  create mode 100644 tools/testing/selftests/drivers/net/perf/lib/py/__init__.py
>  create mode 100644 tools/testing/selftests/drivers/net/perf/settings
>
> diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
> index 07c3719f8b04..c55d7792b27c 100644
> --- a/tools/testing/selftests/Makefile
> +++ b/tools/testing/selftests/Makefile
> @@ -159,7 +159,7 @@ TARGETS_HOTPLUG = cpu-hotplug
>  TARGETS_HOTPLUG += memory-hotplug
>  
>  # Networking tests want the net/lib target, include it automatically
> -ifneq ($(filter net drivers/net drivers/net/hw,$(TARGETS)),)
> +ifneq ($(filter net drivers/net drivers/net/hw drivers/net/perf,$(TARGETS)),)
>  ifeq ($(filter net/lib,$(TARGETS)),)
>  	INSTALL_DEP_TARGETS := net/lib
>  endif
> diff --git a/tools/testing/selftests/drivers/net/lib/py/load.py b/tools/testing/selftests/drivers/net/lib/py/load.py
> index e24660e5c27f..f337b576c4b4 100644
> --- a/tools/testing/selftests/drivers/net/lib/py/load.py
> +++ b/tools/testing/selftests/drivers/net/lib/py/load.py
> @@ -3,10 +3,18 @@
>  import re
>  import time
>  import json
> +from dataclasses import dataclass
>  
>  from lib.py import ksft_pr, cmd, ip, rand_port, wait_port_listen
>  
>  
> +@dataclass
> +class Iperf3Result:
> +    """Average throughput (Gbps) of an iperf3 run."""
> +
> +    throughput: float
> +
> +
>  class Iperf3Runner:
>      """
>      Sets up and runs iperf3 traffic.
> @@ -53,9 +61,9 @@ from lib.py import ksft_pr, cmd, ip, rand_port, wait_port_listen
>              kwargs["timeout"] = duration + 5
>          return cmd(cmdline, **kwargs)
>  
> -    def measure_bandwidth(self, reverse=False):
> +    def measure(self, reverse=False):
>          """
> -        Runs an iperf3 measurement and returns the average bandwidth (Gbps).
> +        Runs an iperf3 measurement and returns an Iperf3Result.
>          Discards the first and last few reporting intervals and uses only the
>          middle part of the run where throughput is typically stable.
>          """
> @@ -78,7 +86,13 @@ from lib.py import ksft_pr, cmd, ip, rand_port, wait_port_listen
>  
>          avg = sum(stable) / len(stable)
>  
> -        return avg
> +        return Iperf3Result(avg)
> +
> +    def measure_bandwidth(self, reverse=False):
> +        """
> +        Runs an iperf3 measurement and returns the average bandwidth (Gbps).
> +        """
> +        return self.measure(reverse=reverse).throughput
>  
>  
>  class GenerateTraffic:
> diff --git a/tools/testing/selftests/drivers/net/perf/Makefile b/tools/testing/selftests/drivers/net/perf/Makefile
> new file mode 100644
> index 000000000000..31b0b60de4f6
> --- /dev/null
> +++ b/tools/testing/selftests/drivers/net/perf/Makefile
> @@ -0,0 +1,11 @@
> +# SPDX-License-Identifier: GPL-2.0
> +
> +TEST_PROGS := \
> +	iperf.py \
> +# end of TEST_PROGS
> +
> +TEST_INCLUDES := \
> +	$(wildcard lib/py/*.py ../lib/py/*.py) \
> +# end of TEST_INCLUDES
> +
> +include ../../../lib.mk
> diff --git a/tools/testing/selftests/drivers/net/perf/config b/tools/testing/selftests/drivers/net/perf/config
> new file mode 100644
> index 000000000000..6450f34933e6
> --- /dev/null
> +++ b/tools/testing/selftests/drivers/net/perf/config
> @@ -0,0 +1 @@
> +CONFIG_IPV6=y
> diff --git a/tools/testing/selftests/drivers/net/perf/iperf.py b/tools/testing/selftests/drivers/net/perf/iperf.py
> new file mode 100755
> index 000000000000..5b6f0bce2a03
> --- /dev/null
> +++ b/tools/testing/selftests/drivers/net/perf/iperf.py
> @@ -0,0 +1,23 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: GPL-2.0
> +
> +"""Measure TCP throughput with iperf3."""
> +
> +from lib.py import Iperf3Runner, NetDrvEpEnv, ksft_exit, ksft_metric, ksft_run
> +
> +
> +def tcp(cfg):
> +    """Measure TCP throughput."""
> +    result = Iperf3Runner(cfg).measure()
> +    ksft_metric("throughput", result.throughput, goal="higher")
> +
> +
> +def main():
> +    """Run the iperf performance test."""
> +    with NetDrvEpEnv(__file__, nsim_test=False) as cfg:
> +        ksft_run([tcp], args=(cfg,))
> +    ksft_exit()
> +
> +
> +if __name__ == "__main__":
> +    main()
> diff --git a/tools/testing/selftests/drivers/net/perf/lib/py/__init__.py b/tools/testing/selftests/drivers/net/perf/lib/py/__init__.py
> new file mode 100644
> index 000000000000..2bb2cd08ff7d
> --- /dev/null
> +++ b/tools/testing/selftests/drivers/net/perf/lib/py/__init__.py
> @@ -0,0 +1,57 @@
> +# SPDX-License-Identifier: GPL-2.0
> +
> +"""
> +Driver test environment (performance tests).
> +NetDrvEnv and NetDrvEpEnv are the main environment classes.
> +NetDrvContEnv extends NetDrvEpEnv with netkit container support.
> +Former is for local host only tests, latter creates / connects
> +to a remote endpoint. See NIPA wiki for more information about
> +running and writing driver tests.
> +"""
> +
> +import sys
> +from pathlib import Path
> +
> +KSFT_DIR = (Path(__file__).parent / "../../../../..").resolve()
> +
> +try:
> +    sys.path.append(KSFT_DIR.as_posix())
> +
> +    # Import one by one to avoid pylint false positives
> +    from net.lib.py import NetNS, NetNSEnter, NetdevSimDev, UserNetNS
> +    from net.lib.py import EthtoolFamily, NetdevFamily, NetshaperFamily, \
> +        NlError, RtnlFamily, DevlinkFamily, PSPFamily, Netlink
> +    from net.lib.py import CmdExitFailure
> +    from net.lib.py import bkg, cmd, bpftool, bpftrace, defer, ethtool, \
> +        fd_read_timeout, ip, rand_port, rand_ports, wait_port_listen, \
> +        wait_file, ctl_file_write, tool
> +    from net.lib.py import bpf_map_set, bpf_map_dump, bpf_prog_map_ids
> +    from net.lib.py import KsftSkipEx, KsftFailEx, KsftXfailEx
> +    from net.lib.py import ksft_disruptive, ksft_exit, ksft_metric, ksft_pr, \
> +        ksft_run, ksft_setup, ksft_variants, KsftNamedVariant
> +    from net.lib.py import ksft_eq, ksft_ge, ksft_in, ksft_is, ksft_lt, \
> +        ksft_ne, ksft_not_in, ksft_raises, ksft_true, ksft_gt, ksft_not_none
> +    from drivers.net.lib.py import set_ethtool_feat
> +    from drivers.net.lib.py import GenerateTraffic, Remote, Iperf3Runner
> +    from drivers.net.lib.py import NetDrvEnv, NetDrvEpEnv, NetDrvContEnv
> +
> +    __all__ = ["NetNS", "NetNSEnter", "NetdevSimDev", "UserNetNS",
> +               "EthtoolFamily", "NetdevFamily", "NetshaperFamily",
> +               "NlError", "RtnlFamily", "DevlinkFamily", "PSPFamily", "Netlink",
> +               "CmdExitFailure",
> +               "bkg", "cmd", "bpftool", "bpftrace", "defer", "ethtool",
> +               "fd_read_timeout", "ip", "rand_port", "rand_ports",
> +               "wait_port_listen", "wait_file", "ctl_file_write", "tool",
> +               "bpf_map_set", "bpf_map_dump", "bpf_prog_map_ids",
> +               "KsftSkipEx", "KsftFailEx", "KsftXfailEx",
> +               "ksft_disruptive", "ksft_exit", "ksft_metric", "ksft_pr",
> +               "ksft_run", "ksft_setup", "ksft_variants", "KsftNamedVariant",
> +               "ksft_eq", "ksft_ge", "ksft_in", "ksft_is", "ksft_lt",
> +               "ksft_ne", "ksft_not_in", "ksft_raises", "ksft_true", "ksft_gt",
> +               "ksft_not_none",
> +               "NetDrvEnv", "NetDrvEpEnv", "NetDrvContEnv", "GenerateTraffic",
> +               "Remote", "Iperf3Runner", "set_ethtool_feat"]
> +except ModuleNotFoundError as e:
> +    print("Failed importing `net` library from kernel sources")
> +    print(str(e))
> +    sys.exit(4)

nit: is there a KSFT_SKIP symbol?

Reviewed-by: Björn Töpel <bjorn@kernel.org>

Re: [PATCH net-next v3 3/4] selftests: drv-net: add a kperf runner

[Björn Töpel]

Stanislav Fomichev <sdf.kernel@gmail.com> writes:

> Add a helper to manage kperf servers, run the client, and return the
> aggregate throughput of both ends. Use a random control port and a unique
> PID file per server, as the hosts may share /tmp, and report server cleanup
> failures. Stopping the servers waits for their sessions to exit, as they
> may still be releasing devmem and RX steering, which the next test needs. A
> server which can log does so to a file next to its PID file, and the end of
> both logs is printed when the client fails, as the client can only tell
> that a server went away.
>
> Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>

Reviewed-by: Björn Töpel <bjorn@kernel.org>

Re: [PATCH net-next v3 4/4] selftests: drv-net: measure devmem performance with kperf

[Björn Töpel]

Stanislav Fomichev <sdf.kernel@gmail.com> writes:

> Add a separate udmabuf-backed devmem performance test using KperfRunner.
> Keep correctness cases in drivers/net/hw. Skip the test when kperf or
> /dev/udmabuf is missing, and fail it when the kperf run fails.
>
> Example metric output:
>
>   # # ktap-metric-json: {"goal":"higher","name":"throughput","value":389.127}
>
> Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>

Reviewed-by: Björn Töpel <bjorn@kernel.org>

Re: [PATCH net-next v3 2/4] selftests: drv-net: add an iperf performance test

[Stanislav Fomichev]

On 11/03, Björn Töpel wrote:
> Stanislav Fomichev <sdf.kernel@gmail.com> writes:
> 
> > Add the drivers/net/perf target, with a lib/py of its own like the one of
> > drivers/net/hw. Add Iperf3Runner.measure(), which returns an Iperf3Result
> > that other results can join later, and report its throughput from the test.
> >
> > Single-stream iperf3 throughput is flaky and can change considerably
> > between back-to-back runs on the same hosts.
> >
> > Tested on two mlx5 hosts. Captured metric output:
> >
> >   # # ktap-metric-json: {"goal":"higher","name":"throughput","value":49.92071602072054}
> >
> > Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
> > ---
> >  tools/testing/selftests/Makefile              |  2 +-
> >  .../selftests/drivers/net/lib/py/load.py      | 20 ++++++-
> >  .../selftests/drivers/net/perf/Makefile       | 11 ++++
> >  .../testing/selftests/drivers/net/perf/config |  1 +
> >  .../selftests/drivers/net/perf/iperf.py       | 23 ++++++++
> >  .../drivers/net/perf/lib/py/__init__.py       | 57 +++++++++++++++++++
> >  .../selftests/drivers/net/perf/settings       |  1 +
> >  7 files changed, 111 insertions(+), 4 deletions(-)
> >  create mode 100644 tools/testing/selftests/drivers/net/perf/Makefile
> >  create mode 100644 tools/testing/selftests/drivers/net/perf/config
> >  create mode 100755 tools/testing/selftests/drivers/net/perf/iperf.py
> >  create mode 100644 tools/testing/selftests/drivers/net/perf/lib/py/__init__.py
> >  create mode 100644 tools/testing/selftests/drivers/net/perf/settings
> >
> > diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
> > index 07c3719f8b04..c55d7792b27c 100644
> > --- a/tools/testing/selftests/Makefile
> > +++ b/tools/testing/selftests/Makefile
> > @@ -159,7 +159,7 @@ TARGETS_HOTPLUG = cpu-hotplug
> >  TARGETS_HOTPLUG += memory-hotplug
> >  
> >  # Networking tests want the net/lib target, include it automatically
> > -ifneq ($(filter net drivers/net drivers/net/hw,$(TARGETS)),)
> > +ifneq ($(filter net drivers/net drivers/net/hw drivers/net/perf,$(TARGETS)),)
> >  ifeq ($(filter net/lib,$(TARGETS)),)
> >  	INSTALL_DEP_TARGETS := net/lib
> >  endif
> > diff --git a/tools/testing/selftests/drivers/net/lib/py/load.py b/tools/testing/selftests/drivers/net/lib/py/load.py
> > index e24660e5c27f..f337b576c4b4 100644
> > --- a/tools/testing/selftests/drivers/net/lib/py/load.py
> > +++ b/tools/testing/selftests/drivers/net/lib/py/load.py
> > @@ -3,10 +3,18 @@
> >  import re
> >  import time
> >  import json
> > +from dataclasses import dataclass
> >  
> >  from lib.py import ksft_pr, cmd, ip, rand_port, wait_port_listen
> >  
> >  
> > +@dataclass
> > +class Iperf3Result:
> > +    """Average throughput (Gbps) of an iperf3 run."""
> > +
> > +    throughput: float
> > +
> > +
> >  class Iperf3Runner:
> >      """
> >      Sets up and runs iperf3 traffic.
> > @@ -53,9 +61,9 @@ from lib.py import ksft_pr, cmd, ip, rand_port, wait_port_listen
> >              kwargs["timeout"] = duration + 5
> >          return cmd(cmdline, **kwargs)
> >  
> > -    def measure_bandwidth(self, reverse=False):
> > +    def measure(self, reverse=False):
> >          """
> > -        Runs an iperf3 measurement and returns the average bandwidth (Gbps).
> > +        Runs an iperf3 measurement and returns an Iperf3Result.
> >          Discards the first and last few reporting intervals and uses only the
> >          middle part of the run where throughput is typically stable.
> >          """
> > @@ -78,7 +86,13 @@ from lib.py import ksft_pr, cmd, ip, rand_port, wait_port_listen
> >  
> >          avg = sum(stable) / len(stable)
> >  
> > -        return avg
> > +        return Iperf3Result(avg)
> > +
> > +    def measure_bandwidth(self, reverse=False):
> > +        """
> > +        Runs an iperf3 measurement and returns the average bandwidth (Gbps).
> > +        """
> > +        return self.measure(reverse=reverse).throughput
> >  
> >  
> >  class GenerateTraffic:
> > diff --git a/tools/testing/selftests/drivers/net/perf/Makefile b/tools/testing/selftests/drivers/net/perf/Makefile
> > new file mode 100644
> > index 000000000000..31b0b60de4f6
> > --- /dev/null
> > +++ b/tools/testing/selftests/drivers/net/perf/Makefile
> > @@ -0,0 +1,11 @@
> > +# SPDX-License-Identifier: GPL-2.0
> > +
> > +TEST_PROGS := \
> > +	iperf.py \
> > +# end of TEST_PROGS
> > +
> > +TEST_INCLUDES := \
> > +	$(wildcard lib/py/*.py ../lib/py/*.py) \
> > +# end of TEST_INCLUDES
> > +
> > +include ../../../lib.mk
> > diff --git a/tools/testing/selftests/drivers/net/perf/config b/tools/testing/selftests/drivers/net/perf/config
> > new file mode 100644
> > index 000000000000..6450f34933e6
> > --- /dev/null
> > +++ b/tools/testing/selftests/drivers/net/perf/config
> > @@ -0,0 +1 @@
> > +CONFIG_IPV6=y
> > diff --git a/tools/testing/selftests/drivers/net/perf/iperf.py b/tools/testing/selftests/drivers/net/perf/iperf.py
> > new file mode 100755
> > index 000000000000..5b6f0bce2a03
> > --- /dev/null
> > +++ b/tools/testing/selftests/drivers/net/perf/iperf.py
> > @@ -0,0 +1,23 @@
> > +#!/usr/bin/env python3
> > +# SPDX-License-Identifier: GPL-2.0
> > +
> > +"""Measure TCP throughput with iperf3."""
> > +
> > +from lib.py import Iperf3Runner, NetDrvEpEnv, ksft_exit, ksft_metric, ksft_run
> > +
> > +
> > +def tcp(cfg):
> > +    """Measure TCP throughput."""
> > +    result = Iperf3Runner(cfg).measure()
> > +    ksft_metric("throughput", result.throughput, goal="higher")
> > +
> > +
> > +def main():
> > +    """Run the iperf performance test."""
> > +    with NetDrvEpEnv(__file__, nsim_test=False) as cfg:
> > +        ksft_run([tcp], args=(cfg,))
> > +    ksft_exit()
> > +
> > +
> > +if __name__ == "__main__":
> > +    main()
> > diff --git a/tools/testing/selftests/drivers/net/perf/lib/py/__init__.py b/tools/testing/selftests/drivers/net/perf/lib/py/__init__.py
> > new file mode 100644
> > index 000000000000..2bb2cd08ff7d
> > --- /dev/null
> > +++ b/tools/testing/selftests/drivers/net/perf/lib/py/__init__.py
> > @@ -0,0 +1,57 @@
> > +# SPDX-License-Identifier: GPL-2.0
> > +
> > +"""
> > +Driver test environment (performance tests).
> > +NetDrvEnv and NetDrvEpEnv are the main environment classes.
> > +NetDrvContEnv extends NetDrvEpEnv with netkit container support.
> > +Former is for local host only tests, latter creates / connects
> > +to a remote endpoint. See NIPA wiki for more information about
> > +running and writing driver tests.
> > +"""
> > +
> > +import sys
> > +from pathlib import Path
> > +
> > +KSFT_DIR = (Path(__file__).parent / "../../../../..").resolve()
> > +
> > +try:
> > +    sys.path.append(KSFT_DIR.as_posix())
> > +
> > +    # Import one by one to avoid pylint false positives
> > +    from net.lib.py import NetNS, NetNSEnter, NetdevSimDev, UserNetNS
> > +    from net.lib.py import EthtoolFamily, NetdevFamily, NetshaperFamily, \
> > +        NlError, RtnlFamily, DevlinkFamily, PSPFamily, Netlink
> > +    from net.lib.py import CmdExitFailure
> > +    from net.lib.py import bkg, cmd, bpftool, bpftrace, defer, ethtool, \
> > +        fd_read_timeout, ip, rand_port, rand_ports, wait_port_listen, \
> > +        wait_file, ctl_file_write, tool
> > +    from net.lib.py import bpf_map_set, bpf_map_dump, bpf_prog_map_ids
> > +    from net.lib.py import KsftSkipEx, KsftFailEx, KsftXfailEx
> > +    from net.lib.py import ksft_disruptive, ksft_exit, ksft_metric, ksft_pr, \
> > +        ksft_run, ksft_setup, ksft_variants, KsftNamedVariant
> > +    from net.lib.py import ksft_eq, ksft_ge, ksft_in, ksft_is, ksft_lt, \
> > +        ksft_ne, ksft_not_in, ksft_raises, ksft_true, ksft_gt, ksft_not_none
> > +    from drivers.net.lib.py import set_ethtool_feat
> > +    from drivers.net.lib.py import GenerateTraffic, Remote, Iperf3Runner
> > +    from drivers.net.lib.py import NetDrvEnv, NetDrvEpEnv, NetDrvContEnv
> > +
> > +    __all__ = ["NetNS", "NetNSEnter", "NetdevSimDev", "UserNetNS",
> > +               "EthtoolFamily", "NetdevFamily", "NetshaperFamily",
> > +               "NlError", "RtnlFamily", "DevlinkFamily", "PSPFamily", "Netlink",
> > +               "CmdExitFailure",
> > +               "bkg", "cmd", "bpftool", "bpftrace", "defer", "ethtool",
> > +               "fd_read_timeout", "ip", "rand_port", "rand_ports",
> > +               "wait_port_listen", "wait_file", "ctl_file_write", "tool",
> > +               "bpf_map_set", "bpf_map_dump", "bpf_prog_map_ids",
> > +               "KsftSkipEx", "KsftFailEx", "KsftXfailEx",
> > +               "ksft_disruptive", "ksft_exit", "ksft_metric", "ksft_pr",
> > +               "ksft_run", "ksft_setup", "ksft_variants", "KsftNamedVariant",
> > +               "ksft_eq", "ksft_ge", "ksft_in", "ksft_is", "ksft_lt",
> > +               "ksft_ne", "ksft_not_in", "ksft_raises", "ksft_true", "ksft_gt",
> > +               "ksft_not_none",
> > +               "NetDrvEnv", "NetDrvEpEnv", "NetDrvContEnv", "GenerateTraffic",
> > +               "Remote", "Iperf3Runner", "set_ethtool_feat"]
> > +except ModuleNotFoundError as e:
> > +    print("Failed importing `net` library from kernel sources")
> > +    print(str(e))
> > +    sys.exit(4)
> 
> nit: is there a KSFT_SKIP symbol?
> 
> Reviewed-by: Björn Töpel <bjorn@kernel.org>

It is a copy-paste from selftests/drivers/net/hw/lib/py/__init__.py (and there
is a similar code in net/lib/py/ynl.py). I guess I can add "# KSFT_SKIP"
here instead of doing the real import.