[RFC PATCH 0/7] Add script for real-time telemetry monitoring

[RFC PATCH 0/7] Add script for real-time telemetry monitoring

[Bruce Richardson]

TL;DR
------

For a  quick demo, apply patces, run e.g. testpmd and then in a separate
terminal run:

  ./usertools/dpdk-telemetry-watcher.py -d1T eth.tx

Output, updated once per second, will be traffic rate per port e.g.:

Connected to application: "dpdk-testpmd"
Time       /ethdev/stats,0.opackets /ethdev/stats,1.opackets        Total
16:29:12                  5,213,119                5,214,304   10,427,423


Fuller details
--------------

While we have the dpdk-telemetry.py CLI app for interactive querying of
telemetry on the commandline, and a telemetry exporter script for
sending telemetry to external tools for real-time monitoring, we don't
have an app that can print real-time stats for DPDK apps on the
terminal. This patchset adds such a script, developed with the help of
Github copilot to fill a need that I found in my testing. Submitting it
here in the hopes that others find it of use.

The script acts as a wrapper around the existing dpdk-telemetry.py
script, and pipes the commands to that script and reads the responses,
querying it once per second. It takes a number of flag parameters, such
as the ones above:
 - "-d" for delta values, i.e. PPS rather than total packets
 - "-1" for single-line output, i.e. no scrolling up the screen
 - "-T" to display a total column

Other flag parameters can be seen by looking at the help output.

Beyond the flags, the script also takes a number of positional
parameters, which refer to specific stats to display. These stats must
be numeric values, and should take the form of the telemetry command to
send, followed by a "." and the stat within the result which is to be
tracked. As above, a stat would be e.g. "/ethdev/stats,0.opackets",
where we send "/ethdev/stats,0" to telemetry and extract the "opackets"
part of the result.

However, specifying individual stats can be awkward, so some shortcuts
are provided too for the common case of monitoring ethernet ports. Any
positional arg starting with "eth" will be replaced by the set of
equivalent values for each port, e.g. "eth.imissed" will track the
imissed value on all ports in use in the app. The ipackets and opackets
values, as common metrics, are also available as shortened values as
just "rx" and "tx", so in the example above, "eth.tx" means to track the
opackets stat for every ethdev port.

Finally, the script also has reconnection support so you can leave it
running while you start and stop your application in another terminal.
The watcher will try and reconnect to a running instance every second.


Bruce Richardson (7):
  usertools: add new script to monitor telemetry on terminal
  usertools/telemetry-watcher: add displaying stats
  usertools/telemetry-watcher: add delta and timeout opts
  usertools/telemetry-watcher: add total and one-line opts
  usertools/telemetry-watcher: add thousands separator
  usertools/telemetry-watcher: add eth name shortcuts
  usertools/telemetry-watcher: support reconnection

 usertools/dpdk-telemetry-watcher.py | 429 ++++++++++++++++++++++++++++
 usertools/meson.build               |   1 +
 2 files changed, 430 insertions(+)
 create mode 100755 usertools/dpdk-telemetry-watcher.py

--
2.51.0

[RFC PATCH 1/7] usertools: add new script to monitor telemetry on terminal

[Bruce Richardson]

The dpdk-telemetry.py script is useful for getting telemetry
interactively, but sometimes we want to monitor stats over time, so add
a telemetry-watcher script to do so.

Start off such a script with basic arg processing, and connecting to
dpdk-telemetry as a subprocess, so that we can send-receive commands
from it.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 usertools/dpdk-telemetry-watcher.py | 200 ++++++++++++++++++++++++++++
 usertools/meson.build               |   1 +
 2 files changed, 201 insertions(+)
 create mode 100755 usertools/dpdk-telemetry-watcher.py

diff --git a/usertools/dpdk-telemetry-watcher.py b/usertools/dpdk-telemetry-watcher.py
new file mode 100755
index 0000000000..01c8683d33
--- /dev/null
+++ b/usertools/dpdk-telemetry-watcher.py
@@ -0,0 +1,200 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2025 Intel Corporation
+
+"""
+Script to monitor DPDK telemetry statistics on the command line.
+Wraps dpdk-telemetry.py to provide continuous monitoring capabilities.
+"""
+
+import argparse
+import subprocess
+import sys
+import os
+import shutil
+import errno
+import json
+
+
+def get_app_name(pid):
+    """return the app name for a given PID, for printing"""
+    proc_cmdline = os.path.join("/proc", str(pid), "cmdline")
+    try:
+        with open(proc_cmdline) as f:
+            argv0 = f.read(1024).split("\0")[0]
+            return os.path.basename(argv0)
+    except IOError as e:
+        # ignore file not found errors
+        if e.errno != errno.ENOENT:
+            raise
+    return None
+
+
+def find_telemetry_script():
+    """Find the dpdk-telemetry.py script in the script directory or PATH.
+
+    Returns:
+        str: Path to the dpdk-telemetry.py script
+
+    Exits:
+        If the script cannot be found
+    """
+    # First, try to find it in the same directory as this script
+    script_dir = os.path.dirname(os.path.abspath(__file__))
+    telemetry_script = os.path.join(script_dir, "dpdk-telemetry.py")
+
+    # If not found locally, check if it's in PATH
+    if not os.path.exists(telemetry_script):
+        telemetry_in_path = shutil.which("dpdk-telemetry.py")
+        if telemetry_in_path:
+            telemetry_script = telemetry_in_path
+        else:
+            print(
+                "Error: dpdk-telemetry.py not found in script directory or PATH",
+                file=sys.stderr,
+            )
+            sys.exit(1)
+
+    return telemetry_script
+
+
+def create_telemetry_process(telemetry_script, args_list):
+    """Create a subprocess for dpdk-telemetry.py with pipes.
+
+    Args:
+        telemetry_script: Path to the dpdk-telemetry.py script
+        args_list: List of arguments to pass to the script
+
+    Returns:
+        subprocess.Popen: Process handle with stdin/stdout/stderr pipes
+
+    Exits:
+        If the process cannot be created
+    """
+    # Build the command
+    cmd = [sys.executable, telemetry_script] + args_list
+
+    try:
+        process = subprocess.Popen(
+            cmd,
+            stdin=subprocess.PIPE,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+            text=True,
+            bufsize=1,  # Line buffered
+        )
+        return process
+    except FileNotFoundError:
+        print(f"Error: Python interpreter or script not found", file=sys.stderr)
+        sys.exit(1)
+    except Exception as e:
+        print(f"Error running dpdk-telemetry.py: {e}", file=sys.stderr)
+        sys.exit(1)
+
+
+def query_telemetry(process, command):
+    """Send a telemetry command and return the parsed JSON response.
+
+    Args:
+        process: The subprocess.Popen handle to the telemetry process
+        command: The telemetry command to send (e.g., "/info" or "/ethdev/stats,0")
+
+    Returns:
+        dict: The parsed JSON response with the command wrapper stripped,
+              or None if there was an error
+    """
+    # Send the command
+    process.stdin.write(f"{command}\n")
+    process.stdin.flush()
+
+    # Read the JSON response
+    response = process.stdout.readline()
+    try:
+        data = json.loads(response)
+        # When run non-interactively, the response is wrapped with the command
+        # e.g., {"/info": {"version": ..., "pid": ...}}
+        # or {"/ethdev/stats,0": {...}}
+        # The response should have exactly one key which is the command
+        if len(data) == 1:
+            # Extract the value, ignoring the key
+            return next(iter(data.values()))
+        else:
+            return data
+    except (json.JSONDecodeError, KeyError):
+        return None
+
+
+def print_connected_app(process):
+    """Query and print the name of the connected DPDK application.
+
+    Args:
+        process: The subprocess.Popen handle to the telemetry process
+    """
+    info = query_telemetry(process, "/info")
+    if info and "pid" in info:
+        app_name = get_app_name(info["pid"])
+        if app_name:
+            print(f'Connected to application: "{app_name}"')
+
+
+def main():
+    """Main function to parse arguments and run dpdk-telemetry.py with a pipe"""
+
+    # Parse command line arguments - matching dpdk-telemetry.py parameters
+    parser = argparse.ArgumentParser(
+        description="Monitor DPDK telemetry statistics on the command line"
+    )
+    parser.add_argument(
+        "-f",
+        "--file-prefix",
+        default="rte",
+        help="Provide file-prefix for DPDK runtime directory",
+    )
+    parser.add_argument(
+        "-i",
+        "--instance",
+        default=0,
+        type=int,
+        help="Provide instance number for DPDK application",
+    )
+    parser.add_argument(
+        "-l",
+        "--list",
+        action="store_true",
+        default=False,
+        help="List all possible file-prefixes and exit",
+    )
+
+    args = parser.parse_args()
+
+    # Find the dpdk-telemetry.py script
+    telemetry_script = find_telemetry_script()
+
+    # Build arguments list
+    args_list = ["-f", args.file_prefix, "-i", str(args.instance)]
+
+    if args.list:
+        args_list.append("-l")
+        # For --list, just run the command directly without pipes
+        cmd = [sys.executable, telemetry_script] + args_list
+        return subprocess.run(cmd).returncode
+
+    # Run dpdk-telemetry.py with pipes for stdin and stdout
+    process = create_telemetry_process(telemetry_script, args_list)
+
+    # Get and display the connected application name
+    print_connected_app(process)
+
+    # TODO: Add monitoring logic here
+
+    # Clean up
+    process.stdin.close()
+    process.stdout.close()
+    process.stderr.close()
+    process.wait()
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())
diff --git a/usertools/meson.build b/usertools/meson.build
index eb48e2f440..114d0a65b4 100644
--- a/usertools/meson.build
+++ b/usertools/meson.build
@@ -12,6 +12,7 @@ install_data([
             'dpdk-hugepages.py',
             'dpdk-rss-flows.py',
             'dpdk-telemetry-exporter.py',
+            'dpdk-telemetry-watcher.py',
         ],
         install_dir: 'bin')
 
-- 
2.51.0

[RFC PATCH 2/7] usertools/telemetry-watcher: add displaying stats

[Bruce Richardson]

Add support for printing out particular stats once every second. The
stats to be queried are given on the commandline, in the format of
<command>.<result-field>, for example /ethdev/stats,0.ipackets.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 usertools/dpdk-telemetry-watcher.py | 101 +++++++++++++++++++++++++++-
 1 file changed, 100 insertions(+), 1 deletion(-)

diff --git a/usertools/dpdk-telemetry-watcher.py b/usertools/dpdk-telemetry-watcher.py
index 01c8683d33..824c9a37be 100755
--- a/usertools/dpdk-telemetry-watcher.py
+++ b/usertools/dpdk-telemetry-watcher.py
@@ -14,6 +14,7 @@
 import shutil
 import errno
 import json
+import time
 
 
 def get_app_name(pid):
@@ -137,6 +138,91 @@ def print_connected_app(process):
             print(f'Connected to application: "{app_name}"')
 
 
+def validate_stats(process, stat_specs):
+    """Validate stat specifications and check that fields are numeric.
+
+    Args:
+        process: The subprocess.Popen handle to the telemetry process
+        stat_specs: List of stat specifications in format "command.field"
+
+    Returns:
+        List of tuples (spec, command, field) for valid specs, or None on error
+    """
+    parsed_specs = []
+    for spec in stat_specs:
+        # Parse the stat specification
+        if "." not in spec:
+            print(f"Error: Invalid stat specification '{spec}'", file=sys.stderr)
+            print(
+                "Expected format: 'command.field' (e.g., /ethdev/stats,0.ipackets)",
+                file=sys.stderr,
+            )
+            return None
+
+        command, field = spec.rsplit(".", 1)
+        if not command or not field:
+            print(f"Error: Invalid stat specification '{spec}'", file=sys.stderr)
+            print(
+                "Expected format: 'command.field' (e.g., /ethdev/stats,0.ipackets)",
+                file=sys.stderr,
+            )
+            return None
+
+        # Query the stat once to validate it exists and is numeric
+        data = query_telemetry(process, command)
+        if not isinstance(data, dict):
+            print(f"Error: Command '{command}' did not return a dictionary", file=sys.stderr)
+            return None
+        if field not in data:
+            print(f"Error: Field '{field}' not found in '{command}' response", file=sys.stderr)
+            return None
+        value = data[field]
+        if not isinstance(value, (int, float)):
+            print(
+                f"Error: Field '{field}' in '{command}' is not numeric (got {type(value).__name__})",
+                file=sys.stderr,
+            )
+            return None
+
+        parsed_specs.append((spec, command, field))
+
+    return parsed_specs
+
+
+def monitor_stats(process, stat_specs):
+    """Monitor and display statistics in columns.
+
+    Args:
+        process: The subprocess.Popen handle to the telemetry process
+        stat_specs: List of stat specifications in format "command.field"
+    """
+    # Validate all stat specifications
+    parsed_specs = validate_stats(process, stat_specs)
+    if not parsed_specs:
+        return
+
+    # Print header
+    header = "Time".ljust(10)
+    for spec, _, _ in parsed_specs:
+        header += spec.rjust(25)
+    print(header)
+
+    # Monitor loop - once per second
+    try:
+        while True:
+            timestamp = time.strftime("%H:%M:%S")
+            row = timestamp.ljust(10)
+
+            for spec, command, field in parsed_specs:
+                data = query_telemetry(process, command)
+                row += str(data[field]).rjust(25)
+
+            print(row)
+            time.sleep(1)
+    except KeyboardInterrupt:
+        print("\nMonitoring stopped")
+
+
 def main():
     """Main function to parse arguments and run dpdk-telemetry.py with a pipe"""
 
@@ -164,6 +250,11 @@ def main():
         default=False,
         help="List all possible file-prefixes and exit",
     )
+    parser.add_argument(
+        "stats",
+        nargs="*",
+        help="Statistics to monitor in format 'command.field' (e.g., /ethdev/stats,0.ipackets)",
+    )
 
     args = parser.parse_args()
 
@@ -179,13 +270,21 @@ def main():
         cmd = [sys.executable, telemetry_script] + args_list
         return subprocess.run(cmd).returncode
 
+    # Check if stats were provided
+    if not args.stats:
+        print("Error: No statistics to monitor specified", file=sys.stderr)
+        print("Usage: dpdk-telemetry-watcher.py [options] stat1 stat2 ...", file=sys.stderr)
+        print("Example: dpdk-telemetry-watcher.py /ethdev/stats,0.ipackets", file=sys.stderr)
+        return 1
+
     # Run dpdk-telemetry.py with pipes for stdin and stdout
     process = create_telemetry_process(telemetry_script, args_list)
 
     # Get and display the connected application name
     print_connected_app(process)
 
-    # TODO: Add monitoring logic here
+    # Monitor the requested statistics
+    monitor_stats(process, args.stats)
 
     # Clean up
     process.stdin.close()
-- 
2.51.0

[RFC PATCH 3/7] usertools/telemetry-watcher: add delta and timeout opts

[Bruce Richardson]

Add options to the script to time out and quit after a certain number of
seconds. Also, add an option to display delta instead of absolute
values, allowing easy display of e.g. PPS.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 usertools/dpdk-telemetry-watcher.py | 64 +++++++++++++++++++++--------
 1 file changed, 48 insertions(+), 16 deletions(-)

diff --git a/usertools/dpdk-telemetry-watcher.py b/usertools/dpdk-telemetry-watcher.py
index 824c9a37be..8af9af867c 100755
--- a/usertools/dpdk-telemetry-watcher.py
+++ b/usertools/dpdk-telemetry-watcher.py
@@ -146,9 +146,13 @@ def validate_stats(process, stat_specs):
         stat_specs: List of stat specifications in format "command.field"
 
     Returns:
-        List of tuples (spec, command, field) for valid specs, or None on error
+        Tuple of (parsed_specs, initial_values) where:
+            parsed_specs: List of tuples (spec, command, field) for valid specs
+            initial_values: List of initial values for each stat
+        Returns (None, None) on error
     """
     parsed_specs = []
+    initial_values = []
     for spec in stat_specs:
         # Parse the stat specification
         if "." not in spec:
@@ -157,7 +161,7 @@ def validate_stats(process, stat_specs):
                 "Expected format: 'command.field' (e.g., /ethdev/stats,0.ipackets)",
                 file=sys.stderr,
             )
-            return None
+            return None, None
 
         command, field = spec.rsplit(".", 1)
         if not command or not field:
@@ -166,38 +170,39 @@ def validate_stats(process, stat_specs):
                 "Expected format: 'command.field' (e.g., /ethdev/stats,0.ipackets)",
                 file=sys.stderr,
             )
-            return None
+            return None, None
 
         # Query the stat once to validate it exists and is numeric
         data = query_telemetry(process, command)
         if not isinstance(data, dict):
             print(f"Error: Command '{command}' did not return a dictionary", file=sys.stderr)
-            return None
+            return None, None
         if field not in data:
             print(f"Error: Field '{field}' not found in '{command}' response", file=sys.stderr)
-            return None
+            return None, None
         value = data[field]
         if not isinstance(value, (int, float)):
             print(
                 f"Error: Field '{field}' in '{command}' is not numeric (got {type(value).__name__})",
                 file=sys.stderr,
             )
-            return None
+            return None, None
 
         parsed_specs.append((spec, command, field))
+        initial_values.append(value)
 
-    return parsed_specs
+    return parsed_specs, initial_values
 
 
-def monitor_stats(process, stat_specs):
+def monitor_stats(process, args):
     """Monitor and display statistics in columns.
 
     Args:
         process: The subprocess.Popen handle to the telemetry process
-        stat_specs: List of stat specifications in format "command.field"
+        args: Parsed command line arguments
     """
-    # Validate all stat specifications
-    parsed_specs = validate_stats(process, stat_specs)
+    # Validate all stat specifications and get initial values
+    parsed_specs, prev_values = validate_stats(process, args.stats)
     if not parsed_specs:
         return
 
@@ -208,17 +213,30 @@ def monitor_stats(process, stat_specs):
     print(header)
 
     # Monitor loop - once per second
+    count = 0
     try:
-        while True:
+        while args.timeout is None or count < args.timeout:
+            time.sleep(1)
+            count += 1
+
             timestamp = time.strftime("%H:%M:%S")
             row = timestamp.ljust(10)
 
-            for spec, command, field in parsed_specs:
+            current_values = []
+            for i, (spec, command, field) in enumerate(parsed_specs):
                 data = query_telemetry(process, command)
-                row += str(data[field]).rjust(25)
+                current_value = data[field]
+                current_values.append(current_value)
+
+                if args.delta:
+                    display_value = current_value - prev_values[i]
+                else:
+                    display_value = current_value
+
+                row += str(display_value).rjust(25)
 
             print(row)
-            time.sleep(1)
+            prev_values = current_values
     except KeyboardInterrupt:
         print("\nMonitoring stopped")
 
@@ -250,6 +268,20 @@ def main():
         default=False,
         help="List all possible file-prefixes and exit",
     )
+    parser.add_argument(
+        "-t",
+        "--timeout",
+        type=int,
+        default=None,
+        help="Number of iterations to run before stopping (default: run indefinitely)",
+    )
+    parser.add_argument(
+        "-d",
+        "--delta",
+        action="store_true",
+        default=False,
+        help="Display delta values instead of absolute values",
+    )
     parser.add_argument(
         "stats",
         nargs="*",
@@ -284,7 +316,7 @@ def main():
     print_connected_app(process)
 
     # Monitor the requested statistics
-    monitor_stats(process, args.stats)
+    monitor_stats(process, args)
 
     # Clean up
     process.stdin.close()
-- 
2.51.0

[RFC PATCH 4/7] usertools/telemetry-watcher: add total and one-line opts

[Bruce Richardson]

Add options to the script to print out totals at the end of each line,
and to print each line replacing the previous one, saving the output
scrolling up the screen constantly.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 usertools/dpdk-telemetry-watcher.py | 27 ++++++++++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)

diff --git a/usertools/dpdk-telemetry-watcher.py b/usertools/dpdk-telemetry-watcher.py
index 8af9af867c..bdf1423dd3 100755
--- a/usertools/dpdk-telemetry-watcher.py
+++ b/usertools/dpdk-telemetry-watcher.py
@@ -210,10 +210,13 @@ def monitor_stats(process, args):
     header = "Time".ljust(10)
     for spec, _, _ in parsed_specs:
         header += spec.rjust(25)
+    if args.total:
+        header += "Total".rjust(25)
     print(header)
 
     # Monitor loop - once per second
     count = 0
+    line_ending = "\r" if args.single_line else "\n"
     try:
         while args.timeout is None or count < args.timeout:
             time.sleep(1)
@@ -223,6 +226,7 @@ def monitor_stats(process, args):
             row = timestamp.ljust(10)
 
             current_values = []
+            total = 0
             for i, (spec, command, field) in enumerate(parsed_specs):
                 data = query_telemetry(process, command)
                 current_value = data[field]
@@ -233,11 +237,17 @@ def monitor_stats(process, args):
                 else:
                     display_value = current_value
 
+                total += display_value
                 row += str(display_value).rjust(25)
 
-            print(row)
+            if args.total:
+                row += str(total).rjust(25)
+
+            print(row, end=line_ending, flush=True)
             prev_values = current_values
     except KeyboardInterrupt:
+        if args.single_line:
+            print()  # Add newline before exit message
         print("\nMonitoring stopped")
 
 
@@ -282,6 +292,21 @@ def main():
         default=False,
         help="Display delta values instead of absolute values",
     )
+    parser.add_argument(
+        "-T",
+        "--total",
+        action="store_true",
+        default=False,
+        help="Display a total column at the end of each row",
+    )
+    parser.add_argument(
+        "-1",
+        "--single-line",
+        action="store_true",
+        default=False,
+        dest="single_line",
+        help="Display output on a single line, replacing the previous output",
+    )
     parser.add_argument(
         "stats",
         nargs="*",
-- 
2.51.0

[RFC PATCH 5/7] usertools/telemetry-watcher: add thousands separator

[Bruce Richardson]

When printing large values of packets, generally in the millions, things
are far more readable with thousands and millions separators.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 usertools/dpdk-telemetry-watcher.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/usertools/dpdk-telemetry-watcher.py b/usertools/dpdk-telemetry-watcher.py
index bdf1423dd3..59ec4c85b6 100755
--- a/usertools/dpdk-telemetry-watcher.py
+++ b/usertools/dpdk-telemetry-watcher.py
@@ -15,6 +15,7 @@
 import errno
 import json
 import time
+import locale
 
 
 def get_app_name(pid):
@@ -238,10 +239,10 @@ def monitor_stats(process, args):
                     display_value = current_value
 
                 total += display_value
-                row += str(display_value).rjust(25)
+                row += f"{display_value:n}".rjust(25)
 
             if args.total:
-                row += str(total).rjust(25)
+                row += f"{total:n}".rjust(25)
 
             print(row, end=line_ending, flush=True)
             prev_values = current_values
@@ -254,6 +255,9 @@ def monitor_stats(process, args):
 def main():
     """Main function to parse arguments and run dpdk-telemetry.py with a pipe"""
 
+    # Set locale for number formatting
+    locale.setlocale(locale.LC_ALL, "")
+
     # Parse command line arguments - matching dpdk-telemetry.py parameters
     parser = argparse.ArgumentParser(
         description="Monitor DPDK telemetry statistics on the command line"
-- 
2.51.0

[RFC PATCH 6/7] usertools/telemetry-watcher: add eth name shortcuts

[Bruce Richardson]

Since an expected main use of the script is to monitor ethdev packet
stats, provide a shortened form of parameters to make it easier to
monitor all ports on the system. Any stat starting with "eth." is taken
not as a direct command, but instead as a shortcut for getting the stats
for all ports on the system. For example: eth.ibytes shows the byte
counts for all ports.

Beyond that, provide a shortcut for ipackets and opackets as just rx and
tx respectively. Therefore, to monitor the output rate of an app, one
can use "dpdk-telemetry-watcher -dT eth.tx"

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 usertools/dpdk-telemetry-watcher.py | 49 ++++++++++++++++++++++++++++-
 1 file changed, 48 insertions(+), 1 deletion(-)

diff --git a/usertools/dpdk-telemetry-watcher.py b/usertools/dpdk-telemetry-watcher.py
index 59ec4c85b6..6beb67d29a 100755
--- a/usertools/dpdk-telemetry-watcher.py
+++ b/usertools/dpdk-telemetry-watcher.py
@@ -139,6 +139,48 @@ def print_connected_app(process):
             print(f'Connected to application: "{app_name}"')
 
 
+def expand_shortcuts(process, stat_specs):
+    """Expand special shortcuts like eth.rx and eth.tx into actual stat specifications.
+
+    Args:
+        process: The subprocess.Popen handle to the telemetry process
+        stat_specs: List of stat specifications, possibly including shortcuts
+
+    Returns:
+        List of expanded stat specifications
+    """
+    expanded = []
+    for spec in stat_specs:
+        if not spec.startswith("eth."):
+            expanded.append(spec)
+            continue
+
+        # Extract the field name after "eth."
+        field = spec[4:]  # Remove "eth." prefix
+        if not field:
+            print(f"Error: Invalid shortcut '{spec}' - missing field name", file=sys.stderr)
+            return None
+
+        # Map common shortcuts to actual field names
+        field_map = {
+            "rx": "ipackets",
+            "tx": "opackets",
+        }
+        field = field_map.get(field, field)
+
+        # Get list of ethernet devices
+        port_list = query_telemetry(process, "/ethdev/list")
+        if not isinstance(port_list, list):
+            print(f"Error: Failed to get ethernet device list", file=sys.stderr)
+            return None
+
+        # Create stat specs for each port
+        for port in port_list:
+            expanded.append(f"/ethdev/stats,{port}.{field}")
+
+    return expanded
+
+
 def validate_stats(process, stat_specs):
     """Validate stat specifications and check that fields are numeric.
 
@@ -202,8 +244,13 @@ def monitor_stats(process, args):
         process: The subprocess.Popen handle to the telemetry process
         args: Parsed command line arguments
     """
+    # Expand any shortcuts like eth-rx, eth-tx
+    expanded_stats = expand_shortcuts(process, args.stats)
+    if not expanded_stats:
+        return
+
     # Validate all stat specifications and get initial values
-    parsed_specs, prev_values = validate_stats(process, args.stats)
+    parsed_specs, prev_values = validate_stats(process, expanded_stats)
     if not parsed_specs:
         return
 
-- 
2.51.0

[RFC PATCH 7/7] usertools/telemetry-watcher: support reconnection

[Bruce Richardson]

Allow the watcher binary to run even when there is no DPDK process
running. In that case, wait for a suitable process to start and begin
monitoring then. In case of disconnection, keep trying to reconnect and
resume once reconnection succeeds.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 usertools/dpdk-telemetry-watcher.py | 46 +++++++++++++++++++++--------
 1 file changed, 34 insertions(+), 12 deletions(-)

diff --git a/usertools/dpdk-telemetry-watcher.py b/usertools/dpdk-telemetry-watcher.py
index 6beb67d29a..e5f470e08c 100755
--- a/usertools/dpdk-telemetry-watcher.py
+++ b/usertools/dpdk-telemetry-watcher.py
@@ -85,6 +85,12 @@ def create_telemetry_process(telemetry_script, args_list):
             text=True,
             bufsize=1,  # Line buffered
         )
+
+        # Get and display the connected application name
+        if not print_connected_app(process):
+            return None
+        process.script = telemetry_script  # Store script path for reference
+        process.args = args_list  # Store args for reference
         return process
     except FileNotFoundError:
         print(f"Error: Python interpreter or script not found", file=sys.stderr)
@@ -102,15 +108,28 @@ def query_telemetry(process, command):
         command: The telemetry command to send (e.g., "/info" or "/ethdev/stats,0")
 
     Returns:
-        dict: The parsed JSON response with the command wrapper stripped,
+        (process, dict): The process handle, in case of reconnection, and the
+              parsed JSON response with the command wrapper stripped,
               or None if there was an error
     """
     # Send the command
     process.stdin.write(f"{command}\n")
     process.stdin.flush()
 
-    # Read the JSON response
+    # Read the JSON response, reconnecting if necessary
     response = process.stdout.readline()
+    while not response:
+        script = process.script
+        args_list = process.args
+        process = None
+        print("Application disconnected, retrying...", file=sys.stderr)
+        while not process:
+            time.sleep(1)
+            process = create_telemetry_process(script, args_list)
+        process.stdin.write(f"{command}\n")
+        process.stdin.flush()
+        response = process.stdout.readline()
+
     try:
         data = json.loads(response)
         # When run non-interactively, the response is wrapped with the command
@@ -119,11 +138,11 @@ def query_telemetry(process, command):
         # The response should have exactly one key which is the command
         if len(data) == 1:
             # Extract the value, ignoring the key
-            return next(iter(data.values()))
+            return (process, next(iter(data.values())))
         else:
-            return data
+            return (process, data)
     except (json.JSONDecodeError, KeyError):
-        return None
+        return (None, None)
 
 
 def print_connected_app(process):
@@ -132,11 +151,12 @@ def print_connected_app(process):
     Args:
         process: The subprocess.Popen handle to the telemetry process
     """
-    info = query_telemetry(process, "/info")
+    process, info = query_telemetry(process, "/info")
     if info and "pid" in info:
         app_name = get_app_name(info["pid"])
         if app_name:
             print(f'Connected to application: "{app_name}"')
+    return process
 
 
 def expand_shortcuts(process, stat_specs):
@@ -169,7 +189,7 @@ def expand_shortcuts(process, stat_specs):
         field = field_map.get(field, field)
 
         # Get list of ethernet devices
-        port_list = query_telemetry(process, "/ethdev/list")
+        process, port_list = query_telemetry(process, "/ethdev/list")
         if not isinstance(port_list, list):
             print(f"Error: Failed to get ethernet device list", file=sys.stderr)
             return None
@@ -216,7 +236,7 @@ def validate_stats(process, stat_specs):
             return None, None
 
         # Query the stat once to validate it exists and is numeric
-        data = query_telemetry(process, command)
+        process, data = query_telemetry(process, command)
         if not isinstance(data, dict):
             print(f"Error: Command '{command}' did not return a dictionary", file=sys.stderr)
             return None, None
@@ -276,7 +296,7 @@ def monitor_stats(process, args):
             current_values = []
             total = 0
             for i, (spec, command, field) in enumerate(parsed_specs):
-                data = query_telemetry(process, command)
+                process, data = query_telemetry(process, command)
                 current_value = data[field]
                 current_values.append(current_value)
 
@@ -387,9 +407,11 @@ def main():
 
     # Run dpdk-telemetry.py with pipes for stdin and stdout
     process = create_telemetry_process(telemetry_script, args_list)
-
-    # Get and display the connected application name
-    print_connected_app(process)
+    if not process:
+        print("Waiting for connection to DPDK application...", file=sys.stderr)
+        while not process:
+            time.sleep(1)
+            process = create_telemetry_process(telemetry_script, args_list)
 
     # Monitor the requested statistics
     monitor_stats(process, args)
-- 
2.51.0

Re: [RFC PATCH 0/7] Add script for real-time telemetry monitoring

[Stephen Hemminger]

On Wed, 10 Dec 2025 16:55:25 +0000
Bruce Richardson <bruce.richardson@intel.com> wrote:

> TL;DR
> ------
> 
> For a  quick demo, apply patces, run e.g. testpmd and then in a separate
> terminal run:
> 
>   ./usertools/dpdk-telemetry-watcher.py -d1T eth.tx
> 
> Output, updated once per second, will be traffic rate per port e.g.:
> 
> Connected to application: "dpdk-testpmd"
> Time       /ethdev/stats,0.opackets /ethdev/stats,1.opackets        Total
> 16:29:12                  5,213,119                5,214,304   10,427,423
> 
> 
> Fuller details
> --------------
> 
> While we have the dpdk-telemetry.py CLI app for interactive querying of
> telemetry on the commandline, and a telemetry exporter script for
> sending telemetry to external tools for real-time monitoring, we don't
> have an app that can print real-time stats for DPDK apps on the
> terminal. This patchset adds such a script, developed with the help of
> Github copilot to fill a need that I found in my testing. Submitting it
> here in the hopes that others find it of use.
> 
> The script acts as a wrapper around the existing dpdk-telemetry.py
> script, and pipes the commands to that script and reads the responses,
> querying it once per second. It takes a number of flag parameters, such
> as the ones above:
>  - "-d" for delta values, i.e. PPS rather than total packets
>  - "-1" for single-line output, i.e. no scrolling up the screen
>  - "-T" to display a total column
> 
> Other flag parameters can be seen by looking at the help output.
> 
> Beyond the flags, the script also takes a number of positional
> parameters, which refer to specific stats to display. These stats must
> be numeric values, and should take the form of the telemetry command to
> send, followed by a "." and the stat within the result which is to be
> tracked. As above, a stat would be e.g. "/ethdev/stats,0.opackets",
> where we send "/ethdev/stats,0" to telemetry and extract the "opackets"
> part of the result.
> 
> However, specifying individual stats can be awkward, so some shortcuts
> are provided too for the common case of monitoring ethernet ports. Any
> positional arg starting with "eth" will be replaced by the set of
> equivalent values for each port, e.g. "eth.imissed" will track the
> imissed value on all ports in use in the app. The ipackets and opackets
> values, as common metrics, are also available as shortened values as
> just "rx" and "tx", so in the example above, "eth.tx" means to track the
> opackets stat for every ethdev port.
> 
> Finally, the script also has reconnection support so you can leave it
> running while you start and stop your application in another terminal.
> The watcher will try and reconnect to a running instance every second.
> 


There was a previous submission of a nice telemetry TUI which seems to have
been abandoned, is there overlap with this?

Re: [RFC PATCH 0/7] Add script for real-time telemetry monitoring

[Bruce Richardson]

On Thu, Dec 11, 2025 at 10:09:10AM +0900, Stephen Hemminger wrote:
> On Wed, 10 Dec 2025 16:55:25 +0000
> Bruce Richardson <bruce.richardson@intel.com> wrote:
> 
> > TL;DR
> > ------
> > 
> > For a  quick demo, apply patces, run e.g. testpmd and then in a separate
> > terminal run:
> > 
> >   ./usertools/dpdk-telemetry-watcher.py -d1T eth.tx
> > 
> > Output, updated once per second, will be traffic rate per port e.g.:
> > 
> > Connected to application: "dpdk-testpmd"
> > Time       /ethdev/stats,0.opackets /ethdev/stats,1.opackets        Total
> > 16:29:12                  5,213,119                5,214,304   10,427,423
> > 
> > 
> > Fuller details
> > --------------
> > 
> > While we have the dpdk-telemetry.py CLI app for interactive querying of
> > telemetry on the commandline, and a telemetry exporter script for
> > sending telemetry to external tools for real-time monitoring, we don't
> > have an app that can print real-time stats for DPDK apps on the
> > terminal. This patchset adds such a script, developed with the help of
> > Github copilot to fill a need that I found in my testing. Submitting it
> > here in the hopes that others find it of use.
> > 
> > The script acts as a wrapper around the existing dpdk-telemetry.py
> > script, and pipes the commands to that script and reads the responses,
> > querying it once per second. It takes a number of flag parameters, such
> > as the ones above:
> >  - "-d" for delta values, i.e. PPS rather than total packets
> >  - "-1" for single-line output, i.e. no scrolling up the screen
> >  - "-T" to display a total column
> > 
> > Other flag parameters can be seen by looking at the help output.
> > 
> > Beyond the flags, the script also takes a number of positional
> > parameters, which refer to specific stats to display. These stats must
> > be numeric values, and should take the form of the telemetry command to
> > send, followed by a "." and the stat within the result which is to be
> > tracked. As above, a stat would be e.g. "/ethdev/stats,0.opackets",
> > where we send "/ethdev/stats,0" to telemetry and extract the "opackets"
> > part of the result.
> > 
> > However, specifying individual stats can be awkward, so some shortcuts
> > are provided too for the common case of monitoring ethernet ports. Any
> > positional arg starting with "eth" will be replaced by the set of
> > equivalent values for each port, e.g. "eth.imissed" will track the
> > imissed value on all ports in use in the app. The ipackets and opackets
> > values, as common metrics, are also available as shortened values as
> > just "rx" and "tx", so in the example above, "eth.tx" means to track the
> > opackets stat for every ethdev port.
> > 
> > Finally, the script also has reconnection support so you can leave it
> > running while you start and stop your application in another terminal.
> > The watcher will try and reconnect to a running instance every second.
> > 
> 
> 
> There was a previous submission of a nice telemetry TUI which seems to have
> been abandoned, is there overlap with this?
> 

Possibly. I didn't remember that being submitted in the past. This was just
something I came up with a while back for my own use, and find it really
handy for quickly getting some perf numbers. Therefore I thought others
might find it of use too.

/Bruce

Re: [RFC PATCH 0/7] Add script for real-time telemetry monitoring

[Stephen Hemminger]

On Wed, 10 Dec 2025 16:55:25 +0000
Bruce Richardson <bruce.richardson@intel.com> wrote:

> TL;DR
> ------
> 
> For a  quick demo, apply patces, run e.g. testpmd and then in a separate
> terminal run:
> 
>   ./usertools/dpdk-telemetry-watcher.py -d1T eth.tx
> 
> Output, updated once per second, will be traffic rate per port e.g.:
> 
> Connected to application: "dpdk-testpmd"
> Time       /ethdev/stats,0.opackets /ethdev/stats,1.opackets        Total
> 16:29:12                  5,213,119                5,214,304   10,427,423
> 
> 
> Fuller details
> --------------
> 
> While we have the dpdk-telemetry.py CLI app for interactive querying of
> telemetry on the commandline, and a telemetry exporter script for
> sending telemetry to external tools for real-time monitoring, we don't
> have an app that can print real-time stats for DPDK apps on the
> terminal. This patchset adds such a script, developed with the help of
> Github copilot to fill a need that I found in my testing. Submitting it
> here in the hopes that others find it of use.
> 
> The script acts as a wrapper around the existing dpdk-telemetry.py
> script, and pipes the commands to that script and reads the responses,
> querying it once per second. It takes a number of flag parameters, such
> as the ones above:
>  - "-d" for delta values, i.e. PPS rather than total packets
>  - "-1" for single-line output, i.e. no scrolling up the screen
>  - "-T" to display a total column
> 
> Other flag parameters can be seen by looking at the help output.
> 
> Beyond the flags, the script also takes a number of positional
> parameters, which refer to specific stats to display. These stats must
> be numeric values, and should take the form of the telemetry command to
> send, followed by a "." and the stat within the result which is to be
> tracked. As above, a stat would be e.g. "/ethdev/stats,0.opackets",
> where we send "/ethdev/stats,0" to telemetry and extract the "opackets"
> part of the result.
> 
> However, specifying individual stats can be awkward, so some shortcuts
> are provided too for the common case of monitoring ethernet ports. Any
> positional arg starting with "eth" will be replaced by the set of
> equivalent values for each port, e.g. "eth.imissed" will track the
> imissed value on all ports in use in the app. The ipackets and opackets
> values, as common metrics, are also available as shortened values as
> just "rx" and "tx", so in the example above, "eth.tx" means to track the
> opackets stat for every ethdev port.
> 
> Finally, the script also has reconnection support so you can leave it
> running while you start and stop your application in another terminal.
> The watcher will try and reconnect to a running instance every second.
> 
> 
> Bruce Richardson (7):
>   usertools: add new script to monitor telemetry on terminal
>   usertools/telemetry-watcher: add displaying stats
>   usertools/telemetry-watcher: add delta and timeout opts
>   usertools/telemetry-watcher: add total and one-line opts
>   usertools/telemetry-watcher: add thousands separator
>   usertools/telemetry-watcher: add eth name shortcuts
>   usertools/telemetry-watcher: support reconnection
> 
>  usertools/dpdk-telemetry-watcher.py | 429 ++++++++++++++++++++++++++++
>  usertools/meson.build               |   1 +
>  2 files changed, 430 insertions(+)
>  create mode 100755 usertools/dpdk-telemetry-watcher.py
> 
> --
> 2.51.0


The TUI patch was
https://patchwork.dpdk.org/project/dpdk/patch/20220831115250.362189-2-conor.walsh@intel.com/

Re: [RFC PATCH 0/7] Add script for real-time telemetry monitoring

[Bruce Richardson]

On Fri, Dec 12, 2025 at 02:32:49PM +0900, Stephen Hemminger wrote:
> On Wed, 10 Dec 2025 16:55:25 +0000
> Bruce Richardson <bruce.richardson@intel.com> wrote:
> 
> > TL;DR
> > ------
> > 
> > For a  quick demo, apply patces, run e.g. testpmd and then in a separate
> > terminal run:
> > 
> >   ./usertools/dpdk-telemetry-watcher.py -d1T eth.tx
> > 
> > Output, updated once per second, will be traffic rate per port e.g.:
> > 
> > Connected to application: "dpdk-testpmd"
> > Time       /ethdev/stats,0.opackets /ethdev/stats,1.opackets        Total
> > 16:29:12                  5,213,119                5,214,304   10,427,423
> > 
> > 
<snip>
> > 
> >  usertools/dpdk-telemetry-watcher.py | 429 ++++++++++++++++++++++++++++
> >  usertools/meson.build               |   1 +
> >  2 files changed, 430 insertions(+)
> >  create mode 100755 usertools/dpdk-telemetry-watcher.py
> > 
> > --
> > 2.51.0
> 
> 
> The TUI patch was
> https://patchwork.dpdk.org/project/dpdk/patch/20220831115250.362189-2-conor.walsh@intel.com/
>

Took a look at that patch proposal, and here are some of the main
differences between the two scripts:

* the telemetry-tui is a full-screen app which is a lot prettier and
  probably much more suitable for non-experts.
* the script I provided is FAR less user friendly and operates at a much
  more "techie" level
* on the other hand, it's more flexible because of that:
  - can be used to display any numeric stats - the tui is hard-coded to show
    particular NIC stats - including e.g. cryptodev, or dmadev ones
  - can show deltas and totals as desired
  - can quit after a certain fixed amount of time
* my script is probably more limited in terms of what can be displayed,
  since it assumes that you want one-line of output per update. TUI
  displays lots of (fixed)stats simultaneously
* TUI as proposed has a few glitches - got a divide by zero error when run
  against a testpmd instance that didn't have traffic running, so all stats
  were zero. (That's an easy fix though)
* The new script I propose has one major feature that I would look to add
  to the TUI if we merged that - the reconnection support. I find it very
  handy to run the watcher script and leave it running while I start and
  stop whatever app(s) I'm using.
* My script has no additional dependencies beyond stock python, while the
  TUI requires one additional dependency "rich" to manage the TUI. It also
  can use a second dependency "plotext" (not Ubuntu packaged) to display
  charts if so desired.

Just my quick assessment of the two from a quick try of the TUI one.

/Bruce

Re: [RFC PATCH 0/7] Add script for real-time telemetry monitoring

[Stephen Hemminger]

On Fri, 12 Dec 2025 17:52:15 +0000
Bruce Richardson <bruce.richardson@intel.com> wrote:

> On Fri, Dec 12, 2025 at 02:32:49PM +0900, Stephen Hemminger wrote:
> > On Wed, 10 Dec 2025 16:55:25 +0000
> > Bruce Richardson <bruce.richardson@intel.com> wrote:
> >   
> > > TL;DR
> > > ------
> > > 
> > > For a  quick demo, apply patces, run e.g. testpmd and then in a separate
> > > terminal run:
> > > 
> > >   ./usertools/dpdk-telemetry-watcher.py -d1T eth.tx
> > > 
> > > Output, updated once per second, will be traffic rate per port e.g.:
> > > 
> > > Connected to application: "dpdk-testpmd"
> > > Time       /ethdev/stats,0.opackets /ethdev/stats,1.opackets        Total
> > > 16:29:12                  5,213,119                5,214,304   10,427,423
> > > 
> > >   
> <snip>
> > > 
> > >  usertools/dpdk-telemetry-watcher.py | 429 ++++++++++++++++++++++++++++
> > >  usertools/meson.build               |   1 +
> > >  2 files changed, 430 insertions(+)
> > >  create mode 100755 usertools/dpdk-telemetry-watcher.py
> > > 
> > > --
> > > 2.51.0  
> > 
> > 
> > The TUI patch was
> > https://patchwork.dpdk.org/project/dpdk/patch/20220831115250.362189-2-conor.walsh@intel.com/
> >  
> 
> Took a look at that patch proposal, and here are some of the main
> differences between the two scripts:
> 
> * the telemetry-tui is a full-screen app which is a lot prettier and
>   probably much more suitable for non-experts.
> * the script I provided is FAR less user friendly and operates at a much
>   more "techie" level
> * on the other hand, it's more flexible because of that:
>   - can be used to display any numeric stats - the tui is hard-coded to show
>     particular NIC stats - including e.g. cryptodev, or dmadev ones
>   - can show deltas and totals as desired
>   - can quit after a certain fixed amount of time
> * my script is probably more limited in terms of what can be displayed,
>   since it assumes that you want one-line of output per update. TUI
>   displays lots of (fixed)stats simultaneously
> * TUI as proposed has a few glitches - got a divide by zero error when run
>   against a testpmd instance that didn't have traffic running, so all stats
>   were zero. (That's an easy fix though)
> * The new script I propose has one major feature that I would look to add
>   to the TUI if we merged that - the reconnection support. I find it very
>   handy to run the watcher script and leave it running while I start and
>   stop whatever app(s) I'm using.
> * My script has no additional dependencies beyond stock python, while the
>   TUI requires one additional dependency "rich" to manage the TUI. It also
>   can use a second dependency "plotext" (not Ubuntu packaged) to display
>   charts if so desired.
> 
> Just my quick assessment of the two from a quick try of the TUI one.
> 
> /Bruce

It is not a case of A or B. Both tools are useful.
I can see uses for both. But original submission of TUI seems to have
been a one off patch with no long term followup