[PATCH net-next 0/3] selftests: Use a master ssh connection for remote commands

[PATCH net-next 0/3] selftests: Use a master ssh connection for remote commands

[Cosmin Ratiu]

This series switches selftests to run remote ssh commands over an ssh
master connection, significantly speeding up test execution in most
environments by avoiding the need to repeatedly authenticate.

The first patch fixes an issue with stdin inheritance, the 2nd patch
wraps some remote command executions in PTYs to avoid a termination
issue and the 3rd patch finally converts remote command execution to use
an ssh master connection. For details, please see individual patches.

Cosmin Ratiu (3):
  selftests: Disable stdin for remote commands over ssh
  selftests: Wrap remote command executions in PTYs
  selftests: Use a master ssh connection for remote commands

 .../drivers/net/lib/py/remote_netns.py        |  2 +-
 .../drivers/net/lib/py/remote_ssh.py          | 52 ++++++++++++++++---
 tools/testing/selftests/net/lib/py/utils.py   |  4 +-
 3 files changed, 50 insertions(+), 8 deletions(-)

-- 
2.53.0

[PATCH net-next 1/3] selftests: Disable stdin for remote commands over ssh

[Cosmin Ratiu]

Local ssh clients executing remote commands get their stdin inherited
from the test process, but stdin isn't needed or useful, as remote
commands shouldn't read stdin in any case.

Explicitly disable stdin for remote ssh commands to avoid executing
remote commands which hang on stdin input.

Signed-off-by: Cosmin Ratiu <cratiu@nvidia.com>
---
 tools/testing/selftests/drivers/net/lib/py/remote_ssh.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tools/testing/selftests/drivers/net/lib/py/remote_ssh.py b/tools/testing/selftests/drivers/net/lib/py/remote_ssh.py
index 924addde19a3..b8c042689d20 100644
--- a/tools/testing/selftests/drivers/net/lib/py/remote_ssh.py
+++ b/tools/testing/selftests/drivers/net/lib/py/remote_ssh.py
@@ -21,6 +21,7 @@ class Remote:
 
     def cmd(self, comm):
         return subprocess.Popen(["ssh", "-q", self.name, comm],
+                                stdin=subprocess.DEVNULL,
                                 stdout=subprocess.PIPE, stderr=subprocess.PIPE)
 
     def _mktmp(self):
-- 
2.53.0

[PATCH net-next 2/3] selftests: Wrap remote command executions in PTYs

[Cosmin Ratiu]

Remote commands (executed via ssh) left running in background are killed
with a Popen.terminate(), which kills the local ssh client, terminates
the ssh connection, and eventually the remote side cleans up the forked
remote process.

The next patch makes use of an ssh master connection to speed up test
execution. Terminating a remote command over the master connection
behaves differently: Popen.terminate() hangs while the remote sshd
politely asks the forked command to exit by closing its stdin. Most
backgrounded commands (e.g. iperf3) don't care about stdin and never
exit, hanging until the timeout expires.

In order to make that work, use pseudo-terminals for remote commands
which are backgrounded. Then, terminating the local ssh client using the
master connection results in the same result as today: sshd closes the
master PTY fd and the remote process receives a SIGHUP, which kills it.

This is done by extending the cmd() interface for both remote types with
a 'pty' argument (needed but does nothing for netns). When True, it
results in '-tt' being passed to the ssh client, which forces a PTY to
be used on the remove end.

Using PTYs results in 3 changes. none of which matter in practice:
1. PTYs merge stdout and stderr. None of the tests expect things in
   stderr, it's only printed alongside stdout when things go south.
2. PTY line discipline uses \r\n\ instead of \n. Nothing in selftests
   matches on full lines including line endings.
3. There are only /proc/sys/kernel/pty/max PTYs on a system, default
   4096. This limits the number of concurrent remote commands being
   executed on a single machine from selftests, but the limit is
   probably unreachable in practice.

So this change should not result in any behavior change for any remote
command execution, but the next patch needs this to switch to using a
master ssh connection.

Signed-off-by: Cosmin Ratiu <cratiu@nvidia.com>
---
 tools/testing/selftests/drivers/net/lib/py/remote_netns.py | 2 +-
 tools/testing/selftests/drivers/net/lib/py/remote_ssh.py   | 7 +++++--
 tools/testing/selftests/net/lib/py/utils.py                | 4 +++-
 3 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/tools/testing/selftests/drivers/net/lib/py/remote_netns.py b/tools/testing/selftests/drivers/net/lib/py/remote_netns.py
index 7d5eeb0271bc..e010b3137e41 100644
--- a/tools/testing/selftests/drivers/net/lib/py/remote_netns.py
+++ b/tools/testing/selftests/drivers/net/lib/py/remote_netns.py
@@ -11,7 +11,7 @@ class Remote:
         self.name = name
         self.dir_path = dir_path
 
-    def cmd(self, comm):
+    def cmd(self, comm, pty=False):  # pty arg needed but ignored, netns is local
         return subprocess.Popen(["ip", "netns", "exec", self.name, "bash", "-c", comm],
                                  stdout=subprocess.PIPE, stderr=subprocess.PIPE)
 
diff --git a/tools/testing/selftests/drivers/net/lib/py/remote_ssh.py b/tools/testing/selftests/drivers/net/lib/py/remote_ssh.py
index b8c042689d20..427082561d00 100644
--- a/tools/testing/selftests/drivers/net/lib/py/remote_ssh.py
+++ b/tools/testing/selftests/drivers/net/lib/py/remote_ssh.py
@@ -19,8 +19,11 @@ class Remote:
             cmd("rm -rf " + self._tmpdir, host=self)
             self._tmpdir = None
 
-    def cmd(self, comm):
-        return subprocess.Popen(["ssh", "-q", self.name, comm],
+    def cmd(self, comm, pty=False):
+        args = ["ssh", "-q"]
+        if pty:
+            args += ["-tt"]
+        return subprocess.Popen(args + [self.name, comm],
                                 stdin=subprocess.DEVNULL,
                                 stdout=subprocess.PIPE, stderr=subprocess.PIPE)
 
diff --git a/tools/testing/selftests/net/lib/py/utils.py b/tools/testing/selftests/net/lib/py/utils.py
index be9408a77168..7d400290aa0b 100644
--- a/tools/testing/selftests/net/lib/py/utils.py
+++ b/tools/testing/selftests/net/lib/py/utils.py
@@ -58,7 +58,9 @@ class cmd:
         self.comm = comm
 
         if host:
-            self.proc = host.cmd(comm)
+            # Remote commands get a PTY, which forces the remote process to
+            # exit in all circumstances when the local ssh client is terminated.
+            self.proc = host.cmd(comm, pty=background)
         else:
             # If user doesn't explicitly request shell try to avoid it.
             if shell is None and isinstance(comm, str) and ' ' in comm:
-- 
2.53.0

[PATCH net-next 3/3] selftests: Use a master ssh connection for remote commands

[Cosmin Ratiu]

Every remote test command done via remote_ssh has to exchange ssh keys
and authenticate, paying a (sometimes significant) setup cost. This cost
is compounded when several remote commands are run in sequence, and made
even worse when the remote configuration is usually restored on defer().

Fix that by using a master ssh connection per remote host, allowing
subsequent connections to skip these steps and be almost as fast as
running those commands locally. This will reduce infrastructure usage by
these tests because they will finish much faster.

This finally implements this part of
tools/testing/selftests/drivers/net/README.rst:
"Using persistent SSH connections is strongly encouraged to avoid
the latency of SSH connection setup on every command."

Significant improvements are seen in a b2b setup involving NIS, here's
an example with the ping.py test, with remote cmd execution instrumented
with timing info:

Before (one ssh connection per remote command):
  # ./tools/testing/selftests/drivers/net/ping.py
  [...]
  remote: 0.769s  ping -c 1 -W0.5 1.0.0.1
  remote: 0.790s  ping -s 65000 -c 1 -W0.5 1.0.0.1
  remote: 0.775s  command -v -- socat
  remote: 0.784s  echo dwlelyrvaimmzirvhlfrgxtjxpxjdwcxmzj...
  remote: 1.238s  cat /proc/net/tcp*
  remote: 0.761s  ping -c 1 -W0.5 1.0.0.1
  remote: 0.753s  ping -s 65000 -c 1 -W0.5 1.0.0.1
  remote: 0.787s  echo zmhdqnnwidygdmoylhpdimcfmtgubbvygkc...
  remote: 1.034s  cat /proc/net/tcp*
  ok 1 ping.test_default_v4
  [...]
  # Totals: pass:5 fail:0 xfail:0 xpass:0 skip:2 error:0
  # Total remote cmd time: 55.416s

After (a single master ssh connection):
  # ./tools/testing/selftests/drivers/net/ping.py
  [...]
  remote: 0.017s  ping -c 1 -W0.5 1.0.0.1
  remote: 0.017s  ping -s 65000 -c 1 -W0.5 1.0.0.1
  remote: 0.016s  command -v -- socat
  remote: 0.020s  echo njbgxukbquvxqxynzyelisajhilshglkqmv...
  remote: 0.022s  cat /proc/net/tcp*
  remote: 0.017s  ping -c 1 -W0.5 1.0.0.1
  remote: 0.017s  ping -s 65000 -c 1 -W0.5 1.0.0.1
  remote: 0.020s  echo rmjpqfbfwfdtzesiqpmeljpdipiopvcfrhs...
  remote: 0.022s  cat /proc/net/tcp*
  ok 1 ping.test_default_v4
  [...]
  # Totals: pass:5 fail:0 xfail:0 xpass:0 skip:2 error:0
  # Total remote cmd time: 2.563s

So a reduction in total time spent executing remote commands from 55s to
2.5s. Similar speedups should apply across all tests using remote_ssh.

Signed-off-by: Cosmin Ratiu <cratiu@nvidia.com>
---
 .../drivers/net/lib/py/remote_ssh.py          | 48 ++++++++++++++++---
 1 file changed, 42 insertions(+), 6 deletions(-)

diff --git a/tools/testing/selftests/drivers/net/lib/py/remote_ssh.py b/tools/testing/selftests/drivers/net/lib/py/remote_ssh.py
index 427082561d00..8f2e57ccbe2f 100644
--- a/tools/testing/selftests/drivers/net/lib/py/remote_ssh.py
+++ b/tools/testing/selftests/drivers/net/lib/py/remote_ssh.py
@@ -1,9 +1,11 @@
 # SPDX-License-Identifier: GPL-2.0
 
 import os
+import shutil
 import string
 import subprocess
 import random
+import tempfile
 
 from lib.py import cmd
 
@@ -13,17 +15,50 @@ class Remote:
         self.name = name
         self.dir_path = dir_path
         self._tmpdir = None
+        self._ctl_dir = tempfile.mkdtemp(prefix="ssh-ksft-")
+        self._ctl = os.path.join(self._ctl_dir, "ctl")
+        self._ssh_opts = ["-o", "BatchMode=yes",
+                          "-o", "ConnectTimeout=20",
+                          "-o", "ControlMaster=auto",
+                          "-o", f"ControlPath={self._ctl}",
+                          "-o", "ControlPersist=30"]
+        self._ssh_base = ["ssh", "-q"] + self._ssh_opts
+        # Open the persistent connection
+        self._master_started = False
+        try:
+            r = subprocess.run(self._ssh_base + ["-fNM", self.name],
+                               stdin=subprocess.DEVNULL,
+                               stdout=subprocess.DEVNULL,
+                               stderr=subprocess.PIPE, timeout=30)
+            if r.returncode:
+                raise OSError(f"SSH master to {name} failed: {r.stderr.decode().strip()}")
+            self._master_started = True
+        finally:
+            if not self._master_started:
+                shutil.rmtree(self._ctl_dir, ignore_errors=True)
 
     def __del__(self):
-        if self._tmpdir:
-            cmd("rm -rf " + self._tmpdir, host=self)
-            self._tmpdir = None
+        try:
+            if self._tmpdir:
+                cmd("rm -rf " + self._tmpdir, host=self)
+                self._tmpdir = None
+        except Exception:
+            pass
+        try:
+            if self._master_started:
+                self._master_started = False
+                subprocess.run(self._ssh_base + ["-O", "exit", self.name],
+                               stdout=subprocess.DEVNULL,
+                               stderr=subprocess.DEVNULL, timeout=10)
+        except Exception:
+            pass
+        shutil.rmtree(self._ctl_dir, ignore_errors=True)
 
     def cmd(self, comm, pty=False):
-        args = ["ssh", "-q"]
+        args = []
         if pty:
             args += ["-tt"]
-        return subprocess.Popen(args + [self.name, comm],
+        return subprocess.Popen(self._ssh_base + args + [self.name, comm],
                                 stdin=subprocess.DEVNULL,
                                 stdout=subprocess.PIPE, stderr=subprocess.PIPE)
 
@@ -39,5 +74,6 @@ class Remote:
         if not os.path.isabs(what):
             what = os.path.abspath(self.dir_path + "/" + what)
 
-        cmd(f"scp {what} {self.name}:{file_name}")
+        cmd(["scp", "-q"] + self._ssh_opts +
+             [what, f"{self.name}:{file_name}"])
         return file_name
-- 
2.53.0

Re: [PATCH net-next 3/3] selftests: Use a master ssh connection for remote commands

[Jakub Kicinski]

On Mon, 18 May 2026 12:36:53 +0300 Cosmin Ratiu wrote:
> This finally implements this part of
> tools/testing/selftests/drivers/net/README.rst:
> "Using persistent SSH connections is strongly encouraged to avoid
> the latency of SSH connection setup on every command."

As this part of the README indicates, the expectation was that this
part of the setup will be handled outside of the harness itself.

After all something also has to set up the keys to make sure that
the hosts can actually log into each other, right?

I'm a bit on the fence, some extra Review / support tags would be
useful, and it'd be useful to make the code in patch 3 less ugly.
Patches 1 and 2 make sense.