[PATCH net-next 3/3] selftests: Use a master ssh connection for remote commands

[Cosmin Ratiu <cratiu@nvidia.com>]

Every remote test command done via remote_ssh has to exchange ssh keys
and authenticate, paying a (sometimes significant) setup cost. This cost
is compounded when several remote commands are run in sequence, and made
even worse when the remote configuration is usually restored on defer().

Fix that by using a master ssh connection per remote host, allowing
subsequent connections to skip these steps and be almost as fast as
running those commands locally. This will reduce infrastructure usage by
these tests because they will finish much faster.

This finally implements this part of
tools/testing/selftests/drivers/net/README.rst:
"Using persistent SSH connections is strongly encouraged to avoid
the latency of SSH connection setup on every command."

Significant improvements are seen in a b2b setup involving NIS, here's
an example with the ping.py test, with remote cmd execution instrumented
with timing info:

Before (one ssh connection per remote command):
  # ./tools/testing/selftests/drivers/net/ping.py
  [...]
  remote: 0.769s  ping -c 1 -W0.5 1.0.0.1
  remote: 0.790s  ping -s 65000 -c 1 -W0.5 1.0.0.1
  remote: 0.775s  command -v -- socat
  remote: 0.784s  echo dwlelyrvaimmzirvhlfrgxtjxpxjdwcxmzj...
  remote: 1.238s  cat /proc/net/tcp*
  remote: 0.761s  ping -c 1 -W0.5 1.0.0.1
  remote: 0.753s  ping -s 65000 -c 1 -W0.5 1.0.0.1
  remote: 0.787s  echo zmhdqnnwidygdmoylhpdimcfmtgubbvygkc...
  remote: 1.034s  cat /proc/net/tcp*
  ok 1 ping.test_default_v4
  [...]
  # Totals: pass:5 fail:0 xfail:0 xpass:0 skip:2 error:0
  # Total remote cmd time: 55.416s

After (a single master ssh connection):
  # ./tools/testing/selftests/drivers/net/ping.py
  [...]
  remote: 0.017s  ping -c 1 -W0.5 1.0.0.1
  remote: 0.017s  ping -s 65000 -c 1 -W0.5 1.0.0.1
  remote: 0.016s  command -v -- socat
  remote: 0.020s  echo njbgxukbquvxqxynzyelisajhilshglkqmv...
  remote: 0.022s  cat /proc/net/tcp*
  remote: 0.017s  ping -c 1 -W0.5 1.0.0.1
  remote: 0.017s  ping -s 65000 -c 1 -W0.5 1.0.0.1
  remote: 0.020s  echo rmjpqfbfwfdtzesiqpmeljpdipiopvcfrhs...
  remote: 0.022s  cat /proc/net/tcp*
  ok 1 ping.test_default_v4
  [...]
  # Totals: pass:5 fail:0 xfail:0 xpass:0 skip:2 error:0
  # Total remote cmd time: 2.563s

So a reduction in total time spent executing remote commands from 55s to
2.5s. Similar speedups should apply across all tests using remote_ssh.

Signed-off-by: Cosmin Ratiu <cratiu@nvidia.com>
---
 .../drivers/net/lib/py/remote_ssh.py          | 48 ++++++++++++++++---
 1 file changed, 42 insertions(+), 6 deletions(-)

diff --git a/tools/testing/selftests/drivers/net/lib/py/remote_ssh.py b/tools/testing/selftests/drivers/net/lib/py/remote_ssh.py
index 427082561d00..8f2e57ccbe2f 100644
--- a/tools/testing/selftests/drivers/net/lib/py/remote_ssh.py
+++ b/tools/testing/selftests/drivers/net/lib/py/remote_ssh.py
@@ -1,9 +1,11 @@
 # SPDX-License-Identifier: GPL-2.0
 
 import os
+import shutil
 import string
 import subprocess
 import random
+import tempfile
 
 from lib.py import cmd
 
@@ -13,17 +15,50 @@ class Remote:
         self.name = name
         self.dir_path = dir_path
         self._tmpdir = None
+        self._ctl_dir = tempfile.mkdtemp(prefix="ssh-ksft-")
+        self._ctl = os.path.join(self._ctl_dir, "ctl")
+        self._ssh_opts = ["-o", "BatchMode=yes",
+                          "-o", "ConnectTimeout=20",
+                          "-o", "ControlMaster=auto",
+                          "-o", f"ControlPath={self._ctl}",
+                          "-o", "ControlPersist=30"]
+        self._ssh_base = ["ssh", "-q"] + self._ssh_opts
+        # Open the persistent connection
+        self._master_started = False
+        try:
+            r = subprocess.run(self._ssh_base + ["-fNM", self.name],
+                               stdin=subprocess.DEVNULL,
+                               stdout=subprocess.DEVNULL,
+                               stderr=subprocess.PIPE, timeout=30)
+            if r.returncode:
+                raise OSError(f"SSH master to {name} failed: {r.stderr.decode().strip()}")
+            self._master_started = True
+        finally:
+            if not self._master_started:
+                shutil.rmtree(self._ctl_dir, ignore_errors=True)
 
     def __del__(self):
-        if self._tmpdir:
-            cmd("rm -rf " + self._tmpdir, host=self)
-            self._tmpdir = None
+        try:
+            if self._tmpdir:
+                cmd("rm -rf " + self._tmpdir, host=self)
+                self._tmpdir = None
+        except Exception:
+            pass
+        try:
+            if self._master_started:
+                self._master_started = False
+                subprocess.run(self._ssh_base + ["-O", "exit", self.name],
+                               stdout=subprocess.DEVNULL,
+                               stderr=subprocess.DEVNULL, timeout=10)
+        except Exception:
+            pass
+        shutil.rmtree(self._ctl_dir, ignore_errors=True)
 
     def cmd(self, comm, pty=False):
-        args = ["ssh", "-q"]
+        args = []
         if pty:
             args += ["-tt"]
-        return subprocess.Popen(args + [self.name, comm],
+        return subprocess.Popen(self._ssh_base + args + [self.name, comm],
                                 stdin=subprocess.DEVNULL,
                                 stdout=subprocess.PIPE, stderr=subprocess.PIPE)
 
@@ -39,5 +74,6 @@ class Remote:
         if not os.path.isabs(what):
             what = os.path.abspath(self.dir_path + "/" + what)
 
-        cmd(f"scp {what} {self.name}:{file_name}")
+        cmd(["scp", "-q"] + self._ssh_opts +
+             [what, f"{self.name}:{file_name}"])
         return file_name
-- 
2.53.0