[PATCH 0/4] oeqa: target: ssh: log SSH errors during tests

[PATCH 0/4] oeqa: target: ssh: log SSH errors during tests

[Mathieu Dubois-Briand]

Most tests running SSH commands ask for no error to be raised when the
returned status is not 0, and handle the returned status themselves. But
most of the tests do not check if the non-zero status is caused by a
fail of the command run on the target or by a fail of SSH itself.

This can lead to confusing test results, such as the one described in
bugzilla issue #15981.

This series introduce a check for SSH failures by default, only bypassed
in some specific cases.

Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
---
Mathieu Dubois-Briand (4):
      oeqa: target: ssh: Fail on SSH error even when errors are ignored
      oeqa: runtime: Ignore SSH errors during setup and tear down
      oeqa: postactions: Ignore SSH errors
      oeqa: runtime: ssh: Manage any SSH failure locally

 meta/lib/oeqa/core/target/ssh.py         | 12 ++++++++----
 meta/lib/oeqa/runtime/cases/logrotate.py |  7 ++++---
 meta/lib/oeqa/runtime/cases/ssh.py       |  2 +-
 meta/lib/oeqa/runtime/cases/weston.py    |  2 +-
 meta/lib/oeqa/utils/postactions.py       |  4 ++--
 5 files changed, 16 insertions(+), 11 deletions(-)
---
base-commit: 9dfd4f44d4f40e7926dc88cb564baa2345c2a24f
change-id: 20251007-mathieu-ssh-fails-0c01b1865511

Best regards,
-- 
Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>

[PATCH 1/4] oeqa: target: ssh: Fail on SSH error even when errors are ignored

[Mathieu Dubois-Briand]

Most tests running SSH commands ask for no error to be raised when the
returned status is not 0. As run() will return this status, they may
later use its value to do a similar check on their own, or completely
ignore it. But most of the tests do not check if the non-zero status is
caused by a fail of the command run on the target or by a fail of SSH
itself.

This can lead to confusion when the error does not come from the command
executed on the target but from SSH itself: test might wrongfully be
marked as PASSED or might fail with incoherent errors.

As SSH errors are always reported with exit code 255, we can easily
filter these.

Modify OESSHTarget.run() behaviour so an AssertionError is raised on SSH
failures, even when ignore_status parameter is True. Still allow to
explicitly ignore this error for the rare cases where this can be
needed.

Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
---
 meta/lib/oeqa/core/target/ssh.py | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/meta/lib/oeqa/core/target/ssh.py b/meta/lib/oeqa/core/target/ssh.py
index 8b5c450a058fd172f36af6cbd9569983aa3d2d51..0ac3ae438895c97e89599b2dfb2797dabbf383dd 100644
--- a/meta/lib/oeqa/core/target/ssh.py
+++ b/meta/lib/oeqa/core/target/ssh.py
@@ -55,7 +55,7 @@ class OESSHTarget(OETarget):
     def stop(self, **kwargs):
         pass
 
-    def _run(self, command, timeout=None, ignore_status=True, raw=False):
+    def _run(self, command, timeout=None, ignore_status=True, raw=False, ignore_ssh_fails=False):
         """
             Runs command in target using SSHProcess.
         """
@@ -66,13 +66,17 @@ class OESSHTarget(OETarget):
         self.logger.debug("[Command returned '%d' after %.2f seconds]"
                  "" % (status, time.time() - starttime))
 
-        if status and not ignore_status:
+        if status == 255 and not ignore_ssh_fails:
+            raise AssertionError("ssh exited with status '255' for command "
+                                 "'%s': this is likely an SSH failure\n%s"
+                                 % (command, output))
+        elif status and not ignore_status:
             raise AssertionError("Command '%s' returned non-zero exit "
                                  "status %d:\n%s" % (command, status, output))
 
         return (status, output)
 
-    def run(self, command, timeout=None, ignore_status=True, raw=False):
+    def run(self, command, timeout=None, ignore_status=True, raw=False, ignore_ssh_fails=False):
         """
             Runs command in target.
 
@@ -91,7 +95,7 @@ class OESSHTarget(OETarget):
         else:
             processTimeout = self.timeout
 
-        status, output = self._run(sshCmd, processTimeout, ignore_status, raw)
+        status, output = self._run(sshCmd, processTimeout, ignore_status, raw, ignore_ssh_fails)
         if len(output) > (64 * 1024):
             self.logger.debug('Command: %s\nStatus: %d Output length:  %s\n' % (command, status, len(output)))
         else:

-- 
2.47.3

[PATCH 2/4] oeqa: runtime: Ignore SSH errors during setup and tear down

[Mathieu Dubois-Briand]

Tests using SSH will fail when no SSH server is present on the target.
These tests are disabled in these cases, by being marked with a
dependency on ssh.SSHTest.test_ssh, which in turns has a dependency on
having either dropbear or openssh-sshd in the image.

But setUpClass() and tearDownClass() functions are always executed, even
on tests failing the dependency checks, leading to unexpected failed
tests.

Ignoring SSH errors in setup and tear down allows to avoid these test
errors.

Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
---
 meta/lib/oeqa/runtime/cases/logrotate.py | 7 ++++---
 meta/lib/oeqa/runtime/cases/weston.py    | 2 +-
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/meta/lib/oeqa/runtime/cases/logrotate.py b/meta/lib/oeqa/runtime/cases/logrotate.py
index 6ad980cb6adc0cc1660989228442633db3a9827d..7cb43d98c56d73865838ed8418a6a88732840a28 100644
--- a/meta/lib/oeqa/runtime/cases/logrotate.py
+++ b/meta/lib/oeqa/runtime/cases/logrotate.py
@@ -15,12 +15,13 @@ class LogrotateTest(OERuntimeTestCase):
 
     @classmethod
     def setUpClass(cls):
-        cls.tc.target.run('cp /etc/logrotate.d/wtmp $HOME/wtmp.oeqabak')
+        cls.tc.target.run('cp /etc/logrotate.d/wtmp $HOME/wtmp.oeqabak',
+                          ignore_ssh_fails=True)
 
     @classmethod
     def tearDownClass(cls):
-        cls.tc.target.run('mv -f $HOME/wtmp.oeqabak /etc/logrotate.d/wtmp && rm -rf /var/log//logrotate_dir')
-        cls.tc.target.run('rm -rf /var/log/logrotate_testfile && rm -rf /etc/logrotate.d/logrotate_testfile')
+        cls.tc.target.run('mv -f $HOME/wtmp.oeqabak /etc/logrotate.d/wtmp && rm -rf /var/log//logrotate_dir', ignore_ssh_fails=True)
+        cls.tc.target.run('rm -rf /var/log/logrotate_testfile && rm -rf /etc/logrotate.d/logrotate_testfile', ignore_ssh_fails=True)
 
     @OETestDepends(['ssh.SSHTest.test_ssh'])
     @OEHasPackage(['logrotate'])
diff --git a/meta/lib/oeqa/runtime/cases/weston.py b/meta/lib/oeqa/runtime/cases/weston.py
index ee4d336482e5e7b4e61625599d38ab2fbf09ace0..e2cecffe83536c9bfeaae62892bad832f75cb541 100644
--- a/meta/lib/oeqa/runtime/cases/weston.py
+++ b/meta/lib/oeqa/runtime/cases/weston.py
@@ -16,7 +16,7 @@ class WestonTest(OERuntimeTestCase):
 
     @classmethod
     def tearDownClass(cls):
-        cls.tc.target.run('rm %s' % cls.weston_log_file)
+        cls.tc.target.run('rm %s' % cls.weston_log_file, ignore_ssh_fails=True)
 
     @OETestDepends(['ssh.SSHTest.test_ssh'])
     @OEHasPackage(['weston'])

-- 
2.47.3

Re: [OE-core] [PATCH 2/4] oeqa: runtime: Ignore SSH errors during setup and tear down

[Ross Burton]

On 7 Oct 2025, at 18:38, Mathieu Dubois-Briand via lists.openembedded.org <mathieu.dubois-briand=bootlin.com@lists.openembedded.org> wrote:
> 
> Tests using SSH will fail when no SSH server is present on the target.
> These tests are disabled in these cases, by being marked with a
> dependency on ssh.SSHTest.test_ssh, which in turns has a dependency on
> having either dropbear or openssh-sshd in the image.
> 
> But setUpClass() and tearDownClass() functions are always executed, even
> on tests failing the dependency checks, leading to unexpected failed
> tests.

Well that’s just silly.  I wonder if OEHasPackage() works on the class scope too and does actually avoid doing the setup/teardown?

However, this does demonstrate that doing non-trivial setup/teardown is a bad idea in test cases, and copying files to the target is definitely non-trivial.

I suggest merging all of the logrotate tests and setup/teardown into a single test that uses finally: to cleanup, and just removing the teardown from the weston test.

Cheers,
Ross

Re: [OE-core] [PATCH 2/4] oeqa: runtime: Ignore SSH errors during setup and tear down

[Mathieu Dubois-Briand]

On Thu Oct 9, 2025 at 4:26 PM CEST, Ross Burton via lists.openembedded.org wrote:
> On 7 Oct 2025, at 18:38, Mathieu Dubois-Briand via lists.openembedded.org <mathieu.dubois-briand=bootlin.com@lists.openembedded.org> wrote:
>> 
>> Tests using SSH will fail when no SSH server is present on the target.
>> These tests are disabled in these cases, by being marked with a
>> dependency on ssh.SSHTest.test_ssh, which in turns has a dependency on
>> having either dropbear or openssh-sshd in the image.
>> 
>> But setUpClass() and tearDownClass() functions are always executed, even
>> on tests failing the dependency checks, leading to unexpected failed
>> tests.
>
> Well that’s just silly.  I wonder if OEHasPackage() works on the class scope too and does actually avoid doing the setup/teardown?
>
> However, this does demonstrate that doing non-trivial setup/teardown is a bad idea in test cases, and copying files to the target is definitely non-trivial.
>
> I suggest merging all of the logrotate tests and setup/teardown into a single test that uses finally: to cleanup, and just removing the teardown from the weston test.
>

I am going to send some patches to clean this.

On the logrotate one, I choose to keep two separate tests, but we can
highly simplify this cleanup part and integrate that in the tests. On
weston, let's remove the log file before starting weston, so we can
still be sure the log is clean.

-- 
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[PATCH 3/4] oeqa: postactions: Ignore SSH errors

[Mathieu Dubois-Briand]

Postactions are not part of the tests but allow to retrieve useful data
from the target. They try to do this using SSH, but this can fail when
no SSH server is present on the target. Ignore these fails.

Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
---
 meta/lib/oeqa/utils/postactions.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/utils/postactions.py b/meta/lib/oeqa/utils/postactions.py
index c69481db6cbf29b5e3f267170151d85e73baa0d0..7b967edaad0f0a418347f18a7fa85073672ceeff 100644
--- a/meta/lib/oeqa/utils/postactions.py
+++ b/meta/lib/oeqa/utils/postactions.py
@@ -22,7 +22,7 @@ from oeqa.utils import get_artefact_dir
 def get_target_disk_usage(d, tc, artifacts_list, outputdir):
     output_file = os.path.join(outputdir, "target_disk_usage.txt")
     try:
-        (status, output) = tc.target.run('df -h')
+        (status, output) = tc.target.run('df -h', ignore_ssh_fails=True)
         with open(output_file, 'w') as f:
             f.write(output)
             f.write("\n")
@@ -49,7 +49,7 @@ def get_artifacts_list(target, raw_list):
     for raw_path in raw_list.split():
         cmd = f"for p in {raw_path}; do if [ -e $p ]; then echo $p; fi; done"
         try:
-            status, output = target.run(cmd)
+            status, output = target.run(cmd, ignore_ssh_fails=True)
             if status != 0 or not output:
                 raise Exception()
             result += output.split()

-- 
2.47.3

[PATCH 4/4] oeqa: runtime: ssh: Manage any SSH failure locally

[Mathieu Dubois-Briand]

This is the SSH test, it makes sense to ignore SSH failures in the SSH
helper and manage them in the test body.

Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
---
 meta/lib/oeqa/runtime/cases/ssh.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/runtime/cases/ssh.py b/meta/lib/oeqa/runtime/cases/ssh.py
index b632a29a0178bf1c0015b84bca8e3fc3fd18fa74..3e9503277e39b281ffa7dc5f08f975ba2e96cee5 100644
--- a/meta/lib/oeqa/runtime/cases/ssh.py
+++ b/meta/lib/oeqa/runtime/cases/ssh.py
@@ -17,7 +17,7 @@ class SSHTest(OERuntimeTestCase):
     @OEHasPackage(['dropbear', 'openssh-sshd'])
     def test_ssh(self):
         for i in range(5):
-          status, output = self.target.run("uname -a", timeout=30)
+          status, output = self.target.run("uname -a", timeout=30, ignore_ssh_fails=True)
           if status == 0:
               break
           elif status == 255 or status == -signal.SIGTERM:

-- 
2.47.3