[PATCH 2/2] gcc: Oe-selftest failure analysis - allowing multiple ssh sessions

[PATCH 2/2] gcc: Oe-selftest failure analysis - allowing multiple ssh sessions

[Harish.Sadineni]

From: Harish Sadineni <Harish.Sadineni@windriver.com>

Reusing SSH Connection to speed up remote login process using multiplexing to reduce time taken while running oe-selftest for gcc.

Signed-off-by: Harish Sadineni <Harish.Sadineni@windriver.com>
---
 meta/lib/oeqa/selftest/cases/gcc.py                   | 10 ++++++++++
 meta/recipes-connectivity/openssh/openssh/sshd_config |  2 +-
 meta/recipes-devtools/gcc/gcc-testsuite.inc           |  3 ++-
 3 files changed, 13 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/gcc.py b/meta/lib/oeqa/selftest/cases/gcc.py
index 4cc0894d42..992a5b74be 100644
--- a/meta/lib/oeqa/selftest/cases/gcc.py
+++ b/meta/lib/oeqa/selftest/cases/gcc.py
@@ -5,6 +5,7 @@
 #
 import os
 import time
+import subprocess
 from oeqa.core.decorator import OETestTag
 from oeqa.core.case import OEPTestResultTestCase
 from oeqa.selftest.case import OESelftestTestCase
@@ -83,6 +84,15 @@ class GccSelfTestBase(OESelftestTestCase, OEPTestResultTestCase):
             # validate that SSH is working
             status, _ = qemu.run("uname")
             self.assertEqual(status, 0)
+            qemu_ip = qemu.ip
+            ssh_command = [
+            "ssh", "-o", "StrictHostKeyChecking=no",
+            "-o", "UserKnownHostsFile=/dev/null",
+            "-o", "ControlMaster=auto",
+            "-o", "ControlPath=~/.ssh/control-%r@%h:%p",
+            "-o", "Controlpersist=yes", "root@{}".format(qemu_ip),
+            "exit" ]
+            subprocess.run(ssh_command, check=True)
 
             return self.run_check(*args, ssh=qemu.ip, **kwargs)
 
diff --git a/meta/recipes-connectivity/openssh/openssh/sshd_config b/meta/recipes-connectivity/openssh/openssh/sshd_config
index b9357ef6c5..37ffe28142 100644
--- a/meta/recipes-connectivity/openssh/openssh/sshd_config
+++ b/meta/recipes-connectivity/openssh/openssh/sshd_config
@@ -34,7 +34,7 @@ Include /etc/ssh/sshd_config.d/*.conf
 #PermitRootLogin prohibit-password
 #StrictModes yes
 #MaxAuthTries 6
-#MaxSessions 10
+MaxSessions 1000
 
 #PubkeyAuthentication yes
 
diff --git a/meta/recipes-devtools/gcc/gcc-testsuite.inc b/meta/recipes-devtools/gcc/gcc-testsuite.inc
index 0a950b6c9e..62e9691a99 100644
--- a/meta/recipes-devtools/gcc/gcc-testsuite.inc
+++ b/meta/recipes-devtools/gcc/gcc-testsuite.inc
@@ -87,7 +87,8 @@ python check_prepare() {
         content.append("set_board_info username {0}".format(d.getVar("TOOLCHAIN_TEST_HOST_USER")))
 
         port = d.getVar("TOOLCHAIN_TEST_HOST_PORT")
-        content.append("set_board_info rsh_prog \"/usr/bin/ssh -p {0} -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no\"".format(port))
+        qemuip_address = d.getVar("TOOLCHAIN_TEST_HOST")
+        content.append("set_board_info rsh_prog \"/usr/bin/ssh -S ~/.ssh/control-root@{0}:{1} \"".format( qemuip_address, port))
         content.append("set_board_info rcp_prog \"/usr/bin/scp -P {0} -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no\"".format(port))
 
         return "\n".join(content)
-- 
2.43.0

Re: [PATCH 2/2] gcc: Oe-selftest failure analysis - allowing multiple ssh sessions

[Sadineni, Harish]

[-- Attachment #1: Type: text/plain, Size: 216 bytes --]

Time taken to run gcc testsuite w/o multiple ssh sessions -> 16-18 hrs
Time taken to run gcc testsuite w/ multiple ssh sessions -> 11-12 hrs
There is an improvement of ~30% in execution time.

Thanks,

Harish

[-- Attachment #2: Type: text/html, Size: 1634 bytes --]

[PATCH 1/2] gcc: fix for host key verfication & kex exchange identification failures

[Harish.Sadineni]

From: Harish Sadineni <Harish.Sadineni@windriver.com>

while runnig oe-selftest for gcc, testcases that need to be run on qemu are not running due to below failures.
- Executing on ssh: mkdir -p /tmp/runtest.3549641   (timeout = 300)
   spawn [open ...]
   Host key verification failed.
   ERROR: Couldn't create remote directory /tmp/runtest.3549641 on ssh
- kex_exchange_identification: read: Connection reset by peer^M
   Connection reset by 192.168.7.2 port 22^M
   ERROR: Couldn't create remote directory /tmp/runtest.3549814 on ssh

Host key verification failure is happening when ssh board config file name is defined as "ssh.exp" and there are multiple ssh.exp files generated during the build and a wrong ssh config was taken. To resolve this changed the board config file name to "linux-ssh.exp" which ensures correct ssh settings are used.

To resolve kex exchange identification error increased the MaxStartups.

Signed-off-by: Harish Sadineni <Harish.Sadineni@windriver.com>
---
 meta/lib/oeqa/selftest/cases/gcc.py                   | 2 +-
 meta/recipes-connectivity/openssh/openssh/sshd_config | 2 +-
 meta/recipes-devtools/gcc/gcc-testsuite.inc           | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/gcc.py b/meta/lib/oeqa/selftest/cases/gcc.py
index 89360178fe..4cc0894d42 100644
--- a/meta/lib/oeqa/selftest/cases/gcc.py
+++ b/meta/lib/oeqa/selftest/cases/gcc.py
@@ -37,7 +37,7 @@ class GccSelfTestBase(OESelftestTestCase, OEPTestResultTestCase):
         features = []
         features.append('MAKE_CHECK_TARGETS = "{0}"'.format(" ".join(targets)))
         if ssh is not None:
-            features.append('TOOLCHAIN_TEST_TARGET = "ssh"')
+            features.append('TOOLCHAIN_TEST_TARGET = "linux-ssh"')
             features.append('TOOLCHAIN_TEST_HOST = "{0}"'.format(ssh))
             features.append('TOOLCHAIN_TEST_HOST_USER = "root"')
             features.append('TOOLCHAIN_TEST_HOST_PORT = "22"')
diff --git a/meta/recipes-connectivity/openssh/openssh/sshd_config b/meta/recipes-connectivity/openssh/openssh/sshd_config
index e9eaf93157..b9357ef6c5 100644
--- a/meta/recipes-connectivity/openssh/openssh/sshd_config
+++ b/meta/recipes-connectivity/openssh/openssh/sshd_config
@@ -100,7 +100,7 @@ ClientAliveInterval 15
 ClientAliveCountMax 4
 #UseDNS no
 #PidFile /var/run/sshd.pid
-#MaxStartups 10:30:100
+MaxStartups 75:30:100
 #PermitTunnel no
 #ChrootDirectory none
 #VersionAddendum none
diff --git a/meta/recipes-devtools/gcc/gcc-testsuite.inc b/meta/recipes-devtools/gcc/gcc-testsuite.inc
index f16d471478..0a950b6c9e 100644
--- a/meta/recipes-devtools/gcc/gcc-testsuite.inc
+++ b/meta/recipes-devtools/gcc/gcc-testsuite.inc
@@ -101,7 +101,7 @@ python check_prepare() {
         f.write(generate_qemu_linux_user_config(d))
 
     # write out target ssh board config
-    with open(os.path.join(dejagnudir, "ssh.exp"), "w") as f:
+    with open(os.path.join(dejagnudir, "linux-ssh.exp"), "w") as f:
         f.write(generate_remote_ssh_linux_config(d))
 
     # generate site.exp to provide boards
-- 
2.43.0

[PATCH 2/2] gcc: Oe-selftest failure analysis - allowing multiple ssh sessions

[Harish.Sadineni]

From: Harish Sadineni <Harish.Sadineni@windriver.com>

Reusing SSH Connection to speed up remote login process using multiplexing to reduce time taken while running oe-selftest for gcc.

Signed-off-by: Harish Sadineni <Harish.Sadineni@windriver.com>
---
 meta/lib/oeqa/selftest/cases/gcc.py                   | 10 ++++++++++
 meta/recipes-connectivity/openssh/openssh/sshd_config |  2 +-
 meta/recipes-devtools/gcc/gcc-testsuite.inc           |  3 ++-
 3 files changed, 13 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/gcc.py b/meta/lib/oeqa/selftest/cases/gcc.py
index 4cc0894d42..992a5b74be 100644
--- a/meta/lib/oeqa/selftest/cases/gcc.py
+++ b/meta/lib/oeqa/selftest/cases/gcc.py
@@ -5,6 +5,7 @@
 #
 import os
 import time
+import subprocess
 from oeqa.core.decorator import OETestTag
 from oeqa.core.case import OEPTestResultTestCase
 from oeqa.selftest.case import OESelftestTestCase
@@ -83,6 +84,15 @@ class GccSelfTestBase(OESelftestTestCase, OEPTestResultTestCase):
             # validate that SSH is working
             status, _ = qemu.run("uname")
             self.assertEqual(status, 0)
+            qemu_ip = qemu.ip
+            ssh_command = [
+            "ssh", "-o", "StrictHostKeyChecking=no",
+            "-o", "UserKnownHostsFile=/dev/null",
+            "-o", "ControlMaster=auto",
+            "-o", "ControlPath=~/.ssh/control-%r@%h:%p",
+            "-o", "Controlpersist=yes", "root@{}".format(qemu_ip),
+            "exit" ]
+            subprocess.run(ssh_command, check=True)
 
             return self.run_check(*args, ssh=qemu.ip, **kwargs)
 
diff --git a/meta/recipes-connectivity/openssh/openssh/sshd_config b/meta/recipes-connectivity/openssh/openssh/sshd_config
index b9357ef6c5..37ffe28142 100644
--- a/meta/recipes-connectivity/openssh/openssh/sshd_config
+++ b/meta/recipes-connectivity/openssh/openssh/sshd_config
@@ -34,7 +34,7 @@ Include /etc/ssh/sshd_config.d/*.conf
 #PermitRootLogin prohibit-password
 #StrictModes yes
 #MaxAuthTries 6
-#MaxSessions 10
+MaxSessions 1000
 
 #PubkeyAuthentication yes
 
diff --git a/meta/recipes-devtools/gcc/gcc-testsuite.inc b/meta/recipes-devtools/gcc/gcc-testsuite.inc
index 0a950b6c9e..62e9691a99 100644
--- a/meta/recipes-devtools/gcc/gcc-testsuite.inc
+++ b/meta/recipes-devtools/gcc/gcc-testsuite.inc
@@ -87,7 +87,8 @@ python check_prepare() {
         content.append("set_board_info username {0}".format(d.getVar("TOOLCHAIN_TEST_HOST_USER")))
 
         port = d.getVar("TOOLCHAIN_TEST_HOST_PORT")
-        content.append("set_board_info rsh_prog \"/usr/bin/ssh -p {0} -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no\"".format(port))
+        qemuip_address = d.getVar("TOOLCHAIN_TEST_HOST")
+        content.append("set_board_info rsh_prog \"/usr/bin/ssh -S ~/.ssh/control-root@{0}:{1} \"".format( qemuip_address, port))
         content.append("set_board_info rcp_prog \"/usr/bin/scp -P {0} -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no\"".format(port))
 
         return "\n".join(content)
-- 
2.43.0

Re: [PATCH 1/2] gcc: fix for host key verfication & kex exchange identification failures

[Sadineni, Harish]

[-- Attachment #1: Type: text/plain, Size: 2425 bytes --]

testresults before applying patch for x86_64 :
|  gcc            |   g++           |  libatomic    |   libgomp  |    libitm      |   libstdc++ |  Total (Sum of all modules)
expected passes         |  148313     |   218545      |  27              |    3369      |    24           |    9693     |  379971
unexpected failures     |  15             |   20              |                    |                  |                   |                 |  35
expected failures         |  1419         |   2052          |                    |    30          |                   |    6           |  3507
unresolved testcases  |  25130        |   10816       |  27              |    1579      |     22          |    4869      |  42443
unsupported tests       |  2686          |   10420       |                    |    410        |     2             |    643       |  14161

testresults after applying patch for x86_64:
|  gcc          |   g++           |  libatomic    |   libgomp  |    libitm      |   libstdc++ |  Total (Sum of all modules)
expected passes         |  177610    |   230158     |  54              |    5033      |    24          |    13695    |  426574
unexpected failures     |  99           |   20              |                    |    1            |    19          |    7            |  146
expected failures         |  1422       |   2063          |                    |    32          |    3            |    106        |  3626
unresolved testcases  |  4              |                    |                    |                  |                  |    1125       |  1129
unsupported tests       |  2147        |   10350       |                    |    429        |    2            |    805         |  13733

Unresolved cases are reduced to ~1000 (i.e., More than 41K tests are newly executed)
More than 45000 tests cases are passed newly.

Thanks,
Harish

[-- Attachment #2: Type: text/html, Size: 5311 bytes --]

Re: [PATCH 2/2] gcc: Oe-selftest failure analysis - allowing multiple ssh sessions

[Sadineni, Harish]

[-- Attachment #1: Type: text/plain, Size: 216 bytes --]

Time taken to run gcc testsuite w/o multiple ssh sessions -> 16-18 hrs
Time taken to run gcc testsuite w/ multiple ssh sessions -> 11-12 hrs
There is an improvement of ~30% in execution time.

Thanks,

Harish

[-- Attachment #2: Type: text/html, Size: 1258 bytes --]