[PATCH 1/2] gcc: Oe-selftest failure analysis - fix for host key verfication & kex exchange identification failures

[PATCH 1/2] gcc: Oe-selftest failure analysis - fix for host key verfication & kex exchange identification failures

[Harish.Sadineni]

From: Harish Sadineni <Harish.Sadineni@windriver.com>

while runnig oe-selftest for gcc, testcases that need to be run on qemu are not running due to below failures.
- Executing on ssh: mkdir -p /tmp/runtest.3549641   (timeout = 300)
   spawn [open ...]
   Host key verification failed.
   ERROR: Couldn't create remote directory /tmp/runtest.3549641 on ssh
- kex_exchange_identification: read: Connection reset by peer^M
   Connection reset by 192.168.7.2 port 22^M
   ERROR: Couldn't create remote directory /tmp/runtest.3549814 on ssh

Host key verification failure is happening when ssh board config file name is defined as "ssh.exp" and there are multiple ssh.exp files generated during the build and a wrong ssh config was taken. To resolve this changed the board config file name to "linux-ssh.exp" which ensures correct ssh settings are used.

To resolve kex exchange identification error increased the MaxStartups.

Signed-off-by: Harish Sadineni <Harish.Sadineni@windriver.com>
---
 meta/lib/oeqa/selftest/cases/gcc.py                   | 2 +-
 meta/recipes-connectivity/openssh/openssh/sshd_config | 2 +-
 meta/recipes-devtools/gcc/gcc-testsuite.inc           | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/gcc.py b/meta/lib/oeqa/selftest/cases/gcc.py
index 89360178fe..4cc0894d42 100644
--- a/meta/lib/oeqa/selftest/cases/gcc.py
+++ b/meta/lib/oeqa/selftest/cases/gcc.py
@@ -37,7 +37,7 @@ class GccSelfTestBase(OESelftestTestCase, OEPTestResultTestCase):
         features = []
         features.append('MAKE_CHECK_TARGETS = "{0}"'.format(" ".join(targets)))
         if ssh is not None:
-            features.append('TOOLCHAIN_TEST_TARGET = "ssh"')
+            features.append('TOOLCHAIN_TEST_TARGET = "linux-ssh"')
             features.append('TOOLCHAIN_TEST_HOST = "{0}"'.format(ssh))
             features.append('TOOLCHAIN_TEST_HOST_USER = "root"')
             features.append('TOOLCHAIN_TEST_HOST_PORT = "22"')
diff --git a/meta/recipes-connectivity/openssh/openssh/sshd_config b/meta/recipes-connectivity/openssh/openssh/sshd_config
index e9eaf93157..b9357ef6c5 100644
--- a/meta/recipes-connectivity/openssh/openssh/sshd_config
+++ b/meta/recipes-connectivity/openssh/openssh/sshd_config
@@ -100,7 +100,7 @@ ClientAliveInterval 15
 ClientAliveCountMax 4
 #UseDNS no
 #PidFile /var/run/sshd.pid
-#MaxStartups 10:30:100
+MaxStartups 75:30:100
 #PermitTunnel no
 #ChrootDirectory none
 #VersionAddendum none
diff --git a/meta/recipes-devtools/gcc/gcc-testsuite.inc b/meta/recipes-devtools/gcc/gcc-testsuite.inc
index f16d471478..0a950b6c9e 100644
--- a/meta/recipes-devtools/gcc/gcc-testsuite.inc
+++ b/meta/recipes-devtools/gcc/gcc-testsuite.inc
@@ -101,7 +101,7 @@ python check_prepare() {
         f.write(generate_qemu_linux_user_config(d))
 
     # write out target ssh board config
-    with open(os.path.join(dejagnudir, "ssh.exp"), "w") as f:
+    with open(os.path.join(dejagnudir, "linux-ssh.exp"), "w") as f:
         f.write(generate_remote_ssh_linux_config(d))
 
     # generate site.exp to provide boards
-- 
2.43.0

[PATCH 2/2] gcc: Oe-selftest failure analysis - allowing multiple ssh sessions

[Harish.Sadineni]

From: Harish Sadineni <Harish.Sadineni@windriver.com>

Reusing SSH Connection to speed up remote login process using multiplexing to reduce time taken while running oe-selftest for gcc.

Signed-off-by: Harish Sadineni <Harish.Sadineni@windriver.com>
---
 meta/lib/oeqa/selftest/cases/gcc.py                   | 10 ++++++++++
 meta/recipes-connectivity/openssh/openssh/sshd_config |  2 +-
 meta/recipes-devtools/gcc/gcc-testsuite.inc           |  3 ++-
 3 files changed, 13 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/gcc.py b/meta/lib/oeqa/selftest/cases/gcc.py
index 4cc0894d42..992a5b74be 100644
--- a/meta/lib/oeqa/selftest/cases/gcc.py
+++ b/meta/lib/oeqa/selftest/cases/gcc.py
@@ -5,6 +5,7 @@
 #
 import os
 import time
+import subprocess
 from oeqa.core.decorator import OETestTag
 from oeqa.core.case import OEPTestResultTestCase
 from oeqa.selftest.case import OESelftestTestCase
@@ -83,6 +84,15 @@ class GccSelfTestBase(OESelftestTestCase, OEPTestResultTestCase):
             # validate that SSH is working
             status, _ = qemu.run("uname")
             self.assertEqual(status, 0)
+            qemu_ip = qemu.ip
+            ssh_command = [
+            "ssh", "-o", "StrictHostKeyChecking=no",
+            "-o", "UserKnownHostsFile=/dev/null",
+            "-o", "ControlMaster=auto",
+            "-o", "ControlPath=~/.ssh/control-%r@%h:%p",
+            "-o", "Controlpersist=yes", "root@{}".format(qemu_ip),
+            "exit" ]
+            subprocess.run(ssh_command, check=True)
 
             return self.run_check(*args, ssh=qemu.ip, **kwargs)
 
diff --git a/meta/recipes-connectivity/openssh/openssh/sshd_config b/meta/recipes-connectivity/openssh/openssh/sshd_config
index b9357ef6c5..37ffe28142 100644
--- a/meta/recipes-connectivity/openssh/openssh/sshd_config
+++ b/meta/recipes-connectivity/openssh/openssh/sshd_config
@@ -34,7 +34,7 @@ Include /etc/ssh/sshd_config.d/*.conf
 #PermitRootLogin prohibit-password
 #StrictModes yes
 #MaxAuthTries 6
-#MaxSessions 10
+MaxSessions 1000
 
 #PubkeyAuthentication yes
 
diff --git a/meta/recipes-devtools/gcc/gcc-testsuite.inc b/meta/recipes-devtools/gcc/gcc-testsuite.inc
index 0a950b6c9e..62e9691a99 100644
--- a/meta/recipes-devtools/gcc/gcc-testsuite.inc
+++ b/meta/recipes-devtools/gcc/gcc-testsuite.inc
@@ -87,7 +87,8 @@ python check_prepare() {
         content.append("set_board_info username {0}".format(d.getVar("TOOLCHAIN_TEST_HOST_USER")))
 
         port = d.getVar("TOOLCHAIN_TEST_HOST_PORT")
-        content.append("set_board_info rsh_prog \"/usr/bin/ssh -p {0} -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no\"".format(port))
+        qemuip_address = d.getVar("TOOLCHAIN_TEST_HOST")
+        content.append("set_board_info rsh_prog \"/usr/bin/ssh -S ~/.ssh/control-root@{0}:{1} \"".format( qemuip_address, port))
         content.append("set_board_info rcp_prog \"/usr/bin/scp -P {0} -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no\"".format(port))
 
         return "\n".join(content)
-- 
2.43.0

Re: [PATCH 1/2] gcc: Oe-selftest failure analysis - fix for host key verfication & kex exchange identification failures

[Sadineni, Harish]

[-- Attachment #1: Type: text/plain, Size: 2406 bytes --]

testresults before applying patch for x86_64 :
|  gcc            |   g++           |  libatomic    |   libgomp  |    libitm      |   libstdc++ |  Total (Sum of all modules)
expected passes         |  148313     |   218545      |  27              |    3369      |    24           |    9693     |  379971
unexpected failures     |  15             |   20              |                    |                  |                   |                 |  35
expected failures         |  1419         |   2052          |                    |    30          |                   |    6           |  3507
unresolved testcases  |  25130        |   10816       |  27              |    1579      |     22          |    4869      |  42443
unsupported tests       |  2686          |   10420       |                    |    410        |     2             |    643       |  14161

testresults after applying patch for x86_64:
|  gcc          |   g++           |  libatomic    |   libgomp  |    libitm      |   libstdc++ |  Total (Sum of all modules)
expected passes         |  177610    |   230158     |  54              |    5033      |    24          |    13695    |  426574
unexpected failures     |  99           |   20              |                    |    1            |    19          |    7            |  146
expected failures         |  1422       |   2063          |                    |    32          |    3            |    106        |  3626
unresolved testcases  |  4              |                    |                    |                  |                  |    1125       |  1129
unsupported tests       |  2147        |   10350       |                    |    429        |    2            |    805         |  13733

Unresolved cases are reduced to ~1000 (i.e., More than 41K tests are newly executed)
More than 45000 tests cases are passed newly.

[-- Attachment #2: Type: text/html, Size: 5234 bytes --]

Re: [PATCH 2/2] gcc: Oe-selftest failure analysis - allowing multiple ssh sessions

[Sadineni, Harish]

[-- Attachment #1: Type: text/plain, Size: 216 bytes --]

Time taken to run gcc testsuite w/o multiple ssh sessions -> 16-18 hrs
Time taken to run gcc testsuite w/ multiple ssh sessions -> 11-12 hrs
There is an improvement of ~30% in execution time.

Thanks,

Harish

[-- Attachment #2: Type: text/html, Size: 1634 bytes --]

Patchtest results for [PATCH 1/2] gcc: Oe-selftest failure analysis - fix for host key verfication & kex exchange identification failures

[patchtest]

[-- Attachment #1: Type: text/plain, Size: 3129 bytes --]

Thank you for your submission. Patchtest identified one
or more issues with the patch. Please see the log below for
more information:

---
Testing patch /home/patchtest/share/mboxes/1-2-gcc-Oe-selftest-failure-analysis---fix-for-host-key-verfication-kex-exchange-identification-failures.patch

FAIL: test shortlog length: Edit shortlog so that it is 90 characters or less (currently 103 characters) (test_mbox.TestMbox.test_shortlog_length)

PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test max line length (test_metadata.TestMetadata.test_max_line_length)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test pylint (test_python_pylint.PyLint.test_pylint)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)

SKIP: pretest src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE check ignore: No modified recipes or older target branch, skipping test (test_metadata.TestMetadata.test_cve_check_ignore)
SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)
SKIP: test target mailing list: Series merged, no reason to check other mailing lists (test_mbox.TestMbox.test_target_mailing_list)

---

Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this
email with an explanation of why the patch should be accepted. If you
believe these results are due to an error in patchtest, please submit a
bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category
under 'Yocto Project Subprojects'). For more information on specific
failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank
you!

Re: [OE-core] [PATCH 1/2] gcc: Oe-selftest failure analysis - fix for host key verfication & kex exchange identification failures

[Richard Purdie]

Hi Harish,

On Thu, 2024-04-18 at 03:50 -0700, Sadineni, Harish via
lists.openembedded.org wrote:
> From: Harish Sadineni <Harish.Sadineni@windriver.com>
> 
> while runnig oe-selftest for gcc, testcases that need to be run on
> qemu are not running due to below failures.
> - Executing on ssh: mkdir -p /tmp/runtest.3549641   (timeout = 300)
>    spawn [open ...]
>    Host key verification failed.
>    ERROR: Couldn't create remote directory /tmp/runtest.3549641 on
> ssh
> - kex_exchange_identification: read: Connection reset by peer^M
>    Connection reset by 192.168.7.2 port 22^M
>    ERROR: Couldn't create remote directory /tmp/runtest.3549814 on
> ssh
> 
> Host key verification failure is happening when ssh board config file
> name is defined as "ssh.exp" and there are multiple ssh.exp files
> generated during the build and a wrong ssh config was taken. To
> resolve this changed the board config file name to "linux-ssh.exp"
> which ensures correct ssh settings are used.
> 
> To resolve kex exchange identification error increased the
> MaxStartups.
> 
> Signed-off-by: Harish Sadineni <Harish.Sadineni@windriver.com>
> ---
>  meta/lib/oeqa/selftest/cases/gcc.py                   | 2 +-
>  meta/recipes-connectivity/openssh/openssh/sshd_config | 2 +-
>  meta/recipes-devtools/gcc/gcc-testsuite.inc           | 2 +-
>  3 files changed, 3 insertions(+), 3 deletions(-)

Thanks for these. They did pass testing on the autobuilder but I there
are some style tweaks needed before they can merge and we have to
answer some questions about the sshd_config changes.

Style wise, the shortlog of the patch (subject line) needs to be
improved. It should start "oeqa/selftest/gcc:" to make it clear this
isn't a gcc recipe change but a selftest change.

The shortlog should also be a short summary, so for example:

oeqa/selftest/gcc: Fix ssh tests to run correctly

The longer log can contain contain information about the host key and
key exchange pieces.

Moving on to the patch content, the openssh piece needs to be separated
out into a separate patch as it is changing a quite key separate part
of the system.

I noticed the second patch also has an openssh change, so perhaps
combine those two changes together into a separate patch. The second
patch also needs the shortlog improving similar to the above.

The rest of the patch content is good.

The remaining issue is where/when to apply the openssh changes. My
worry is that this does have potential DoS implications on real target
devices as the config change is being made globally. It made me wonder
if we should have a "qemuall" openssh config that applies to our qemu
machines we use for testing as way of handling this?

Cheers,

Richard

Re: [OE-core] [PATCH 1/2] gcc: Oe-selftest failure analysis - fix for host key verfication & kex exchange identification failures

[Richard Purdie]

I did compare a build with this patch in it with the 5.0 rc4 test
report:

http://autobuilder.yocto.io/pub/non-release/20240424-25/testresults/testresult-report.txt

vs

http://autobuilder.yocto.io/pub/releases/yocto-5.0.rc4/testreport.txt

which shows:

gcc                          | 149932       | 15        | 27534       | 1548
gcc-g++                      | 219372       | 22        | 21074       | 1350
gcc-libatomic                | 27           | 0         | 27          | 513
gcc-libgomp                  | 3428         | 0         | 1960        | 582
gcc-libitm                   | 24           | 0         | 24          | 430
gcc-libstdc++-v3             | 9722         | 0         | 5447        | 2202

vs

gcc                          | 149794       | 31        | 27672       | 1446
gcc-g++                      | 219325       | 31        | 21128       | 1100
gcc-libatomic                | 27           | 1         | 27          | 517
gcc-libgomp                  | 3411         | 15        | 1977        | 782
gcc-libitm                   | 24           | 1         | 24          | 397
gcc-libstdc++-v3             | 9705         | 32        | 5464        | 2669


so I'm not sure this is fixing the large number of skipped tests that
it is supposed to fix?

Cheers,

Richard

Re: [PATCH 1/2] gcc: Oe-selftest failure analysis - fix for host key verfication & kex exchange identification failures

[Sadineni, Harish]

[-- Attachment #1: Type: text/plain, Size: 1046 bytes --]

In fedora distro while running oe-selftest for gcc, owner permessions of files in /etc/ssh/ is getting changed to "nobody nobody" as following "-rw-r--r--. 1 nobody nobody 581 Jan 10  2024 /etc/ssh/ssh_config.d/50-redhat.conf". Because of this we are getting bad owner permission's on /etc/ssh/ssh-config.d/5-redhat.conf.
But in Ubuntu while running oe-selftest for gcc, owner permissions of files in /etc/ssh/  are  "root root" that why its working in Ubuntu.

when checked with upstream they suggested this can happen if the tests run through systemd unit and if systemd unit have ProtectHome=yes or PrivateUsers=yes then this can make the ownership of the file appear to be nobody instead of root .
https://www.reddit.com/r/Fedora/comments/1dimh0i/bad_owner_or_permissions_on_etcsshssh/

we have tried in fedora docker and there also while running oe-selftest for gcc, we are having owner permissions of files in /etc/ssh/  are  "root root".

May be this is to be changed on fedora host machine(autobuilder?).

Thanks,
Harish

[-- Attachment #2: Type: text/html, Size: 1300 bytes --]