[XEN PATCH v2 0/3] CI: Fixes for tools/tests and junit and other

[XEN PATCH v2 0/3] CI: Fixes for tools/tests and junit and other

[Anthony PERARD]

From: Anthony PERARD <anthony.perard@vates.tech>

Patch series available in this git branch:
https://xenbits.xenproject.org/git-http/people/aperard/xen-unstable.git br.ci-tools-tests-junit-fixes-v2

All the *-tools-tests-* are currently only checking automatically if the
machine as booted. The only way to find out if one of the tools/tests failed is
to read the console output by hand. Fix that.

Cheers,

Anthony PERARD (3):
  CI: Rework run-tools-test exit path
  CI: Upload junit result as artefact
  CI: Workaround extra content in junit

 automation/gitlab-ci/test.yaml     |  1 +
 automation/scripts/qubes-x86-64.sh | 19 +++++++++++++++++--
 2 files changed, 18 insertions(+), 2 deletions(-)

-- 
Anthony PERARD

[XEN PATCH v2 1/3] CI: Rework run-tools-test exit path

[Anthony PERARD]

From: Anthony PERARD <anthony.perard@vates.tech>

The main script expect to find the string "$passed" or it just timeout
and doesn't try to download the junit file in this case. So we ignore
the return value of run-tools-test to always print "$passed" and
instead look for failure in the generated junit file. If the junit
report is incomplete, this will also result in a failure of the job.

Signed-off-by: Anthony PERARD <anthony.perard@vates.tech>
---

Notes:
    Changes in v2:
    - This squash both patch "CI: Ignore run-tools-test return value" and
      "CI: Have the gitlab job fail on tools/tests failure"
    - grep for '<failure type="failure"' instead of '</failure>'

 automation/scripts/qubes-x86-64.sh | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
index 2750d24eba..21dcd9b063 100755
--- a/automation/scripts/qubes-x86-64.sh
+++ b/automation/scripts/qubes-x86-64.sh
@@ -135,10 +135,11 @@ done
     ### tests: tools-tests-pv, tools-tests-pvh
     "tools-tests-pv"|"tools-tests-pvh")
         retrieve_xml=1
-        passed="test passed"
+        passed="run-tools-test over"
         domU_check=""
         dom0_check="
-/root/run-tools-tests /usr/lib/xen/tests /tmp/tests-junit.xml && echo \"${passed}\"
+/root/run-tools-tests /usr/lib/xen/tests /tmp/tests-junit.xml ||:
+echo \"${passed}\"
 nc -l -p 8080 < /tmp/tests-junit.xml >/dev/null &
 "
         if [ "${test_variant}" = "tools-tests-pvh" ]; then
@@ -297,6 +298,14 @@ TEST_RESULT=$?
 
 if [ -n "$retrieve_xml" ]; then
     nc -w 10 "$SUT_ADDR" 8080 > tests-junit.xml </dev/null
+    # Findout if one of the test failed
+    if ! grep -q '</testsuites>' tests-junit.xml; then
+        echo "ERROR: tests-junit.xml is incomplete or missing."
+        TEST_RESULT=1
+    # Only match "type=failure" to allow to "tolerable" for example.
+    elif grep -q '<failure type="failure"' tests-junit.xml; then
+        TEST_RESULT=1
+    fi
 fi
 
 exit "$TEST_RESULT"
-- 
Anthony PERARD

[XEN PATCH v2 2/3] CI: Upload junit result as artefact

[Anthony PERARD]

From: Anthony PERARD <anthony.perard@vates.tech>

This allows to investigate the junit file in cases of parse error.

Signed-off-by: Anthony PERARD <anthony.perard@vates.tech>
---

Notes:
    Changes in v2:
    - more concise patch description.

 automation/gitlab-ci/test.yaml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index 1f0b27b237..feb03cc7ed 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -155,6 +155,7 @@
     paths:
       - smoke.serial
       - '*.log'
+      - tests-junit.xml
     when: always
   rules:
     - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
-- 
Anthony PERARD

[XEN PATCH v2 3/3] CI: Workaround extra content in junit

[Anthony PERARD]

From: Anthony PERARD <anthony.perard@vates.tech>

Signed-off-by: Anthony PERARD <anthony.perard@vates.tech>
---
 automation/scripts/qubes-x86-64.sh | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
index 21dcd9b063..f1b7e02e63 100755
--- a/automation/scripts/qubes-x86-64.sh
+++ b/automation/scripts/qubes-x86-64.sh
@@ -298,6 +298,12 @@ TEST_RESULT=$?
 
 if [ -n "$retrieve_xml" ]; then
     nc -w 10 "$SUT_ADDR" 8080 > tests-junit.xml </dev/null
+    # Workaround duplicated data been received
+    sed -i.old '/^<\/testsuites>/q' tests-junit.xml > /dev/null
+    extra_line_in_junit=$(($(wc -l < tests-junit.xml.old) - $(wc -l < tests-junit.xml)))
+    if [ $extra_line_in_junit -gt 0 ]; then
+        echo "WARNING: Found $extra_line_in_junit too many lines in junit."
+    fi
     # Findout if one of the test failed
     if ! grep -q '</testsuites>' tests-junit.xml; then
         echo "ERROR: tests-junit.xml is incomplete or missing."
-- 
Anthony PERARD

Re: [XEN PATCH v2 1/3] CI: Rework run-tools-test exit path

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 2863 bytes --]

On Wed, Jul 30, 2025 at 05:26:00PM +0200, Anthony PERARD wrote:
> From: Anthony PERARD <anthony.perard@vates.tech>
> 
> The main script expect to find the string "$passed" or it just timeout
> and doesn't try to download the junit file in this case. So we ignore
> the return value of run-tools-test to always print "$passed" and
> instead look for failure in the generated junit file. If the junit
> report is incomplete, this will also result in a failure of the job.
> 
> Signed-off-by: Anthony PERARD <anthony.perard@vates.tech>
> ---
> 
> Notes:
>     Changes in v2:
>     - This squash both patch "CI: Ignore run-tools-test return value" and
>       "CI: Have the gitlab job fail on tools/tests failure"
>     - grep for '<failure type="failure"' instead of '</failure>'
> 
>  automation/scripts/qubes-x86-64.sh | 13 +++++++++++--
>  1 file changed, 11 insertions(+), 2 deletions(-)
> 
> diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
> index 2750d24eba..21dcd9b063 100755
> --- a/automation/scripts/qubes-x86-64.sh
> +++ b/automation/scripts/qubes-x86-64.sh
> @@ -135,10 +135,11 @@ done
>      ### tests: tools-tests-pv, tools-tests-pvh
>      "tools-tests-pv"|"tools-tests-pvh")
>          retrieve_xml=1
> -        passed="test passed"
> +        passed="run-tools-test over"
>          domU_check=""
>          dom0_check="
> -/root/run-tools-tests /usr/lib/xen/tests /tmp/tests-junit.xml && echo \"${passed}\"
> +/root/run-tools-tests /usr/lib/xen/tests /tmp/tests-junit.xml ||:
> +echo \"${passed}\"
>  nc -l -p 8080 < /tmp/tests-junit.xml >/dev/null &
>  "
>          if [ "${test_variant}" = "tools-tests-pvh" ]; then
> @@ -297,6 +298,14 @@ TEST_RESULT=$?
>  
>  if [ -n "$retrieve_xml" ]; then
>      nc -w 10 "$SUT_ADDR" 8080 > tests-junit.xml </dev/null
> +    # Findout if one of the test failed
> +    if ! grep -q '</testsuites>' tests-junit.xml; then
> +        echo "ERROR: tests-junit.xml is incomplete or missing."
> +        TEST_RESULT=1
> +    # Only match "type=failure" to allow to "tolerable" for example.
> +    elif grep -q '<failure type="failure"' tests-junit.xml; then

Maybe drop -q here, or add some message? Otherwise it's not obvious why it failed. See for example:
https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/10968674458

Yes, you can click "Test summary" link, but it isn't obvious by looking
at the final part of the log. Alternatively, this can be combined with
my "CI: list failed tests at the end of tools job" patch (already
reviewed by Andrew), but I had an impression this series is supposed to
be included instead of that one.

> +        TEST_RESULT=1
> +    fi
>  fi
>  
>  exit "$TEST_RESULT"
> -- 
> Anthony PERARD
> 

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [XEN PATCH v2 2/3] CI: Upload junit result as artefact

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 1021 bytes --]

On Wed, Jul 30, 2025 at 05:26:01PM +0200, Anthony PERARD wrote:
> From: Anthony PERARD <anthony.perard@vates.tech>
> 
> This allows to investigate the junit file in cases of parse error.
> 
> Signed-off-by: Anthony PERARD <anthony.perard@vates.tech>

Reviewed-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> ---
> 
> Notes:
>     Changes in v2:
>     - more concise patch description.
> 
>  automation/gitlab-ci/test.yaml | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index 1f0b27b237..feb03cc7ed 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -155,6 +155,7 @@
>      paths:
>        - smoke.serial
>        - '*.log'
> +      - tests-junit.xml
>      when: always
>    rules:
>      - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
> -- 
> Anthony PERARD
> 

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [XEN PATCH v2 3/3] CI: Workaround extra content in junit

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 1360 bytes --]

On Wed, Jul 30, 2025 at 05:26:02PM +0200, Anthony PERARD wrote:
> From: Anthony PERARD <anthony.perard@vates.tech>
> 
> Signed-off-by: Anthony PERARD <anthony.perard@vates.tech>

Is it still an issue? AFAIR similar issue was fixed together with fixing
xml upload hanging.

> ---
>  automation/scripts/qubes-x86-64.sh | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
> index 21dcd9b063..f1b7e02e63 100755
> --- a/automation/scripts/qubes-x86-64.sh
> +++ b/automation/scripts/qubes-x86-64.sh
> @@ -298,6 +298,12 @@ TEST_RESULT=$?
>  
>  if [ -n "$retrieve_xml" ]; then
>      nc -w 10 "$SUT_ADDR" 8080 > tests-junit.xml </dev/null
> +    # Workaround duplicated data been received
> +    sed -i.old '/^<\/testsuites>/q' tests-junit.xml > /dev/null
> +    extra_line_in_junit=$(($(wc -l < tests-junit.xml.old) - $(wc -l < tests-junit.xml)))
> +    if [ $extra_line_in_junit -gt 0 ]; then
> +        echo "WARNING: Found $extra_line_in_junit too many lines in junit."
> +    fi
>      # Findout if one of the test failed
>      if ! grep -q '</testsuites>' tests-junit.xml; then
>          echo "ERROR: tests-junit.xml is incomplete or missing."
> -- 
> Anthony PERARD
> 

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [XEN PATCH v2 1/3] CI: Rework run-tools-test exit path

[Anthony PERARD]

On Sat, Aug 09, 2025 at 06:22:33PM +0200, Marek Marczykowski-Górecki wrote:
> On Wed, Jul 30, 2025 at 05:26:00PM +0200, Anthony PERARD wrote:
> > From: Anthony PERARD <anthony.perard@vates.tech>
> > 
> > The main script expect to find the string "$passed" or it just timeout
> > and doesn't try to download the junit file in this case. So we ignore
> > the return value of run-tools-test to always print "$passed" and
> > instead look for failure in the generated junit file. If the junit
> > report is incomplete, this will also result in a failure of the job.
> > 
> > Signed-off-by: Anthony PERARD <anthony.perard@vates.tech>
> > ---
> > 
> > Notes:
> >     Changes in v2:
> >     - This squash both patch "CI: Ignore run-tools-test return value" and
> >       "CI: Have the gitlab job fail on tools/tests failure"
> >     - grep for '<failure type="failure"' instead of '</failure>'
> > 
> >  automation/scripts/qubes-x86-64.sh | 13 +++++++++++--
> >  1 file changed, 11 insertions(+), 2 deletions(-)
> > 
> > diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
> > index 2750d24eba..21dcd9b063 100755
> > --- a/automation/scripts/qubes-x86-64.sh
> > +++ b/automation/scripts/qubes-x86-64.sh
> > @@ -135,10 +135,11 @@ done
> >      ### tests: tools-tests-pv, tools-tests-pvh
> >      "tools-tests-pv"|"tools-tests-pvh")
> >          retrieve_xml=1
> > -        passed="test passed"
> > +        passed="run-tools-test over"
> >          domU_check=""
> >          dom0_check="
> > -/root/run-tools-tests /usr/lib/xen/tests /tmp/tests-junit.xml && echo \"${passed}\"
> > +/root/run-tools-tests /usr/lib/xen/tests /tmp/tests-junit.xml ||:
> > +echo \"${passed}\"
> >  nc -l -p 8080 < /tmp/tests-junit.xml >/dev/null &
> >  "
> >          if [ "${test_variant}" = "tools-tests-pvh" ]; then
> > @@ -297,6 +298,14 @@ TEST_RESULT=$?
> >  
> >  if [ -n "$retrieve_xml" ]; then
> >      nc -w 10 "$SUT_ADDR" 8080 > tests-junit.xml </dev/null
> > +    # Findout if one of the test failed
> > +    if ! grep -q '</testsuites>' tests-junit.xml; then
> > +        echo "ERROR: tests-junit.xml is incomplete or missing."
> > +        TEST_RESULT=1
> > +    # Only match "type=failure" to allow to "tolerable" for example.
> > +    elif grep -q '<failure type="failure"' tests-junit.xml; then
> 
> Maybe drop -q here, or add some message? Otherwise it's not obvious why it failed. See for example:
> https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/10968674458
> 
> Yes, you can click "Test summary" link, but it isn't obvious by looking
> at the final part of the log. Alternatively, this can be combined with
> my "CI: list failed tests at the end of tools job" patch (already
> reviewed by Andrew), but I had an impression this series is supposed to
> be included instead of that one.

Dropping "-q" sounds good here, it mean in your pipeline example we
would have the following in the logs:
   <failure type="failure" message="binary /usr/lib/xen/tests/test_vpci exited with code 134">
Which should be could enough for now.

Thanks,

-- 
Anthony PERARD

Re: [XEN PATCH v2 3/3] CI: Workaround extra content in junit

[Anthony PERARD]

On Sat, Aug 09, 2025 at 06:25:28PM +0200, Marek Marczykowski-Górecki wrote:
> On Wed, Jul 30, 2025 at 05:26:02PM +0200, Anthony PERARD wrote:
> > From: Anthony PERARD <anthony.perard@vates.tech>
> > 
> > Signed-off-by: Anthony PERARD <anthony.perard@vates.tech>
> 
> Is it still an issue? AFAIR similar issue was fixed together with fixing
> xml upload hanging.

It was an issue in June:
    https://lore.kernel.org/xen-devel/20250603124222.52057-9-anthony@xenproject.org/

And it still is:
    https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/10969191455
    > WARNING: Found 75 too many lines in junit.

Thanks,

-- 
Anthony PERARD