Re: [XEN PATCH 5/9] CI: Have the gitlab job fail on tools/tests failure

["Marek Marczykowski-Górecki" <marmarek@invisiblethingslab.com>]

[-- Attachment #1: Type: text/plain, Size: 2575 bytes --]

On Tue, Jun 03, 2025 at 02:41:50PM +0100, Andrew Cooper wrote:
> On 03/06/2025 1:42 pm, Anthony PERARD wrote:
> > From: Anthony PERARD <anthony.perard@vates.tech>
> >
> > We can't rely on an exit value from `run-tools-tests` since we only
> > have the console output. `console.exp` only look for success or it
> > times out. We could parse the console output, but the junit is more
> > concise. Also check if we have it or fail as well.
> >
> > Signed-off-by: Anthony PERARD <anthony.perard@vates.tech>
> > ---
> >  automation/scripts/qubes-x86-64.sh | 7 +++++++
> >  1 file changed, 7 insertions(+)
> >
> > diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
> > index 046137a4a6..7a4c5ae489 100755
> > --- a/automation/scripts/qubes-x86-64.sh
> > +++ b/automation/scripts/qubes-x86-64.sh
> > @@ -298,6 +298,13 @@ TEST_RESULT=$?
> >  
> >  if [ -n "$retrieve_xml" ]; then
> >      nc -w 10 "$SUT_ADDR" 8080 > tests-junit.xml </dev/null
> > +    # Findout if one of the test failed
> > +    if ! grep -q '</testsuites>' tests-junit.xml; then
> > +        echo "ERROR: tests-junit.xml is incomplete or missing."
> > +        TEST_RESULT=1
> > +    elif grep -q '</failure>' tests-junit.xml; then
> > +        TEST_RESULT=1
> > +    fi
> >  fi
> >  
> >  exit "$TEST_RESULT"
> 
> A couple of things.
> 
> From my experimentation with junit,
> https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1849342222/test_report?job_name=kbl-xtf-x86-64-gcc-debug
> we can also use </error> for classification.  I'm also very disappointed
> in Gitlab classifying <warning> as success.
> 
> Not for this patch, but for XTF I need to be able to express "tolerable
> failure".  (All branches of Xen will run the same tests, and we don't
> have OSSTest to deem "fail never passed" as non-blocking.)
> 
> Even if the job passes overall, I want tolerable failures to show up in
> the UI, so I have to use <failure> in junit.xml.  But that means needing
> to be more selective, and I don't have a good idea of how to do this. 
> (I have one terrible idea, which is </failure type=tolerable"> which
> will escape that grep, but it feels like (ab)buse of XML.)

But that automation/ dir (including the run-tools-tests script) is
per-branch, so you can specify there which tests should be considered
failure and which just warning, no? It will require few more bits in the
script, but fundamentally shouldn't be a problem?

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]