Re: [XEN PATCH 5/9] CI: Have the gitlab job fail on tools/tests failure

[Anthony PERARD <anthony@xenproject.org>]

On Tue, Jun 03, 2025 at 02:41:50PM +0100, Andrew Cooper wrote:
> On 03/06/2025 1:42 pm, Anthony PERARD wrote:
> >  if [ -n "$retrieve_xml" ]; then
> >      nc -w 10 "$SUT_ADDR" 8080 > tests-junit.xml </dev/null
> > +    # Findout if one of the test failed
> > +    if ! grep -q '</testsuites>' tests-junit.xml; then
> > +        echo "ERROR: tests-junit.xml is incomplete or missing."
> > +        TEST_RESULT=1
> > +    elif grep -q '</failure>' tests-junit.xml; then
> > +        TEST_RESULT=1
> > +    fi
> >  fi
> >  
> >  exit "$TEST_RESULT"
> 
> A couple of things.
> 
> From my experimentation with junit,
> https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1849342222/test_report?job_name=kbl-xtf-x86-64-gcc-debug
> we can also use </error> for classification.  I'm also very disappointed
> in Gitlab classifying <warning> as success.

According to the documentation [1] which point to this junit xml format [2]
the only elements (and path) are:
    testsuites.testsuite.testcase.failure
"error" or "warning" don't exist.
There's the attributes `type` in <failure> but this isn't explained how
it's used.

But I guess if we follow the link in [2], go through web.archive.org, we
can find [3] which has "skipped", "error", "failure", but still no
"warning".


[1] https://docs.gitlab.com/ci/testing/unit_test_reports/#unit-test-reporting-workflow
[2] https://www.ibm.com/docs/en/developer-for-zos/16.0?topic=formats-junit-xml-format
[3] https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd


> Not for this patch, but for XTF I need to be able to express "tolerable
> failure".  (All branches of Xen will run the same tests, and we don't
> have OSSTest to deem "fail never passed" as non-blocking.)

According to [1], there's a notion of "Existing failures", but that
might show up only on merge request.

> Even if the job passes overall, I want tolerable failures to show up in
> the UI, so I have to use <failure> in junit.xml.  But that means needing
> to be more selective, and I don't have a good idea of how to do this. 
> (I have one terrible idea, which is </failure type=tolerable"> which
> will escape that grep, but it feels like (ab)buse of XML.)

At the moment, `run-tools-tests` write '<failure type="failure"' on
failure, so we could grep on that instead event if it is sligtly more
fragile. I've choosen to grep on '</failure>' at first because that's
much less likely to be written differently, while the attributes in the
tag could be written in a different order.

Then, we can always use `sed` and extract the "type" to check it:
sed -n 's/.*<failure \(\|.* \)type="\([^"]\+\)" .*/\2/p' tests-junit.xml | while read type; do
  case $type in
    failure) echo fail;;
    tolerable) echo ok;;
    *) echo error unknwon type $type;;
  esac
done
But that maybe going a bit too far :-)

Cheers,

-- 
Anthony PERARD