Re: [XEN PATCH 4/9] CI: Ignore run-tools-test return value

["Marek Marczykowski-Górecki" <marmarek@invisiblethingslab.com>]

[-- Attachment #1: Type: text/plain, Size: 2491 bytes --]

On Tue, Jun 03, 2025 at 02:26:31PM +0100, Andrew Cooper wrote:
> On 03/06/2025 1:42 pm, Anthony PERARD wrote:
> > diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
> > index 2750d24eba..046137a4a6 100755
> > --- a/automation/scripts/qubes-x86-64.sh
> > +++ b/automation/scripts/qubes-x86-64.sh
> > @@ -135,10 +135,11 @@ done
> >      ### tests: tools-tests-pv, tools-tests-pvh
> >      "tools-tests-pv"|"tools-tests-pvh")
> >          retrieve_xml=1
> > -        passed="test passed"
> > +        passed="run-tools-test over"
> >          domU_check=""
> >          dom0_check="
> > -/root/run-tools-tests /usr/lib/xen/tests /tmp/tests-junit.xml && echo \"${passed}\"
> > +/root/run-tools-tests /usr/lib/xen/tests /tmp/tests-junit.xml ||:
> > +echo \"${passed}\"
> >  nc -l -p 8080 < /tmp/tests-junit.xml >/dev/null &
> >  "
> >          if [ "${test_variant}" = "tools-tests-pvh" ]; then
> 
> I noticed this too while hacking on XTF support.  Also of note, I'm not
> sure we want to be saying done to the expect script before sending the
> results.

No, expect needs to be notified before, otherwise retrieving the file
won't start.

> The underlying problem is that ${passed} isn't really a pass message;
> it's a done+pass message, and there is no way to say "failed" to the
> expect script, leaving timeout as the only option of signalling a failure.

Well, that can be changed, by adding yet another message that it looks
for, and handling it with exit 1 instead of exit 0.

> I don't think we want to be ignoring errors from run-tools-tests
> specifically, but arranging this is hard.  You always need something at
> an outer level judging whether there was a good test result.

For those tests specifically, if changing to always say success, you
need to ensure that both finding "failure" in xml and not getting the
xml at all (or getting it malformed) is treated as a failure.

> For XTF, the way I did this was to declare that anything didn't match
> "^(Success|Error|Crash|Fail).*" on the final line of the console log to
> be deemed to be Crash (because it's usually an uncaught exception).
> 
> For this, the best I can think of is to have a fixed outer script which
> runs "./test/script && echo $success || echo $failure".  The internal
> set -e will cause most unexpected conditions to be an error.
> 
> ~Andrew

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]