From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A5508C5AE59 for ; Tue, 3 Jun 2025 15:59:04 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.1004642.1384357 (Exim 4.92) (envelope-from ) id 1uMU2K-0003Xj-KT; Tue, 03 Jun 2025 15:58:48 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 1004642.1384357; Tue, 03 Jun 2025 15:58:48 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uMU2K-0003Xc-Hp; Tue, 03 Jun 2025 15:58:48 +0000 Received: by outflank-mailman (input) for mailman id 1004642; Tue, 03 Jun 2025 15:58:47 +0000 Received: from mail.xenproject.org ([104.130.215.37]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uMU2J-0003XG-DK for xen-devel@lists.xenproject.org; Tue, 03 Jun 2025 15:58:47 +0000 Received: from xenbits.xenproject.org ([104.239.192.120]) by mail.xenproject.org with esmtp (Exim 4.96) (envelope-from ) id 1uMU2I-000GMx-2D; Tue, 03 Jun 2025 15:58:46 +0000 Received: from [2a01:e0a:1da:8420:b77:bd5:6e45:7633] (helo=l14) by xenbits.xenproject.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1uMU2I-006BcW-2J; Tue, 03 Jun 2025 15:58:46 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=xenproject.org; s=20200302mail; h=In-Reply-To:Content-Transfer-Encoding: Content-Type:MIME-Version:References:Message-ID:Subject:Cc:To:From:Date; bh=WgWMHehosbmoZ5s9vhs7cuJMPBtHntjnuJFYBdCbYLU=; b=qTBwXa7aRB5QN44d3OTwvxfYMr gz0HUcD1wMoV6IYLijnDi1g7MjoTnrMJtGr8bBGfg/LLUDqPYSfdMIBVnJddHMNd71DXV0KLpVvqH ISnhwV/A3oWxO3e4C0kBY49wmpWF4BXDDbVasas81obfxStia0kvA8mgWh+IhRzkScFU=; Date: Tue, 3 Jun 2025 17:58:44 +0200 From: Anthony PERARD To: Andrew Cooper Cc: xen-devel@lists.xenproject.org, Marek =?iso-8859-1?Q?Marczykowski-G=F3recki?= , Anthony PERARD , Doug Goldstein , Stefano Stabellini Subject: Re: [XEN PATCH 5/9] CI: Have the gitlab job fail on tools/tests failure Message-ID: References: <20250603124222.52057-1-anthony@xenproject.org> <20250603124222.52057-6-anthony@xenproject.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: On Tue, Jun 03, 2025 at 02:41:50PM +0100, Andrew Cooper wrote: > On 03/06/2025 1:42 pm, Anthony PERARD wrote: > > if [ -n "$retrieve_xml" ]; then > > nc -w 10 "$SUT_ADDR" 8080 > tests-junit.xml > + # Findout if one of the test failed > > + if ! grep -q '' tests-junit.xml; then > > + echo "ERROR: tests-junit.xml is incomplete or missing." > > + TEST_RESULT=1 > > + elif grep -q '' tests-junit.xml; then > > + TEST_RESULT=1 > > + fi > > fi > > > > exit "$TEST_RESULT" > > A couple of things. > > From my experimentation with junit, > https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1849342222/test_report?job_name=kbl-xtf-x86-64-gcc-debug > we can also use for classification.  I'm also very disappointed > in Gitlab classifying as success. According to the documentation [1] which point to this junit xml format [2] the only elements (and path) are: testsuites.testsuite.testcase.failure "error" or "warning" don't exist. There's the attributes `type` in but this isn't explained how it's used. But I guess if we follow the link in [2], go through web.archive.org, we can find [3] which has "skipped", "error", "failure", but still no "warning". [1] https://docs.gitlab.com/ci/testing/unit_test_reports/#unit-test-reporting-workflow [2] https://www.ibm.com/docs/en/developer-for-zos/16.0?topic=formats-junit-xml-format [3] https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd > Not for this patch, but for XTF I need to be able to express "tolerable > failure".  (All branches of Xen will run the same tests, and we don't > have OSSTest to deem "fail never passed" as non-blocking.) According to [1], there's a notion of "Existing failures", but that might show up only on merge request. > Even if the job passes overall, I want tolerable failures to show up in > the UI, so I have to use in junit.xml.  But that means needing > to be more selective, and I don't have a good idea of how to do this.  > (I have one terrible idea, which is which > will escape that grep, but it feels like (ab)buse of XML.) At the moment, `run-tools-tests` write '' at first because that's much less likely to be written differently, while the attributes in the tag could be written in a different order. Then, we can always use `sed` and extract the "type" to check it: sed -n 's/.*