From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E0F8EEB64DD for ; Sun, 16 Jul 2023 18:18:08 +0000 (UTC) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id BF98286366; Sun, 16 Jul 2023 20:18:06 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=konsulko.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=konsulko.com header.i=@konsulko.com header.b="HMF34lUs"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 0785386332; Sun, 16 Jul 2023 20:18:06 +0200 (CEST) Received: from mail-yb1-xb36.google.com (mail-yb1-xb36.google.com [IPv6:2607:f8b0:4864:20::b36]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 32BA58630D for ; Sun, 16 Jul 2023 20:18:02 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=konsulko.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=trini@konsulko.com Received: by mail-yb1-xb36.google.com with SMTP id 3f1490d57ef6-cc7863e7b82so1777963276.0 for ; Sun, 16 Jul 2023 11:18:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; t=1689531481; x=1692123481; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=iCzi//8bISV+1eP1SdyC/09x7bCHpdRoNXa9HGdNQI8=; b=HMF34lUsgLythS5XwaLw++gH9lm/ukYrxtRVb24j3HaEXXjqgxD4E7iqemhtv+jYOi aNrIz2tJTf89DqhvN9qT/nCqFO4Xcfz01oUEhkNp/4fPIosEKpIHMB/y5UAwfVHzaLjB yc1UGESNOgBnVZlLFUDWhjtn6xM+w6TTLXzsw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689531481; x=1692123481; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=iCzi//8bISV+1eP1SdyC/09x7bCHpdRoNXa9HGdNQI8=; b=W3Io6i86qTNfZGoaZWf0T6g02wZYUhKYAS3ELwRMunQl3yp7po18UMUwwv9lc22+Xf elViyZT9Gu775WP12RQJA7lPdnabglPnlGLni6DE1ONJkfhgoBHE8VTQMS7wGis6oSSH SF16EvyyMxaJF30KaF7a9Wj58qcr4GPtVFyt38CnJvUB/iV4jWPdYG0sqINJouRlyRu/ VYT4ec5pewHXYD/2B2rFlppaOhdZ42kIoXrqX5g4NULOJysuE8hoHZvAhsd1nYK1VI63 Feer7Ye8CcP0nXEhF3nERZK2CBmYH5gQvin3sTYe5/1L04gfliUSjvPnFDBsAH5LLIDZ /Ilg== X-Gm-Message-State: ABy/qLatDgxGB8MBku0jmwhKTURhGFkHtzgalMBVAtqHY0eREQhEQm3b lu0cWtMG5fGa5RRX6ncoRHX1VnqNdkcPuk6LINFhKw== X-Google-Smtp-Source: APBJJlESiaMBc2HqUQtYkP+gHxk9XqmgZpIfuHWDm+FD75H1ygKiTqMQZ6p46wfYaPQK4W/9bmylGw== X-Received: by 2002:a05:6902:311:b0:cc1:95c2:9130 with SMTP id b17-20020a056902031100b00cc195c29130mr6553531ybs.4.1689531480689; Sun, 16 Jul 2023 11:18:00 -0700 (PDT) Received: from bill-the-cat (2603-6081-7b00-6400-5e84-cf0c-0c9a-a3e2.res6.spectrum.com. [2603:6081:7b00:6400:5e84:cf0c:c9a:a3e2]) by smtp.gmail.com with ESMTPSA id 2-20020a251302000000b00bdd085da648sm2705959ybt.48.2023.07.16.11.18.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Jul 2023 11:18:00 -0700 (PDT) Date: Sun, 16 Jul 2023 14:17:58 -0400 From: Tom Rini To: Simon Glass Cc: u-boot@lists.denx.de Subject: Re: [PATCH] CI: Add automatic retry for test.py jobs Message-ID: <20230716181758.GT2066889@bill-the-cat> References: <20230712023303.1969966-1-trini@konsulko.com> <20230712170859.GA2066889@bill-the-cat> <20230712203839.GJ2066889@bill-the-cat> <20230713215740.GP2066889@bill-the-cat> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="MFp5tuWu527IM0gL" Content-Disposition: inline In-Reply-To: X-Clacks-Overhead: GNU Terry Pratchett X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.8 at phobos.denx.de X-Virus-Status: Clean --MFp5tuWu527IM0gL Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jul 15, 2023 at 05:40:25PM -0600, Simon Glass wrote: > Hi Tom, >=20 > On Thu, 13 Jul 2023 at 15:57, Tom Rini wrote: > > > > On Thu, Jul 13, 2023 at 03:03:57PM -0600, Simon Glass wrote: > > > Hi Tom, > > > > > > On Wed, 12 Jul 2023 at 14:38, Tom Rini wrote: > > > > > > > > On Wed, Jul 12, 2023 at 02:32:18PM -0600, Simon Glass wrote: > > > > > Hi Tom, > > > > > > > > > > On Wed, 12 Jul 2023 at 11:09, Tom Rini wrote: > > > > > > > > > > > > On Wed, Jul 12, 2023 at 08:00:23AM -0600, Simon Glass wrote: > > > > > > > Hi Tom, > > > > > > > > > > > > > > On Tue, 11 Jul 2023 at 20:33, Tom Rini w= rote: > > > > > > > > > > > > > > > > It is not uncommon for some of the QEMU-based jobs to fail = not because > > > > > > > > of a code issue but rather because of a timing issue or sim= ilar problem > > > > > > > > that is out of our control. Make use of the keywords that A= zure and > > > > > > > > GitLab provide so that we will automatically re-run these w= hen they fail > > > > > > > > 2 times. If they fail that often it is likely we have found= a real issue > > > > > > > > to investigate. > > > > > > > > > > > > > > > > Signed-off-by: Tom Rini > > > > > > > > --- > > > > > > > > .azure-pipelines.yml | 1 + > > > > > > > > .gitlab-ci.yml | 1 + > > > > > > > > 2 files changed, 2 insertions(+) > > > > > > > > > > > > > > This seems like a slippery slope. Do we know why things fail?= I wonder > > > > > > > if we should disable the tests / builders instead, until it c= an be > > > > > > > corrected? > > > > > > > > > > > > It happens in Azure, so it's not just the broken runner problem= we have > > > > > > in GitLab. And the problem is timing, as I said in the commit. > > > > > > Sometimes we still get the RTC test failing. Other times we don= 't get > > > > > > QEMU + U-Boot spawned in time (most often m68k, but sometimes x= 86). > > > > > > > > > > How do we keep this list from growing? > > > > > > > > Do we need to? The problem is in essence since we rely on free > > > > resources, sometimes some heavy lifts take longer. That's what this > > > > flag is for. > > > > > > I'm fairly sure the RTC thing could be made deterministic. > > > > We've already tried that once, and it happens a lot less often. If we > > make it even looser we risk making the test itself useless. >=20 > For sleep, yes, but for rtc it should be deterministic now...next time > you get a failure could you send me the trace? Found one: https://dev.azure.com/u-boot/u-boot/_build/results?buildId=3D6592&view=3Dlo= gs&j=3Db6c47816-145c-5bfe-20a7-c6a2572e6c41&t=3D0929c28c-6e32-5635-9624-54e= aa917d713&l=3D599 And note that we have a different set of timeout problems that may or may n= ot be configurable, which is in the upload of the pytest results. I haven't se= en if there's a knob for this one yet, within Azure (or the python project we'= re adding for it). > > > The spawning thing...is there a timeout for that? What actually fails? > > > > It doesn't spawn in time for the framework to get to the prompt. We > > could maybe increase the timeout value. It's always the version test > > that fails. >=20 > Ah OK, yes increasing the timeout makes sense. >=20 > > > > > > > > > I'll note that we don't have this problem with sandbox tests. > > > > > > > > > > > > OK, but that's not relevant? > > > > > > > > > > It is relevant to the discussion about using QEMU instead of sand= box, > > > > > e.g. with the TPM. I recall a discussion with Ilias a while back. > > > > > > > > I'm sure we could make sandbox take too long to start as well, if e= nough > > > > other things are going on with the system. And sandbox has its own= set > > > > of super frustrating issues instead, so I don't think this is a gre= at > > > > argument to have right here (I have to run it in docker, to get aro= und > > > > some application version requirements and exclude event_dump, bootm= gr, > > > > abootimg and gpt tests, which could otherwise run, but fail for me). > > > > > > I haven't heard about this before. Is there anything that could be do= ne? > > > > I have no idea what could be done about it since I believe all of them > > run fine in CI, including on this very host, when gitlab invokes it > > rather than when I invoke it. My point here is that sandbox tests are > > just a different kind of picky about things and need their own kind of > > "just hit retry". >=20 > Perhaps this is Python dependencies? I'm not sure, but if you see it > again, please let me know in case we can actually fix this. Alright. So the first pass I took at running sandbox pytest with as little hand-holding as possible I hit the known issue of /boot/vmlinu* being 0400 in Ubuntu. I fixed that and then re-ran and: test/py/tests/test_cleanup_build.py F =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D FAILURES =3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D _________________________________________ test_clean ______________________= ____________________ test/py/tests/test_cleanup_build.py:94: in test_clean assert not leftovers, f"leftovers: {', '.join(map(str, leftovers))}" E AssertionError: leftovers: fdt-out.dtb, sha1-pad/sandbox-u-boot.dtb, sh= a1-pad/sandbox-kernel.dtb, sha1-basic/sandbox-u-boot.dtb, sha1-basic/sandbo= x-kernel.dtb, sha384-basic/sandbox-u-boot.dtb, sha384-basic/sandbox-kernel.= dtb, algo-arg/sandbox-u-boot.dtb, algo-arg/sandbox-kernel.dtb, sha1-pss/san= dbox-u-boot.dtb, sha1-pss/sandbox-kernel.dtb, sha256-pad/sandbox-u-boot.dtb= , sha256-pad/sandbox-kernel.dtb, sha256-global-sign/sandbox-binman.dtb, sha= 256-global-sign/sandbox-u-boot.dtb, sha256-global-sign/sandbox-u-boot-globa= l.dtb, sha256-global-sign/sandbox-kernel.dtb, sha256-global-sign-pss/sandbo= x-binman-pss.dtb, sha256-global-sign-pss/sandbox-u-boot.dtb, sha256-global-= sign-pss/sandbox-kernel.dtb, sha256-global-sign-pss/sandbox-u-boot-global-p= ss.dtb, auto_fit/dt-1.dtb, auto_fit/dt-2.dtb, sha256-pss/sandbox-u-boot.dtb= , sha256-pss/sandbox-kernel.dtb, sha256-pss-pad/sandbox-u-boot.dtb, sha256-= pss-pad/sandbox-kernel.dtb, hashes/sandbox-kernel.dtb, sha256-basic/sandbox= -u-boot.dtb, sha256-basic/sandbox-kernel.dtb, sha1-pss-pad/sandbox-u-boot.d= tb, sha1-pss-pad/sandbox-kernel.dtb, sha384-pad/sandbox-u-boot.dtb, sha384-= pad/sandbox-kernel.dtb, sha256-pss-pad-required/sandbox-u-boot.dtb, sha256-= pss-pad-required/sandbox-kernel.dtb, ecdsa/sandbox-kernel.dtb, sha256-pss-r= equired/sandbox-u-boot.dtb, sha256-pss-required/sandbox-kernel.dtb E assert not [PosixPath('fdt-out.dtb'), PosixPath('sha1-pad/sandbox-u-boo= t.dtb'), PosixPath('sha1-pad/sandbox-kernel.dtb'), PosixPa...ic/sandbox-u-b= oot.dtb'), PosixPath('sha1-basic/sandbox-kernel.dtb'), PosixPath('sha384-ba= sic/sandbox-u-boot.dtb'), ...] ------------------------------------ Captured stdout call -----------------= -------------------- +make O=3D/tmp/pytest-of-trini/pytest-231/test_clean0 clean make[1]: Entering directory '/tmp/pytest-of-trini/pytest-231/test_clean0' CLEAN cmd CLEAN dts/../arch/sandbox/dts CLEAN dts CLEAN lib CLEAN tools CLEAN tools/generated CLEAN include/bmp_logo.h include/bmp_logo_data.h include/generated/env.= in include/generated/env.txt drivers/video/u_boot_logo.S u-boot u-boot-dtb.= bin u-boot-initial-env u-boot-nodtb.bin u-boot.bin u-boot.cfg u-boot.dtb u-= boot.dtb.gz u-boot.dtb.out u-boot.dts u-boot.lds u-boot.map u-boot.srec u-b= oot.sym System.map image.map keep-syms-lto.c lib/efi_loader/helloworld_efi.S make[1]: Leaving directory '/tmp/pytest-of-trini/pytest-231/test_clean0' =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D short test summary info =3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D FAILED test/py/tests/test_cleanup_build.py::test_clean - AssertionError: le= ftovers: fdt-out.... !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!= !!!!!!!!!!!!!!!!!!!! =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D 1 failed, 6 passed in 6.42s =3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D Fixing that manually with an rm -rf of /tmp/pytest-of-trini and now it's stuck. I've rm -rf'd that and git clean -dfx and just repeat that failure. I'm hopeful that when I reboot whatever magic is broken will be cleaned out. Moving things in to a docker container again, I get: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D ERRORS =3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D _______________________________ ERROR at setup of test_gpt_read ___________= ____________________ /home/trini/work/u-boot/u-boot/test/py/tests/test_gpt.py:74: in state_disk_= image ??? /home/trini/work/u-boot/u-boot/test/py/tests/test_gpt.py:37: in __init__ ??? test/py/u_boot_utils.py:279: in __enter__ self.module_filename =3D module.__file__ E AttributeError: 'NoneType' object has no attribute '__file__' =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D short test summary info =3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D ERROR test/py/tests/test_gpt.py::test_gpt_read - AttributeError: 'NoneType'= object has no at... !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!= !!!!!!!!!!!!!!!!!!!! =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D 41 passed, 45 skipped, 1 error in 19.29s =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D And then ignoring that one with "-k not gpt": test/py/tests/test_android/test_ab.py E =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D ERRORS =3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D __________________________________ ERROR at setup of test_ab ______________= ____________________ /home/trini/work/u-boot/u-boot/test/py/tests/test_android/test_ab.py:54: in= ab_disk_image ??? /home/trini/work/u-boot/u-boot/test/py/tests/test_android/test_ab.py:28: in= __init__ ??? test/py/u_boot_utils.py:279: in __enter__ self.module_filename =3D module.__file__ E AttributeError: 'NoneType' object has no attribute '__file__' =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D short test summary info =3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D ERROR test/py/tests/test_android/test_ab.py::test_ab - AttributeError: 'Non= eType' object has... !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!= !!!!!!!!!!!!!!!!!!!! =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 908 passed, 75 skipped, 10 deselect= ed, 1 error in 159.17s (0:02:39) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Now, funny things. If I git clean -dfx, I can then get that test to pass. So I guess something else isn't cleaning up / is writing to a common area? I intentionally build within the source tree, but in a subdirectory of that, and indeed a lot of tests write to the source directory itself. --=20 Tom --MFp5tuWu527IM0gL Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGzBAABCgAdFiEEGjx/cOCPqxcHgJu/FHw5/5Y0tywFAmS0NFMACgkQFHw5/5Y0 tyybGwv/d8f7OpkyTQBKNYcF966/8wf1L6oymMubwmjFtkHz71uPUp+ZEd0ri129 LWNiK39Ar7DDNX2WiiestLDI4/XpMm7EcDmjMeodNtoNF5wTVZBDAYkonP2+sdQO deACSnL7w0sKAaHQ82IhhZkxn2BvsMST4a2xEIytE+y/7bzSGwwTPmFUjkZg2ZEe inTYLF4vO2A7Hdmpf36Nur0S/5uwv1X+7llQr0AbsxSaszuOlG8aqEKctkYkx73M rhc0yj2cjh4XUpbx3VhE5o3+rq2g/4/Ey666gsfGTAAo9WEL1pTUoJWhhpHXjE2C cZnP61sLs90YCyl2RItE1Ppyj3FSiiRI7ei4/EU1BgZCWh6YDQ4BHi5akBmc2BWK 4bbdPCHDDhjnHf1Gmdvq1PK6V52tkSuV2eSrehvhsVAkt0+lyyB6FyBJoQMrFiFT Bgh00sVNpMPjS1C7EfT9YcgW3Dtzk1/I5/OBbSc8piq2iibXCLqTratIp4rGxIot JjQJ6C21 =o3UO -----END PGP SIGNATURE----- --MFp5tuWu527IM0gL--