From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 69115C001B0 for ; Thu, 13 Jul 2023 21:57:49 +0000 (UTC) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 6C08D86CD7; Thu, 13 Jul 2023 23:57:47 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=konsulko.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=konsulko.com header.i=@konsulko.com header.b="Mzk01kat"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id C7F7286CCF; Thu, 13 Jul 2023 23:57:46 +0200 (CEST) Received: from mail-yb1-xb2c.google.com (mail-yb1-xb2c.google.com [IPv6:2607:f8b0:4864:20::b2c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 667C186CE5 for ; Thu, 13 Jul 2023 23:57:44 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=konsulko.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=trini@konsulko.com Received: by mail-yb1-xb2c.google.com with SMTP id 3f1490d57ef6-cb4de3bd997so1233807276.1 for ; Thu, 13 Jul 2023 14:57:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; t=1689285463; x=1691877463; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=hSPTqHrYwV12ruQuASfL0CLAENqQQ9uRdED2wmVmBYA=; b=Mzk01katr2usIlBGJSCeIX9hIq54JkSDvRMRrchMNmYNwDTv+uJd5+rmzukoEgW5h8 mP0YmoJ0qie1U+D69JGaC5Il/Le6ev7hoSWgWIO5UWtwsYna9cih1MO4d2x0DHHZoWlw W7pD8E80H/2addtZluDllrT3WdX9039w7eEns= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689285463; x=1691877463; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=hSPTqHrYwV12ruQuASfL0CLAENqQQ9uRdED2wmVmBYA=; b=ce8V01cYNKc061W+DdPBwU6wsPdbLgszw9HWvXWnSWTkK5UV3E2ATBHv7Ag7t54THF f6vcprk+3c2gGQRMYfiBF96IbHI/tI8Ac/zLvASI6ydJlhcx7IxYj5fisCKx+8iZNQnn R4TowotlEcBo4qlfljsCb/U8ir9QlGvPgvvmU2qWcfRkty+QariWH5P6A+DameW/dowW l6d4oQGKPfpynvTuXhQ3mlisaFoiyty8lYH+0cKTG261Z/jmXlISO2CbebnIN1e1cGbm mPs9xN9hf0soIv35Z+qKtBWXPYTFIbhJvOA9BvzpZoqkLpNPTHJDNCeJ7be/P2Gim7VF KJlQ== X-Gm-Message-State: ABy/qLYkrByY8Jf1+WHjh4XCk2YCd5ziyeXMoioC4lfqJVTOGRORFcTG /89xC98EFQ0qh5SC3cv9rl8rB40+deyhdVrjurznJg== X-Google-Smtp-Source: APBJJlHZcvpLymXdQ6G18ISfFYVgE+dv1r3KZfQlqJHcv0/C7v2kSn7wg2aVZmuKmJXgmbkSX2h7LA== X-Received: by 2002:a81:6983:0:b0:56d:2e66:4a3f with SMTP id e125-20020a816983000000b0056d2e664a3fmr1280661ywc.0.1689285462903; Thu, 13 Jul 2023 14:57:42 -0700 (PDT) Received: from bill-the-cat (2603-6081-7b00-6400-cb70-e592-de93-e6d2.res6.spectrum.com. [2603:6081:7b00:6400:cb70:e592:de93:e6d2]) by smtp.gmail.com with ESMTPSA id v186-20020a8161c3000000b005612fc707bfsm1963862ywb.120.2023.07.13.14.57.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jul 2023 14:57:42 -0700 (PDT) Date: Thu, 13 Jul 2023 17:57:40 -0400 From: Tom Rini To: Simon Glass Cc: u-boot@lists.denx.de Subject: Re: [PATCH] CI: Add automatic retry for test.py jobs Message-ID: <20230713215740.GP2066889@bill-the-cat> References: <20230712023303.1969966-1-trini@konsulko.com> <20230712170859.GA2066889@bill-the-cat> <20230712203839.GJ2066889@bill-the-cat> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="kdLkXJqHH47tQkp0" Content-Disposition: inline In-Reply-To: X-Clacks-Overhead: GNU Terry Pratchett X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.8 at phobos.denx.de X-Virus-Status: Clean --kdLkXJqHH47tQkp0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jul 13, 2023 at 03:03:57PM -0600, Simon Glass wrote: > Hi Tom, >=20 > On Wed, 12 Jul 2023 at 14:38, Tom Rini wrote: > > > > On Wed, Jul 12, 2023 at 02:32:18PM -0600, Simon Glass wrote: > > > Hi Tom, > > > > > > On Wed, 12 Jul 2023 at 11:09, Tom Rini wrote: > > > > > > > > On Wed, Jul 12, 2023 at 08:00:23AM -0600, Simon Glass wrote: > > > > > Hi Tom, > > > > > > > > > > On Tue, 11 Jul 2023 at 20:33, Tom Rini wrote: > > > > > > > > > > > > It is not uncommon for some of the QEMU-based jobs to fail not = because > > > > > > of a code issue but rather because of a timing issue or similar= problem > > > > > > that is out of our control. Make use of the keywords that Azure= and > > > > > > GitLab provide so that we will automatically re-run these when = they fail > > > > > > 2 times. If they fail that often it is likely we have found a r= eal issue > > > > > > to investigate. > > > > > > > > > > > > Signed-off-by: Tom Rini > > > > > > --- > > > > > > .azure-pipelines.yml | 1 + > > > > > > .gitlab-ci.yml | 1 + > > > > > > 2 files changed, 2 insertions(+) > > > > > > > > > > This seems like a slippery slope. Do we know why things fail? I w= onder > > > > > if we should disable the tests / builders instead, until it can be > > > > > corrected? > > > > > > > > It happens in Azure, so it's not just the broken runner problem we = have > > > > in GitLab. And the problem is timing, as I said in the commit. > > > > Sometimes we still get the RTC test failing. Other times we don't g= et > > > > QEMU + U-Boot spawned in time (most often m68k, but sometimes x86). > > > > > > How do we keep this list from growing? > > > > Do we need to? The problem is in essence since we rely on free > > resources, sometimes some heavy lifts take longer. That's what this > > flag is for. >=20 > I'm fairly sure the RTC thing could be made deterministic. We've already tried that once, and it happens a lot less often. If we make it even looser we risk making the test itself useless. > The spawning thing...is there a timeout for that? What actually fails? It doesn't spawn in time for the framework to get to the prompt. We could maybe increase the timeout value. It's always the version test that fails. > > > > > I'll note that we don't have this problem with sandbox tests. > > > > > > > > OK, but that's not relevant? > > > > > > It is relevant to the discussion about using QEMU instead of sandbox, > > > e.g. with the TPM. I recall a discussion with Ilias a while back. > > > > I'm sure we could make sandbox take too long to start as well, if enough > > other things are going on with the system. And sandbox has its own set > > of super frustrating issues instead, so I don't think this is a great > > argument to have right here (I have to run it in docker, to get around > > some application version requirements and exclude event_dump, bootmgr, > > abootimg and gpt tests, which could otherwise run, but fail for me). >=20 > I haven't heard about this before. Is there anything that could be done? I have no idea what could be done about it since I believe all of them run fine in CI, including on this very host, when gitlab invokes it rather than when I invoke it. My point here is that sandbox tests are just a different kind of picky about things and need their own kind of "just hit retry". --=20 Tom --kdLkXJqHH47tQkp0 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGzBAABCgAdFiEEGjx/cOCPqxcHgJu/FHw5/5Y0tywFAmSwc1QACgkQFHw5/5Y0 tyzHZAv+OXZTMxm9/YeSaCHKSib9BBGO8ysnSx28ozD5cN16UUPEcjKTYIZoEPjF D46iYvndS2nPEGg/MfdsbZc9UwmWNTyT7DDI+HbJYk5z7b+ejmtriB2Vu2l4McUV Z2mulQjp4V10Nr33u7ue3+m3cIy2NvE8xEkDkNVI/xVqNbZlmgAND7pth9xYbijM fIeX5MRrTF+UyKJmHKhYy++RuPDG3Lp8BtCxwaD6/jgFGCMzuh57cjF/pFpsyIwp RYWwo/Up5NnbL14Io/3Jn9q7tM3WmvFZGS5qcWeLxGoMy09K+rV/B4zp8+3O1sme r57dIPhfHSnkSfqSVCSO3OwoZOsdfEJAyz8TAk7S6IOF1LXf6mLWo68wkvjD0nmt 1n3BX402PmQI3gPpuT1JuUr/py6AXnmQYcVVXafd6ZDXzggZ0xNGg0PRJGQ171zt y1+rpL275m/M6qrdnkvxz1QCl3adxEY71ayrpurlTIpA0UaEHGfzZ0NCFlYpwO4m J6vwGJRq =jeva -----END PGP SIGNATURE----- --kdLkXJqHH47tQkp0--