From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B67D1C433F5 for ; Mon, 27 Sep 2021 13:36:29 +0000 (UTC) Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1EC2E61058 for ; Mon, 27 Sep 2021 13:36:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 1EC2E61058 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=konsulko.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.denx.de Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id C046581E47; Mon, 27 Sep 2021 15:36:25 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=none (p=none dis=none) header.from=konsulko.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=konsulko.com header.i=@konsulko.com header.b="ovhJGZu7"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 1896B82903; Mon, 27 Sep 2021 15:36:23 +0200 (CEST) Received: from mail-qk1-x733.google.com (mail-qk1-x733.google.com [IPv6:2607:f8b0:4864:20::733]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 972F081BC8 for ; Mon, 27 Sep 2021 15:36:17 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=none (p=none dis=none) header.from=konsulko.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=trini@konsulko.com Received: by mail-qk1-x733.google.com with SMTP id p4so36905754qki.3 for ; Mon, 27 Sep 2021 06:36:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=r2rPkmC5L3NSorA/zNoIzR1NRvf9aWsmyRtDnC4OUg0=; b=ovhJGZu7offge45Fmvgs0OdzgrmbOfDxMb/RNuDxo4QKi6XvY4IauvWFtt2X+9jyVJ 9QVTtlMzsauIz1bW0sji1+qh6sSygVv3gz7kXlg0bADCTfjb6vsM4W00BLzffOMJIgDl +OEHA6X0hQv28VCbwcIhCkBs4AaToZpH1hz0c= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=r2rPkmC5L3NSorA/zNoIzR1NRvf9aWsmyRtDnC4OUg0=; b=FVtcWu231m3x57Pdo2hZIIiEYUoVJM24bD9t1pCMUqUaWbIiRDzZ5S9HqC9iIfsCGF fI0CLrAx7K+OinTDGgn/Q5VNch/hwfmIF9Ff1c9WHod8h/uZmVeXvR83obHPxVQ4WQ42 n/WTJYUVqfbCc35djWrtB3QxK4kOhUAMEmf5C/CK3jmuZuiG8n0Bstn4CVEY9SoSGxuO KdSnpiOfOnUNE7NH5kwPtm0UqEQbnYkgoOb+/hLXolWgfuEdRrSXsuaK3iXZ8c18pElf hu1XGwELmXgXB5vYNG2bFsVX1mohPz6V3yPHVQYGSUDN2xqYM60YRoUmLN0hmaytriz7 ed2A== X-Gm-Message-State: AOAM531i1BUG3VESa9e4YczezA5SJJfFf/ek8/Ml0uEpHvLbAXAaczLZ J3luynwqLTMdWLUBLcXAfH2bNA== X-Google-Smtp-Source: ABdhPJxN6re+Be3GaoxVZT5w69gYqUylWlDhXg0xPcO8KtIHyGOCudfE1vJGqyTuzfHN3yzeiQud2g== X-Received: by 2002:a37:93c5:: with SMTP id v188mr19302560qkd.57.1632749775893; Mon, 27 Sep 2021 06:36:15 -0700 (PDT) Received: from bill-the-cat (2603-6081-7b01-cbda-ac93-cd04-265f-76da.res6.spectrum.com. [2603:6081:7b01:cbda:ac93:cd04:265f:76da]) by smtp.gmail.com with ESMTPSA id r139sm12444497qke.84.2021.09.27.06.36.14 (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 27 Sep 2021 06:36:15 -0700 (PDT) Date: Mon, 27 Sep 2021 09:36:13 -0400 From: Tom Rini To: Simon Glass Cc: Harald Seiler , U-Boot Mailing List Subject: Re: Two jobs at once on denx-vulcan? Message-ID: <20210927133613.GH31748@bill-the-cat> References: <654d5288acac8259d2fe10abb882a66941b742ef.camel@denx.de> <20210924142009.GE31748@bill-the-cat> <20210924145553.GF31748@bill-the-cat> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="tJbPEdXFtLQcWY9B" Content-Disposition: inline In-Reply-To: X-Clacks-Overhead: GNU Terry Pratchett User-Agent: Mutt/1.9.4 (2018-02-28) X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.2 at phobos.denx.de X-Virus-Status: Clean --tJbPEdXFtLQcWY9B Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Sep 24, 2021 at 05:36:31PM -0600, Simon Glass wrote: > Hi Tom, >=20 > On Fri, 24 Sept 2021 at 08:55, Tom Rini wrote: > > > > On Fri, Sep 24, 2021 at 08:38:49AM -0600, Simon Glass wrote: > > > Hi Tom, > > > > > > On Fri, 24 Sept 2021 at 08:20, Tom Rini wrote: > > > > > > > > On Fri, Sep 24, 2021 at 04:01:21PM +0200, Harald Seiler wrote: > > > > > Hi Simon, > > > > > > > > > > On Mon, 2021-09-20 at 08:06 -0600, Simon Glass wrote: > > > > > > Hi Harald, > > > > > > > > > > > > On Mon, 20 Sept 2021 at 02:12, Harald Seiler wrot= e: > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > On Sat, 2021-09-18 at 10:37 -0600, Simon Glass wrote: > > > > > > > > Hi, > > > > > > > > > > > > > > > > Is there something screwy with this? It seems that denx-vul= can does > > > > > > > > two builds at once? > > > > > > > > > > > > > > > > https://source.denx.de/u-boot/custodians/u-boot-dm/-/jobs/3= 23540 > > > > > > > > > > > > > > Hm, I did some changes to the vulcan runner which might have = caused > > > > > > > this... But still, even if it is running multiple jobs in par= allel, they > > > > > > > should still be isolated, so how does this lead to a build fa= ilure? > > > > > > > > > > > > I'm not sure that it does, but I do see this at the above link: > > > > > > > > > > > > Error: Unable to create > > > > > > '/builds/u-boot/custodians/u-boot-dm/.git/logs/HEAD.lock': File > > > > > > exists. > > > > > > > > > > This is super strange... Each build should be running in its own > > > > > container so there should never be a way for such a race to occur= =2E No > > > > > clue what is going on here... > > > > > > > > I know this from having to track down a different oddball failure w= ith > > > > konsulko-bootbake. It comes down to something along the lines of > > > > volumes being re-used. Good in that it means that every job every = time > > > > isn't doing a whole clone of the u-boot tree. Bad in that just in = case > > > > the job gets wedged/killed in a crazy spot you end up with problems= like > > > > this. If you run a 'find' on vulcan you'll figure out which overla= y has > > > > a problem. Or you can stop the runner for a moment and tell docker= to > > > > purge unused volumes and it'll clear it up. > > > > > > > > > > Re doing multiple builds, have you set it up so it doesn't take= on the > > > > > > very large builds? I would love to enable multiple builds for t= he qemu > > > > > > steps since they mostly use a single CPU, but am not sure how t= o do > > > > > > it. > > > > > > > > > > Actually, this was more a mistake than an intentional change. I = updated > > > > > the runner on vulcan to also take jobs for some other repos and w= anted > > > > > those jobs to run in parallel. It looks like I just forgot setti= ng the > > > > > `limit =3D 1` option for the U-Boot runner. > > > > > > > > > > Now, I think doing what you suggest is possible. We need to tag = build > > > > > and "test" jobs differently and then define multiple runners with > > > > > different limits. E.g. in `.gitlab-ci.yml`: > > > > > > > > > > build all 32bit ARM platforms: > > > > > stage: world build > > > > > tags: > > > > > - build > > > > > > > > > > cppcheck: > > > > > stage: testsuites > > > > > tags: > > > > > - test > > > > > > > > > > And then define two runners in `/etc/gitlab-runner/config.toml`: > > > > > > > > > > concurrent =3D 4 > > > > > > > > > > [[runners]] > > > > > name =3D "u-boot builder on vulcan" > > > > > limit =3D 1 > > > > > ... > > > > > > > > > > [[runners]] > > > > > name =3D "u-boot tester on vulcan" > > > > > limit =3D 4 > > > > > ... > > > > > > > > > > and during registration they get the `build` and `test` tags > > > > > respectively. This would allow running (in this example) up to 4= test > > > > > jobs concurrently, but only ever one large build job at once. > > > > > > > > Yes, but this would also make it harder for people to use the CI as= -is > > > > with their own runners. For example, the only thing stopping people > > > > from using the free gitlab CI runners on their own is that squashfs > > > > test being broken. > > > > > > Thanks for the info Harald. > > > > > > Would it just mean that they would need to add both 'build' and 'test' > > > tags to their running? If so that does not sound onerous. > > > > Along with not being able to use the gitlab free runners. > > > > > I believe it would speed up CI quite a bit. > > > > I'm not sure? First, did you upgrade your runners recently? I started > > by looking at > > https://source.denx.de/u-boot/u-boot/-/pipelines/9238/builds and all of > > the last stage jobs went super quick. But second, assuming the time >=20 > They are the same as ever: tui did about 1 build per second on average > and kaki did 0.5 builds per second, but this has slowed by about 15% > recently. They are both have quite a few cores. It could just be that > the other two runners were busy so kaki and tui did everything. >=20 > > there includes spinning up the runner, sandbox+clang took 2x as long to > > run as regular sandbox, to run less tests: > > https://source.denx.de/u-boot/u-boot/-/jobs/326772 > > https://source.denx.de/u-boot/u-boot/-/jobs/326773 >=20 > Yes but tui is 2x as fast as kaki (both in terms of number of CPUs and > single-threaded performance) so that might explain it. >=20 > > > > But we might save a minute, or two, if all of the other much quicker > > tests ran to completion sooner, but we'd still be stuck waiting on the > > longest running test. >=20 > Yes, which can be many minutes. But each qemu run takes a good minute > and we have about 30 of them now. Even if all four runners are running > on them, then that is 7 minutes. In parallel it might only take a > minute or two. >=20 > > > > So while I think splitting the job in to stages, such that if something > > fails early we call it all off, a time test where we just have a single > > stage would mean more stuff in parallel and maybe would be quicker, > > especially when we have more free runners. And to me, sadly, that's our > > biggest gating factor and the one that can be solved with money rather > > than technical wizardry. >=20 > Make sense. The other problem is that, to run the tests in parallel, > we might need to clean some of them up (the series I sent is a start > on that). But I think tui could probably run all the qemu jobs in > parallel at once, for example. >=20 > So perhaps we can come back to this when we get parallel tests > running. It definitely is not efficient at present, in the second > (qemu) stage. OK. And I guess the other part of this would be that you could take tui/kaki/etc out of general rotation for a bit and run some pipelines to see what the time change is with your ideas in place. --=20 Tom --tJbPEdXFtLQcWY9B Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGzBAABCgAdFiEEGjx/cOCPqxcHgJu/FHw5/5Y0tywFAmFRyM0ACgkQFHw5/5Y0 tyxrawwAn2AVLmlXSx3CYIpwWR9Id0NbdLhxpe64DQQ8EZ0H4fejUylFFKFIkHW9 beoVpB8kV8IF1qB22mITFvepUfBG6HxfaAKcjszhOhZ8naWg+DtYssJEq/Xu9p83 1WIN8fvYeMh771BIbTegI4sL+LXN720WUndcTqpb5ENx4nlcBM3j287EqvAFaPM9 9NVRaiciArIgX5NB5HcejWq5CznB0xD0FzH7ZAY9UmVjVYoPaydHGt2C8qt0bwbM s4Vu9TxjwoS69G88xavhQFWnwGx3bgK44e90kmAnC7TpvXcf7UN+XjcUszYIBWBg gnD0MIxdrerjgzX0fplrhYZWnSuAnZDS6ZDhnUIpm2xJw7+7UF5OtwJnCwcRa2As Gufj4aNRBgiUB0BDeoXmiB3c0+SpDhhAbNyTzNi0qpctwLTubAUlhCMM3++mhl3S fe4P2Y9pHeU1upxSxoG3RX5KtvVhN3c+nnO9OETUiQlZZzT7pBsfqWsbWiFJEdcD yIzxkToS =9uay -----END PGP SIGNATURE----- --tJbPEdXFtLQcWY9B--