From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20022C433EF for ; Tue, 19 Oct 2021 22:53:37 +0000 (UTC) Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8CDE5610E5 for ; Tue, 19 Oct 2021 22:53:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 8CDE5610E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=konsulko.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.denx.de Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id EC51F81725; Wed, 20 Oct 2021 00:53:34 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=none (p=none dis=none) header.from=konsulko.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=konsulko.com header.i=@konsulko.com header.b="oCeYJs90"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 200F0829FC; Wed, 20 Oct 2021 00:53:33 +0200 (CEST) Received: from mail-qt1-x82b.google.com (mail-qt1-x82b.google.com [IPv6:2607:f8b0:4864:20::82b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 08B9C82DF7 for ; Wed, 20 Oct 2021 00:53:29 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=none (p=none dis=none) header.from=konsulko.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=trini@konsulko.com Received: by mail-qt1-x82b.google.com with SMTP id w2so1500521qtn.0 for ; Tue, 19 Oct 2021 15:53:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=vcFdqw5VLR79bFw2MPgXO9wHzQXouIUiUrVp0UTX+lE=; b=oCeYJs90iD4pZFPy0OCEK+1T1ngItfB2EqrBUHAOZy+sRU0I1IzdGj0g1OpLgYUT5D bxnlfWTYN5srdldgYjhTBRjX+EOdE2FVEqUsPt0Vit1q2XDRUk1oIZ9CLVI06RAV72Hg dUOsxKY1e3z87DQoFVPDQSFwKg9df+IvJ/8Ug= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=vcFdqw5VLR79bFw2MPgXO9wHzQXouIUiUrVp0UTX+lE=; b=Pc9KQKto1SMQdmYofhHxWYLrs6aFIo3HBwzG7p+oIz+jhfqkyt4J9H2xqNGoUfeHyK z1wC3zcEj7li7AtMJToSIfXXDGakf662f2XRvtW7avVHAYdvhYIT2PMmM5xaPlzLuJOi /geWBVAswlNWczHJ3HmRTNgijWcecReXZDDMQ7uqcnCtODLcMyx/Ogaec9BZRDorB9qt Jhadom3Zf5FcZlS1RXpuG+Gi9g+O052jw8D2lkhrLGu61dACuLURZzgpn3CLqVuSWo3h SBPJ5N8GHKyJLQgNGeZG4iC/k7sI8nbjtdfRx2EMi0gnq9Q0p2bOy0Ayl0lNbsLf0cfH f6AA== X-Gm-Message-State: AOAM532Dg3VG3iDdpf2zORAijhSXfuYBImEgbu6O44Pa8WZVfpmPCfZf 3e5G4X0elXBicCPHvHMPwH+E/g== X-Google-Smtp-Source: ABdhPJydIYxJHWq3RjGqVMPuoc0VtWwG4eXdGy8TY7DSzv/NHpLlWLNDN3nUCD7LzcJ3Bi0ZL4UR1Q== X-Received: by 2002:ac8:5cc5:: with SMTP id s5mr3105149qta.347.1634684007716; Tue, 19 Oct 2021 15:53:27 -0700 (PDT) Received: from bill-the-cat (2603-6081-7b01-cbda-815c-b889-de4c-aa50.res6.spectrum.com. [2603:6081:7b01:cbda:815c:b889:de4c:aa50]) by smtp.gmail.com with ESMTPSA id x125sm220792qkd.8.2021.10.19.15.53.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Oct 2021 15:53:26 -0700 (PDT) Date: Tue, 19 Oct 2021 18:53:25 -0400 From: Tom Rini To: Stefano Babic , Simon Glass Cc: "U-Boot@lists.denx.de" Subject: Re: buildman stops (crashed) on current master Message-ID: <20211019225325.GK7964@bill-the-cat> References: <720a1fe2-894d-0e6c-ece5-b3c737857dd7@denx.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="ti5ZTCIkqg9s5lPK" Content-Disposition: inline In-Reply-To: X-Clacks-Overhead: GNU Terry Pratchett X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.2 at phobos.denx.de X-Virus-Status: Clean --ti5ZTCIkqg9s5lPK Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Oct 19, 2021 at 05:39:12PM +0200, Stefano Babic wrote: > Hi Simon, >=20 > On 07.10.21 15:43, Simon Glass wrote: > > Hi Stefano, > >=20 > > On Thu, 7 Oct 2021 at 04:37, Stefano Babic wrote: > > >=20 > > > Hi all, > > >=20 > > > CI stops by building aarch64 without notice, for reference: > > >=20 > > > https://source.denx.de/u-boot/custodians/u-boot-imx/-/jobs/332319 > > >=20 > > > There is no error, just process is killed. It looks like it stops at > > > xilinx_zynqmp_virt, > > >=20 > > > ./tools/buildman/buildman -o /tmp -P -E -W aarch64but board can be bu= ilt > > > without issues. > > >=20 > > > If I build on my host (not in docker, anyway), it generally builds fi= ne > > > - but it crashes sometimes, too. On gitlab instance , it crashes. > > > Issue does not seem that depends on merged patches, and introduces > > > boards were already built successfully. Any hint ? I have also no idea > > > what I should look as what I see is just > > >=20 > > > "usr/bin/bash: line 104: 24 Killed > > > ./tools/buildman/buildman -o /tmp -P -E -W aarch64" > >=20 > > I cannot see that link. I am not sure what is going on. Does it say > > what signal killed it? >=20 > Pipelines on our server were not public - I have enbaled now for u-boot-i= mx. >=20 > >=20 > > Does it sit there for an hour and timeout? If so, then I did see that > > myself once recently, when the Kconfig needed stdin, but I could not > > quitetie it down. I think buildman would provide it, but sometimes > > not, apparently. So it can happen when there is an existing build > > there and your new one which adds Kconfig options that don't have > > defaults, or something like that? > >=20 >=20 > I have investigated further, and I can reproduce it on my host outside the > gitlab server. buildman causes a OOM, but I cannot find the cause. >=20 > Strange enough, this happens with the "aarch64" target, and I cannot > reproduce it with Tom's master. So it seems that -master is ok, and somet= hin > on u-boot-imx generates the OOM. >=20 > However.... >=20 > The OOM happens always when -2 (two boards remain) appears. I can see with > htop that buildman starts to allocate memory until it is exhausted (64GB = RAM > + 8 GB swap). Then the kernel decides that it is enough and kills buildma= n - > this is what I see on Ci. >=20 > You can see now the pipelines: >=20 > https://source.denx.de/u-boot/custodians/u-boot-imx/-/pipelines/9520 >=20 > I have then split aarch64 and I built imx8 separately - same result. The > pipeline stops with xilinx board, but they have nothing to do. In fact, I > can build all xilinx board separately. If I run buildman -W aarch64 -x > xilinx, OOM is shown by another board. >=20 > Strange enough, I can build each single board with buildman without issue= s, > neither errors nor warnongs. Just when buildman runs all together (aarch6= 4, > 308 boards), the OOM is generated. >=20 > Bisect does not help: I started bisect, and at the end this commit was > presented: >=20 > commit 53a24dee86fb72ae41e7579607bafe13442616f2 > Author: Fabio Estevam > Date: Mon Aug 23 21:11:09 2021 -0300 >=20 > imx8mm-cl-iot-gate: Split the defconfigs I strongly suspect what's going on here is that these new defconfigs are out of sync with changes now in Kconfig. The build itself will just sit there, waiting for the "oldconfig" prompt to be answered. I want to say the problem here is that stdin is open, rather than pointing to something closed and would lead to the build failing immediately, rather than once a timeout is hit, or OOM kicks in due to kconfig chewing up all the memory. --=20 Tom --ti5ZTCIkqg9s5lPK Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGzBAABCgAdFiEEGjx/cOCPqxcHgJu/FHw5/5Y0tywFAmFvTGEACgkQFHw5/5Y0 tyz8Ugv+Pbteml+t6u3mwhSAi1Zwvwa/3YJd4bK4qXYaU1QR0+FFYjfMXhHGZiCh Lp8UZGgaIpa/AEl59T5hyAlgiwu8Cif3xJh9kImWfUn+Q52M7jSCprTIrKrK4qce 8JWYoKRxmlg1WldxlJViWCVXue54zTPjOkntMjAxtC18slsjMB/bTe1TDZV52uiu sKtw+BKKiToqDlGFqlEtxl/9/7Sa46G5MoweEegKLUAbd9mGPMiUJns2FpawWF3u BN8SDXTJLLGcUQpp3+3XWBFibTqBBND/WvIp8reFa6FnCT71eohDUM/8b4RxKO0S 6FC4k2RiloCMf0NZSYJtiPR/lfSAVx98kizNgj1mscRg9wnfAS+f4K8GyiYshNum r4kJ0Ec370jCAmLYmGx/lr8OvTSTK6tNlZ9qPyr2dKcnqoX6r78mFJhQ7xZjqxcV ScmA+iaivdLan+c9TS1UJyFdHnwAu3p5suY3DalI/k1wTrjhI/V7MdiR8uUyk+ZO 4iStbbVt =1m6K -----END PGP SIGNATURE----- --ti5ZTCIkqg9s5lPK--