From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0DBFC433FE for ; Tue, 19 Oct 2021 15:39:18 +0000 (UTC) Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 891AD60FDA for ; Tue, 19 Oct 2021 15:39:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 891AD60FDA Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=denx.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.denx.de Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id BDB158347D; Tue, 19 Oct 2021 17:39:15 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=none (p=none dis=none) header.from=denx.de Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=denx.de; s=phobos-20191101; t=1634657956; bh=mVVjMmNEI2N5/6V5Ko4xGAOGQsyL99PYB8U/gd/d09I=; h=Subject:To:Cc:References:From:Date:In-Reply-To:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From; b=XU8xL5B36H/PruVQ3A5D/wf8DiwYTzF6XbhxxZ66yfi9iGZcPPhaE0xCICS+d9AVu yBABjeisXcMAMY8D95DwIAhba7zTGfGKEr62CSRxCi/4wEFreqlOmKdPhVmcVc+XWN j1bqlirWfPviScrU1Y39HzUcIgmQZJxl4XXD8fqf5kbzmv/iwsRrsXCpZ3tucjUnBp bI71CobNmdLwm52ZsAY35ud2zq4FXrJDXB+F+LToblDrpYy2VBHuxrmPkem5ek4ECM uYjiMQvX6qWF3q1qeXcynjJ2TeP/FJTTqZ7dUo5CVOUU4LDsw40wLJEbqHf6pTfB8+ 0+pJT3VhCFMNg== Received: from [IPv6:2001:a61:603e:fa01:5d22:b781:1dd1:ca44] (unknown [IPv6:2001:a61:603e:fa01:5d22:b781:1dd1:ca44]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: sbabic@denx.de) by phobos.denx.de (Postfix) with ESMTPSA id 66B3B83434; Tue, 19 Oct 2021 17:39:13 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=denx.de; s=phobos-20191101; t=1634657953; bh=mVVjMmNEI2N5/6V5Ko4xGAOGQsyL99PYB8U/gd/d09I=; h=Subject:To:Cc:References:From:Date:In-Reply-To:From; b=nnq3C5L9HyJ2syEq+GZxKiyWAw2JNC+k26g+QSQIpttc9+GiKx+zGy8b2P/YJAjRQ YPNa3o7QEcKJ7owbJ2TBYHAP4a8rezY7VrGwdLXwdRO8gXQ6LkHl/V53iGUeQP5MCL qyCQhV7LhSP45HJOaQGqi8VXZEPm/h5UgLsW6HI3/htgsdvRutSi3J99n/GY4Y3C85 pj4bARr0ivsNkxDDSGDgINjEFCyldHDylnFGTMTAjk+GFoe3iYz11jX7SHpePbyhRS jq+bt+DNKXB0eTiuqeP/N9bcoKRFJ8HW5DuR7hHZaJ1g8wW451OhgO0yby/ic7VelB kqPAevB10tCug== Subject: Re: buildman stops (crashed) on current master To: Simon Glass , Stefano Babic Cc: "U-Boot@lists.denx.de" References: <720a1fe2-894d-0e6c-ece5-b3c737857dd7@denx.de> From: Stefano Babic Message-ID: Date: Tue, 19 Oct 2021 17:39:12 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: de-DE Content-Transfer-Encoding: 7bit X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.2 at phobos.denx.de X-Virus-Status: Clean Hi Simon, On 07.10.21 15:43, Simon Glass wrote: > Hi Stefano, > > On Thu, 7 Oct 2021 at 04:37, Stefano Babic wrote: >> >> Hi all, >> >> CI stops by building aarch64 without notice, for reference: >> >> https://source.denx.de/u-boot/custodians/u-boot-imx/-/jobs/332319 >> >> There is no error, just process is killed. It looks like it stops at >> xilinx_zynqmp_virt, >> >> ./tools/buildman/buildman -o /tmp -P -E -W aarch64but board can be built >> without issues. >> >> If I build on my host (not in docker, anyway), it generally builds fine >> - but it crashes sometimes, too. On gitlab instance , it crashes. >> Issue does not seem that depends on merged patches, and introduces >> boards were already built successfully. Any hint ? I have also no idea >> what I should look as what I see is just >> >> "usr/bin/bash: line 104: 24 Killed >> ./tools/buildman/buildman -o /tmp -P -E -W aarch64" > > I cannot see that link. I am not sure what is going on. Does it say > what signal killed it? Pipelines on our server were not public - I have enbaled now for u-boot-imx. > > Does it sit there for an hour and timeout? If so, then I did see that > myself once recently, when the Kconfig needed stdin, but I could not > quitetie it down. I think buildman would provide it, but sometimes > not, apparently. So it can happen when there is an existing build > there and your new one which adds Kconfig options that don't have > defaults, or something like that? > I have investigated further, and I can reproduce it on my host outside the gitlab server. buildman causes a OOM, but I cannot find the cause. Strange enough, this happens with the "aarch64" target, and I cannot reproduce it with Tom's master. So it seems that -master is ok, and somethin on u-boot-imx generates the OOM. However.... The OOM happens always when -2 (two boards remain) appears. I can see with htop that buildman starts to allocate memory until it is exhausted (64GB RAM + 8 GB swap). Then the kernel decides that it is enough and kills buildman - this is what I see on Ci. You can see now the pipelines: https://source.denx.de/u-boot/custodians/u-boot-imx/-/pipelines/9520 I have then split aarch64 and I built imx8 separately - same result. The pipeline stops with xilinx board, but they have nothing to do. In fact, I can build all xilinx board separately. If I run buildman -W aarch64 -x xilinx, OOM is shown by another board. Strange enough, I can build each single board with buildman without issues, neither errors nor warnongs. Just when buildman runs all together (aarch64, 308 boards), the OOM is generated. Bisect does not help: I started bisect, and at the end this commit was presented: commit 53a24dee86fb72ae41e7579607bafe13442616f2 Author: Fabio Estevam Date: Mon Aug 23 21:11:09 2021 -0300 imx8mm-cl-iot-gate: Split the defconfigs But it is a fake: I can revert it, I get the issue again. And the patch has nothing to do. It looks to me it is something in binman, maybe triggered by some changes in tree, but all boards can be built separately without issues. I supposed to find the cause in code due to applied patches, but because each board can be built and no help from bisect, I am quite puzzled. I avoid to send a PR to Tom, else I guess the problem goes into -master, but I do not know how to proceed, and I have a lot of patches to be applied. What can be done ? > If that is it, you can repeat it by clearing out your .bm-work On gitlab, the build starts from scratch. > directory then building just that board for one commit, then the next > (with the Kconfig change). I have run buildman for each single board, all of them were successuful. With aarch64, I get OOM from buildman. > > Buildman is supposed to handle this, of course. I'm not sure what has changed. > Regards, Stefano -- ===================================================================== DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sbabic@denx.de =====================================================================