From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DB4C9C19F2A for ; Sun, 7 Aug 2022 15:47:53 +0000 (UTC) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 9262584858; Sun, 7 Aug 2022 17:47:51 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="ncw2cK2g"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 23F9384856; Sun, 7 Aug 2022 17:47:50 +0200 (CEST) Received: from mail-oi1-x22a.google.com (mail-oi1-x22a.google.com [IPv6:2607:f8b0:4864:20::22a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 763FC83FEE for ; Sun, 7 Aug 2022 17:47:47 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-oi1-x22a.google.com with SMTP id q184so7886639oif.1 for ; Sun, 07 Aug 2022 08:47:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=pNkdppRoGixU8FByA4N+LcKQ08sXtX0e7ZmNmRG4BgA=; b=ncw2cK2g4f6jrcCLinRsocEuPFU6UN52oTouata/QNjaWsjgfu9HOdZ20EoHoxFjEO TaBO56Y/uze5datsDGwERLOAskcT6snm+k77McO3FBk9RIhM0mJWHjRGBQhLC4UGlD/r lg68nFfetiDWlLK7An3P7s4y8IiZ44nuPmxR4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=pNkdppRoGixU8FByA4N+LcKQ08sXtX0e7ZmNmRG4BgA=; b=PhCyPNDxQgy4ynx5d7GYDY/np1iuuYtOBcICmtL3893n+rfT0RbKEm9zMwV24b1oPQ tHsHIEB7QnFSr6VyysjqkMSfI2tWwc3i4GsCZDG2iIs0hG49UQaueKkWm01GVazUyxK5 ZnBtLn9JEEarHdQHSiAK3wHBQp45IVIM34UXQfbF9EopWqlUL1OgbC9YwuXPVkY/nJ+C 2UAkdRjoiYE2dWp8yaT+BvA3yI4q43trKeGcQL8eR4VbrvNOhIp7gVID06kP05v29TUS qYyJ1ttH1AJ194idMGgVlXyUtsM21dyPkOGJ/ZqLKEAiwKjioNGFEutEt9KUmaWBwsuk rbkg== X-Gm-Message-State: ACgBeo1zZKX6P7f1x9TMm/cYldi2rMGvJWBmcvXDB+LDSQoKoS7W9/Sq laWedZq56xq5Lln0DCCp2uabC20P1nRSzg== X-Google-Smtp-Source: AA6agR74P6ZVXeS19Vd/iz+CiTPa+ST2sCmMMlZOAdiv9NQDQJ8cPyasTt3AcoPhgecApEuacN//Ow== X-Received: by 2002:aca:1307:0:b0:33a:9eba:c8bd with SMTP id e7-20020aca1307000000b0033a9ebac8bdmr10061523oii.276.1659887265876; Sun, 07 Aug 2022 08:47:45 -0700 (PDT) Received: from sjg1.roam.corp.google.com (c-67-190-102-125.hsd1.co.comcast.net. [67.190.102.125]) by smtp.gmail.com with ESMTPSA id y18-20020a056808131200b0034290c4ed01sm1553412oiv.41.2022.08.07.08.47.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 07 Aug 2022 08:47:45 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Tom Rini , Heinrich Schuchardt , Simon Glass , Alper Nebi Yasak Subject: [PATCH 01/23] binman: Put fake files in a subdirectory Date: Sun, 7 Aug 2022 09:46:46 -0600 Message-Id: <20220807154708.1418967-2-sjg@chromium.org> X-Mailer: git-send-email 2.37.1.559.g78731f0fdb-goog In-Reply-To: <20220807154708.1418967-1-sjg@chromium.org> References: <20220807154708.1418967-1-sjg@chromium.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.6 at phobos.denx.de X-Virus-Status: Clean At present fake files from a previous build appear to be real files for a subsequent build, since they sit in the output directory. This can cause problems, since binman may need to parse the file, e.g. with the Intel description.bin files. Fix this by putting them in a 'binman-fake' subdirectory. Keep a track of the fake filename so we only create it once. Subsequent builds will still see that the file is missing and mark it as fake. Update a few tests to check the behaviour. Signed-off-by: Simon Glass --- tools/binman/control.py | 10 +++++++++- tools/binman/entry.py | 25 ++++++++++++++++++++----- tools/binman/ftest.py | 30 ++++++++++++++++++++++++------ 3 files changed, 53 insertions(+), 12 deletions(-) diff --git a/tools/binman/control.py b/tools/binman/control.py index ce57dc7efc7..8eea864d45b 100644 --- a/tools/binman/control.py +++ b/tools/binman/control.py @@ -16,8 +16,9 @@ from patman import tools from binman import bintool from binman import cbfs_util -from binman import elf from patman import command +from binman import elf +from binman import entry from patman import tout # These are imported if needed since they import libfdt @@ -717,6 +718,13 @@ def Binman(args): bintool.Bintool.set_missing_list( args.force_missing_bintools.split(',') if args.force_missing_bintools else None) + + # Create the directory here instead of Entry.check_fake_fname() + # since that is called from a threaded context so different threads + # may race to create the directory + if args.fake_ext_blobs: + entry.Entry.create_fake_dir() + for image in images.values(): invalid |= ProcessImage(image, args.update_fdt, args.map, allow_missing=args.allow_missing, diff --git a/tools/binman/entry.py b/tools/binman/entry.py index a07a5888643..41f0eb58ae0 100644 --- a/tools/binman/entry.py +++ b/tools/binman/entry.py @@ -9,6 +9,7 @@ import importlib import os import pathlib import sys +import time from binman import bintool from binman import comp_util @@ -82,7 +83,10 @@ class Entry(object): missing_bintools: List of missing bintools for this entry update_hash: True if this entry's "hash" subnode should be updated with a hash of the entry contents + fake_fname: Fake filename, if one was created, else None """ + fake_dir = None + def __init__(self, section, etype, node, name_prefix=''): # Put this here to allow entry-docs and help to work without libfdt global state @@ -116,6 +120,7 @@ class Entry(object): self.bintools = {} self.missing_bintools = [] self.update_hash = True + self.fake_fname = None @staticmethod def FindEntryClass(etype, expanded): @@ -1009,12 +1014,14 @@ features to produce new behaviours. bool: True if the blob was faked, False if not """ if self.allow_fake and not pathlib.Path(fname).is_file(): - outfname = tools.get_output_filename(os.path.basename(fname)) - with open(outfname, "wb") as out: - out.truncate(size) + if not self.fake_fname: + outfname = os.path.join(self.fake_dir, os.path.basename(fname)) + with open(outfname, "wb") as out: + out.truncate(size) + tout.info(f"Entry '{self._node.path}': Faked blob '{outfname}'") + self.fake_fname = outfname self.faked = True - tout.info(f"Entry '{self._node.path}': Faked file '{outfname}'") - return outfname, True + return self.fake_fname, True return fname, False def CheckFakedBlobs(self, faked_blobs_list): @@ -1164,3 +1171,11 @@ features to produce new behaviours. fname = tools.get_output_filename(f'{prefix}.{uniq}') tools.write_file(fname, data) return data, fname, uniq + + @classmethod + def create_fake_dir(cls): + """Create the directory for fake files""" + cls.fake_dir = tools.get_output_filename('binman-fake') + if not os.path.exists(cls.fake_dir): + os.mkdir(cls.fake_dir) + tout.notice(f"Fake-blob dir is '{cls.fake_dir}'") diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py index fa1f421c052..c8bab5c9416 100644 --- a/tools/binman/ftest.py +++ b/tools/binman/ftest.py @@ -5453,7 +5453,16 @@ fdt fdtmap Extract the devicetree blob from the fdtmap err) def checkFitSplitElf(self, **kwargs): - """Test an split-elf FIT with a missing ELF file""" + """Test an split-elf FIT with a missing ELF file + + Args: + kwargs (dict of str): Arguments to pass to _DoTestFile() + + Returns: + tuple: + str: stdout result + str: stderr result + """ entry_args = { 'of-list': 'test-fdt1 test-fdt2', 'default-dt': 'test-fdt2', @@ -5464,23 +5473,32 @@ fdt fdtmap Extract the devicetree blob from the fdtmap with test_util.capture_sys_output() as (stdout, stderr): self._DoTestFile( '226_fit_split_elf.dts', entry_args=entry_args, - extra_indirs=[test_subdir], **kwargs) - err = stderr.getvalue() - return err + extra_indirs=[test_subdir], verbosity=3, **kwargs) + out = stdout.getvalue() + err = stderr.getvalue() + return out, err def testFitSplitElfMissing(self): """Test an split-elf FIT with a missing ELF file""" - err = self.checkFitSplitElf(allow_missing=True) + out, err = self.checkFitSplitElf(allow_missing=True) self.assertRegex( err, "Image '.*' is missing external blobs and is non-functional: .*") + self.assertNotRegex(out, '.*Faked blob.*') + fname = tools.get_output_filename('binman-fake/missing.elf') + self.assertFalse(os.path.exists(fname)) def testFitSplitElfFaked(self): """Test an split-elf FIT with faked ELF file""" - err = self.checkFitSplitElf(allow_missing=True, allow_fake_blobs=True) + out, err = self.checkFitSplitElf(allow_missing=True, allow_fake_blobs=True) self.assertRegex( err, "Image '.*' is missing external blobs and is non-functional: .*") + self.assertRegex( + out, + "Entry '/binman/fit/images/@tee-SEQ/tee-os': Faked blob '.*binman-fake/missing.elf") + fname = tools.get_output_filename('binman-fake/missing.elf') + self.assertTrue(os.path.exists(fname)) def testPreLoad(self): """Test an image with a pre-load header""" -- 2.37.1.559.g78731f0fdb-goog