From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 37315C00528 for ; Thu, 3 Aug 2023 19:29:55 +0000 (UTC) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id E7E72867D6; Thu, 3 Aug 2023 21:29:53 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="a0209+EZ"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id A9C038682C; Thu, 3 Aug 2023 21:29:52 +0200 (CEST) Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 68EB886672 for ; Thu, 3 Aug 2023 21:29:50 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=alpernebiyasak@gmail.com Received: by mail-wr1-x42b.google.com with SMTP id ffacd0b85a97d-317798b359aso1120631f8f.1 for ; Thu, 03 Aug 2023 12:29:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691090990; x=1691695790; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=/4m/a2fM4UjKcl4pfp2/oyLzP+9nfMDpWjdKzCHWZ6Q=; b=a0209+EZPdnT2jNMEkegvuClkQKcKUTNUGQxc53sRj/gdXM6zF8UZl1CyKRbVtyJXd 6tdTScUUG/8Ksv0xfAC5tFK4qdtwF6Fa+0ITYH3U1Kx7OMSuwUXw1DMoHCkNqw0nIrIV uu/l53ElxPL+4QEg7vNdP3TYkLXjkvcPVbF+n3Jty17sNQYv5klVvj8iXyIxQRM45z04 Z6kaTCUNghQTDCMgZNgzuk2mgdTiWT8LCFOVfnBHtEW9iVpisFIIvQ8Sa0+bnbUb6cLT cZNZWpiWenjYxr7NM5+ZwMqpyIBKTKl44pQIcFyQaBbU0SmxDCs7xQmxuDggJ92Vx5kN LxEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691090990; x=1691695790; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/4m/a2fM4UjKcl4pfp2/oyLzP+9nfMDpWjdKzCHWZ6Q=; b=Ng3aI3dpF15zkRkJ/ApaK8zb94GIBHXwCwPo4AA8og3l2p9cszW9/kaeEkdknMvLFw LzCiZmi+dk6oFh2OtJlxV6jOEHHujDvZnebDe5XjJAOwDiScN0uzfE6IM6Bz/gJ3h1aB r9YLx38cKGR22ilW6StvahkeAlHEVjUUEorm/rlgOg+HoQ+ZaWyBu0cfHLPQoiXl36BG Els7QHzK6vf6KjMfHMqjyz/FpML7dbJig3neYhYwwZrKh8jdLWVI32Jgz/L1bwOXkEDK Vizb60nPWBYc7/v0Hc1T14vql+KY5LUv9/najFLxTankoL+y5ViPm+Mwck2Nk0ZoaFCH cWRQ== X-Gm-Message-State: ABy/qLbc0XRt7CAmYiJoL7oVR/F5Tqo5nvjgadJI8K6QkbIqXg4ZeUXL F1YaKevNd383p/je1WDQm9MmqFFDb0Y= X-Google-Smtp-Source: APBJJlHE7p+4FBlhIM4E1LvhapT5WWPHruRNImAD2QYWVzNuJJ++VCDqjKRTghCbrehzrRiVKhT6PQ== X-Received: by 2002:a5d:61cd:0:b0:313:f347:eea0 with SMTP id q13-20020a5d61cd000000b00313f347eea0mr7257968wrv.60.1691090989476; Thu, 03 Aug 2023 12:29:49 -0700 (PDT) Received: from [192.168.0.79] ([178.233.24.1]) by smtp.gmail.com with ESMTPSA id x18-20020a5d60d2000000b003143cb109d5sm610123wrt.14.2023.08.03.12.29.48 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 03 Aug 2023 12:29:48 -0700 (PDT) Message-ID: Date: Thu, 3 Aug 2023 22:29:47 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 2/2] schemas: Add a schema for binman To: Simon Glass Cc: Tom Rini , U-Boot Mailing List References: <20230711211810.4172447-1-sjg@chromium.org> <20230711211810.4172447-2-sjg@chromium.org> Content-Language: en-US, tr, en-GB From: Alper Nebi Yasak In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.8 at phobos.denx.de X-Virus-Status: Clean (I think I've strayed far away from schema/dtb side of things towards binman-the-program side, so I'm dropping Rob and devicetree@ from Cc). On 2023-07-20 22:55 +03:00, Simon Glass wrote: > On Thu, 20 Jul 2023 at 09:23, Alper Nebi Yasak wrote: >> There's also the issue of binman having multiple different device-trees: >> its input, the ones in fdtmaps per image, the ones injected to U-Boot >> dtb files per image. I'd say each has different needs, and those >> differences have to be figured out before specified upstream. I can only >> guess this is about the one that'll be in a u-boot.dtb. > > Well, there is really only one. The fdtmap is actually the same > schema, except that it mentions only the image that it is embedded in, > i.e. if the fdtmap is for the SPI image, then the fdtmap in SPI flash > only contains the definition for the SPI image, not the MMC image > which is in a different device. The input is the same schema, albeit > that things like 'offset' may be filled in by Binman automatically > when it packs things. I was thinking of things like generator nodes and templates and "filename"s in blob entries that (IMO) only make sense as binman inputs and never should be in a fdtmap. Trying to highlight a difference between "how to build this image" and "what this image contains". But I guess it's OK to call them the same schema unless something has two conflicting meanings in input-domain and output-domain, and I can't think of any counter-examples now. >> I want to go through binman more thoroughly, but a lot of changes >> happened since I last looked at it and I'm a bit slow at writing things, >> so won't exactly be soon. That being said, here's some ideas off the top >> of my head, for inspiration on both this schema and binman itself. > > Do you mean the code? There are definitely some abstractions that are > stretching a bit, but it is mostly holding together for now. I mean both the implementation and the design. There's a lot more on my mind, some more examples: - Precise structures for data instead of thinking of them as black boxes - Heuristically parsing arbitrary images/data for e.g. binman extract - Deconstructing input files to reuse their pieces for building images - Explicit dependency tracking and resolution for data and layout - Making things act more like native Python objects - In fact, making it entirely operable with just Python code - Decoupling internals from the control dtbs ("entry._node" etc.) - Ensuring reproducible builds and testing for it - Fuzzing as a replacement for most tests - ... I think it has the beginnings of a very nice declarative-style tool, but has to embrace that style to get there. (And I guess I'll have to do the work to properly express myself on some points...) > [...] >> Or maybe even better, I think we should make it like FIT: allow a "data" >> property that has it inside the dtb, or a pair of "data-offset/position" >> and "data-size" properties if it's outside. > > Eh? The point of the entry is to declare the position of actual data. > If the data is elsewhere, then the entry will be too. Sorry, I'm jumping into a different contexts here without explaining it. I'm seeing a similarity between images that start with a fdtmap and the images built by "mkimage -T fit -E". Then I'm extrapolating to the non-"-E" case. Then extending to make it possible for a "fdtmap + data" binman description to result in something almost backwards-compatible with FIT, which could replace most uses of a fit etype. (Of course, backwards compatible only if you add config nodes, and flatten or don't nest entries, and whatnot. And fit etype would still stay.) >> Inlining data inside the dtb >> could help us do nice things in binman, like hashes/signatures as entry >> types instead of special-casing them. > > We already do that though, right? See testHash() for example. This is > a powerful feature of a firmware-layout description / Binman, IMO, > since all the metadata you need is right there. Having that functionality in state.py and having to work around it for mkimage/fit etypes (because we want to embed them into the data instead of the binman dtb) is what I mean by special-casing. If we had them as etypes, and could embed arbitrary entries as "data" in binman dtb, I think we would have the best of both worlds -- avoid calling mkimage just for hash/signature embedding, have it still possible to put those in binman metadata, and enable a foundation for other sign-verify flows (leaning towards replacing custom signing tools/scripts with binman). >> In fact it could be possible to turn binman images into a FIT 2.0 if we >> do some more work on top of that, instead of nesting a FIT inside a >> binman image. > > Well binman needs to produce things which are not FITs. Bear in mind > that the output can be a binary file in any format. It doesn't > necessarily have to have any metadata in it at all, certainly not a > FIT header. I know, I don't mean it as the only mode of operation, I hope I have explained better this time. > Thanks again for your encouraging comments. I'll do a v2 with the above changes. I'm glad to hear that you appreciate my comments, because trying to review your work always feels quite hubristic to me, haha.