Re: DTC Version and Reproducible Builds

Re: DTC Version and Reproducible Builds

[david]

[-- Attachment #1: Type: text/plain, Size: 2301 bytes --]

On Mon, Sep 16, 2024 at 07:51:31PM +0000, Anthony Ortega wrote:
> Hi David,
> 
> I had a question behind the intent of the versioning information
> that gets compiled into the DTC programs. Apologies if a thread
> exists on this already, but I had trouble finding one.

I don't believe there's an existing discussion, but there is a mailing
list it would be appropriate for: <devicetree-compiler@vger.kernel.org>.
I've added it to the CC.

> I am trying to generate reproducible builds for a Linux environment
> using Buildroot. One of the last packages that is giving me trouble
> is the DTC programs.

By "the DTC programs" I'm guessing you mean fdtget, fdtput and
fdtoverlay primarily?  Or do you mean the actual dtc binary itself
too?

> They all seem to intend to encode the git
> version of the package they were pulled from. However, in a build
> system such as Buildroot the DTC source is staged in such a way that
> the git information is stripped away. While that in of itself is not
> a problem, the problem is that it may be getting built in an output
> directory that is within a larger git repository. Therefore, when
> the setlocalversion script runs, it ends up pulling the version of
> this larger git repository.

Huh.

> Is there any way that you recommend being able to override this
> behavior in the DTC project? At the moment it appears to be the only
> project with this unique issue.

Which is a bit weird, since I'm pretty sure we copied that
setlocalversion gunk from somewhere else (thuogh I forget where).

> This is resulting in a different
> binary executable for each commit of the build system. For
> reference, I am referring to the following section of the Makefile:
> 
> https://github.com/dgibson/dtc/blob/main/Makefile#L118-L119

I don't off hand know of a workaround for this.. but let's try a
possible easy approach first.  What happens if you build using meson
instead of make?  They're supposed to give the same results, but don't
always, and I think meson has its own version logic which might be
more sensible.

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

RE: DTC Version and Reproducible Builds

[Anthony Ortega]

David,

When I say DTC programs, I am referring to: convert-dtsv0, dtc, fdtdump, fdtget, fdtoverlay, fdtput

One other thing that wasn't immediately clear to me is if those tools are needed on the Linux side or just for U-Boot. In Buildroot it seems that U-Boot requires it when using Linux, which puts them in the Linux image. That could be another potential path forward.

I'd be interested in seeing where the versioning logic was copied from. Since this seems to be the only package we are using that is doing this sort of versioning.

Unfortunately, I don't think our system can be configured to use meson without a massive overhaul. In the short term I've applied a patch in our build that removes the line that grabs the commit tag during build.

-Anthony

-----Original Message-----
From: david@gibson.dropbear.id.au <david@gibson.dropbear.id.au>
Sent: Tuesday, September 17, 2024 7:49 PM
To: Anthony Ortega <anthony.ortega@terranorbital.com>
Cc: devicetree-compiler@vger.kernel.org
Subject: Re: DTC Version and Reproducible Builds

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.


This e-mail message and any attachments are intended solely for the specified addressee(s) and may be confidential, proprietary, privileged, and/or U.S. export controlled. If you are not the intended recipient, you are hereby notified that any use, disclosure, distribution, copying, or storage of this message or its attachments is strictly prohibited. Please immediately notify the sender by return e-mail and delete this message and any attachments.

Re: DTC Version and Reproducible Builds

[david]

[-- Attachment #1: Type: text/plain, Size: 2683 bytes --]

On Thu, Sep 19, 2024 at 09:45:04PM +0000, Anthony Ortega wrote:
> David,
> 
> When I say DTC programs, I am referring to: convert-dtsv0, dtc,
> fdtdump, fdtget, fdtoverlay, fdtput

Ok.

> One other thing that wasn't immediately clear to me is if those
> tools are needed on the Linux side or just for U-Boot. In Buildroot
> it seems that U-Boot requires it when using Linux, which puts them
> in the Linux image. That could be another potential path forward.

I don't really understand the question.  They're userspace tools, so
whether you need them in any particular context depends on what you're
doing there.

> I'd be interested in seeing where the versioning logic was copied
> from. Since this seems to be the only package we are using that is
> doing this sort of versioning.

According to commit a657ce8f ("Add DTC release version information.")
it came from the Linux kernel... circa 2007.  The kernel's probably
doing something entirely different nowadays.

> Unfortunately, I don't think our system can be configured to use
> meson without a massive overhaul. In the short term I've applied a
> patch in our build that removes the line that grabs the commit tag
> during build.

That.. seems like a pretty serious limitation, given that more and
more packages are moving away from make.

But, in any case, I wasn't suggesting this as a practical workaround,
but more as a diagnostic.  If it works with meson we can look closer
at what it's doing and maybe duplicate it with make.

> 
> -Anthony
> 
> -----Original Message-----
> From: david@gibson.dropbear.id.au <david@gibson.dropbear.id.au>
> Sent: Tuesday, September 17, 2024 7:49 PM
> To: Anthony Ortega <anthony.ortega@terranorbital.com>
> Cc: devicetree-compiler@vger.kernel.org
> Subject: Re: DTC Version and Reproducible Builds
> 
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> 
> 
> This e-mail message and any attachments are intended solely for the specified addressee(s) and may be confidential, proprietary, privileged, and/or U.S. export controlled. If you are not the intended recipient, you are hereby notified that any use, disclosure, distribution, copying, or storage of this message or its attachments is strictly prohibited. Please immediately notify the sender by return e-mail and delete this message and any attachments.
> 

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: DTC Version and Reproducible Builds

[Andreas Gnau]

On Mon, Sep 16, 2024 at 07:51:31PM +0000, Anthony Ortega wrote:
 > I am trying to generate reproducible builds for a Linux environment
 > using Buildroot. One of the last packages that is giving me trouble
 > is the DTC programs.
 > [...]
 > They all seem to intend to encode the git
 > version of the package they were pulled from. However, in a build
 > system such as Buildroot the DTC source is staged in such a way that
 > the git information is stripped away. While that in of itself is not
 > a problem, the problem is that it may be getting built in an output
 > directory that is within a larger git repository. Therefore, when
 > the setlocalversion script runs, it ends up pulling the version of
 > this larger git repository.

Yocto/OpenEmbedded[1] and OpenWrt[2] avoid such issues by setting 
GIT_CEILING_DIRECTORIES. Maybe buildroot could do the same and fix it 
for all packages, not just dtc tools?

[1] 
https://git.openembedded.org/openembedded-core/tree/meta/conf/bitbake.conf?id=bb3101fffceacfa4c021c33affb7c785da8d859f#n788

[2] 
https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=include/package.mk;h=7fbecf98dc7c9a5b11ce1958e254578f7e4d9c73;hb=7850e197670afff38426075953ba6a71f0f431c8#l202

On 2024-09-20 13:05, david@gibson.dropbear.id.au wrote:
 > Unfortunately, I don't think our system can be configured to use
 > meson without a massive overhaul. In the short term I've applied a
 > patch in our build that removes the line that grabs the commit tag
 > during build.

On Thu, Sep 19, 2024 at 09:45:04PM +0000, Anthony Ortega wrote:
 > But, in any case, I wasn't suggesting this as a practical workaround,
 > but more as a diagnostic.  If it works with meson we can look closer
 > at what it's doing and maybe duplicate it with make.

I do not know what other projects do, but most likely those that do not 
exhibit this issue, set the git-dir explicitly (or use GIT_CEILING_DIRS) 
to avoid git taking some repository above the source directory.

I still think as a first thing, buildroot should implement a generic 
solution and then we can look into fixing dtc and tools.

Regards, Andreas

Re: DTC Version and Reproducible Builds

[david]

[-- Attachment #1: Type: text/plain, Size: 2579 bytes --]

On Mon, Sep 23, 2024 at 12:46:20PM +0200, Andreas Gnau wrote:
> On Mon, Sep 16, 2024 at 07:51:31PM +0000, Anthony Ortega wrote:
> > I am trying to generate reproducible builds for a Linux environment
> > using Buildroot. One of the last packages that is giving me trouble
> > is the DTC programs.
> > [...]
> > They all seem to intend to encode the git
> > version of the package they were pulled from. However, in a build
> > system such as Buildroot the DTC source is staged in such a way that
> > the git information is stripped away. While that in of itself is not
> > a problem, the problem is that it may be getting built in an output
> > directory that is within a larger git repository. Therefore, when
> > the setlocalversion script runs, it ends up pulling the version of
> > this larger git repository.
> 
> Yocto/OpenEmbedded[1] and OpenWrt[2] avoid such issues by setting
> GIT_CEILING_DIRECTORIES. Maybe buildroot could do the same and fix it for
> all packages, not just dtc tools?

Oh, I didn't know about GIT_CEILING_DIRECTORIES.  That does indeed
sound look a good idea.

> [1] https://git.openembedded.org/openembedded-core/tree/meta/conf/bitbake.conf?id=bb3101fffceacfa4c021c33affb7c785da8d859f#n788
> 
> [2] https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=include/package.mk;h=7fbecf98dc7c9a5b11ce1958e254578f7e4d9c73;hb=7850e197670afff38426075953ba6a71f0f431c8#l202
> 
> On 2024-09-20 13:05, david@gibson.dropbear.id.au wrote:
> > Unfortunately, I don't think our system can be configured to use
> > meson without a massive overhaul. In the short term I've applied a
> > patch in our build that removes the line that grabs the commit tag
> > during build.
> 
> On Thu, Sep 19, 2024 at 09:45:04PM +0000, Anthony Ortega wrote:
> > But, in any case, I wasn't suggesting this as a practical workaround,
> > but more as a diagnostic.  If it works with meson we can look closer
> > at what it's doing and maybe duplicate it with make.
> 
> I do not know what other projects do, but most likely those that do not
> exhibit this issue, set the git-dir explicitly (or use GIT_CEILING_DIRS) to
> avoid git taking some repository above the source directory.
> 
> I still think as a first thing, buildroot should implement a generic
> solution and then we can look into fixing dtc and tools.
> 
> Regards, Andreas
> 

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]