From mboxrd@z Thu Jan 1 00:00:00 1970 From: Gilles Chanteperdrix Date: Sat, 14 May 2016 15:34:42 +0200 Subject: [Buildroot] [PATCH 09/34] reproducibility/libglib2: allow removing codegen In-Reply-To: References: <20160430074358.GE1781@hermes.click-hack.org> <1462002570-14706-1-git-send-email-gilles.chanteperdrix@xenomai.org> <1462002570-14706-9-git-send-email-gilles.chanteperdrix@xenomai.org> <88864677-f612-bd6b-3158-5c2980bb9825@mind.be> <20160508202511.GT13285@hermes.click-hack.org> <00b9819d-dbe3-52ce-75f7-6d7a8217f47a@mind.be> <20160510194257.GE13285@hermes.click-hack.org> Message-ID: <20160514133442.GE27354@hermes.click-hack.org> List-Id: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: buildroot@busybox.net On Thu, May 12, 2016 at 10:05:36PM +0200, Arnout Vandecappelle wrote: > On 05/10/16 21:42, Gilles Chanteperdrix wrote: > > On Tue, May 10, 2016 at 01:40:30AM +0200, Arnout Vandecappelle wrote: > >> On 05/08/16 22:25, Gilles Chanteperdrix wrote: > >>> On Sat, May 07, 2016 at 11:04:25PM +0200, Arnout Vandecappelle wrote: > >>>> On 04/30/16 09:49, Gilles Chanteperdrix wrote: > >>>>> But this is not sufficient, compiling the python bytecode with the > >>>>> same interpreter in different environments yields different binaries, > >>>> > >>>> Er, this is worrisome... You are saying that we don't have a chance in hell of > >>>> generating reproducible python bytecode? > >>> > >>> I am not a python specialist, but it seems to me four bytes in the > >>> python generated bytecode are the build timestamp, so unless there > >>> is a way to override it with SOURCE_DATE_EPOCH, I do not see that > >>> possible. > >> > >> I've checked the docs. What is saved is the timestamp of the .py file, not the > >> build time. > > > > Mmmm I don't remember. I would run a compilation manually, twice at > > a different time, to make sure that the problem is only the file > > date. > > I did that - admittedly with just a few seconds difference. Both in > python2.7.11 and python3.5, they were identical when compiling a second time, > and different after touch'ing. > > > > >> > >> I think it would make sense to run 'touch -d @$(SOURCE_DATE_EPOCH)' on all > >> files after patching to capture this aspect. This was also proposed for > >> Fedora[1] (though there it was only for the .py files). Not sure what happened > >> with that proposal in the end. > > > > I think I tried running "touch" before compiling, unfortunately > > playing with file dates before running "make" is a bit like playing > > with fire. For instance with autotools based projects for which > > autoreconf is not run, the project may use versions of the autotools > > not installed on the user machine, and because of file dates may > > want to rerun autoconf or autmake and whine because the right > > versions are not available. Doing this only for .py files is much > > more reasonable. > > I tested this as well, with make-3.81 and make-4.04. Both of them do _not_ > rebuild if the timestamps are identical. And since the idea is to use touch -d > @$(SOURCE_DATE_EPOCH), all timestamps will be identical. You checked both the pyo and the pyc? > > But it does indeed mean that if a package has a generated file with an earlier > date than the source files, it will now suddenly no longer be > rebuilt. Yes, that is another problem. But I tried it, and this is not the one I had, the one I had was the contrary: the dates made make want to rebuild some files (the autotools/automake files), whereas the right versions of autoconf and automake were not installed, this was a package that did not run autoreconf. The "make" tool is completely based on file dates, so again, I think messing with file dates before running make is a bad idea. -- Gilles. https://click-hack.org