Re: [RFC][PATCH 2/2] buildhistory: support generating md5sum of files

[<Mikko.Rapeli@bmw.de>]

On Tue, Jan 08, 2019 at 11:36:24AM +0100, Jacob Kroon wrote:
> On Tue, Jan 8, 2019 at 11:32 AM <Mikko.Rapeli@bmw.de> wrote:
> >
> > On Mon, Jan 07, 2019 at 03:17:00PM +0100, Jacob Kroon wrote:
> > > On Sun, Jan 6, 2019 at 7:14 PM Jacob Kroon <jacob.kroon@gmail.com> wrote:
> > > >
> > > > Introduce 'md5' in BUILDHISTORY_FEATURES and enable it by default
> > > > when doing reproducible builds.
> > > >
> > > > When enabled this will additionally create:
> > > >
> > > >   files-in-package-md5.txt
> > > >   files-in-image-md5.txt
> > > >   files-in-sdk-md5.txt
> > > >
> > > > containing the md5 checksums of regular files.
> > > >
> > > > Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com>
> > > > ---
> > > >  meta/classes/buildhistory.bbclass | 10 ++++++++--
> > > >  1 file changed, 8 insertions(+), 2 deletions(-)
> > > >
> > > > diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
> > > > index 33eb1b00f6..00f0701dec 100644
> > > > --- a/meta/classes/buildhistory.bbclass
> > > > +++ b/meta/classes/buildhistory.bbclass
> > > > @@ -7,7 +7,8 @@
> > > >  # Copyright (C) 2007-2011 Koen Kooi <koen@openembedded.org>
> > > >  #
> > > >
> > > > -BUILDHISTORY_FEATURES ?= "image package sdk"
> > > > +BUILDHISTORY_FEATURES ?= "image package sdk \
> > > > +  ${@ "md5" if bb.utils.to_boolean(d.getVar('BUILD_REPRODUCIBLE_BINARIES')) else ""}"
> > > >  BUILDHISTORY_DIR ?= "${TOPDIR}/buildhistory"
> > > >  BUILDHISTORY_DIR_IMAGE = "${BUILDHISTORY_DIR}/images/${MACHINE_ARCH}/${TCLIBC}/${IMAGE_BASENAME}"
> > > >  BUILDHISTORY_DIR_PACKAGE = "${BUILDHISTORY_DIR}/packages/${MULTIMACH_TARGET_SYS}/${PN}"
> > > > @@ -526,7 +527,12 @@ buildhistory_list_files() {
> > > >                 eval ${FAKEROOTENV} ${FAKEROOTCMD} $find_cmd
> > > >         else
> > > >                 eval $find_cmd
> > > > -       fi | sort -k5 | sed 's/ * -> $//' > $2 )
> > > > +       fi | sort -k5 | sed 's/ * -> $//' > $2
> > > > +       if [ "${@bb.utils.contains('BUILDHISTORY_FEATURES', 'md5', '1', '0', d)}" = "1" ] ; then
> > > > +               md5filename=$(echo $2 | sed 's/\.txt$/-md5.txt/')
> > > > +               find -type f | xargs -I{} -n1 md5sum {} | sort -k2 > $md5filename
> > > > +               [ -s $md5filename ] || rm $md5filename # remove result if empty
> > >
> > > I added this remove because I thought it didn't make sense to keep
> > > empty files around, but I now realize that the "files-in-package.txt"
> > > file is kept around, even if empty. Is there a preference on what to
> > > do here ?
> >
> > FWIW, I'm wiping the all buildhistory data with external scripts before doing a
> > clean build. Basically a "git rm -rf *" in buildhistory directory. Otherwise
> > stale data about images, packages etc which are no longer built remain in
> > buildhistory.
> 
> I think one can also do:
> 
> BB_ENV_EXTRAWHITE=BUILDHISTORY_RESET BUILDHISTORY_RESET=1 bitbake <build>
> 
> (see buildhistory.bbclass)

Thanks for the hint!

-Mikko