[PATCH 0/1] Initial QA test for reproducible builds

[Joshua Watt <jpewhacker@gmail.com>]

Implements an initial QA check for reproducible builds. This check is
sufficient for an initial implementation, and will catch a wide variety
of reproducible problems, but it does have the following problems:

 1) It doesn't pass. Currently, about 800 packages fail to build
    in a reproducible manner for core-image-minimal. I've found two
    major sources of non-reproducibility so far:
     a) The perl-module packages don't have a consistent
        SOURCE_DATE_EPOCH which means when they are packaged the
        timestamps on all the files are different. Thankfully, this
        accounts for several hundred of the packages, so fixing this
        should remove a lot of the failures
     b) Debug package strings aren't consistent. It appears that in some
        of the -dbg packages, the linker changes the order of the merged
        .debug_strings section. This trickles down into the packages
        that contain the executables because it changes the hash the
        executable contains to ensure the debug symbols match up.

 2) It's not easy to debug issues when there are reproducibility
    problems. I had initially intended to run diffoscope on the
    resulting files but this takes much longer than I think we are
    willing to run on the autobuilder and also generates far too much
    output to be really useful. I think a better long term route is to
    have the test dump the list of non-reproducible packages and then
    write a helper script that can consumer this list, allow the user to
    select a package, then run diffoscope to examine it.

 3) This test currently is incomplete and won't catch all classes of
    reproducibility problems. At the least, I know that it won't
    consistently catch the use of the __DATE__ macro in source code,
    since that requires the builds to be done on two separate dates (on
    the other hand, use of __TIME__ will be caught pretty reliably since
    the builds are done serially). I suspect the correct solution to
    this is to borrow from Debian and use something like faketime to
    fake out the system time to some suitable future date when doing the
    test build, but this will require some though to how it should be
    implemented.

 4) It currently only tests Debian packages and core-image-minimal. The
    test case has support for building the other package formats and
    other images at the same time, the idea being that the long step in
    this test is building everything from scratch, and building multiple
    package formats and images at the same time will be much faster
    overall than having multiple tests that have to do from-scratch
    builds (although, there might be a way to serialize multiple tests
    and have them share the test build TMPDIR). Until at least 1 package
    format and image are passing, I don't see a huge motivation to
    enable more.

Joshua Watt (1):
  oeqa: Add reproducible build selftest

 meta/lib/oeqa/selftest/cases/reproducible.py | 159 +++++++++++++++++++
 1 file changed, 159 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/cases/reproducible.py

-- 
2.21.0