[OE-core][PATCH v2 0/2] Fixes for reproducible build and externalsrc

[OE-core][PATCH v2 0/2] Fixes for reproducible build and externalsrc

[Mark Hatle]

I've reworked the patches completely based on the feedback and a discusion
with RP on IRC.

The first patch both simplifies and expands the description of what the
class is doing.  (To fully disable the behavior for a single recipe that
recipe would need to delVar SOURCE_DATE_EPOCH.  With this set, all sorts
of programs pay attention to the value, if it's blank it may error,
otherwise it falls back to either 0, or another date.)

The documentation is expanded to make it clear how to override the behavior
but it was not desired to explain how to 'disable' the behavior for a
single recipe.

For the externalsrc, the code was moved from the reproducible_build into
the externalsrc and made contingent on the do_unpack being removed.

Mark Hatle (2):
  reproducible_build: Remove BUILD_REPRODUCIBLE_BINARIES checking
  externalsrc: Work with reproducible_build

 meta/classes/externalsrc.bbclass        | 10 +++++
 meta/classes/reproducible_build.bbclass | 53 ++++++++++++++++---------
 2 files changed, 44 insertions(+), 19 deletions(-)

-- 
2.17.1

[OE-core][PATCH v2 1/2] reproducible_build: Remove BUILD_REPRODUCIBLE_BINARIES checking

[Mark Hatle]

From: Mark Hatle <mark.hatle@xilinx.com>

Previously if BUILD_REPRODUCIBLE_BINARIES was set to 0, the system would
fall back and select the default epoch (April 2011), but still perform
the reproducible build actions.  This resulted in binaries that had an
unusually old date.

Simplify the functions and remove the anonymous python as no longer
necessary.

Also improve the documentation to better explain what the class is doing
and how a recipe can override the behavior if necessary.

Signed-off-by: Mark Hatle <mark.hatle@xilinx.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
---
 meta/classes/reproducible_build.bbclass | 53 ++++++++++++++++---------
 1 file changed, 34 insertions(+), 19 deletions(-)

diff --git a/meta/classes/reproducible_build.bbclass b/meta/classes/reproducible_build.bbclass
index 378121903d..96bb012243 100644
--- a/meta/classes/reproducible_build.bbclass
+++ b/meta/classes/reproducible_build.bbclass
@@ -1,17 +1,38 @@
 # reproducible_build.bbclass
 #
-# Sets SOURCE_DATE_EPOCH in each component's build environment.
+# Sets the default SOURCE_DATE_EPOCH in each component's build environment.
+# The format is number of seconds since the system epoch.
+#
 # Upstream components (generally) respect this environment variable,
 # using it in place of the "current" date and time.
 # See https://reproducible-builds.org/specs/source-date-epoch/
 #
-# After sources are unpacked but before they are patched, we set a reproducible value for SOURCE_DATE_EPOCH.
-# This value should be reproducible for anyone who builds the same revision from the same sources.
+# The default value of SOURCE_DATE_EPOCH comes from the function
+# get_source_date_epoch_value which reads from the SDE_FILE, or if the file
+# is not available (or set to 0) will use the fallback of
+# SOURCE_DATE_EPOCH_FALLBACK.
+#
+# The SDE_FILE is normally constructed from the function
+# create_source_date_epoch_stamp which is typically added as a postfuncs to
+# the do_unpack task.  If a recipe does NOT have do_unpack, it should be added
+# to a task that runs after the source is available and before the
+# do_deploy_source_date_epoch task is executed.
+#
+# If a recipe wishes to override the default behavior it should set it's own
+# SOURCE_DATE_EPOCH or override the do_deploy_source_date_epoch_stamp task
+# with recipe-specific functionality to write the appropriate
+# SOURCE_DATE_EPOCH into the SDE_FILE.
+#
+# SOURCE_DATE_EPOCH is intended to be a reproducible value.  This value should
+# be reproducible for anyone who builds the same revision from the same
+# sources.
 #
-# There are 4 ways we determine SOURCE_DATE_EPOCH:
+# There are 4 ways the create_source_date_epoch_stamp function determines what
+# becomes SOURCE_DATE_EPOCH:
 #
 # 1. Use the value from __source_date_epoch.txt file if this file exists.
-#    This file was most likely created in the previous build by one of the following methods 2,3,4.
+#    This file was most likely created in the previous build by one of the
+#    following methods 2,3,4.
 #    Alternatively, it can be provided by a recipe via SRC_URI.
 #
 # If the file does not exist:
@@ -22,20 +43,16 @@
 # 3. Use the mtime of "known" files such as NEWS, CHANGLELOG, ...
 #    This works for well-kept repositories distributed via tarball.
 #
-# 4. Use the modification time of the youngest file in the source tree, if there is one.
+# 4. Use the modification time of the youngest file in the source tree, if
+#    there is one.
 #    This will be the newest file from the distribution tarball, if any.
 #
-# 5. Fall back to a fixed timestamp.
+# 5. Fall back to a fixed timestamp (SOURCE_DATE_EPOCH_FALLBACK).
 #
-# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's SDE_FILE.
-# If none of these mechanisms are suitable, replace the do_deploy_source_date_epoch task
-# with recipe-specific functionality to write the appropriate SOURCE_DATE_EPOCH into the SDE_FILE.
-#
-# If this file is found by other tasks, the value is exported in the SOURCE_DATE_EPOCH variable.
-# SOURCE_DATE_EPOCH is set for all tasks that might use it (do_configure, do_compile, do_package, ...)
+# Once the value is determined, it is stored in the recipe's SDE_FILE.
 
 BUILD_REPRODUCIBLE_BINARIES ??= '1'
-inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'reproducible_build_simple', '')}
+inherit reproducible_build_simple
 
 SDE_DIR = "${WORKDIR}/source-date-epoch"
 SDE_FILE = "${SDE_DIR}/__source_date_epoch.txt"
@@ -92,6 +109,9 @@ python create_source_date_epoch_stamp() {
     os.rename(tmp_file, epochfile)
 }
 
+# Generate the stamp after do_unpack runs
+do_unpack[postfuncs] += "create_source_date_epoch_stamp"
+
 def get_source_date_epoch_value(d):
     cached = d.getVar('__CACHED_SOURCE_DATE_EPOCH')
     if cached:
@@ -120,8 +140,3 @@ def get_source_date_epoch_value(d):
 
 export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}"
 BB_HASHBASE_WHITELIST += "SOURCE_DATE_EPOCH"
-
-python () {
-    if d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1':
-        d.appendVarFlag("do_unpack", "postfuncs", " create_source_date_epoch_stamp")
-}
-- 
2.17.1

Re: [OE-core][PATCH v2 1/2] reproducible_build: Remove BUILD_REPRODUCIBLE_BINARIES checking

[Douglas]

[-- Attachment #1: Type: text/plain, Size: 5610 bytes --]

On 10/09/21 1:29 pm, Mark Hatle wrote:
> From: Mark Hatle <mark.hatle@xilinx.com>
>
> Previously if BUILD_REPRODUCIBLE_BINARIES was set to 0, the system would
> fall back and select the default epoch (April 2011), but still perform
> the reproducible build actions.  This resulted in binaries that had an
> unusually old date.
>
> Simplify the functions and remove the anonymous python as no longer
> necessary.
>
> Also improve the documentation to better explain what the class is doing
> and how a recipe can override the behavior if necessary.
>
> Signed-off-by: Mark Hatle <mark.hatle@xilinx.com>
> Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
> ---
>   meta/classes/reproducible_build.bbclass | 53 ++++++++++++++++---------
>   1 file changed, 34 insertions(+), 19 deletions(-)
>
> diff --git a/meta/classes/reproducible_build.bbclass b/meta/classes/reproducible_build.bbclass
> index 378121903d..96bb012243 100644
> --- a/meta/classes/reproducible_build.bbclass
> +++ b/meta/classes/reproducible_build.bbclass
> @@ -1,17 +1,38 @@
>   # reproducible_build.bbclass
>   #
> -# Sets SOURCE_DATE_EPOCH in each component's build environment.
> +# Sets the default SOURCE_DATE_EPOCH in each component's build environment.
> +# The format is number of seconds since the system epoch.
> +#
>   # Upstream components (generally) respect this environment variable,
>   # using it in place of the "current" date and time.
>   # See https://reproducible-builds.org/specs/source-date-epoch/
>   #
> -# After sources are unpacked but before they are patched, we set a reproducible value for SOURCE_DATE_EPOCH.
> -# This value should be reproducible for anyone who builds the same revision from the same sources.
> +# The default value of SOURCE_DATE_EPOCH comes from the function
> +# get_source_date_epoch_value which reads from the SDE_FILE, or if the file
> +# is not available (or set to 0) will use the fallback of
> +# SOURCE_DATE_EPOCH_FALLBACK.
> +#
> +# The SDE_FILE is normally constructed from the function
> +# create_source_date_epoch_stamp which is typically added as a postfuncs to
> +# the do_unpack task.  If a recipe does NOT have do_unpack, it should be added
> +# to a task that runs after the source is available and before the
> +# do_deploy_source_date_epoch task is executed.
> +#
> +# If a recipe wishes to override the default behavior it should set it's own
> +# SOURCE_DATE_EPOCH or override the do_deploy_source_date_epoch_stamp task
> +# with recipe-specific functionality to write the appropriate
> +# SOURCE_DATE_EPOCH into the SDE_FILE.
> +#
> +# SOURCE_DATE_EPOCH is intended to be a reproducible value.  This value should
> +# be reproducible for anyone who builds the same revision from the same
> +# sources.
>   #
> -# There are 4 ways we determine SOURCE_DATE_EPOCH:
> +# There are 4 ways the create_source_date_epoch_stamp function determines what
> +# becomes SOURCE_DATE_EPOCH:
>   #
>   # 1. Use the value from __source_date_epoch.txt file if this file exists.
> -#    This file was most likely created in the previous build by one of the following methods 2,3,4.
> +#    This file was most likely created in the previous build by one of the
> +#    following methods 2,3,4.
>   #    Alternatively, it can be provided by a recipe via SRC_URI.
>   #
>   # If the file does not exist:
> @@ -22,20 +43,16 @@
>   # 3. Use the mtime of "known" files such as NEWS, CHANGLELOG, ...
>   #    This works for well-kept repositories distributed via tarball.
>   #
> -# 4. Use the modification time of the youngest file in the source tree, if there is one.
> +# 4. Use the modification time of the youngest file in the source tree, if
> +#    there is one.
>   #    This will be the newest file from the distribution tarball, if any.
>   #
> -# 5. Fall back to a fixed timestamp.
> +# 5. Fall back to a fixed timestamp (SOURCE_DATE_EPOCH_FALLBACK).
>   #
> -# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's SDE_FILE.
> -# If none of these mechanisms are suitable, replace the do_deploy_source_date_epoch task
> -# with recipe-specific functionality to write the appropriate SOURCE_DATE_EPOCH into the SDE_FILE.
> -#
> -# If this file is found by other tasks, the value is exported in the SOURCE_DATE_EPOCH variable.
> -# SOURCE_DATE_EPOCH is set for all tasks that might use it (do_configure, do_compile, do_package, ...)
> +# Once the value is determined, it is stored in the recipe's SDE_FILE.
>   
>   BUILD_REPRODUCIBLE_BINARIES ??= '1'
> -inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'reproducible_build_simple', '')}
> +inherit reproducible_build_simple
>   
>   SDE_DIR = "${WORKDIR}/source-date-epoch"
>   SDE_FILE = "${SDE_DIR}/__source_date_epoch.txt"
> @@ -92,6 +109,9 @@ python create_source_date_epoch_stamp() {
>       os.rename(tmp_file, epochfile)
>   }
>   
> +# Generate the stamp after do_unpack runs
> +do_unpack[postfuncs] += "create_source_date_epoch_stamp"
> +
>   def get_source_date_epoch_value(d):
>       cached = d.getVar('__CACHED_SOURCE_DATE_EPOCH')
>       if cached:
> @@ -120,8 +140,3 @@ def get_source_date_epoch_value(d):
>   
>   export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}"
>   BB_HASHBASE_WHITELIST += "SOURCE_DATE_EPOCH"
> -
> -python () {
> -    if d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1':
> -        d.appendVarFlag("do_unpack", "postfuncs", " create_source_date_epoch_stamp")
> -}


Tidy. Nice to delete some bewilderment.


[-- Attachment #2: Type: text/html, Size: 6286 bytes --]

[OE-core][PATCH v2 2/2] externalsrc: Work with reproducible_build

[Mark Hatle]

From: Mark Hatle <mark.hatle@xilinx.com>

Externalsrc removes do_fetch, do_unpack, and do_patch.  The system normally
discovers the correct reproducible date as a postfuncs of do_unpack, so this
date is never found, so it falls back to the default epoch.

Instead we can move the discovery function to a prefuncs on the epoch
deploy task.  This task will run before do_configure, and since the source
is already available can run safely at anytime.

Signed-off-by: Mark Hatle <mark.hatle@xilinx.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
---
 meta/classes/externalsrc.bbclass | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index 54b08adf62..34921b39a2 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -110,6 +110,16 @@ python () {
                 continue
             bb.build.deltask(task, d)
 
+        if bb.data.inherits_class('reproducible_build', d) and 'do_unpack' in d.getVar("SRCTREECOVEREDTASKS").split():
+            # The reproducible_build's create_source_date_epoch_stamp function must
+            # be run after the source is available and before the
+            # do_deploy_source_date_epoch task.  In the normal case, it's attached
+            # to do_unpack as a postfuncs, but since we removed do_unpack (above)
+            # we need to move the function elsewhere.  The easiest thing to do is
+            # move it into the prefuncs of the do_deploy_source_date_epoch task.
+            # This is safe, as externalsrc runs with the source is already unpacked.
+            d.prependVarFlag('do_deploy_source_date_epoch', 'prefuncs', 'create_source_date_epoch_stamp ')
+
         d.prependVarFlag('do_compile', 'prefuncs', "externalsrc_compile_prefunc ")
         d.prependVarFlag('do_configure', 'prefuncs', "externalsrc_configure_prefunc ")
 
-- 
2.17.1