[PATCH 0/2] python3: Reproducible build fixes

[PATCH 0/2] python3: Reproducible build fixes

[Joshua Watt]

Two fixes to make the Python3 build reproducible

Joshua Watt (2):
  python3: Reformat sysconfig
  python3: Disable PGO for reproducible builds

 .../python/python3/reformat_sysconfig.py      | 21 +++++++++++++++++++
 meta/recipes-devtools/python/python3_3.7.3.bb | 20 +++++++++++++++++-
 2 files changed, 40 insertions(+), 1 deletion(-)
 create mode 100644 meta/recipes-devtools/python/python3/reformat_sysconfig.py

-- 
2.21.0

[PATCH 1/2] python3: Reformat sysconfig

[Joshua Watt]

Reformats the sysconfig file when packaging. This file is output by
using the python pprint function. This function will wrap long lines at
80 characters by default, and will even split strings at whitespace
boundaries to do so, e.g.:

 'A': 'B is really'
    ' long'

This causes a problem for reproducibility however because there might be
lines of differing lengths depending on the build path. These
non-reproducible paths are removed, but their effect on string wrapping
from pprint remains.

To correct this, reformat the entire sysconfig file by re-printing using
pprint with an (effectively) unlimited line length.

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 .../python/python3/reformat_sysconfig.py      | 21 +++++++++++++++++++
 meta/recipes-devtools/python/python3_3.7.3.bb | 10 +++++++++
 2 files changed, 31 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3/reformat_sysconfig.py

diff --git a/meta/recipes-devtools/python/python3/reformat_sysconfig.py b/meta/recipes-devtools/python/python3/reformat_sysconfig.py
new file mode 100644
index 00000000000..c4164313e8b
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/reformat_sysconfig.py
@@ -0,0 +1,21 @@
+#! /usr/bin/env python3
+#
+# SPDX-License-Identifier: MIT
+#
+# Copyright 2019 by Garmin Ltd. or its subsidiaries
+#
+# A script to reformat python sysconfig
+
+import sys
+import pprint
+l = {}
+g = {}
+with open(sys.argv[1], 'r') as f:
+    exec(f.read(), g, l)
+
+with open(sys.argv[1], 'w') as f:
+    for k in sorted(l.keys()):
+        f.write('%s = ' % k)
+        pprint.pprint(l[k], stream=f, width=sys.maxsize)
+        f.write('\n')
+
diff --git a/meta/recipes-devtools/python/python3_3.7.3.bb b/meta/recipes-devtools/python/python3_3.7.3.bb
index 24442961421..d14c93880fb 100644
--- a/meta/recipes-devtools/python/python3_3.7.3.bb
+++ b/meta/recipes-devtools/python/python3_3.7.3.bb
@@ -27,6 +27,10 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
 	   file://crosspythonpath.patch \
            "
 
+SRC_URI_append_class-target = " \
+           file://reformat_sysconfig.py \
+           "
+
 SRC_URI_append_class-native = " \
            file://0001-distutils-sysconfig-append-STAGING_LIBDIR-python-sys.patch \
            file://12-distutils-prefix-is-inside-staging-area.patch \
@@ -157,6 +161,12 @@ py_package_preprocess () {
                 ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata*.py \
                 ${PKGD}/${bindir}/python${PYTHON_BINABI}-config
 
+        # Reformat _sysconfigdata after modifying it so that it remains
+        # reproducible
+        for c in ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata*.py; do
+            python3 ${WORKDIR}/reformat_sysconfig.py $c
+        done
+
         # Recompile _sysconfigdata after modifying it
         cd ${PKGD}
         sysconfigfile=`find . -name _sysconfigdata_*.py`
-- 
2.21.0

Re: [PATCH 1/2] python3: Reformat sysconfig

[Richard Purdie]

On Thu, 2019-06-20 at 10:43 -0500, Joshua Watt wrote:
> Reformats the sysconfig file when packaging. This file is output by
> using the python pprint function. This function will wrap long lines
> at
> 80 characters by default, and will even split strings at whitespace
> boundaries to do so, e.g.:
> 
>  'A': 'B is really'
>     ' long'
> 
> This causes a problem for reproducibility however because there might
> be
> lines of differing lengths depending on the build path. These
> non-reproducible paths are removed, but their effect on string
> wrapping
> from pprint remains.
> 
> To correct this, reformat the entire sysconfig file by re-printing
> using
> pprint with an (effectively) unlimited line length.
> 
> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>

Fails on nativesdk:

https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/969

Have aborted that build.

Cheers,

Richard

[PATCH v2] python3: Reformat sysconfig

[Joshua Watt]

Reformats the sysconfig file when packaging. This file is output by
using the python pprint function. This function will wrap long lines at
80 characters by default, and will even split strings at whitespace
boundaries to do so, e.g.:

 'A': 'B is really'
    ' long'

This causes a problem for reproducibility however because there might be
lines of differing lengths depending on the build path. These
non-reproducible paths are removed, but their effect on string wrapping
from pprint remains.

To correct this, reformat the entire sysconfig file by re-printing using
pprint with an (effectively) unlimited line length.

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 .../python/python3/reformat_sysconfig.py      | 21 +++++++++++++++++++
 meta/recipes-devtools/python/python3_3.7.3.bb |  7 +++++++
 2 files changed, 28 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3/reformat_sysconfig.py

diff --git a/meta/recipes-devtools/python/python3/reformat_sysconfig.py b/meta/recipes-devtools/python/python3/reformat_sysconfig.py
new file mode 100644
index 00000000000..c4164313e8b
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/reformat_sysconfig.py
@@ -0,0 +1,21 @@
+#! /usr/bin/env python3
+#
+# SPDX-License-Identifier: MIT
+#
+# Copyright 2019 by Garmin Ltd. or its subsidiaries
+#
+# A script to reformat python sysconfig
+
+import sys
+import pprint
+l = {}
+g = {}
+with open(sys.argv[1], 'r') as f:
+    exec(f.read(), g, l)
+
+with open(sys.argv[1], 'w') as f:
+    for k in sorted(l.keys()):
+        f.write('%s = ' % k)
+        pprint.pprint(l[k], stream=f, width=sys.maxsize)
+        f.write('\n')
+
diff --git a/meta/recipes-devtools/python/python3_3.7.3.bb b/meta/recipes-devtools/python/python3_3.7.3.bb
index 8e77dbe9595..3409d94ba08 100644
--- a/meta/recipes-devtools/python/python3_3.7.3.bb
+++ b/meta/recipes-devtools/python/python3_3.7.3.bb
@@ -25,6 +25,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
            file://0001-Makefile-fix-Issue36464-parallel-build-race-problem.patch \
 	   file://0001-bpo-36852-proper-detection-of-mips-architecture-for-.patch \
 	   file://crosspythonpath.patch \
+           file://reformat_sysconfig.py \
            "
 
 SRC_URI_append_class-native = " \
@@ -165,6 +166,12 @@ py_package_preprocess () {
                 ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata*.py \
                 ${PKGD}/${bindir}/python${PYTHON_BINABI}-config
 
+        # Reformat _sysconfigdata after modifying it so that it remains
+        # reproducible
+        for c in ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata*.py; do
+            python3 ${WORKDIR}/reformat_sysconfig.py $c
+        done
+
         # Recompile _sysconfigdata after modifying it
         cd ${PKGD}
         sysconfigfile=`find . -name _sysconfigdata_*.py`
-- 
2.21.0

[PATCH 2/2] python3: Disable PGO for reproducible builds

[Joshua Watt]

Enabling PGO for python current causes it to not be reproducible when
building, so disable it for now.

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/recipes-devtools/python/python3_3.7.3.bb | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/python/python3_3.7.3.bb b/meta/recipes-devtools/python/python3_3.7.3.bb
index d14c93880fb..40e2229a165 100644
--- a/meta/recipes-devtools/python/python3_3.7.3.bb
+++ b/meta/recipes-devtools/python/python3_3.7.3.bb
@@ -79,8 +79,16 @@ CACHED_CONFIGUREVARS = " \
                 ac_cv_file__dev_ptc=no \
                 ac_cv_working_tzset=yes \
 "
+python() {
+    # PGO currently causes builds to not be reproducible, so disable it for
+    # now. See YOCTO #13407
+    if bb.utils.contains('MACHINE_FEATURES', 'qemu-usermode', True, False, d) and d.getVar('BUILD_REPRODUCIBLE_BINARIES') != '1':
+        d.setVar('PACKAGECONFIG_PGO', 'pgo')
+    else:
+        d.setVar('PACKAGECONFIG_PGO', '')
+}
 
-PACKAGECONFIG_class-target ??= "readline ${@bb.utils.contains('MACHINE_FEATURES', 'qemu-usermode', 'pgo', '', d)}"
+PACKAGECONFIG_class-target ??= "readline ${PACKAGECONFIG_PGO}"
 PACKAGECONFIG_class-native ??= "readline"
 PACKAGECONFIG_class-nativesdk ??= "readline"
 PACKAGECONFIG[readline] = ",,readline"
-- 
2.21.0

✗ patchtest: failure for python3: Reproducible build fixes

[Patchwork]

== Series Details ==

Series: python3: Reproducible build fixes
Revision: 1
URL   : https://patchwork.openembedded.org/series/18272/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Errors in your Python code were encountered [test_pylint] 
  Suggested fix    Correct the lines introduced by your patch
  Output           Please, fix the listed issues:
                   meta/recipes-devtools/python/python3/reformat_sysconfig.py does not exist



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

✗ patchtest: failure for python3: Reproducible build fixes (rev2)

[Patchwork]

== Series Details ==

Series: python3: Reproducible build fixes (rev2)
Revision: 2
URL   : https://patchwork.openembedded.org/series/18272/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
  Suggested fix    Rebase your series on top of targeted branch
  Targeted branch  master (currently at c724a2feae)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe