[PATCH 0/1] rpm_4.14.0: clamp timestamps

[PATCH 0/1] rpm_4.14.0: clamp timestamps

[Juro Bystricky]

The recipe for RPM has been upgraded to version 4.14.0.
This version introduced a new macro "clamp_mtime_to_source_date_epoch".
The macro allows the RPM packages to set buildtime and mtime of files to
values as specified by SOURCE_DATE_EPOCH. This, however, is not the enabled by default:
The default setting for "clamp_mtime_to_source_date_epoch" is "0".
This patch sets the macro to "1", thus allowing clamping of the timestamps.
As the user needs to jump through some hoops to set SOURCE_DATE_EPOCH in
the environment, clamping the timestamps is very likely what the user wants.

I am aware that the macro can be set on command line.
This would involve patching the file package_rpm.bbclass, something like:

cmd = cmd + " --define 'clamp_mtime_to_source_date_epoch 1'"

However, I think a better approach was to patch the upgraded recipe for the RPM,
in order to keep package_rpm.bbclass as RPM version agnostic as possible.

I tested this patch by a simple test, adding to local.conf:

export SOURCE_DATE_EPOCH
SOURCE_DATE_EPOCH="0"
PACKAGE_CLASSES = "package_rpm package_deb"

Then I ran twice (in two different folders, same machine, different times, no SSTATE):

$ bitbake core-image-minimal

The build created 3952 target packages (RPM and Debian each).

Without this patch the binary package comparison for RPM packages was:

Same: 538
Different: 3414
Total: 3952

With the patch the results were substantially better (same numbers for RPM and Debian):

Same: 3917
Different: 35
Total: 3952

Few caveats:
You need two additional patches by Alex Kanavin:
"rpm: update to 4.14.0"
"package.bbclass: replace rpm/debugedit with dwarfsrcfiles"

The resulting numbers need to be taken with a grain of salt, as there are
addional factors that can cause different results.
I realize SOURCE_DATE_EPOCH should be determined based on source code,
instead of setting it to "0", but that is not crucial to test the patch.

Finally, RPM 4.14.0 supports yet another macro "_buildhost" which should be used
to define some kind of a "standard" buildhost name in order to improve
reproducibility. (Otherwise, I think, buildost is taken from /etc/hosts).
But that's a subject of a separate patch.


Juro Bystricky (1):
  rpm_4.14.0: clamp timestamps by default

 .../recipes-devtools/rpm/files/0001-support-sde.patch | 19 +++++++++++++++++++
 meta/recipes-devtools/rpm/rpm_4.14.0.bb               |  1 +
 2 files changed, 20 insertions(+)
 create mode 100644 meta/recipes-devtools/rpm/files/0001-support-sde.patch

-- 
2.7.4

[PATCH 1/1] rpm_4.14.0: clamp timestamps by default

[Juro Bystricky]

Improve reproducibility by making sure that timestamps
in built rpms are not later than the value of SOURCE_DATE_EPOCH as
found in the environment.
Timestamps as usual when SOURCE_DATE_EPOCH is not set.

[YOCTO #12425]

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
 .../recipes-devtools/rpm/files/0001-support-sde.patch | 19 +++++++++++++++++++
 meta/recipes-devtools/rpm/rpm_4.14.0.bb               |  1 +
 2 files changed, 20 insertions(+)
 create mode 100644 meta/recipes-devtools/rpm/files/0001-support-sde.patch

diff --git a/meta/recipes-devtools/rpm/files/0001-support-sde.patch b/meta/recipes-devtools/rpm/files/0001-support-sde.patch
new file mode 100644
index 0000000..b947c62d
--- /dev/null
+++ b/meta/recipes-devtools/rpm/files/0001-support-sde.patch
@@ -0,0 +1,19 @@
+
+Improve reproducibility: clamp timestamps by default.
+
+Upstream-Status: Inappropriate [oe-core specific]
+Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
+
+diff --git a/macros.in b/macros.in
+index d086248..af39c4a 100644
+--- a/macros.in
++++ b/macros.in
+@@ -248,7 +248,7 @@ package or when debugging this package.\
+ #	If true, make sure that timestamps in built rpms
+ #	are not later than the value of SOURCE_DATE_EPOCH.
+ #	Is ignored when SOURCE_DATE_EPOCH is not set.
+-%clamp_mtime_to_source_date_epoch 0
++%clamp_mtime_to_source_date_epoch 1
+ 
+ #	The directory where newly built binary packages will be written.
+ %_rpmdir		%{_topdir}/RPMS
diff --git a/meta/recipes-devtools/rpm/rpm_4.14.0.bb b/meta/recipes-devtools/rpm/rpm_4.14.0.bb
index e4e9c3e..83af18e 100644
--- a/meta/recipes-devtools/rpm/rpm_4.14.0.bb
+++ b/meta/recipes-devtools/rpm/rpm_4.14.0.bb
@@ -40,6 +40,7 @@ SRC_URI = "git://github.com/rpm-software-management/rpm;branch=rpm-4.14.x \
            file://0003-rpmstrpool.c-make-operations-over-string-pools-threa.patch \
            file://0004-build-pack.c-remove-static-local-variables-from-buil.patch \
            file://0001-perl-disable-auto-reqs.patch \
+           file://0001-support-sde.patch \
            "
 
 PE = "1"
-- 
2.7.4

Re: [PATCH 1/1] rpm_4.14.0: clamp timestamps by default

[Alexander Kanavin]

On 01/03/2018 01:16 AM, Juro Bystricky wrote:
> Improve reproducibility by making sure that timestamps
> in built rpms are not later than the value of SOURCE_DATE_EPOCH as
> found in the environment.
> Timestamps as usual when SOURCE_DATE_EPOCH is not set.

I'm not sure I understand the necessity of this. What matters for 
reproducibility is that rpms install the same files; why is it important 
that the rpm file itself has exactly same build time and is otherwise 
identical bit by bit?

A technicality: do not patch mnacros.in, set the macro directly from 
package_rpm.bbclass.

Alex

Re: [PATCH 1/1] rpm_4.14.0: clamp timestamps by default

[Richard Purdie]

On Wed, 2018-01-03 at 10:47 +0200, Alexander Kanavin wrote:
> On 01/03/2018 01:16 AM, Juro Bystricky wrote:
> > 
> > Improve reproducibility by making sure that timestamps
> > in built rpms are not later than the value of SOURCE_DATE_EPOCH as
> > found in the environment.
> > Timestamps as usual when SOURCE_DATE_EPOCH is not set.
> I'm not sure I understand the necessity of this. What matters for 
> reproducibility is that rpms install the same files; why is it
> important that the rpm file itself has exactly same build time and is
> otherwise identical bit by bit?

People have different definitions of reproducibility. For some its the
end binaries on the target system, for others its identical rpms.

Its certainly true that producing binaries that are more similar does
help us in a number of ways (e.g. churn in package feeds) so if we can
improve that without causing serious complexity issues, I'm broadly in
favour of it.

Cheers,

Richard

Re: [PATCH 1/1] rpm_4.14.0: clamp timestamps by default

[Bystricky, Juro]

> I'm not sure I understand the necessity of this. What matters for
> reproducibility is that rpms install the same files; why is it important
> that the rpm file itself has exactly same build time and is otherwise
> identical bit by bit?
> 

There is actually a demand for binary reproducible packages.
See for example https://wiki.debian.org/ReproducibleBuilds/About
Debian packages already clamp timestamps by default. I am not even sure you can disable this 
(short of unsetting SOURCE_CODE_EPOCH). The same functionality exists for RPM packages,
except it is not the default behavior.

> A technicality: do not patch mnacros.in, set the macro directly from
> package_rpm.bbclass.
>

Yes, I considered this (see the [patch 0/1]). I chose to patch macros.in in the 
recipe rpm_4.14.0 instead because the new macro is introduced in RPM 4.14.0. 
I assumed we did not want to have RPM version dependencies in package_rpm.bbclass. 
But I have no strong feelings regarding this, I am pretty sure the new macro is here
to stay in the future and it is unlikely anyone would want to use a pre-4.14.0 
version of RPM either. If you think patching package_rpm.bbclass makes more sense,
I can send in another patch.
 
Juro

Re: [PATCH 1/1] rpm_4.14.0: clamp timestamps by default

[Alexander Kanavin]

On 01/03/2018 06:59 PM, Bystricky, Juro wrote:

>> A technicality: do not patch mnacros.in, set the macro directly from
>> package_rpm.bbclass.
>>
> 
> Yes, I considered this (see the [patch 0/1]). I chose to patch macros.in in the
> recipe rpm_4.14.0 instead because the new macro is introduced in RPM 4.14.0.
> I assumed we did not want to have RPM version dependencies in package_rpm.bbclass.
> But I have no strong feelings regarding this, I am pretty sure the new macro is here
> to stay in the future and it is unlikely anyone would want to use a pre-4.14.0
> version of RPM either. If you think patching package_rpm.bbclass makes more sense,
> I can send in another patch.

Yes please. Setting configuration options at runtime is always 
preferable to patching the upstream source code, for maintainability 
reasons. We are already drowning in patches.

Alex