* [oe-core][PATCH] reproducible: fix git SOURCE_DATE_EPOCH randomness
@ 2026-02-13 0:42 rs
2026-02-14 9:53 ` Mathieu Dubois-Briand
0 siblings, 1 reply; 4+ messages in thread
From: rs @ 2026-02-13 0:42 UTC (permalink / raw)
To: raj.khem, richard.purdie, mathieu.dubois-briand, alex, otavio,
kexin.hao
Cc: afd, detheridge, denis, reatmon, openembedded-core, vijayp
From: Randolph Sapp <rs@ti.com>
Anything that defines multiple git sources should have the largest value
taken when calculating the SOURCE_DATE_EPOCH for a package.
The previous iteration actually introduced some degree of randomness, as
it would stop on the first git repository reported by os.walk, which
does not assure any specific ordering by default.
Signed-off-by: Randolph Sapp <rs@ti.com>
---
To address issue reported here:
https://lists.openembedded.org/g/openembedded-core/message/231076
meta/lib/oe/reproducible.py | 64 ++++++++++++++++---------------------
1 file changed, 28 insertions(+), 36 deletions(-)
diff --git a/meta/lib/oe/reproducible.py b/meta/lib/oe/reproducible.py
index 0270024a83..06ceda8d7f 100644
--- a/meta/lib/oe/reproducible.py
+++ b/meta/lib/oe/reproducible.py
@@ -3,6 +3,7 @@
#
# SPDX-License-Identifier: GPL-2.0-only
#
+import glob
import os
import subprocess
import bb
@@ -74,52 +75,43 @@ def get_source_date_epoch_from_known_files(d, sourcedir):
bb.debug(1, "SOURCE_DATE_EPOCH taken from: %s" % newest_file)
return source_date_epoch
-def find_git_folder(d, sourcedir):
- # First guess: UNPACKDIR/BB_GIT_DEFAULT_DESTSUFFIX
- # This is the default git fetcher unpack path
+def find_git_folders(d, sourcedir):
unpackdir = d.getVar('UNPACKDIR')
- default_destsuffix = d.getVar('BB_GIT_DEFAULT_DESTSUFFIX')
- gitpath = os.path.join(unpackdir, default_destsuffix, ".git")
- if os.path.isdir(gitpath):
- return gitpath
-
- # Second guess: ${S}
- gitpath = os.path.join(sourcedir, ".git")
- if os.path.isdir(gitpath):
- return gitpath
-
- # Perhaps there was a subpath or destsuffix specified.
- # Go looking in the UNPACKDIR
- for root, dirs, files in os.walk(unpackdir, topdown=True):
- if '.git' in dirs:
- return os.path.join(root, ".git")
+ git_folders = []
- for root, dirs, files in os.walk(sourcedir, topdown=True):
- if '.git' in dirs:
- return os.path.join(root, ".git")
+ for mainpath in (sourcedir, unpackdir):
+ gitpath_glob = os.path.join(mainpath, "**/.git/")
+ for gitpath in glob.glob(gitpath_glob, recursive=True):
+ git_folders.append(gitpath)
- bb.warn("Failed to find a git repository in UNPACKDIR: %s" % unpackdir)
- return None
+ if not git_folders:
+ bb.warn("Failed to find any git repository in UNPACKDIR or S")
+
+ return git_folders
def get_source_date_epoch_from_git(d, sourcedir):
if not "git://" in d.getVar('SRC_URI') and not "gitsm://" in d.getVar('SRC_URI'):
return None
- gitpath = find_git_folder(d, sourcedir)
- if not gitpath:
- return None
+ # Get an epoch from all valid git repositoies
+ sources_dates = []
+ for gitpath in find_git_folders(d, sourcedir):
+ # Check that the repository has a valid HEAD; it may not if subdir is used
+ # in SRC_URI
+ p = subprocess.run(['git', '--git-dir', gitpath, 'rev-parse', 'HEAD'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+ if p.returncode != 0:
+ bb.debug(1, "%s does not have a valid HEAD: %s" % (gitpath, p.stdout.decode('utf-8')))
+ continue
- # Check that the repository has a valid HEAD; it may not if subdir is used
- # in SRC_URI
- p = subprocess.run(['git', '--git-dir', gitpath, 'rev-parse', 'HEAD'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
- if p.returncode != 0:
- bb.debug(1, "%s does not have a valid HEAD: %s" % (gitpath, p.stdout.decode('utf-8')))
- return None
+ bb.debug(1, "git repository: %s" % gitpath)
+ p = subprocess.run(['git', '-c', 'log.showSignature=false', '--git-dir', gitpath, 'log', '-1', '--pretty=%ct'],
+ check=True, stdout=subprocess.PIPE)
+ sources_dates.append(int(p.stdout.decode('utf-8')))
+
+ if sources_dates:
+ return sorted(sources_dates, reverse=True)[0]
- bb.debug(1, "git repository: %s" % gitpath)
- p = subprocess.run(['git', '-c', 'log.showSignature=false', '--git-dir', gitpath, 'log', '-1', '--pretty=%ct'],
- check=True, stdout=subprocess.PIPE)
- return int(p.stdout.decode('utf-8'))
+ return None
def get_source_date_epoch_from_youngest_file(d, sourcedir):
if sourcedir == d.getVar('UNPACKDIR'):
--
2.52.0
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [oe-core][PATCH] reproducible: fix git SOURCE_DATE_EPOCH randomness
[not found] <1893A7AF46371281.653184@lists.openembedded.org>
@ 2026-02-13 1:36 ` Randolph Sapp
0 siblings, 0 replies; 4+ messages in thread
From: Randolph Sapp @ 2026-02-13 1:36 UTC (permalink / raw)
To: rs, raj.khem, richard.purdie, mathieu.dubois-briand, alex, otavio,
kexin.hao
Cc: afd, detheridge, denis, reatmon, openembedded-core, vijayp
On Thu Feb 12, 2026 at 6:42 PM CST, Randolph Sapp via lists.openembedded.org wrote:
> From: Randolph Sapp <rs@ti.com>
>
> Anything that defines multiple git sources should have the largest value
> taken when calculating the SOURCE_DATE_EPOCH for a package.
>
> The previous iteration actually introduced some degree of randomness, as
> it would stop on the first git repository reported by os.walk, which
> does not assure any specific ordering by default.
>
> Signed-off-by: Randolph Sapp <rs@ti.com>
> ---
>
> To address issue reported here:
> https://lists.openembedded.org/g/openembedded-core/message/231076
>
> meta/lib/oe/reproducible.py | 64 ++++++++++++++++---------------------
> 1 file changed, 28 insertions(+), 36 deletions(-)
>
> diff --git a/meta/lib/oe/reproducible.py b/meta/lib/oe/reproducible.py
> index 0270024a83..06ceda8d7f 100644
> --- a/meta/lib/oe/reproducible.py
> +++ b/meta/lib/oe/reproducible.py
> @@ -3,6 +3,7 @@
> #
> # SPDX-License-Identifier: GPL-2.0-only
> #
> +import glob
> import os
> import subprocess
> import bb
> @@ -74,52 +75,43 @@ def get_source_date_epoch_from_known_files(d, sourcedir):
> bb.debug(1, "SOURCE_DATE_EPOCH taken from: %s" % newest_file)
> return source_date_epoch
>
> -def find_git_folder(d, sourcedir):
> - # First guess: UNPACKDIR/BB_GIT_DEFAULT_DESTSUFFIX
> - # This is the default git fetcher unpack path
> +def find_git_folders(d, sourcedir):
> unpackdir = d.getVar('UNPACKDIR')
> - default_destsuffix = d.getVar('BB_GIT_DEFAULT_DESTSUFFIX')
> - gitpath = os.path.join(unpackdir, default_destsuffix, ".git")
> - if os.path.isdir(gitpath):
> - return gitpath
> -
> - # Second guess: ${S}
> - gitpath = os.path.join(sourcedir, ".git")
> - if os.path.isdir(gitpath):
> - return gitpath
> -
> - # Perhaps there was a subpath or destsuffix specified.
> - # Go looking in the UNPACKDIR
> - for root, dirs, files in os.walk(unpackdir, topdown=True):
> - if '.git' in dirs:
> - return os.path.join(root, ".git")
> + git_folders = []
>
> - for root, dirs, files in os.walk(sourcedir, topdown=True):
> - if '.git' in dirs:
> - return os.path.join(root, ".git")
> + for mainpath in (sourcedir, unpackdir):
> + gitpath_glob = os.path.join(mainpath, "**/.git/")
> + for gitpath in glob.glob(gitpath_glob, recursive=True):
> + git_folders.append(gitpath)
I honestly don't know if recursively searching the deploy directory is any worse
than instantiating a new fetcher and walking the SRC_URI and destsuffix values
directly. They both feel a little heavy-handed.
> - bb.warn("Failed to find a git repository in UNPACKDIR: %s" % unpackdir)
> - return None
> + if not git_folders:
> + bb.warn("Failed to find any git repository in UNPACKDIR or S")
> +
> + return git_folders
>
> def get_source_date_epoch_from_git(d, sourcedir):
> if not "git://" in d.getVar('SRC_URI') and not "gitsm://" in d.getVar('SRC_URI'):
> return None
>
> - gitpath = find_git_folder(d, sourcedir)
> - if not gitpath:
> - return None
> + # Get an epoch from all valid git repositoies
> + sources_dates = []
> + for gitpath in find_git_folders(d, sourcedir):
> + # Check that the repository has a valid HEAD; it may not if subdir is used
> + # in SRC_URI
> + p = subprocess.run(['git', '--git-dir', gitpath, 'rev-parse', 'HEAD'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
> + if p.returncode != 0:
> + bb.debug(1, "%s does not have a valid HEAD: %s" % (gitpath, p.stdout.decode('utf-8')))
> + continue
>
> - # Check that the repository has a valid HEAD; it may not if subdir is used
> - # in SRC_URI
> - p = subprocess.run(['git', '--git-dir', gitpath, 'rev-parse', 'HEAD'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
> - if p.returncode != 0:
> - bb.debug(1, "%s does not have a valid HEAD: %s" % (gitpath, p.stdout.decode('utf-8')))
> - return None
> + bb.debug(1, "git repository: %s" % gitpath)
> + p = subprocess.run(['git', '-c', 'log.showSignature=false', '--git-dir', gitpath, 'log', '-1', '--pretty=%ct'],
> + check=True, stdout=subprocess.PIPE)
> + sources_dates.append(int(p.stdout.decode('utf-8')))
> +
> + if sources_dates:
> + return sorted(sources_dates, reverse=True)[0]
>
> - bb.debug(1, "git repository: %s" % gitpath)
> - p = subprocess.run(['git', '-c', 'log.showSignature=false', '--git-dir', gitpath, 'log', '-1', '--pretty=%ct'],
> - check=True, stdout=subprocess.PIPE)
> - return int(p.stdout.decode('utf-8'))
> + return None
>
> def get_source_date_epoch_from_youngest_file(d, sourcedir):
> if sourcedir == d.getVar('UNPACKDIR'):
> --
> 2.52.0
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [oe-core][PATCH] reproducible: fix git SOURCE_DATE_EPOCH randomness
2026-02-13 0:42 [oe-core][PATCH] reproducible: fix git SOURCE_DATE_EPOCH randomness rs
@ 2026-02-14 9:53 ` Mathieu Dubois-Briand
2026-02-17 19:48 ` Randolph Sapp
0 siblings, 1 reply; 4+ messages in thread
From: Mathieu Dubois-Briand @ 2026-02-14 9:53 UTC (permalink / raw)
To: rs, raj.khem, richard.purdie, alex, otavio, kexin.hao
Cc: afd, detheridge, denis, reatmon, openembedded-core, vijayp
On Fri Feb 13, 2026 at 1:42 AM CET, Randolph Sapp via lists.openembedded.org wrote:
> From: Randolph Sapp <rs@ti.com>
>
> Anything that defines multiple git sources should have the largest value
> taken when calculating the SOURCE_DATE_EPOCH for a package.
>
> The previous iteration actually introduced some degree of randomness, as
> it would stop on the first git repository reported by os.walk, which
> does not assure any specific ordering by default.
>
> Signed-off-by: Randolph Sapp <rs@ti.com>
> ---
Hi Randolph,
Thanks for trying to fix this. However, it looks like this is causing
another intermittent issue.
We got the following error on the autobuilder:
ERROR: libjitterentropy-3.6.3-r0 do_unpack: Error executing a python function in exec_func_python() autogenerated:
...
File: '/srv/pokybuild/yocto-worker/qemux86-world/build/layers/openembedded-core/meta/lib/oe/reproducible.py', lineno: 107, function: get_source_date_epoch_from_git
0103: bb.debug(1, "%s does not have a valid HEAD: %s" % (gitpath, p.stdout.decode('utf-8')))
0104: continue
0105:
0106: bb.debug(1, "git repository: %s" % gitpath)
*** 0107: p = subprocess.run(['git', '-c', 'log.showSignature=false', '--git-dir', gitpath, 'log', '-1', '--pretty=%ct'],
0108: check=True, stdout=subprocess.PIPE)
0109: sources_dates.append(int(p.stdout.decode('utf-8')))
0110:
0111: if sources_dates:
File: '/usr/lib64/python3.13/subprocess.py', lineno: 577, function: run
0573: # We don't call process.wait() as .__exit__ does that for us.
0574: raise
0575: retcode = process.poll()
0576: if check and retcode:
*** 0577: raise CalledProcessError(retcode, process.args,
0578: output=stdout, stderr=stderr)
0579: return CompletedProcess(process.args, retcode, stdout, stderr)
0580:
0581:
Exception: subprocess.CalledProcessError: Command '['git', '-c', 'log.showSignature=false', '--git-dir', '/srv/pokybuild/yocto-worker/qemux86-world/build/build/tmp/work/x86-64-v3-poky-linux/libjitterentropy/3.6.3/sources/libjitterentropy-3.6.3/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/.git/', 'log', '-1', '--pretty=%ct']' returned non-zero exit status 128.
https://autobuilder.yoctoproject.org/valkyrie/#/builders/59/builds/3177
https://autobuilder.yoctoproject.org/valkyrie/#/builders/3/builds/3231
https://autobuilder.yoctoproject.org/valkyrie/#/builders/6/builds/3194
https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/3345
I only saw it on libjitterentropy so far. I quickly tried to reproduce
locally, without any success. Yet I confirm dropping this patch solve
the issue.
Can you have a look?
Thanks,
Mathieu
--
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [oe-core][PATCH] reproducible: fix git SOURCE_DATE_EPOCH randomness
2026-02-14 9:53 ` Mathieu Dubois-Briand
@ 2026-02-17 19:48 ` Randolph Sapp
0 siblings, 0 replies; 4+ messages in thread
From: Randolph Sapp @ 2026-02-17 19:48 UTC (permalink / raw)
To: Mathieu Dubois-Briand, rs, raj.khem, richard.purdie, alex, otavio,
kexin.hao
Cc: afd, detheridge, denis, reatmon, openembedded-core, vijayp
On Sat Feb 14, 2026 at 3:53 AM CST, Mathieu Dubois-Briand wrote:
> On Fri Feb 13, 2026 at 1:42 AM CET, Randolph Sapp via lists.openembedded.org wrote:
>> From: Randolph Sapp <rs@ti.com>
>>
>> Anything that defines multiple git sources should have the largest value
>> taken when calculating the SOURCE_DATE_EPOCH for a package.
>>
>> The previous iteration actually introduced some degree of randomness, as
>> it would stop on the first git repository reported by os.walk, which
>> does not assure any specific ordering by default.
>>
>> Signed-off-by: Randolph Sapp <rs@ti.com>
>> ---
>
> Hi Randolph,
>
> Thanks for trying to fix this. However, it looks like this is causing
> another intermittent issue.
>
> We got the following error on the autobuilder:
>
> ERROR: libjitterentropy-3.6.3-r0 do_unpack: Error executing a python function in exec_func_python() autogenerated:
> ...
> File: '/srv/pokybuild/yocto-worker/qemux86-world/build/layers/openembedded-core/meta/lib/oe/reproducible.py', lineno: 107, function: get_source_date_epoch_from_git
> 0103: bb.debug(1, "%s does not have a valid HEAD: %s" % (gitpath, p.stdout.decode('utf-8')))
> 0104: continue
> 0105:
> 0106: bb.debug(1, "git repository: %s" % gitpath)
> *** 0107: p = subprocess.run(['git', '-c', 'log.showSignature=false', '--git-dir', gitpath, 'log', '-1', '--pretty=%ct'],
> 0108: check=True, stdout=subprocess.PIPE)
> 0109: sources_dates.append(int(p.stdout.decode('utf-8')))
> 0110:
> 0111: if sources_dates:
> File: '/usr/lib64/python3.13/subprocess.py', lineno: 577, function: run
> 0573: # We don't call process.wait() as .__exit__ does that for us.
> 0574: raise
> 0575: retcode = process.poll()
> 0576: if check and retcode:
> *** 0577: raise CalledProcessError(retcode, process.args,
> 0578: output=stdout, stderr=stderr)
> 0579: return CompletedProcess(process.args, retcode, stdout, stderr)
> 0580:
> 0581:
> Exception: subprocess.CalledProcessError: Command '['git', '-c', 'log.showSignature=false', '--git-dir', '/srv/pokybuild/yocto-worker/qemux86-world/build/build/tmp/work/x86-64-v3-poky-linux/libjitterentropy/3.6.3/sources/libjitterentropy-3.6.3/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/tests/raw-entropy/recording_userspace/jitterentropy/.git/', 'log', '-1', '--pretty=%ct']' returned non-zero exit status 128.
>
>
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/59/builds/3177
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/3/builds/3231
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/6/builds/3194
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/3345
>
> I only saw it on libjitterentropy so far. I quickly tried to reproduce
> locally, without any success. Yet I confirm dropping this patch solve
> the issue.
>
> Can you have a look?
>
> Thanks,
> Mathieu
Oh, a recursive symbolic link. Fun. I thought glob didn't traverse links by
default. Guess I was thinking about Pathlib's version. I'll go back to a
modified os.walk version since that supports more python versions and has an
explicit 'followlinks' parameter.
- Randolph
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2026-02-17 19:48 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-02-13 0:42 [oe-core][PATCH] reproducible: fix git SOURCE_DATE_EPOCH randomness rs
2026-02-14 9:53 ` Mathieu Dubois-Briand
2026-02-17 19:48 ` Randolph Sapp
[not found] <1893A7AF46371281.653184@lists.openembedded.org>
2026-02-13 1:36 ` Randolph Sapp
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox