* [yocto-autobuilder-helper][PATCH v2 01/10] scripts/utils: fix stale extraction dir when tarball is updated
2026-06-01 23:17 [yocto-autobuilder-helper][PATCH v2 00/10] Implement 'containers' jobs tim.orling
@ 2026-06-01 23:18 ` tim.orling
2026-06-02 10:22 ` [yocto-patches] " Paul Barker
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 02/10] scripts/utils: warn and force re-download for HTTPS sources without SHA256 tim.orling
` (8 subsequent siblings)
9 siblings, 1 reply; 23+ messages in thread
From: tim.orling @ 2026-06-01 23:18 UTC (permalink / raw)
To: yocto-patches
From: Tim Orling <tim.orling@konsulko.com>
Previously the entire download/cache-validation block in
setup_tools_tarball() was guarded by `if not os.path.exists(btdir)`.
Once the extraction directory existed from a prior build, every
subsequent call was a no-op: the cached tarball was never re-validated,
so a freshly-published SDK (e.g. vcontainer-tarball-latest) was silently
ignored and the stale btdir kept being used.
Fix by moving the lock/download block outside the btdir existence guard
so cache validation always runs. Track tarball_updated to know when the
cached download was actually replaced, then remove the stale btdir when
True so the fresh tarball is re-extracted.
Also add an mtime-based staleness check for local-path (cp) sources so
that a newer source file automatically invalidates the cached copy.
AI-Generated: Claude Cowork Sonnet 4.6
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
scripts/utils.py | 64 +++++++++++++++++++++++++++++++-----------------
1 file changed, 41 insertions(+), 23 deletions(-)
diff --git a/scripts/utils.py b/scripts/utils.py
index 88842f0..87acad6 100644
--- a/scripts/utils.py
+++ b/scripts/utils.py
@@ -481,31 +481,49 @@ def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"):
if ";" in bttarball:
bttarball, sha256 = bttarball.split(";")
btdir = os.path.abspath(btdir)
+ btdlpath = getconfig("BASE_SHAREDDIR", ourconfig) + "/cluster-downloads-cache/" + os.path.basename(bttarball)
+ btlock = btdlpath + ".lock"
+ if not os.path.exists(os.path.dirname(btdlpath)):
+ os.makedirs(os.path.dirname(btdlpath), exist_ok=True)
+ # Always run the cache-validation / download step so that a freshly
+ # published tarball is detected even when btdir already exists from a
+ # previous build. tarball_updated is set to True whenever the cached
+ # download is replaced, which triggers removal of the stale btdir.
+ tarball_updated = False
+ while True:
+ try:
+ with open(btlock, 'a+') as lf:
+ fileno = lf.fileno()
+ fcntl.flock(fileno, fcntl.LOCK_EX)
+ if sha256 and os.path.exists(btdlpath):
+ dl_sha256 = sha256_file(btdlpath)
+ if dl_sha256 != sha256:
+ os.unlink(btdlpath)
+ elif bttarball.startswith("/") and os.path.exists(btdlpath):
+ # For local-path sources (e.g. vcontainer-tarball-latest)
+ # invalidate the cached copy when the source is newer so
+ # that a freshly-published tarball is always picked up.
+ if os.path.getmtime(bttarball) > os.path.getmtime(btdlpath):
+ os.unlink(btdlpath)
+ os.unlink(btdlpath)
+ if not os.path.exists(btdlpath):
+ if bttarball.startswith("/"):
+ subprocess.check_call(["cp", bttarball, btdlpath])
+ else:
+ subprocess.check_call(["wget", "-O", btdlpath, bttarball])
+ os.chmod(btdlpath, 0o775)
+ tarball_updated = True
+ break
+ except OSError:
+ # We raced with someone else, try again
+ pass
+ # If the underlying tarball changed, remove any stale extraction
+ # directory so it is re-extracted below.
+ if tarball_updated and os.path.exists(btdir):
+ print("Removing stale %s extraction at %s" % (name, btdir))
+ subprocess.check_call(["rm", "-rf", btdir])
if not os.path.exists(btdir):
- btdlpath = getconfig("BASE_SHAREDDIR", ourconfig) + "/cluster-downloads-cache/" + os.path.basename(bttarball)
print("Extracting %s %s" % (name, bttarball))
- btlock = btdlpath + ".lock"
- if not os.path.exists(os.path.dirname(btdlpath)):
- os.makedirs(os.path.dirname(btdlpath), exist_ok=True)
- while True:
- try:
- with open(btlock, 'a+') as lf:
- fileno = lf.fileno()
- fcntl.flock(fileno, fcntl.LOCK_EX)
- if sha256 and os.path.exists(btdlpath):
- dl_sha256 = sha256_file(btdlpath)
- if dl_sha256 != sha256:
- os.unlink(btdlpath)
- if not os.path.exists(btdlpath):
- if bttarball.startswith("/"):
- subprocess.check_call(["cp", bttarball, btdlpath])
- else:
- subprocess.check_call(["wget", "-O", btdlpath, bttarball])
- os.chmod(btdlpath, 0o775)
- break
- except OSError:
- # We raced with someone else, try again
- pass
subprocess.check_call(["bash", btdlpath, "-d", btdir, "-y"])
enable_tools_tarball(btdir, name)
--
2.43.0
^ permalink raw reply related [flat|nested] 23+ messages in thread* Re: [yocto-patches] [yocto-autobuilder-helper][PATCH v2 01/10] scripts/utils: fix stale extraction dir when tarball is updated
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 01/10] scripts/utils: fix stale extraction dir when tarball is updated tim.orling
@ 2026-06-02 10:22 ` Paul Barker
2026-06-06 1:19 ` [yocto-autobuilder-helper][PATCH] scripts/utils.py: add timeout check for tarball extraction Tim Orling
2026-06-06 3:15 ` [yocto-patches] [yocto-autobuilder-helper][PATCH v2 01/10] scripts/utils: fix stale extraction dir when tarball is updated Tim Orling
0 siblings, 2 replies; 23+ messages in thread
From: Paul Barker @ 2026-06-02 10:22 UTC (permalink / raw)
To: yocto-patches
[-- Attachment #1: Type: text/plain, Size: 5969 bytes --]
On Mon, 2026-06-01 at 16:18 -0700, Tim Orling via lists.yoctoproject.org
wrote:
> From: Tim Orling <tim.orling@konsulko.com>
>
> Previously the entire download/cache-validation block in
> setup_tools_tarball() was guarded by `if not os.path.exists(btdir)`.
> Once the extraction directory existed from a prior build, every
> subsequent call was a no-op: the cached tarball was never re-validated,
> so a freshly-published SDK (e.g. vcontainer-tarball-latest) was silently
> ignored and the stale btdir kept being used.
>
> Fix by moving the lock/download block outside the btdir existence guard
> so cache validation always runs. Track tarball_updated to know when the
> cached download was actually replaced, then remove the stale btdir when
> True so the fresh tarball is re-extracted.
>
> Also add an mtime-based staleness check for local-path (cp) sources so
> that a newer source file automatically invalidates the cached copy.
>
> AI-Generated: Claude Cowork Sonnet 4.6
> Signed-off-by: Tim Orling <tim.orling@konsulko.com>
> ---
> scripts/utils.py | 64 +++++++++++++++++++++++++++++++-----------------
> 1 file changed, 41 insertions(+), 23 deletions(-)
>
> diff --git a/scripts/utils.py b/scripts/utils.py
> index 88842f0..87acad6 100644
> --- a/scripts/utils.py
> +++ b/scripts/utils.py
> @@ -481,31 +481,49 @@ def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"):
> if ";" in bttarball:
> bttarball, sha256 = bttarball.split(";")
> btdir = os.path.abspath(btdir)
> + btdlpath = getconfig("BASE_SHAREDDIR", ourconfig) + "/cluster-downloads-cache/" + os.path.basename(bttarball)
> + btlock = btdlpath + ".lock"
> + if not os.path.exists(os.path.dirname(btdlpath)):
> + os.makedirs(os.path.dirname(btdlpath), exist_ok=True)
> + # Always run the cache-validation / download step so that a freshly
> + # published tarball is detected even when btdir already exists from a
> + # previous build. tarball_updated is set to True whenever the cached
> + # download is replaced, which triggers removal of the stale btdir.
> + tarball_updated = False
> + while True:
> + try:
> + with open(btlock, 'a+') as lf:
> + fileno = lf.fileno()
> + fcntl.flock(fileno, fcntl.LOCK_EX)
> + if sha256 and os.path.exists(btdlpath):
> + dl_sha256 = sha256_file(btdlpath)
> + if dl_sha256 != sha256:
> + os.unlink(btdlpath)
> + elif bttarball.startswith("/") and os.path.exists(btdlpath):
> + # For local-path sources (e.g. vcontainer-tarball-latest)
> + # invalidate the cached copy when the source is newer so
> + # that a freshly-published tarball is always picked up.
> + if os.path.getmtime(bttarball) > os.path.getmtime(btdlpath):
> + os.unlink(btdlpath)
> + os.unlink(btdlpath)
> + if not os.path.exists(btdlpath):
> + if bttarball.startswith("/"):
> + subprocess.check_call(["cp", bttarball, btdlpath])
> + else:
> + subprocess.check_call(["wget", "-O", btdlpath, bttarball])
> + os.chmod(btdlpath, 0o775)
> + tarball_updated = True
> + break
> + except OSError:
> + # We raced with someone else, try again
> + pass
Hi Tim,
This loop allows us to handle random OSError conditions caused by a
race, but it turns a deterministic OSError into an infinite loop. The
logic already exists, you're just moving it around, but now I have seen
it I cannot unsee it!
Should we add a timeout while we are here?
> + # If the underlying tarball changed, remove any stale extraction
> + # directory so it is re-extracted below.
> + if tarball_updated and os.path.exists(btdir):
> + print("Removing stale %s extraction at %s" % (name, btdir))
> + subprocess.check_call(["rm", "-rf", btdir])
> if not os.path.exists(btdir):
> - btdlpath = getconfig("BASE_SHAREDDIR", ourconfig) + "/cluster-downloads-cache/" + os.path.basename(bttarball)
> print("Extracting %s %s" % (name, bttarball))
> - btlock = btdlpath + ".lock"
> - if not os.path.exists(os.path.dirname(btdlpath)):
> - os.makedirs(os.path.dirname(btdlpath), exist_ok=True)
> - while True:
> - try:
> - with open(btlock, 'a+') as lf:
> - fileno = lf.fileno()
> - fcntl.flock(fileno, fcntl.LOCK_EX)
> - if sha256 and os.path.exists(btdlpath):
> - dl_sha256 = sha256_file(btdlpath)
> - if dl_sha256 != sha256:
> - os.unlink(btdlpath)
> - if not os.path.exists(btdlpath):
> - if bttarball.startswith("/"):
> - subprocess.check_call(["cp", bttarball, btdlpath])
> - else:
> - subprocess.check_call(["wget", "-O", btdlpath, bttarball])
> - os.chmod(btdlpath, 0o775)
> - break
> - except OSError:
> - # We raced with someone else, try again
> - pass
> subprocess.check_call(["bash", btdlpath, "-d", btdir, "-y"])
> enable_tools_tarball(btdir, name)
Best regards,
--
Paul Barker
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 252 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread* [yocto-autobuilder-helper][PATCH] scripts/utils.py: add timeout check for tarball extraction
2026-06-02 10:22 ` [yocto-patches] " Paul Barker
@ 2026-06-06 1:19 ` Tim Orling
2026-06-06 3:15 ` [yocto-patches] [yocto-autobuilder-helper][PATCH v2 01/10] scripts/utils: fix stale extraction dir when tarball is updated Tim Orling
1 sibling, 0 replies; 23+ messages in thread
From: Tim Orling @ 2026-06-06 1:19 UTC (permalink / raw)
To: yocto-patches; +Cc: Tim Orling
The while True, try, except OSError: pass loop has a risk of
getting caught in an infinite loop. Add an overall timeout
to ensure we raise the underlying OSError after we have tried
for 300 seconds/5 minutes.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
scripts/utils.py | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/scripts/utils.py b/scripts/utils.py
index a4dd12e..4aa4468 100644
--- a/scripts/utils.py
+++ b/scripts/utils.py
@@ -506,6 +506,7 @@ def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"):
# previous build. tarball_updated is set to True whenever the cached
# download is replaced, which triggers removal of the stale btdir.
tarball_updated = False
+ tarball_timeout = time.monotonic() + 300 # 5 min, tune as needed
while True:
try:
with open(btlock, 'a+') as lf:
@@ -548,7 +549,10 @@ def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"):
break
except OSError:
# We raced with someone else, try again
- pass
+ if time.monotonic() > tarball_timeout:
+ raise
+ time.sleep(1)
+
# If the underlying tarball changed, remove any stale extraction
# directory so it is re-extracted below.
if tarball_updated and os.path.exists(btdir):
--
2.54.0
^ permalink raw reply related [flat|nested] 23+ messages in thread* Re: [yocto-patches] [yocto-autobuilder-helper][PATCH v2 01/10] scripts/utils: fix stale extraction dir when tarball is updated
2026-06-02 10:22 ` [yocto-patches] " Paul Barker
2026-06-06 1:19 ` [yocto-autobuilder-helper][PATCH] scripts/utils.py: add timeout check for tarball extraction Tim Orling
@ 2026-06-06 3:15 ` Tim Orling
1 sibling, 0 replies; 23+ messages in thread
From: Tim Orling @ 2026-06-06 3:15 UTC (permalink / raw)
To: yocto-patches
[-- Attachment #1: Type: text/plain, Size: 7019 bytes --]
On Tue, Jun 2, 2026 at 3:22 AM Paul Barker via lists.yoctoproject.org <paul=
pbarker.dev@lists.yoctoproject.org> wrote:
> On Mon, 2026-06-01 at 16:18 -0700, Tim Orling via lists.yoctoproject.org
> wrote:
> > From: Tim Orling <tim.orling@konsulko.com>
> >
> > Previously the entire download/cache-validation block in
> > setup_tools_tarball() was guarded by `if not os.path.exists(btdir)`.
> > Once the extraction directory existed from a prior build, every
> > subsequent call was a no-op: the cached tarball was never re-validated,
> > so a freshly-published SDK (e.g. vcontainer-tarball-latest) was silently
> > ignored and the stale btdir kept being used.
> >
> > Fix by moving the lock/download block outside the btdir existence guard
> > so cache validation always runs. Track tarball_updated to know when the
> > cached download was actually replaced, then remove the stale btdir when
> > True so the fresh tarball is re-extracted.
> >
> > Also add an mtime-based staleness check for local-path (cp) sources so
> > that a newer source file automatically invalidates the cached copy.
> >
> > AI-Generated: Claude Cowork Sonnet 4.6
> > Signed-off-by: Tim Orling <tim.orling@konsulko.com>
> > ---
> > scripts/utils.py | 64 +++++++++++++++++++++++++++++++-----------------
> > 1 file changed, 41 insertions(+), 23 deletions(-)
> >
> > diff --git a/scripts/utils.py b/scripts/utils.py
> > index 88842f0..87acad6 100644
> > --- a/scripts/utils.py
> > +++ b/scripts/utils.py
> > @@ -481,31 +481,49 @@ def setup_tools_tarball(ourconfig, btdir,
> bttarball, name="buildtools"):
> > if ";" in bttarball:
> > bttarball, sha256 = bttarball.split(";")
> > btdir = os.path.abspath(btdir)
> > + btdlpath = getconfig("BASE_SHAREDDIR", ourconfig) +
> "/cluster-downloads-cache/" + os.path.basename(bttarball)
> > + btlock = btdlpath + ".lock"
> > + if not os.path.exists(os.path.dirname(btdlpath)):
> > + os.makedirs(os.path.dirname(btdlpath), exist_ok=True)
> > + # Always run the cache-validation / download step so that a
> freshly
> > + # published tarball is detected even when btdir already exists
> from a
> > + # previous build. tarball_updated is set to True whenever the
> cached
> > + # download is replaced, which triggers removal of the stale
> btdir.
> > + tarball_updated = False
> > + while True:
> > + try:
> > + with open(btlock, 'a+') as lf:
> > + fileno = lf.fileno()
> > + fcntl.flock(fileno, fcntl.LOCK_EX)
> > + if sha256 and os.path.exists(btdlpath):
> > + dl_sha256 = sha256_file(btdlpath)
> > + if dl_sha256 != sha256:
> > + os.unlink(btdlpath)
> > + elif bttarball.startswith("/") and
> os.path.exists(btdlpath):
> > + # For local-path sources (e.g.
> vcontainer-tarball-latest)
> > + # invalidate the cached copy when the source is
> newer so
> > + # that a freshly-published tarball is always
> picked up.
> > + if os.path.getmtime(bttarball) >
> os.path.getmtime(btdlpath):
> > + os.unlink(btdlpath)
> > + os.unlink(btdlpath)
> > + if not os.path.exists(btdlpath):
> > + if bttarball.startswith("/"):
> > + subprocess.check_call(["cp", bttarball,
> btdlpath])
> > + else:
> > + subprocess.check_call(["wget", "-O",
> btdlpath, bttarball])
> > + os.chmod(btdlpath, 0o775)
> > + tarball_updated = True
> > + break
> > + except OSError:
> > + # We raced with someone else, try again
> > + pass
>
> Hi Tim,
>
> This loop allows us to handle random OSError conditions caused by a
> race, but it turns a deterministic OSError into an infinite loop. The
> logic already exists, you're just moving it around, but now I have seen
> it I cannot unsee it!
>
> Should we add a timeout while we are here?
>
This patch was already merged, but the timeout concern was submitted
separately:
https://lore.kernel.org/yocto-patches/20260606011918.61582-1-tim.orling@konsulko.com/
>
> > + # If the underlying tarball changed, remove any stale extraction
> > + # directory so it is re-extracted below.
> > + if tarball_updated and os.path.exists(btdir):
> > + print("Removing stale %s extraction at %s" % (name, btdir))
> > + subprocess.check_call(["rm", "-rf", btdir])
> > if not os.path.exists(btdir):
> > - btdlpath = getconfig("BASE_SHAREDDIR", ourconfig) +
> "/cluster-downloads-cache/" + os.path.basename(bttarball)
> > print("Extracting %s %s" % (name, bttarball))
> > - btlock = btdlpath + ".lock"
> > - if not os.path.exists(os.path.dirname(btdlpath)):
> > - os.makedirs(os.path.dirname(btdlpath), exist_ok=True)
> > - while True:
> > - try:
> > - with open(btlock, 'a+') as lf:
> > - fileno = lf.fileno()
> > - fcntl.flock(fileno, fcntl.LOCK_EX)
> > - if sha256 and os.path.exists(btdlpath):
> > - dl_sha256 = sha256_file(btdlpath)
> > - if dl_sha256 != sha256:
> > - os.unlink(btdlpath)
> > - if not os.path.exists(btdlpath):
> > - if bttarball.startswith("/"):
> > - subprocess.check_call(["cp", bttarball,
> btdlpath])
> > - else:
> > - subprocess.check_call(["wget", "-O",
> btdlpath, bttarball])
> > - os.chmod(btdlpath, 0o775)
> > - break
> > - except OSError:
> > - # We raced with someone else, try again
> > - pass
> > subprocess.check_call(["bash", btdlpath, "-d", btdir, "-y"])
> > enable_tools_tarball(btdir, name)
>
> Best regards,
>
> --
> Paul Barker
>
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#4110):
> https://lists.yoctoproject.org/g/yocto-patches/message/4110
> Mute This Topic: https://lists.yoctoproject.org/mt/119603240/924729
> Group Owner: yocto-patches+owner@lists.yoctoproject.org
> Unsubscribe:
> https://lists.yoctoproject.org/g/yocto-patches/leave/13169857/924729/1023951714/xyzzy
> [ticotimo@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>
>
[-- Attachment #2: Type: text/html, Size: 9781 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* [yocto-autobuilder-helper][PATCH v2 02/10] scripts/utils: warn and force re-download for HTTPS sources without SHA256
2026-06-01 23:17 [yocto-autobuilder-helper][PATCH v2 00/10] Implement 'containers' jobs tim.orling
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 01/10] scripts/utils: fix stale extraction dir when tarball is updated tim.orling
@ 2026-06-01 23:18 ` tim.orling
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 03/10] config.json: add sha256sum for BUILDTOOLS URLs tim.orling
` (7 subsequent siblings)
9 siblings, 0 replies; 23+ messages in thread
From: tim.orling @ 2026-06-01 23:18 UTC (permalink / raw)
To: yocto-patches
From: Tim Orling <tim.orling@konsulko.com>
Without a SHA256 checksum there is no way to verify that a cached HTTPS
download is still current. Rather than silently reusing a potentially
stale copy, delete the cached file and force a re-download each run,
and emit a clear WARNING telling the operator how to avoid the overhead
(by appending ;sha256=<hash> to the URL in their config).
AI-Generated: Claude Cowork Sonnet 4.6
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
scripts/utils.py | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/scripts/utils.py b/scripts/utils.py
index 87acad6..ea905d9 100644
--- a/scripts/utils.py
+++ b/scripts/utils.py
@@ -505,6 +505,14 @@ def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"):
# that a freshly-published tarball is always picked up.
if os.path.getmtime(bttarball) > os.path.getmtime(btdlpath):
os.unlink(btdlpath)
+ elif not bttarball.startswith("/") and os.path.exists(btdlpath):
+ # HTTPS/FTP source with no SHA256: there is no way to
+ # verify the cached copy is current, so force a
+ # re-download every run. Add a sha256=<hash> suffix to
+ # the URL in your config to avoid this.
+ print("WARNING: no SHA256 provided for %s source %s; "
+ "forcing re-download to avoid using a stale cached copy"
+ % (name, bttarball))
os.unlink(btdlpath)
if not os.path.exists(btdlpath):
if bttarball.startswith("/"):
--
2.43.0
^ permalink raw reply related [flat|nested] 23+ messages in thread* [yocto-autobuilder-helper][PATCH v2 03/10] config.json: add sha256sum for BUILDTOOLS URLs
2026-06-01 23:17 [yocto-autobuilder-helper][PATCH v2 00/10] Implement 'containers' jobs tim.orling
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 01/10] scripts/utils: fix stale extraction dir when tarball is updated tim.orling
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 02/10] scripts/utils: warn and force re-download for HTTPS sources without SHA256 tim.orling
@ 2026-06-01 23:18 ` tim.orling
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 04/10] scripts/utils: add getconfigdict() for dict-type config values tim.orling
` (6 subsequent siblings)
9 siblings, 0 replies; 23+ messages in thread
From: tim.orling @ 2026-06-01 23:18 UTC (permalink / raw)
To: yocto-patches
From: Tim Orling <tim.orling@konsulko.com>
setup_tools_tarball() now checks the sha256sum of cached tools tarballs
to determine if an update is available and needs to be re-downloaded.
To prevent unnecessary downloads, add ;sha256= to the BUILDTOOLS URLs.
extratools/EXTRATOOLS_URL uses the same utils.setup_tools_tarball(), so
it also needs the ;sha256=.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
config.json | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/config.json b/config.json
index b27fee9..7df4271 100644
--- a/config.json
+++ b/config.json
@@ -7,10 +7,10 @@
"BUILD_HISTORY_REPO" : "ssh://git@push.yoctoproject.org/poky-buildhistory",
"BUILD_HISTORY_FORKPUSH" : {"openembedded-core-contrib:ross/mut" : "openembedded-core:master", "openembedded-core-contrib:abelloni/master-next": "openembedded-core:master", "openembedded-core:master-next" : "openembedded-core:master"},
- "BUILDTOOLS_URL" : "https://downloads.yoctoproject.org/releases/yocto/milestones/yocto-5.2_M2/buildtools/x86_64-buildtools-extended-nativesdk-standalone-5.1.sh",
- "BUILDTOOLS_ARM_URL" : "https://downloads.yoctoproject.org/releases/yocto/milestones/yocto-5.2_M2/buildtools/aarch64-buildtools-extended-nativesdk-standalone-5.1.sh",
- "BUILDTOOLS_MAKE_URL" : "https://downloads.yoctoproject.org/releases/yocto/yocto-5.0.4/buildtools/x86_64-buildtools-make-nativesdk-standalone-5.0.4.sh",
- "EXTRATOOLS_URL" : "https://downloads.yoctoproject.org/tools/buildtools/x86_64-buildtools-imagemagick-nativesdk-standalone-4.3+snapshot-5f2ba20f203114db9a3b11264467f8c23a05041d.sh",
+ "BUILDTOOLS_URL" : "https://downloads.yoctoproject.org/releases/yocto/milestones/yocto-5.2_M2/buildtools/x86_64-buildtools-extended-nativesdk-standalone-5.1.sh;sha256=5af9d92898af17fcc2fca4d07607a59f41b1c39a0a4ff058f64d834b98ec7fd5",
+ "BUILDTOOLS_ARM_URL" : "https://downloads.yoctoproject.org/releases/yocto/milestones/yocto-5.2_M2/buildtools/aarch64-buildtools-extended-nativesdk-standalone-5.1.sh;sha256=8074d582a60a5135fd5606b8326e749185d26f51aca27570447831310bcac187",
+ "BUILDTOOLS_MAKE_URL" : "https://downloads.yoctoproject.org/releases/yocto/yocto-5.0.4/buildtools/x86_64-buildtools-make-nativesdk-standalone-5.0.4.sh;sha256=2eb7a6c013113f4fdb87e800167606dc13af1bd8bbc1c9e2443b7be37fefd124",
+ "EXTRATOOLS_URL" : "https://downloads.yoctoproject.org/tools/buildtools/x86_64-buildtools-imagemagick-nativesdk-standalone-4.3+snapshot-5f2ba20f203114db9a3b11264467f8c23a05041d.sh;sha256=9cbff3a7cf524bdfa7779dce8afaf3453114d8017918d2927f723ea38a36ebdc",
"REPO_STASH_DIR" : "${BASE_HOMEDIR}/git/mirror",
"TRASH_DIR" : "${BASE_HOMEDIR}/git/trash",
--
2.43.0
^ permalink raw reply related [flat|nested] 23+ messages in thread* [yocto-autobuilder-helper][PATCH v2 04/10] scripts/utils: add getconfigdict() for dict-type config values
2026-06-01 23:17 [yocto-autobuilder-helper][PATCH v2 00/10] Implement 'containers' jobs tim.orling
` (2 preceding siblings ...)
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 03/10] config.json: add sha256sum for BUILDTOOLS URLs tim.orling
@ 2026-06-01 23:18 ` tim.orling
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 05/10] scripts: add vcontainer-tarball setup, integration, and publishing tim.orling
` (5 subsequent siblings)
9 siblings, 0 replies; 23+ messages in thread
From: tim.orling @ 2026-06-01 23:18 UTC (permalink / raw)
To: yocto-patches
From: Tim Orling <tim.orling@konsulko.com>
Parallel to getconfiglist(), but for JSON object values. The merge
priority is defaults < target-level < step-level so that more-specific
keys win: a step can override individual entries in a target-level dict
without replacing the whole thing, and both levels refine the defaults.
Used by the upcoming CONTAINER_IMAGES support, where each entry maps a
Yocto recipe name (the on-disk OCI path stem) to an image name (the
name pushed to the container registry).
AI-Generated: Claude Cowork Sonnet 4.6
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
scripts/utils.py | 15 +++++++++++++++
1 file changed, 15 insertions(+)
diff --git a/scripts/utils.py b/scripts/utils.py
index ea905d9..112ebc2 100644
--- a/scripts/utils.py
+++ b/scripts/utils.py
@@ -112,6 +112,21 @@ def getconfiglist(name, config, target, stepnum):
ret.extend(config['defaults'][name])
return expandresult(ret, config)
+# Get a build configuration dict, merging defaults < target < step so that
+# more-specific entries win (step-level keys override target-level, which
+# override defaults).
+def getconfigdict(name, config, target, stepnum):
+ ret = {}
+ step = "step" + str(stepnum)
+ if name in config['defaults']:
+ ret.update(config['defaults'][name])
+ if target in config['overrides']:
+ if name in config['overrides'][target]:
+ ret.update(config['overrides'][target][name])
+ if step in config['overrides'][target] and name in config['overrides'][target][step]:
+ ret.update(config['overrides'][target][step][name])
+ return expandresult(ret, config)
+
# Return only unique configuration values (identified with '=' in them)
def getconfiglistfilter(name, config, target, stepnum):
def merge(main, newvals):
--
2.43.0
^ permalink raw reply related [flat|nested] 23+ messages in thread* [yocto-autobuilder-helper][PATCH v2 05/10] scripts: add vcontainer-tarball setup, integration, and publishing
2026-06-01 23:17 [yocto-autobuilder-helper][PATCH v2 00/10] Implement 'containers' jobs tim.orling
` (3 preceding siblings ...)
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 04/10] scripts/utils: add getconfigdict() for dict-type config values tim.orling
@ 2026-06-01 23:18 ` tim.orling
2026-06-05 16:02 ` [yocto-patches] " Richard Purdie
2026-06-05 16:26 ` Richard Purdie
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 06/10] config.json: add vcontainer-tarball build target tim.orling
` (4 subsequent siblings)
9 siblings, 2 replies; 23+ messages in thread
From: tim.orling @ 2026-06-01 23:18 UTC (permalink / raw)
To: yocto-patches
From: Tim Orling <tim.orling@konsulko.com>
Introduce the vcontainer-tarball SDK plumbing used by container build
jobs. The vcontainer-tarball is a meta-virtualization-derived SDK
(modelled after buildtools-tarball) that ships the container build
toolchain so worker jobs do not need to rebuild it for every step.
* scripts/utils.py: add setup_vcontainer_tarball(), and add an
env_glob keyword argument to setup_tools_tarball() and
enable_tools_tarball() so the vcontainer-tarball can source its
specific environment-setup-ci file rather than the universal
glob.
* scripts/run-config: source the vcontainer-tarball environment
for build-targets / cmds / test-targets / plain-cmds steps,
gated by a new NOVCONTAINER step variable so individual steps
(such as the dashboard indexing step) can opt out independently
of NOBUILDTOOLS.
* scripts/shared-repo-unpack: invoke setup_vcontainer_tarball so
workers extract the SDK during unpack.
* scripts/publish-artefacts: publish the vcontainer-tarball
artefact so downstream test jobs can fetch a stable SDK.
AI-Generated: Claude Cowork Opus 4.7
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
config.json | 1 +
scripts/publish-artefacts | 5 +++++
scripts/run-config | 19 +++++++++++++++++++
scripts/shared-repo-unpack | 1 +
scripts/utils.py | 22 ++++++++++++++++++----
5 files changed, 44 insertions(+), 4 deletions(-)
diff --git a/config.json b/config.json
index 7df4271..16d8a04 100644
--- a/config.json
+++ b/config.json
@@ -1433,6 +1433,7 @@
"step3" : {
"shortname" : "Populate/update dashboard site",
"NOBUILDTOOLS" : true,
+ "NOVCONTAINER" : true,
"EXTRACMDS" : ["${SCRIPTSDIR}/run-dashboard-index ${HELPERBUILDDIR}/../"]
}
},
diff --git a/scripts/publish-artefacts b/scripts/publish-artefacts
index e56e131..0e820e9 100755
--- a/scripts/publish-artefacts
+++ b/scripts/publish-artefacts
@@ -146,5 +146,10 @@ case "$target" in
sha256sums $TMPDIR/deploy/images/qemux86-64
cp -R --no-dereference --preserve=links $TMPDIR/deploy/images/qemux86-64/*qemux86* $DEST/patchtest
;;
+ "vcontainer-tarball")
+ mkdir -p $DEST/vcontainer-tarball
+ sha256sums $TMPDIR/deploy/sdk
+ cp -R --no-dereference --preserve=links $TMPDIR/deploy/sdk/*vcontainer* $DEST/vcontainer-tarball
+ ;;
esac
diff --git a/scripts/run-config b/scripts/run-config
index e896234..0f5a26a 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -153,6 +153,25 @@ else:
if args.phase == "init" and args.stepname == "buildtools":
sys.exit(0)
+if jcfg:
+ vcontainer = utils.setup_vcontainer_tarball(ourconfig, args.workername, None, checkonly=True)
+ if vcontainer:
+ addentry("vcontainer", "Setup vcontainer tarball", "init")
+else:
+ # If we're executing a specific step, check whether vcontainer is disabled for it
+ vcontainer = True
+ if args.stepname in ("build-targets", "cmds", "test-targets", "plain-cmds"):
+ try:
+ vcontainer = not utils.getconfigvar("NOVCONTAINER", ourconfig, args.target, int(args.phase))
+ except ValueError:
+ # Not an integer step phase
+ pass
+
+ if vcontainer:
+ utils.setup_vcontainer_tarball(ourconfig, args.workername, args.builddir + "/../vcontainer-tarball")
+ if args.phase == "init" and args.stepname == "vcontainer":
+ sys.exit(0)
+
extratools = utils.getconfigvar("extratools", ourconfig, args.target)
if jcfg:
if extratools:
diff --git a/scripts/shared-repo-unpack b/scripts/shared-repo-unpack
index 797dec6..869b214 100755
--- a/scripts/shared-repo-unpack
+++ b/scripts/shared-repo-unpack
@@ -77,6 +77,7 @@ for repo in sorted(repos.keys()):
utils.flush()
utils.setup_buildtools_tarball(ourconfig, args.workername, args.abworkdir + "/buildtools")
+utils.setup_vcontainer_tarball(ourconfig, args.workername, args.abworkdir + "/vcontainer-tarball")
if "bitbake" not in repos:
sys.exit(0)
diff --git a/scripts/utils.py b/scripts/utils.py
index 112ebc2..b020a7b 100644
--- a/scripts/utils.py
+++ b/scripts/utils.py
@@ -456,8 +456,8 @@ def sha256_file(filename):
pass
return method.hexdigest()
-def enable_tools_tarball(btdir, name):
- btenv = glob.glob(btdir + "/environment-setup*")
+def enable_tools_tarball(btdir, name, env_glob="/environment-setup*"):
+ btenv = glob.glob(btdir + env_glob)
print("Using %s %s" % (name, btenv))
# We either parse or wrap all our execution calls, rock and a hard place :(
with open(btenv[0], "r") as f:
@@ -474,6 +474,20 @@ def enable_tools_tarball(btdir, name):
if line in os.environ:
del os.environ[line]
+def setup_vcontainer_tarball(ourconfig, workername, vcdir, checkonly=False):
+ vctarball = None
+ if "vcontainer" in ourconfig and workername:
+ vccfg = getconfig("vcontainer", ourconfig)
+ for entry in vccfg:
+ if fnmatch.fnmatch(workername, entry):
+ vctarball = vccfg[entry]
+ break
+
+ if checkonly:
+ return vctarball
+
+ setup_tools_tarball(ourconfig, vcdir, vctarball, name="vcontainer-tarball", env_glob="/environment-setup-ci")
+
def setup_buildtools_tarball(ourconfig, workername, btdir, checkonly=False):
bttarball = None
if "buildtools" in ourconfig and workername:
@@ -488,7 +502,7 @@ def setup_buildtools_tarball(ourconfig, workername, btdir, checkonly=False):
setup_tools_tarball(ourconfig, btdir, bttarball)
-def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"):
+def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools", env_glob="/environment-setup*"):
btenv = None
if bttarball:
@@ -548,7 +562,7 @@ def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"):
if not os.path.exists(btdir):
print("Extracting %s %s" % (name, bttarball))
subprocess.check_call(["bash", btdlpath, "-d", btdir, "-y"])
- enable_tools_tarball(btdir, name)
+ enable_tools_tarball(btdir, name, env_glob)
def get_string_from_version(version, milestone=None, rc=None):
""" Point releases finishing by 0 (e.g 4.0.0, 4.1.0) do no exists,
--
2.43.0
^ permalink raw reply related [flat|nested] 23+ messages in thread* Re: [yocto-patches] [yocto-autobuilder-helper][PATCH v2 05/10] scripts: add vcontainer-tarball setup, integration, and publishing
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 05/10] scripts: add vcontainer-tarball setup, integration, and publishing tim.orling
@ 2026-06-05 16:02 ` Richard Purdie
2026-06-05 16:26 ` Richard Purdie
1 sibling, 0 replies; 23+ messages in thread
From: Richard Purdie @ 2026-06-05 16:02 UTC (permalink / raw)
To: yocto-patches, Tim Orling
On Mon, 2026-06-01 at 16:18 -0700, Tim Orling via lists.yoctoproject.org wrote:
> From: Tim Orling <tim.orling@konsulko.com>
>
> Introduce the vcontainer-tarball SDK plumbing used by container build
> jobs. The vcontainer-tarball is a meta-virtualization-derived SDK
> (modelled after buildtools-tarball) that ships the container build
> toolchain so worker jobs do not need to rebuild it for every step.
>
> * scripts/utils.py: add setup_vcontainer_tarball(), and add an
> env_glob keyword argument to setup_tools_tarball() and
> enable_tools_tarball() so the vcontainer-tarball can source its
> specific environment-setup-ci file rather than the universal
> glob.
> * scripts/run-config: source the vcontainer-tarball environment
> for build-targets / cmds / test-targets / plain-cmds steps,
> gated by a new NOVCONTAINER step variable so individual steps
> (such as the dashboard indexing step) can opt out independently
> of NOBUILDTOOLS.
> * scripts/shared-repo-unpack: invoke setup_vcontainer_tarball so
> workers extract the SDK during unpack.
> * scripts/publish-artefacts: publish the vcontainer-tarball
> artefact so downstream test jobs can fetch a stable SDK.
>
> AI-Generated: Claude Cowork Opus 4.7
> Signed-off-by: Tim Orling <tim.orling@konsulko.com>
> ---
> config.json | 1 +
> scripts/publish-artefacts | 5 +++++
> scripts/run-config | 19 +++++++++++++++++++
> scripts/shared-repo-unpack | 1 +
> scripts/utils.py | 22 ++++++++++++++++++----
> 5 files changed, 44 insertions(+), 4 deletions(-)
>
> diff --git a/config.json b/config.json
> index 7df4271..16d8a04 100644
> --- a/config.json
> +++ b/config.json
> @@ -1433,6 +1433,7 @@
> "step3" : {
> "shortname" : "Populate/update dashboard site",
> "NOBUILDTOOLS" : true,
> + "NOVCONTAINER" : true,
> "EXTRACMDS" : ["${SCRIPTSDIR}/run-dashboard-index ${HELPERBUILDDIR}/../"]
> }
> },
> diff --git a/scripts/publish-artefacts b/scripts/publish-artefacts
> index e56e131..0e820e9 100755
> --- a/scripts/publish-artefacts
> +++ b/scripts/publish-artefacts
> @@ -146,5 +146,10 @@ case "$target" in
> sha256sums $TMPDIR/deploy/images/qemux86-64
> cp -R --no-dereference --preserve=links $TMPDIR/deploy/images/qemux86-64/*qemux86* $DEST/patchtest
> ;;
> + "vcontainer-tarball")
> + mkdir -p $DEST/vcontainer-tarball
> + sha256sums $TMPDIR/deploy/sdk
> + cp -R --no-dereference --preserve=links $TMPDIR/deploy/sdk/*vcontainer* $DEST/vcontainer-tarball
> + ;;
> esac
>
> diff --git a/scripts/run-config b/scripts/run-config
> index e896234..0f5a26a 100755
> --- a/scripts/run-config
> +++ b/scripts/run-config
> @@ -153,6 +153,25 @@ else:
> if args.phase == "init" and args.stepname == "buildtools":
> sys.exit(0)
>
> +if jcfg:
> + vcontainer = utils.setup_vcontainer_tarball(ourconfig, args.workername, None, checkonly=True)
> + if vcontainer:
> + addentry("vcontainer", "Setup vcontainer tarball", "init")
> +else:
> + # If we're executing a specific step, check whether vcontainer is disabled for it
> + vcontainer = True
> + if args.stepname in ("build-targets", "cmds", "test-targets", "plain-cmds"):
> + try:
> + vcontainer = not utils.getconfigvar("NOVCONTAINER", ourconfig, args.target, int(args.phase))
> + except ValueError:
> + # Not an integer step phase
> + pass
I suspect this logic also needs to be in the if jcfg block above, else
it will add the task to all jobs on the autobuilder, then just do
nothing in the task. If there isn't anything to do, we may as well just
not add it all?
Cheers,
Richard
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [yocto-patches] [yocto-autobuilder-helper][PATCH v2 05/10] scripts: add vcontainer-tarball setup, integration, and publishing
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 05/10] scripts: add vcontainer-tarball setup, integration, and publishing tim.orling
2026-06-05 16:02 ` [yocto-patches] " Richard Purdie
@ 2026-06-05 16:26 ` Richard Purdie
2026-06-06 3:08 ` Tim Orling
1 sibling, 1 reply; 23+ messages in thread
From: Richard Purdie @ 2026-06-05 16:26 UTC (permalink / raw)
To: yocto-patches
On Mon, 2026-06-01 at 16:18 -0700, Tim Orling via lists.yoctoproject.org wrote:
> From: Tim Orling <tim.orling@konsulko.com>
>
> Introduce the vcontainer-tarball SDK plumbing used by container build
> jobs. The vcontainer-tarball is a meta-virtualization-derived SDK
> (modelled after buildtools-tarball) that ships the container build
> toolchain so worker jobs do not need to rebuild it for every step.
>
> * scripts/utils.py: add setup_vcontainer_tarball(), and add an
> env_glob keyword argument to setup_tools_tarball() and
> enable_tools_tarball() so the vcontainer-tarball can source its
> specific environment-setup-ci file rather than the universal
> glob.
> * scripts/run-config: source the vcontainer-tarball environment
> for build-targets / cmds / test-targets / plain-cmds steps,
> gated by a new NOVCONTAINER step variable so individual steps
> (such as the dashboard indexing step) can opt out independently
> of NOBUILDTOOLS.
> * scripts/shared-repo-unpack: invoke setup_vcontainer_tarball so
> workers extract the SDK during unpack.
> * scripts/publish-artefacts: publish the vcontainer-tarball
> artefact so downstream test jobs can fetch a stable SDK.
>
> AI-Generated: Claude Cowork Opus 4.7
> Signed-off-by: Tim Orling <tim.orling@konsulko.com>
> ---
> config.json | 1 +
> scripts/publish-artefacts | 5 +++++
> scripts/run-config | 19 +++++++++++++++++++
> scripts/shared-repo-unpack | 1 +
> scripts/utils.py | 22 ++++++++++++++++++----
> 5 files changed, 44 insertions(+), 4 deletions(-)
>
> diff --git a/config.json b/config.json
> index 7df4271..16d8a04 100644
> --- a/config.json
> +++ b/config.json
> @@ -1433,6 +1433,7 @@
> "step3" : {
> "shortname" : "Populate/update dashboard site",
> "NOBUILDTOOLS" : true,
> + "NOVCONTAINER" : true,
> "EXTRACMDS" : ["${SCRIPTSDIR}/run-dashboard-index ${HELPERBUILDDIR}/../"]
> }
> },
Why is this step needing to opt out of a vcontainer? vcontainers are
only enabled for specific steps and off by default?
I looked further and realised you're basically using a vcontainer
everywhere as far as I can tell? Why do we want to have the overhead of
doing that outside of the vcontainer builds?
For buildtools, it makes sense we need it everywhere except where we
configure otherwise but I think for vcontainer, you want to opposite,
you only use it in jobs where we actually need/use it?
Cheers,
Richard
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [yocto-patches] [yocto-autobuilder-helper][PATCH v2 05/10] scripts: add vcontainer-tarball setup, integration, and publishing
2026-06-05 16:26 ` Richard Purdie
@ 2026-06-06 3:08 ` Tim Orling
0 siblings, 0 replies; 23+ messages in thread
From: Tim Orling @ 2026-06-06 3:08 UTC (permalink / raw)
To: yocto-patches
[-- Attachment #1: Type: text/plain, Size: 3601 bytes --]
On Fri, Jun 5, 2026 at 9:26 AM Richard Purdie via lists.yoctoproject.org
<richard.purdie=linuxfoundation.org@lists.yoctoproject.org> wrote:
> On Mon, 2026-06-01 at 16:18 -0700, Tim Orling via lists.yoctoproject.org
> wrote:
> > From: Tim Orling <tim.orling@konsulko.com>
> >
> > Introduce the vcontainer-tarball SDK plumbing used by container build
> > jobs. The vcontainer-tarball is a meta-virtualization-derived SDK
> > (modelled after buildtools-tarball) that ships the container build
> > toolchain so worker jobs do not need to rebuild it for every step.
> >
> > * scripts/utils.py: add setup_vcontainer_tarball(), and add an
> > env_glob keyword argument to setup_tools_tarball() and
> > enable_tools_tarball() so the vcontainer-tarball can source its
> > specific environment-setup-ci file rather than the universal
> > glob.
> > * scripts/run-config: source the vcontainer-tarball environment
> > for build-targets / cmds / test-targets / plain-cmds steps,
> > gated by a new NOVCONTAINER step variable so individual steps
> > (such as the dashboard indexing step) can opt out independently
> > of NOBUILDTOOLS.
> > * scripts/shared-repo-unpack: invoke setup_vcontainer_tarball so
> > workers extract the SDK during unpack.
> > * scripts/publish-artefacts: publish the vcontainer-tarball
> > artefact so downstream test jobs can fetch a stable SDK.
> >
> > AI-Generated: Claude Cowork Opus 4.7
> > Signed-off-by: Tim Orling <tim.orling@konsulko.com>
> > ---
> > config.json | 1 +
> > scripts/publish-artefacts | 5 +++++
> > scripts/run-config | 19 +++++++++++++++++++
> > scripts/shared-repo-unpack | 1 +
> > scripts/utils.py | 22 ++++++++++++++++++----
> > 5 files changed, 44 insertions(+), 4 deletions(-)
> >
> > diff --git a/config.json b/config.json
> > index 7df4271..16d8a04 100644
> > --- a/config.json
> > +++ b/config.json
> > @@ -1433,6 +1433,7 @@
> > "step3" : {
> > "shortname" : "Populate/update dashboard site",
> > "NOBUILDTOOLS" : true,
> > + "NOVCONTAINER" : true,
> > "EXTRACMDS" : ["${SCRIPTSDIR}/run-dashboard-index
> ${HELPERBUILDDIR}/../"]
> > }
> > },
>
> Why is this step needing to opt out of a vcontainer? vcontainers are
> only enabled for specific steps and off by default?
>
> I looked further and realised you're basically using a vcontainer
> everywhere as far as I can tell? Why do we want to have the overhead of
> doing that outside of the vcontainer builds?
>
> For buildtools, it makes sense we need it everywhere except where we
> configure otherwise but I think for vcontainer, you want to opposite,
> you only use it in jobs where we actually need/use it?
>
>
The simple answer is that the 'buildtools' pattern was followed and an
attempt
was made to reuse the 'buildtools' code as much as possible. Upon further
review, the 'extratools' pattern is a better fit. This was addressed in v3.
Cheers,
>
> Richard
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#4153):
> https://lists.yoctoproject.org/g/yocto-patches/message/4153
> Mute This Topic: https://lists.yoctoproject.org/mt/119603244/924729
> Group Owner: yocto-patches+owner@lists.yoctoproject.org
> Unsubscribe:
> https://lists.yoctoproject.org/g/yocto-patches/leave/13169857/924729/1023951714/xyzzy
> [ticotimo@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>
>
[-- Attachment #2: Type: text/html, Size: 5407 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* [yocto-autobuilder-helper][PATCH v2 06/10] config.json: add vcontainer-tarball build target
2026-06-01 23:17 [yocto-autobuilder-helper][PATCH v2 00/10] Implement 'containers' jobs tim.orling
` (4 preceding siblings ...)
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 05/10] scripts: add vcontainer-tarball setup, integration, and publishing tim.orling
@ 2026-06-01 23:18 ` tim.orling
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 07/10] scripts: add run-vcontainer-tests for meta-virtualization tim.orling
` (3 subsequent siblings)
9 siblings, 0 replies; 23+ messages in thread
From: tim.orling @ 2026-06-01 23:18 UTC (permalink / raw)
To: yocto-patches
From: Tim Orling <tim.orling@konsulko.com>
Add the vcontainer-tarball build target which produces the
meta-virtualization SDK used by downstream container build/test
jobs. Modelled after the buildtools-tarball target.
The target uses EXTRACMDS to reset BBMULTICONFIG for the worker
shell, places the extravars in the per-step config, and includes
a publish-artefacts step so the resulting SDK tarball is staged
for reuse by container-tests and other consumers.
Since meta-virtualization is the first layer removed by
'remove-layers', our BBMULTICONFIGs become invalid immediately
after layer removal and throw an error.
In scripts/run-config, EXTRACMDS runs after BBTARGETS, but
before 'remove-layers'. Use 'sed' to reset BBMULTICONFIG to ''.
The error in the 'remove-layers' step can be summarized as:
bb.parse.ParseError: ParseError at /home/pokybuild/yocto-worker/vcontainer-tarball/build/layers/openembedded-core/meta/conf/bitbake.conf:824: Could not include required file conf/multiconfig/vruntime-aarch64.conf
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
config.json | 26 ++++++++++++++++++++++++++
1 file changed, 26 insertions(+)
diff --git a/config.json b/config.json
index 16d8a04..7206c41 100644
--- a/config.json
+++ b/config.json
@@ -1864,6 +1864,32 @@
},
"toaster" : {
"EXTRACMDS" : ["${SCRIPTSDIR}/run-toaster-tests ${HELPERBUILDDIR} ${HELPERBUILDDIR}/../layers/bitbake"]
+ },
+ "vcontainer-tarball": {
+ "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"],
+ "ADDLAYER" : [
+ "${BUILDDIR}/../meta-openembedded/meta-oe",
+ "${BUILDDIR}/../meta-openembedded/meta-python",
+ "${BUILDDIR}/../meta-openembedded/meta-networking",
+ "${BUILDDIR}/../meta-openembedded/meta-filesystems",
+ "${BUILDDIR}/../meta-virtualization"
+ ],
+ "step1" : {
+ "shortname" : "Build vcontainer-tarballs",
+ "BBTARGETS" : "vcontainer-tarball",
+ "extravars" : [
+ "DISTRO_FEATURES:append = ' virtualization vcontainer'",
+ "BBMULTICONFIG = 'vruntime-aarch64 vruntime-x86-64'",
+ "INIT_MANAGER = 'systemd'"
+ ],
+ "EXTRACMDS" : ["sed -i '/vruntime-aarch64 vruntime-x86-64/d' ${HELPERBUILDDIR}/conf/auto.conf"]
+ },
+ "step2" : {
+ "shortname" : "Publish vcontainer SDK for test reuse",
+ "EXTRACMDS" : [
+ "install -d ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest && install -m 0755 ${BUILDDIR}/tmp/deploy/sdk/vcontainer-standalone.sh ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new && mv -f ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh"
+ ]
+ }
}
},
"repo-defaults" : {
--
2.43.0
^ permalink raw reply related [flat|nested] 23+ messages in thread* [yocto-autobuilder-helper][PATCH v2 07/10] scripts: add run-vcontainer-tests for meta-virtualization
2026-06-01 23:17 [yocto-autobuilder-helper][PATCH v2 00/10] Implement 'containers' jobs tim.orling
` (5 preceding siblings ...)
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 06/10] config.json: add vcontainer-tarball build target tim.orling
@ 2026-06-01 23:18 ` tim.orling
2026-06-02 10:50 ` [yocto-patches] " Paul Barker
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 08/10] scripts: add container registry push, auth, tagging, runtime selection tim.orling
` (2 subsequent siblings)
9 siblings, 1 reply; 23+ messages in thread
From: tim.orling @ 2026-06-01 23:18 UTC (permalink / raw)
To: yocto-patches
From: Tim Orling <tim.orling@konsulko.com>
Add scripts/run-vcontainer-tests, the test runner used by the
vcontainer test jobs. It sources the vcontainer-tarball SDK,
discovers the meta-virtualization pytest suite, and runs a
configurable set of suites (vdkr, vpdmn, memres) against the
checked-out layers. Suites can be selected per-step so the
top-level 'vcontainer-tests' job runs the container engine
agnostic tests:
- tests/test_container_cross_install.py
- tests/test_container_registry_script.py
- tests/test_vcontainer_auth_config.py
- tests/test_multiarch_oci.py
- tests/test_multilayer_oci.py
The 'vdkr-tests' and 'vpdmn-tests' jobs run only their respective
suites (including memres for each container engine):
- tests/test_vdkr.py
- tests/test_vdkr_registry.py
and
- tests/test_vpdmn.py
AI-Generated: Claude Cowork Opus 4.7
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
config.json | 34 ++++++++
scripts/run-vcontainer-tests | 164 +++++++++++++++++++++++++++++++++++
2 files changed, 198 insertions(+)
create mode 100755 scripts/run-vcontainer-tests
diff --git a/config.json b/config.json
index 7206c41..79a9d10 100644
--- a/config.json
+++ b/config.json
@@ -1890,6 +1890,40 @@
"install -d ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest && install -m 0755 ${BUILDDIR}/tmp/deploy/sdk/vcontainer-standalone.sh ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new && mv -f ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh"
]
}
+ },
+ "vcontainer-tests": {
+ "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"],
+ "ADDLAYER" : [
+ "${BUILDDIR}/../meta-openembedded/meta-oe",
+ "${BUILDDIR}/../meta-openembedded/meta-python",
+ "${BUILDDIR}/../meta-openembedded/meta-networking",
+ "${BUILDDIR}/../meta-openembedded/meta-filesystems",
+ "${BUILDDIR}/../meta-virtualization"
+ ],
+ "step1" : {
+ "shortname" : "Run vcontainer pytest suite",
+ "NOBUILDTOOLS" : 1,
+ "NOVCONTAINER" : 1,
+ "EXTRACMDS" : [
+ "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vcontainer ${BUILDDIR} ${BUILDDIR}/../meta-virtualization"
+ ]
+ },
+ "step2" : {
+ "shortname" : "Run vdkr pytest suite",
+ "NOBUILDTOOLS" : 1,
+ "NOVCONTAINER" : 1,
+ "EXTRACMDS" : [
+ "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vdkr ${BUILDDIR} ${BUILDDIR}/../meta-virtualization"
+ ]
+ },
+ "step3" : {
+ "shortname" : "Run vpdmn pytest suite",
+ "NOBUILDTOOLS" : 1,
+ "NOVCONTAINER" : 1,
+ "EXTRACMDS" : [
+ "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vpdmn ${BUILDDIR} ${BUILDDIR}/../meta-virtualization"
+ ]
+ }
}
},
"repo-defaults" : {
diff --git a/scripts/run-vcontainer-tests b/scripts/run-vcontainer-tests
new file mode 100755
index 0000000..1394c7c
--- /dev/null
+++ b/scripts/run-vcontainer-tests
@@ -0,0 +1,164 @@
+#!/bin/bash
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Run meta-virtualization pytest test suites against the vcontainer
+# standalone SDK (vdkr/vpdmn) that was built in a previous bitbake
+# step.
+#
+# Arguments:
+# $1 - suite name: one of "vcontainer", "vdkr", "vpdmn"
+# $2 - bitbake build directory (${BUILDDIR})
+# $3 - path to the meta-virtualization layer
+#
+# Optional environment variables:
+# RESULTS_DIR - directory to copy pytest artefacts (junit xml / log) to
+# VCONTAINER_EXTRACT_DIR - where to extract the standalone SDK tarball
+# (default: ${builddir}/vcontainer-test-extracted)
+# TEST_OCI_IMAGE - path to an OCI image directory (enables vdkr/vpdmn
+# import tests)
+# VDKR_ARCH - target architecture for vdkr/vpdmn tests (default: x86_64)
+#
+# The script is intentionally conservative: any pytest tests that cannot run
+# in the CI environment (those marked "slow", "network", "boot") are skipped
+# so that the autobuilder step completes without needing network access. Those
+# can be re-enabled by exporting META_VIRT_PYTEST_MARKERS before invocation.
+#
+# It is assumed that /dev/kvm is writable by the CI user running the tests,
+# since the performance is significantly faster with 'memres'.
+#
+
+set -e
+set -u
+set -o pipefail
+set -x
+
+if [ $# -lt 3 ]; then
+ echo "Usage: $0 <suite> <builddir> <meta-virtualization-dir>" >&2
+ echo " suite: vcontainer | vdkr | vpdmn" >&2
+ exit 2
+fi
+
+suite="$1"
+builddir=$(realpath "$2")
+metavirtdir=$(realpath "$3")
+
+if [ ! -d "$metavirtdir/tests" ]; then
+ echo "ERROR: meta-virtualization tests directory not found at $metavirtdir/tests" >&2
+ exit 1
+fi
+
+# Locate the vcontainer standalone SDK tarball. Prefer an externally-built
+# SDK passed via VCONTAINER_SDK (the autobuilder -tests jobs share the SDK
+# produced by the separate vcontainer-tarball builder), and fall back to
+# looking in the local build's deploy/sdk directory when running stand-alone.
+sdk_tarball=""
+if [ -n "${VCONTAINER_SDK:-}" ]; then
+ if [ -f "$VCONTAINER_SDK" ]; then
+ sdk_tarball="$VCONTAINER_SDK"
+ else
+ echo "ERROR: VCONTAINER_SDK=$VCONTAINER_SDK is set but not a file" >&2
+ exit 1
+ fi
+fi
+if [ -z "$sdk_tarball" ]; then
+ sdk_tarball="$builddir/tmp/deploy/sdk/vcontainer-standalone.sh"
+ if [ ! -f "$sdk_tarball" ]; then
+ # Try to find any matching tarball in case naming changed (e.g. versioned)
+ alt=$(ls -1 "$builddir"/tmp/deploy/sdk/vcontainer-*.sh 2>/dev/null | head -n1 || true)
+ if [ -n "$alt" ]; then
+ sdk_tarball="$alt"
+ else
+ echo "ERROR: vcontainer standalone SDK not found." >&2
+ echo " Set VCONTAINER_SDK to an existing SDK installer, or" >&2
+ echo " build vcontainer-tarball so $builddir/tmp/deploy/sdk/vcontainer-standalone.sh exists." >&2
+ exit 1
+ fi
+ fi
+fi
+
+extract_dir="${VCONTAINER_EXTRACT_DIR:-$builddir/vcontainer-test-extracted}"
+rm -rf "$extract_dir"
+mkdir -p "$(dirname "$extract_dir")"
+
+# Self-extracting installer (silent, -y agrees to license, -d picks dir)
+"$sdk_tarball" -d "$extract_dir" -y
+
+# Prepare a Python venv so we don't pollute the worker's system packages.
+python3 -m venv "$builddir/meta-virt-test-venv"
+# shellcheck disable=SC1091
+source "$builddir/meta-virt-test-venv/bin/activate"
+# Avoid warnings by upgrading pip; install pytest/pexpect into the venv via pip.
+python3 -m pip install --quiet --upgrade pip setuptools wheel
+python3 -m pip install --quiet --upgrade pytest pytest-timeout pexpect
+
+# Default marker filter excludes long running / infrastructure dependent tests.
+marker_filter="${META_VIRT_PYTEST_MARKERS:-not slow and not network and not boot and not incus and not k3s}"
+
+# Per-suite test file selection. Uses -k/-m for fine-grained filtering and
+# keeps the CLI small for logging clarity.
+case "$suite" in
+ vdkr)
+ test_files=(
+ "tests/test_vdkr.py"
+ "tests/test_vdkr_registry.py"
+ )
+ ;;
+ vpdmn)
+ test_files=(
+ "tests/test_vpdmn.py"
+ )
+ ;;
+ vcontainer)
+ # Broad vcontainer/bbclass/tooling coverage that doesn't require the
+ # vdkr/vpdmn CLI harness to be running.
+ test_files=(
+ "tests/test_container_cross_install.py"
+ "tests/test_container_registry_script.py"
+ "tests/test_vcontainer_auth_config.py"
+ "tests/test_multiarch_oci.py"
+ "tests/test_multilayer_oci.py"
+ )
+ ;;
+ *)
+ echo "ERROR: unknown suite '$suite' (expected vcontainer|vdkr|vpdmn)" >&2
+ exit 2
+ ;;
+esac
+
+pytest_args=(
+ -v
+ --tb=short
+ -m "$marker_filter"
+ --vdkr-dir "$extract_dir"
+ --junitxml="$builddir/pytest-$suite-results.xml"
+)
+
+# Allow tests that consume an OCI image (import/save/load) to find one.
+if [ -n "${TEST_OCI_IMAGE:-}" ] && [ -d "${TEST_OCI_IMAGE}" ]; then
+ pytest_args+=(--oci-image "$TEST_OCI_IMAGE")
+fi
+
+# Pass architecture through when set in the environment (default is x86_64).
+if [ -n "${VDKR_ARCH:-}" ]; then
+ pytest_args+=(--arch "$VDKR_ARCH")
+fi
+
+cd "$metavirtdir"
+# Don't let a single failing test kill the whole step - collect the junit
+# report, then surface the exit code via the junit file + exit status.
+set +e
+python3 -m pytest "${pytest_args[@]}" "${test_files[@]}"
+rc=$?
+set -e
+
+# Copy artefacts to the results dir if one was provided.
+if [ -n "${RESULTS_DIR:-}" ]; then
+ mkdir -p "$RESULTS_DIR"
+ cp -f "$builddir/pytest-$suite-results.xml" "$RESULTS_DIR/" 2>/dev/null || true
+ if [ -f /tmp/pytest-vcontainer.log ]; then
+ cp -f /tmp/pytest-vcontainer.log "$RESULTS_DIR/pytest-$suite.log" || true
+ fi
+fi
+
+exit $rc
--
2.43.0
^ permalink raw reply related [flat|nested] 23+ messages in thread* Re: [yocto-patches] [yocto-autobuilder-helper][PATCH v2 07/10] scripts: add run-vcontainer-tests for meta-virtualization
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 07/10] scripts: add run-vcontainer-tests for meta-virtualization tim.orling
@ 2026-06-02 10:50 ` Paul Barker
2026-06-06 3:05 ` Tim Orling
0 siblings, 1 reply; 23+ messages in thread
From: Paul Barker @ 2026-06-02 10:50 UTC (permalink / raw)
To: yocto-patches
[-- Attachment #1: Type: text/plain, Size: 11612 bytes --]
On Mon, 2026-06-01 at 16:18 -0700, Tim Orling via lists.yoctoproject.org
wrote:
> From: Tim Orling <tim.orling@konsulko.com>
>
> Add scripts/run-vcontainer-tests, the test runner used by the
> vcontainer test jobs. It sources the vcontainer-tarball SDK,
> discovers the meta-virtualization pytest suite, and runs a
> configurable set of suites (vdkr, vpdmn, memres) against the
> checked-out layers. Suites can be selected per-step so the
> top-level 'vcontainer-tests' job runs the container engine
> agnostic tests:
>
> - tests/test_container_cross_install.py
> - tests/test_container_registry_script.py
> - tests/test_vcontainer_auth_config.py
> - tests/test_multiarch_oci.py
> - tests/test_multilayer_oci.py
>
> The 'vdkr-tests' and 'vpdmn-tests' jobs run only their respective
> suites (including memres for each container engine):
>
> - tests/test_vdkr.py
> - tests/test_vdkr_registry.py
>
> and
>
> - tests/test_vpdmn.py
>
> AI-Generated: Claude Cowork Opus 4.7
> Signed-off-by: Tim Orling <tim.orling@konsulko.com>
> ---
> config.json | 34 ++++++++
> scripts/run-vcontainer-tests | 164 +++++++++++++++++++++++++++++++++++
> 2 files changed, 198 insertions(+)
> create mode 100755 scripts/run-vcontainer-tests
>
> diff --git a/config.json b/config.json
> index 7206c41..79a9d10 100644
> --- a/config.json
> +++ b/config.json
> @@ -1890,6 +1890,40 @@
> "install -d ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest && install -m 0755 ${BUILDDIR}/tmp/deploy/sdk/vcontainer-standalone.sh ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new && mv -f ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh"
> ]
> }
> + },
> + "vcontainer-tests": {
> + "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"],
> + "ADDLAYER" : [
> + "${BUILDDIR}/../meta-openembedded/meta-oe",
> + "${BUILDDIR}/../meta-openembedded/meta-python",
> + "${BUILDDIR}/../meta-openembedded/meta-networking",
> + "${BUILDDIR}/../meta-openembedded/meta-filesystems",
> + "${BUILDDIR}/../meta-virtualization"
> + ],
> + "step1" : {
> + "shortname" : "Run vcontainer pytest suite",
> + "NOBUILDTOOLS" : 1,
> + "NOVCONTAINER" : 1,
> + "EXTRACMDS" : [
> + "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vcontainer ${BUILDDIR} ${BUILDDIR}/../meta-virtualization"
> + ]
> + },
> + "step2" : {
> + "shortname" : "Run vdkr pytest suite",
> + "NOBUILDTOOLS" : 1,
> + "NOVCONTAINER" : 1,
> + "EXTRACMDS" : [
> + "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vdkr ${BUILDDIR} ${BUILDDIR}/../meta-virtualization"
> + ]
> + },
> + "step3" : {
> + "shortname" : "Run vpdmn pytest suite",
> + "NOBUILDTOOLS" : 1,
> + "NOVCONTAINER" : 1,
> + "EXTRACMDS" : [
> + "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vpdmn ${BUILDDIR} ${BUILDDIR}/../meta-virtualization"
> + ]
> + }
> }
> },
> "repo-defaults" : {
> diff --git a/scripts/run-vcontainer-tests b/scripts/run-vcontainer-tests
> new file mode 100755
> index 0000000..1394c7c
> --- /dev/null
> +++ b/scripts/run-vcontainer-tests
> @@ -0,0 +1,164 @@
> +#!/bin/bash
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +# Run meta-virtualization pytest test suites against the vcontainer
> +# standalone SDK (vdkr/vpdmn) that was built in a previous bitbake
> +# step.
> +#
> +# Arguments:
> +# $1 - suite name: one of "vcontainer", "vdkr", "vpdmn"
> +# $2 - bitbake build directory (${BUILDDIR})
> +# $3 - path to the meta-virtualization layer
> +#
> +# Optional environment variables:
> +# RESULTS_DIR - directory to copy pytest artefacts (junit xml / log) to
> +# VCONTAINER_EXTRACT_DIR - where to extract the standalone SDK tarball
> +# (default: ${builddir}/vcontainer-test-extracted)
> +# TEST_OCI_IMAGE - path to an OCI image directory (enables vdkr/vpdmn
> +# import tests)
> +# VDKR_ARCH - target architecture for vdkr/vpdmn tests (default: x86_64)
Taking some arguments on the command line and others via environment
variables is confusing. I wonder if should use getopts to parse arguments.
> +#
> +# The script is intentionally conservative: any pytest tests that cannot run
> +# in the CI environment (those marked "slow", "network", "boot") are skipped
> +# so that the autobuilder step completes without needing network access. Those
> +# can be re-enabled by exporting META_VIRT_PYTEST_MARKERS before invocation.
> +#
> +# It is assumed that /dev/kvm is writable by the CI user running the tests,
> +# since the performance is significantly faster with 'memres'.
> +#
> +
> +set -e
> +set -u
> +set -o pipefail
> +set -x
> +
> +if [ $# -lt 3 ]; then
> + echo "Usage: $0 <suite> <builddir> <meta-virtualization-dir>" >&2
> + echo " suite: vcontainer | vdkr | vpdmn" >&2
> + exit 2
> +fi
> +
> +suite="$1"
> +builddir=$(realpath "$2")
> +metavirtdir=$(realpath "$3")
> +
> +if [ ! -d "$metavirtdir/tests" ]; then
> + echo "ERROR: meta-virtualization tests directory not found at $metavirtdir/tests" >&2
> + exit 1
> +fi
> +
> +# Locate the vcontainer standalone SDK tarball. Prefer an externally-built
> +# SDK passed via VCONTAINER_SDK (the autobuilder -tests jobs share the SDK
> +# produced by the separate vcontainer-tarball builder), and fall back to
> +# looking in the local build's deploy/sdk directory when running stand-alone.
> +sdk_tarball=""
> +if [ -n "${VCONTAINER_SDK:-}" ]; then
> + if [ -f "$VCONTAINER_SDK" ]; then
> + sdk_tarball="$VCONTAINER_SDK"
> + else
> + echo "ERROR: VCONTAINER_SDK=$VCONTAINER_SDK is set but not a file" >&2
> + exit 1
> + fi
> +fi
> +if [ -z "$sdk_tarball" ]; then
> + sdk_tarball="$builddir/tmp/deploy/sdk/vcontainer-standalone.sh"
> + if [ ! -f "$sdk_tarball" ]; then
> + # Try to find any matching tarball in case naming changed (e.g. versioned)
> + alt=$(ls -1 "$builddir"/tmp/deploy/sdk/vcontainer-*.sh 2>/dev/null | head -n1 || true)
> + if [ -n "$alt" ]; then
> + sdk_tarball="$alt"
> + else
> + echo "ERROR: vcontainer standalone SDK not found." >&2
> + echo " Set VCONTAINER_SDK to an existing SDK installer, or" >&2
> + echo " build vcontainer-tarball so $builddir/tmp/deploy/sdk/vcontainer-standalone.sh exists." >&2
> + exit 1
> + fi
> + fi
> +fi
> +
> +extract_dir="${VCONTAINER_EXTRACT_DIR:-$builddir/vcontainer-test-extracted}"
> +rm -rf "$extract_dir"
> +mkdir -p "$(dirname "$extract_dir")"
> +
> +# Self-extracting installer (silent, -y agrees to license, -d picks dir)
> +"$sdk_tarball" -d "$extract_dir" -y
Patch 5 in this series added support for setting up the vcontainer
tarball for a step. I think the commit message in this patch needs to
explain why we can't use that in the steps that call
run-vcontainer-tests.
> +
> +# Prepare a Python venv so we don't pollute the worker's system packages.
> +python3 -m venv "$builddir/meta-virt-test-venv"
> +# shellcheck disable=SC1091
> +source "$builddir/meta-virt-test-venv/bin/activate"
Note that this will shadow the venv we use when running test jobs on the
autobuilder workers. That may be ok, but we need to be careful.
> +# Avoid warnings by upgrading pip; install pytest/pexpect into the venv via pip.
> +python3 -m pip install --quiet --upgrade pip setuptools wheel
> +python3 -m pip install --quiet --upgrade pytest pytest-timeout pexpect
Do we want to automatically install the latest versions of these tools,
via the network, in each build? It's better to pin versions to ensure
that upgrades are planned rather than automagical.
Also, should we store this list of dependencies in a requirements.txt
file somewhere? Perhaps tests/requirements.txt in meta-virtualization.
> +
> +# Default marker filter excludes long running / infrastructure dependent tests.
> +marker_filter="${META_VIRT_PYTEST_MARKERS:-not slow and not network and not boot and not incus and not k3s}"
Would it be better to have a positive marker, e.g. 'autobuilder', to
mark tests that we do want to run? That would allow us to maintain the
filtering solely in meta-virtualization instead of needing updates here
as well when things change.
> +
> +# Per-suite test file selection. Uses -k/-m for fine-grained filtering and
> +# keeps the CLI small for logging clarity.
> +case "$suite" in
> + vdkr)
> + test_files=(
> + "tests/test_vdkr.py"
> + "tests/test_vdkr_registry.py"
> + )
> + ;;
> + vpdmn)
> + test_files=(
> + "tests/test_vpdmn.py"
> + )
> + ;;
> + vcontainer)
> + # Broad vcontainer/bbclass/tooling coverage that doesn't require the
> + # vdkr/vpdmn CLI harness to be running.
> + test_files=(
> + "tests/test_container_cross_install.py"
> + "tests/test_container_registry_script.py"
> + "tests/test_vcontainer_auth_config.py"
> + "tests/test_multiarch_oci.py"
> + "tests/test_multilayer_oci.py"
> + )
> + ;;
> + *)
> + echo "ERROR: unknown suite '$suite' (expected vcontainer|vdkr|vpdmn)" >&2
> + exit 2
> + ;;
> +esac
> +
> +pytest_args=(
> + -v
> + --tb=short
> + -m "$marker_filter"
> + --vdkr-dir "$extract_dir"
> + --junitxml="$builddir/pytest-$suite-results.xml"
> +)
> +
> +# Allow tests that consume an OCI image (import/save/load) to find one.
> +if [ -n "${TEST_OCI_IMAGE:-}" ] && [ -d "${TEST_OCI_IMAGE}" ]; then
> + pytest_args+=(--oci-image "$TEST_OCI_IMAGE")
> +fi
> +
> +# Pass architecture through when set in the environment (default is x86_64).
> +if [ -n "${VDKR_ARCH:-}" ]; then
> + pytest_args+=(--arch "$VDKR_ARCH")
> +fi
> +
> +cd "$metavirtdir"
> +# Don't let a single failing test kill the whole step - collect the junit
> +# report, then surface the exit code via the junit file + exit status.
> +set +e
> +python3 -m pytest "${pytest_args[@]}" "${test_files[@]}"
> +rc=$?
> +set -e
> +
> +# Copy artefacts to the results dir if one was provided.
> +if [ -n "${RESULTS_DIR:-}" ]; then
> + mkdir -p "$RESULTS_DIR"
> + cp -f "$builddir/pytest-$suite-results.xml" "$RESULTS_DIR/" 2>/dev/null || true
> + if [ -f /tmp/pytest-vcontainer.log ]; then
> + cp -f /tmp/pytest-vcontainer.log "$RESULTS_DIR/pytest-$suite.log" || true
> + fi
> +fi
> +
> +exit $rc
--
Paul Barker
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 252 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [yocto-patches] [yocto-autobuilder-helper][PATCH v2 07/10] scripts: add run-vcontainer-tests for meta-virtualization
2026-06-02 10:50 ` [yocto-patches] " Paul Barker
@ 2026-06-06 3:05 ` Tim Orling
0 siblings, 0 replies; 23+ messages in thread
From: Tim Orling @ 2026-06-06 3:05 UTC (permalink / raw)
To: yocto-patches
[-- Attachment #1: Type: text/plain, Size: 13726 bytes --]
On Tue, Jun 2, 2026 at 3:50 AM Paul Barker via lists.yoctoproject.org <paul=
pbarker.dev@lists.yoctoproject.org> wrote:
> On Mon, 2026-06-01 at 16:18 -0700, Tim Orling via lists.yoctoproject.org
> wrote:
> > From: Tim Orling <tim.orling@konsulko.com>
> >
> > Add scripts/run-vcontainer-tests, the test runner used by the
> > vcontainer test jobs. It sources the vcontainer-tarball SDK,
> > discovers the meta-virtualization pytest suite, and runs a
> > configurable set of suites (vdkr, vpdmn, memres) against the
> > checked-out layers. Suites can be selected per-step so the
> > top-level 'vcontainer-tests' job runs the container engine
> > agnostic tests:
> >
> > - tests/test_container_cross_install.py
> > - tests/test_container_registry_script.py
> > - tests/test_vcontainer_auth_config.py
> > - tests/test_multiarch_oci.py
> > - tests/test_multilayer_oci.py
> >
> > The 'vdkr-tests' and 'vpdmn-tests' jobs run only their respective
> > suites (including memres for each container engine):
> >
> > - tests/test_vdkr.py
> > - tests/test_vdkr_registry.py
> >
> > and
> >
> > - tests/test_vpdmn.py
> >
> > AI-Generated: Claude Cowork Opus 4.7
> > Signed-off-by: Tim Orling <tim.orling@konsulko.com>
> > ---
> > config.json | 34 ++++++++
> > scripts/run-vcontainer-tests | 164 +++++++++++++++++++++++++++++++++++
> > 2 files changed, 198 insertions(+)
> > create mode 100755 scripts/run-vcontainer-tests
> >
> > diff --git a/config.json b/config.json
> > index 7206c41..79a9d10 100644
> > --- a/config.json
> > +++ b/config.json
> > @@ -1890,6 +1890,40 @@
> > "install -d
> ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest && install -m 0755
> ${BUILDDIR}/tmp/deploy/sdk/vcontainer-standalone.sh
> ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/
> vcontainer-standalone.sh.new && mv -f
> ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/
> vcontainer-standalone.sh.new
> ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh"
> > ]
> > }
> > + },
> > + "vcontainer-tests": {
> > + "NEEDREPOS" : ["bitbake", "meta-openembedded",
> "meta-virtualization"],
> > + "ADDLAYER" : [
> > + "${BUILDDIR}/../meta-openembedded/meta-oe",
> > + "${BUILDDIR}/../meta-openembedded/meta-python",
> > + "${BUILDDIR}/../meta-openembedded/meta-networking",
> > + "${BUILDDIR}/../meta-openembedded/meta-filesystems",
> > + "${BUILDDIR}/../meta-virtualization"
> > + ],
> > + "step1" : {
> > + "shortname" : "Run vcontainer pytest suite",
> > + "NOBUILDTOOLS" : 1,
> > + "NOVCONTAINER" : 1,
> > + "EXTRACMDS" : [
> > +
> "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh
> RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests
> vcontainer ${BUILDDIR} ${BUILDDIR}/../meta-virtualization"
> > + ]
> > + },
> > + "step2" : {
> > + "shortname" : "Run vdkr pytest suite",
> > + "NOBUILDTOOLS" : 1,
> > + "NOVCONTAINER" : 1,
> > + "EXTRACMDS" : [
> > +
> "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh
> RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vdkr
> ${BUILDDIR} ${BUILDDIR}/../meta-virtualization"
> > + ]
> > + },
> > + "step3" : {
> > + "shortname" : "Run vpdmn pytest suite",
> > + "NOBUILDTOOLS" : 1,
> > + "NOVCONTAINER" : 1,
> > + "EXTRACMDS" : [
> > +
> "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh
> RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vpdmn
> ${BUILDDIR} ${BUILDDIR}/../meta-virtualization"
> > + ]
> > + }
> > }
> > },
> > "repo-defaults" : {
> > diff --git a/scripts/run-vcontainer-tests b/scripts/run-vcontainer-tests
> > new file mode 100755
> > index 0000000..1394c7c
> > --- /dev/null
> > +++ b/scripts/run-vcontainer-tests
> > @@ -0,0 +1,164 @@
> > +#!/bin/bash
> > +#
> > +# SPDX-License-Identifier: GPL-2.0-only
> > +#
> > +# Run meta-virtualization pytest test suites against the vcontainer
> > +# standalone SDK (vdkr/vpdmn) that was built in a previous bitbake
> > +# step.
> > +#
> > +# Arguments:
> > +# $1 - suite name: one of "vcontainer", "vdkr", "vpdmn"
> > +# $2 - bitbake build directory (${BUILDDIR})
> > +# $3 - path to the meta-virtualization layer
> > +#
> > +# Optional environment variables:
> > +# RESULTS_DIR - directory to copy pytest artefacts (junit xml /
> log) to
> > +# VCONTAINER_EXTRACT_DIR - where to extract the standalone SDK tarball
> > +# (default: ${builddir}/vcontainer-test-extracted)
> > +# TEST_OCI_IMAGE - path to an OCI image directory (enables vdkr/vpdmn
> > +# import tests)
> > +# VDKR_ARCH - target architecture for vdkr/vpdmn tests
> (default: x86_64)
>
> Taking some arguments on the command line and others via environment
> variables is confusing. I wonder if should use getopts to parse arguments.
>
This is addressed in v3.
> > +#
> > +# The script is intentionally conservative: any pytest tests that
> cannot run
> > +# in the CI environment (those marked "slow", "network", "boot") are
> skipped
> > +# so that the autobuilder step completes without needing network
> access. Those
> > +# can be re-enabled by exporting META_VIRT_PYTEST_MARKERS before
> invocation.
> > +#
> > +# It is assumed that /dev/kvm is writable by the CI user running the
> tests,
> > +# since the performance is significantly faster with 'memres'.
> > +#
> > +
> > +set -e
> > +set -u
> > +set -o pipefail
> > +set -x
> > +
> > +if [ $# -lt 3 ]; then
> > + echo "Usage: $0 <suite> <builddir> <meta-virtualization-dir>" >&2
> > + echo " suite: vcontainer | vdkr | vpdmn" >&2
> > + exit 2
> > +fi
> > +
> > +suite="$1"
> > +builddir=$(realpath "$2")
> > +metavirtdir=$(realpath "$3")
> > +
> > +if [ ! -d "$metavirtdir/tests" ]; then
> > + echo "ERROR: meta-virtualization tests directory not found at
> $metavirtdir/tests" >&2
> > + exit 1
> > +fi
> > +
> > +# Locate the vcontainer standalone SDK tarball. Prefer an
> externally-built
> > +# SDK passed via VCONTAINER_SDK (the autobuilder -tests jobs share the
> SDK
> > +# produced by the separate vcontainer-tarball builder), and fall back to
> > +# looking in the local build's deploy/sdk directory when running
> stand-alone.
> > +sdk_tarball=""
> > +if [ -n "${VCONTAINER_SDK:-}" ]; then
> > + if [ -f "$VCONTAINER_SDK" ]; then
> > + sdk_tarball="$VCONTAINER_SDK"
> > + else
> > + echo "ERROR: VCONTAINER_SDK=$VCONTAINER_SDK is set but not a
> file" >&2
> > + exit 1
> > + fi
> > +fi
> > +if [ -z "$sdk_tarball" ]; then
> > + sdk_tarball="$builddir/tmp/deploy/sdk/vcontainer-standalone.sh"
> > + if [ ! -f "$sdk_tarball" ]; then
> > + # Try to find any matching tarball in case naming changed (e.g.
> versioned)
> > + alt=$(ls -1 "$builddir"/tmp/deploy/sdk/vcontainer-*.sh
> 2>/dev/null | head -n1 || true)
> > + if [ -n "$alt" ]; then
> > + sdk_tarball="$alt"
> > + else
> > + echo "ERROR: vcontainer standalone SDK not found." >&2
> > + echo " Set VCONTAINER_SDK to an existing SDK
> installer, or" >&2
> > + echo " build vcontainer-tarball so
> $builddir/tmp/deploy/sdk/vcontainer-standalone.sh exists." >&2
> > + exit 1
> > + fi
> > + fi
> > +fi
> > +
> >
> +extract_dir="${VCONTAINER_EXTRACT_DIR:-$builddir/vcontainer-test-extracted}"
> > +rm -rf "$extract_dir"
> > +mkdir -p "$(dirname "$extract_dir")"
> > +
> > +# Self-extracting installer (silent, -y agrees to license, -d picks dir)
> > +"$sdk_tarball" -d "$extract_dir" -y
>
> Patch 5 in this series added support for setting up the vcontainer
> tarball for a step. I think the commit message in this patch needs to
> explain why we can't use that in the steps that call
> run-vcontainer-tests.
>
> Added information related to this context to v3.
> > +
> > +# Prepare a Python venv so we don't pollute the worker's system
> packages.
> > +python3 -m venv "$builddir/meta-virt-test-venv"
> > +# shellcheck disable=SC1091
> > +source "$builddir/meta-virt-test-venv/bin/activate"
>
> Note that this will shadow the venv we use when running test jobs on the
> autobuilder workers. That may be ok, but we need to be careful.
>
This venv is very very specifically for only the meta-virtualization tests.
Other venvs in
autobuilder workers as far as I can tell do not have 'pip' and do not
install 'pytest' nor
the other requirements.
> > +# Avoid warnings by upgrading pip; install pytest/pexpect into the venv
> via pip.
> > +python3 -m pip install --quiet --upgrade pip setuptools wheel
> > +python3 -m pip install --quiet --upgrade pytest pytest-timeout pexpect
>
> Do we want to automatically install the latest versions of these tools,
> via the network, in each build? It's better to pin versions to ensure
> that upgrades are planned rather than automagical.
>
>
In v3, I altered the pip, setuptools, wheel upgrade to
exporting PIP_DISABLE_PIP_VERSION_CHECK=1
to silence the pip upgrade warning.
> Also, should we store this list of dependencies in a requirements.txt
> file somewhere? Perhaps tests/requirements.txt in meta-virtualization.
>
In v3, I changed the installation of pytest, pytest-timeout and pexpect to
be instead from a
meta-virtualization/tests/requirements.txt file (pending).
> > +
> > +# Default marker filter excludes long running / infrastructure
> dependent tests.
> > +marker_filter="${META_VIRT_PYTEST_MARKERS:-not slow and not network and
> not boot and not incus and not k3s}"
>
> Would it be better to have a positive marker, e.g. 'autobuilder', to
> mark tests that we do want to run? That would allow us to maintain the
> filtering solely in meta-virtualization instead of needing updates here
> as well when things change.
>
>
Making CI related changes in meta-virtualization is STRONGLY rejected by
the maintainer of meta-virtualization and you will have to convince the
upstream layer maintainer.
This pattern was lifted directly from the meta-virtualization README and
existing examples.
> > +
> > +# Per-suite test file selection. Uses -k/-m for fine-grained filtering
> and
> > +# keeps the CLI small for logging clarity.
> > +case "$suite" in
> > + vdkr)
> > + test_files=(
> > + "tests/test_vdkr.py"
> > + "tests/test_vdkr_registry.py"
> > + )
> > + ;;
> > + vpdmn)
> > + test_files=(
> > + "tests/test_vpdmn.py"
> > + )
> > + ;;
> > + vcontainer)
> > + # Broad vcontainer/bbclass/tooling coverage that doesn't
> require the
> > + # vdkr/vpdmn CLI harness to be running.
> > + test_files=(
> > + "tests/test_container_cross_install.py"
> > + "tests/test_container_registry_script.py"
> > + "tests/test_vcontainer_auth_config.py"
> > + "tests/test_multiarch_oci.py"
> > + "tests/test_multilayer_oci.py"
> > + )
> > + ;;
> > + *)
> > + echo "ERROR: unknown suite '$suite' (expected
> vcontainer|vdkr|vpdmn)" >&2
> > + exit 2
> > + ;;
> > +esac
> > +
> > +pytest_args=(
> > + -v
> > + --tb=short
> > + -m "$marker_filter"
> > + --vdkr-dir "$extract_dir"
> > + --junitxml="$builddir/pytest-$suite-results.xml"
> > +)
> > +
> > +# Allow tests that consume an OCI image (import/save/load) to find one.
> > +if [ -n "${TEST_OCI_IMAGE:-}" ] && [ -d "${TEST_OCI_IMAGE}" ]; then
> > + pytest_args+=(--oci-image "$TEST_OCI_IMAGE")
> > +fi
> > +
> > +# Pass architecture through when set in the environment (default is
> x86_64).
> > +if [ -n "${VDKR_ARCH:-}" ]; then
> > + pytest_args+=(--arch "$VDKR_ARCH")
> > +fi
> > +
> > +cd "$metavirtdir"
> > +# Don't let a single failing test kill the whole step - collect the
> junit
> > +# report, then surface the exit code via the junit file + exit status.
> > +set +e
> > +python3 -m pytest "${pytest_args[@]}" "${test_files[@]}"
> > +rc=$?
> > +set -e
> > +
> > +# Copy artefacts to the results dir if one was provided.
> > +if [ -n "${RESULTS_DIR:-}" ]; then
> > + mkdir -p "$RESULTS_DIR"
> > + cp -f "$builddir/pytest-$suite-results.xml" "$RESULTS_DIR/"
> 2>/dev/null || true
> > + if [ -f /tmp/pytest-vcontainer.log ]; then
> > + cp -f /tmp/pytest-vcontainer.log
> "$RESULTS_DIR/pytest-$suite.log" || true
> > + fi
> > +fi
> > +
> > +exit $rc
>
> --
> Paul Barker
>
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#4111):
> https://lists.yoctoproject.org/g/yocto-patches/message/4111
> Mute This Topic: https://lists.yoctoproject.org/mt/119603246/924729
> Group Owner: yocto-patches+owner@lists.yoctoproject.org
> Unsubscribe:
> https://lists.yoctoproject.org/g/yocto-patches/leave/13169857/924729/1023951714/xyzzy
> [ticotimo@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>
>
[-- Attachment #2: Type: text/html, Size: 19001 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* [yocto-autobuilder-helper][PATCH v2 08/10] scripts: add container registry push, auth, tagging, runtime selection
2026-06-01 23:17 [yocto-autobuilder-helper][PATCH v2 00/10] Implement 'containers' jobs tim.orling
` (6 preceding siblings ...)
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 07/10] scripts: add run-vcontainer-tests for meta-virtualization tim.orling
@ 2026-06-01 23:18 ` tim.orling
2026-06-05 16:10 ` [yocto-patches] " Richard Purdie
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 09/10] scripts/run-config: push_containers workaround tim.orling
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 10/10] config.json: add 'containers-library' build job tim.orling
9 siblings, 1 reply; 23+ messages in thread
From: tim.orling @ 2026-06-01 23:18 UTC (permalink / raw)
To: yocto-patches
From: Tim Orling <tim.orling@konsulko.com>
Add the push-containers infrastructure that drives the
post-build steps for the 'containers-library' job. After
each build step the runtime container store is harvested
and pushed to one or more registries with derived per-step
tags.
* config.json: add CONTAINER_REGISTRIES, CONTAINER_AUTH_CONFIG,
CONTAINER_RUNTIME, CONTAINER_TAG_CMDS, CONTAINER_VERSION_RECIPE
and CONTAINER_IMAGE_MAP configuration knobs.
* scripts/run-config: drive push-containers as a post-step
action. Tags are generated from recipe and distro metadata
(yocto- tag uses major.minor on snapshots and full PV on
releases) with CONTAINER_VERSION_RECIPE allowing a step to
source PV from a different recipe than the image itself.
* Registry auth is staged via .../config.json or podman
.../auth.json using CONTAINER_AUTH_CONFIG, replacing an
interactive login that could hang. CONTAINER_RUNTIME picks
between vdkr (Docker-compatible) and vpdmn (Podman) runtimes.
* Robustness: skip gracefully when no registries are configured,
fix the OCI directory path, handle memres already running,
and avoid hanging when memres has not yet come up.
AI-Generated: Claude Cowork Opus 4.7
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
config.json | 5 ++
scripts/run-config | 129 +++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 134 insertions(+)
diff --git a/config.json b/config.json
index 79a9d10..9f85a7d 100644
--- a/config.json
+++ b/config.json
@@ -43,6 +43,11 @@
"BUILDINFOVARS" : ["INHERIT += 'image-buildinfo'", "IMAGE_BUILDINFO_VARS:append = ' IMAGE_BASENAME IMAGE_NAME'"],
"WRITECONFIG" : true,
"SENDERRORS" : true,
+ "CONTAINER_RUNTIME" : "vpdmn",
+ "CONTAINER_REGISTRIES" : [],
+ "CONTAINER_TAGS" : ["latest"],
+ "CONTAINER_TAG_CMDS" : [],
+ "CONTAINER_IMAGE_MAP" : {},
"extravars" : [
"SANITY_TESTED_DISTROS = ''",
"BB_HASHSERVE = '${AUTOBUILDER_HASHSERV}'",
diff --git a/scripts/run-config b/scripts/run-config
index 0f5a26a..0fe0385 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -203,6 +203,8 @@ utils.mkdir(errordir)
errorlogs = set()
+push_containers = properties.get("push_containers", False)
+
def log_file_contents(filename, builddir, stepnum, stepname):
logfile = logname(builddir, stepnum, stepname)
with open(logfile, "a") as outf, open(filename, "r") as f:
@@ -321,6 +323,133 @@ def handle_stepnum(stepnum):
hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, sanitytargets))
bitbakecmd(args.builddir, "bitbake %s -k" % (sanitytargets), report, stepnum, args.stepname)
+ # Push container images to registries when push_containers is enabled
+ container_images = utils.getconfigdict("CONTAINER_IMAGE_MAP", ourconfig, args.target, stepnum)
+ if container_images and push_containers:
+ if jcfg:
+ addstepentry("push-containers", "Push containers", shortdesc, desc, str(container_images), str(stepnum))
+ elif args.stepname == "push-containers":
+ runtime = utils.getconfigvar("CONTAINER_RUNTIME", ourconfig, args.target, stepnum) or "vpdmn"
+ registries = utils.getconfiglist("CONTAINER_REGISTRIES", ourconfig, args.target, stepnum)
+ if not registries:
+ hp.printheader("Step %s/%s: push-containers skipped — CONTAINER_REGISTRIES is empty, no containers pushed" % (stepnum, maxsteps))
+ else:
+ static_tags = utils.getconfiglist("CONTAINER_TAGS", ourconfig, args.target, stepnum)
+ auth_config = utils.getconfigvar("CONTAINER_AUTH_CONFIG", ourconfig, args.target, stepnum)
+ if not auth_config:
+ if runtime == "vpdmn":
+ auth_config = "${HOME}/.config/containers/auth.json"
+ else:
+ auth_config = "${HOME}/.docker/config.json"
+ hp.printheader("Step %s/%s: Pushing container images %s" % (stepnum, maxsteps, list(container_images.keys())))
+ script = [
+ "set -e",
+ "test -w /dev/kvm || { echo 'ERROR: /dev/kvm is not writable, cannot push containers'; exit 1; }",
+ # Always bring up a fresh memres VM in the foreground.
+ #
+ # 'memres status' only checks that the QEMU PID in daemon.pid
+ # is alive (see daemon_is_running()/daemon_status() in
+ # meta-virtualization's vrunner.sh); it returns 0 as soon as
+ # QEMU forks, so a hung/partially-booted VM from a previous
+ # run — or a VM in mid-boot — is reported as healthy. The
+ # subsequent 'login'/'vimport'/'push' commands then hang on
+ # the unresponsive daemon socket.
+ #
+ # 'memres restart' is synchronous: it does stop+start and
+ # runs a PING/PONG readiness probe against the daemon socket
+ # (120s timeout), exiting non-zero if the VM never answers.
+ # Running it in the foreground gives us a trustworthy ready
+ # signal via its exit code, so we can drop the status-poll
+ # loop entirely.
+ #
+ # Install an EXIT trap first so we always tear the daemon
+ # down, even if bitbake -e / vimport / push fails mid-step
+ # under 'set -e'. The trap is armed before the restart so
+ # a restart failure also triggers cleanup.
+ #
+ # Registry auth is staged into the guest at VM boot via
+ # the global '--config' flag — vrunner.sh's setup_auth_share()
+ # copies $AUTH_CONFIG onto a read-only 9p share, and
+ # vdkr-init.sh / vpdmn-init.sh's install_auth_config()
+ # installs it at /root/.docker/config.json (vdkr) or
+ # /run/containers/0/auth.json (vpdmn) inside the guest.
+ # Subsequent 'push' calls use those creds directly, so no
+ # explicit 'login' step is needed. Calling 'login' would
+ # actually hang under the autobuilder (no PTY): when the
+ # memres daemon is running, vcontainer-common.sh dispatches
+ # login via '--daemon-interactive' and blocks reading the
+ # password from stdin (see login case in vcontainer-common.sh).
+ "trap '%s-$(arch) memres stop 2>/dev/null || true' EXIT" % runtime,
+ "%s-$(arch) --config %s memres restart </dev/null" % (runtime, auth_config),
+ ]
+ tag_cmds = utils.getconfiglist("CONTAINER_TAG_CMDS", ourconfig, args.target, stepnum)
+ version_recipe = utils.getconfigvar("CONTAINER_VERSION_RECIPE", ourconfig, args.target, stepnum)
+ for recipe, image in container_images.items():
+ # Extract version metadata from the recipe and distro via
+ # bitbake -e. Steps that need additional derived tags (e.g.
+ # major, major.minor) populate _EXTRA_TAGS via
+ # CONTAINER_TAG_CMDS in their step config.
+ #
+ # PV is sanitized with 'sed s/+.*//' to drop Yocto's
+ # '+git<sha>' suffix on AUTOREV/dev recipes — Docker
+ # reference format does not allow '+' in tags, and the
+ # base PV is what consumers expect.
+ #
+ # DISTRO_VERSION needs context-sensitive handling. Poky's
+ # DISTRO_VERSION resolves to '${PV}+snapshot-${METADATA_REVISION}'
+ # off a tag and just '${PV}' on a release tag. The '+' in
+ # the snapshot form is illegal in a Docker tag, but more
+ # importantly the patch level on a snapshot build (e.g.
+ # '6.0.99' between 6.0 and 6.1) is a moving target that
+ # doesn't correspond to any real release — only the
+ # major.minor line is meaningful. So:
+ # - snapshot build (DISTRO_VERSION contains '+') → tag
+ # with major.minor only, e.g. 'yocto-6.0'.
+ # - release-tag build (no '+') → tag with the full
+ # version, e.g. 'yocto-5.0.5' from the yocto-5.0.5 tag.
+ script += [
+ "_BBENV=$(bitbake -e %s 2>/dev/null) || true" % recipe,
+ "_PV=$(echo \"$_BBENV\" | awk -F'\"' '/^PV=/{ print $2; exit }' | sed 's/+.*//')",
+ "_DISTRO_CODENAME=$(echo \"$_BBENV\" | awk -F'\"' '/^DISTRO_CODENAME=/{ print $2; exit }')",
+ "_DISTRO_VERSION_RAW=$(echo \"$_BBENV\" | awk -F'\"' '/^DISTRO_VERSION=/{ print $2; exit }')",
+ "case \"$_DISTRO_VERSION_RAW\" in",
+ " *+*) _DISTRO_VERSION=$(echo \"${_DISTRO_VERSION_RAW%%+*}\" | cut -d. -f1,2) ;;",
+ " *) _DISTRO_VERSION=\"$_DISTRO_VERSION_RAW\" ;;",
+ "esac",
+ "_DEPLOY_DIR_IMAGE=$(echo \"$_BBENV\" | awk -F'\"' '/^DEPLOY_DIR_IMAGE=/{ print $2; exit }')",
+ "_EXTRA_TAGS=\"\"",
+ ]
+ if version_recipe:
+ # When the image recipe's PV is a wrapper-style
+ # placeholder (e.g. app-container-python_1.0.0.bb,
+ # whose 1.0.0 is meaningless to a downstream user),
+ # CONTAINER_VERSION_RECIPE points at the recipe whose
+ # PV is actually meaningful for the resulting tag —
+ # typically the language runtime or app being packaged
+ # (e.g. python3 -> 3.14.x). Override _PV from that
+ # recipe; image-recipe state still drives
+ # DEPLOY_DIR_IMAGE and DISTRO_* since those are
+ # environment-wide.
+ script += [
+ "_VBBENV=$(bitbake -e %s 2>/dev/null) || true" % version_recipe,
+ "_PV=$(echo \"$_VBBENV\" | awk -F'\"' '/^PV=/{ print $2; exit }' | sed 's/+.*//')",
+ ]
+ script += tag_cmds
+ script.append(
+ "_TAGS=\"%s $_PV $_DISTRO_CODENAME yocto-$_DISTRO_VERSION $_EXTRA_TAGS\"" % " ".join(static_tags)
+ )
+ for registry in registries:
+ # No per-registry 'login': credentials were staged into
+ # the guest by '--config' on 'memres restart' above.
+ script += [
+ "for _tag in $_TAGS; do",
+ " %s-$(arch) vimport ${_DEPLOY_DIR_IMAGE}/%s-latest-oci %s/%s:${_tag}" % (runtime, recipe, registry, image),
+ " %s-$(arch) push %s/%s:${_tag}" % (runtime, registry, image),
+ "done",
+ ]
+ # Tear-down is handled by the EXIT trap installed above.
+ bitbakecmd(args.builddir, "\n".join(script), report, stepnum, args.stepname)
+
# Run any extra commands specified
cmds = utils.getconfiglist("EXTRACMDS", ourconfig, args.target, stepnum)
if jcfg:
--
2.43.0
^ permalink raw reply related [flat|nested] 23+ messages in thread* Re: [yocto-patches] [yocto-autobuilder-helper][PATCH v2 08/10] scripts: add container registry push, auth, tagging, runtime selection
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 08/10] scripts: add container registry push, auth, tagging, runtime selection tim.orling
@ 2026-06-05 16:10 ` Richard Purdie
2026-06-06 3:10 ` Tim Orling
0 siblings, 1 reply; 23+ messages in thread
From: Richard Purdie @ 2026-06-05 16:10 UTC (permalink / raw)
To: yocto-patches
On Mon, 2026-06-01 at 16:18 -0700, Tim Orling via lists.yoctoproject.org wrote:
> From: Tim Orling <tim.orling@konsulko.com>
>
> Add the push-containers infrastructure that drives the
> post-build steps for the 'containers-library' job. After
> each build step the runtime container store is harvested
> and pushed to one or more registries with derived per-step
> tags.
>
> * config.json: add CONTAINER_REGISTRIES, CONTAINER_AUTH_CONFIG,
> CONTAINER_RUNTIME, CONTAINER_TAG_CMDS, CONTAINER_VERSION_RECIPE
> and CONTAINER_IMAGE_MAP configuration knobs.
> * scripts/run-config: drive push-containers as a post-step
> action. Tags are generated from recipe and distro metadata
> (yocto- tag uses major.minor on snapshots and full PV on
> releases) with CONTAINER_VERSION_RECIPE allowing a step to
> source PV from a different recipe than the image itself.
> * Registry auth is staged via .../config.json or podman
> .../auth.json using CONTAINER_AUTH_CONFIG, replacing an
> interactive login that could hang. CONTAINER_RUNTIME picks
> between vdkr (Docker-compatible) and vpdmn (Podman) runtimes.
> * Robustness: skip gracefully when no registries are configured,
> fix the OCI directory path, handle memres already running,
> and avoid hanging when memres has not yet come up.
>
> AI-Generated: Claude Cowork Opus 4.7
> Signed-off-by: Tim Orling <tim.orling@konsulko.com>
> ---
> config.json | 5 ++
> scripts/run-config | 129 +++++++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 134 insertions(+)
This is adding way too much code for a specific use case to run-config.
This needs to be in a separate script (in the same way docs builds or
other things are).
Cheers,
Richard
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [yocto-patches] [yocto-autobuilder-helper][PATCH v2 08/10] scripts: add container registry push, auth, tagging, runtime selection
2026-06-05 16:10 ` [yocto-patches] " Richard Purdie
@ 2026-06-06 3:10 ` Tim Orling
0 siblings, 0 replies; 23+ messages in thread
From: Tim Orling @ 2026-06-06 3:10 UTC (permalink / raw)
To: yocto-patches
[-- Attachment #1: Type: text/plain, Size: 2604 bytes --]
On Fri, Jun 5, 2026 at 9:11 AM Richard Purdie via lists.yoctoproject.org
<richard.purdie=linuxfoundation.org@lists.yoctoproject.org> wrote:
> On Mon, 2026-06-01 at 16:18 -0700, Tim Orling via lists.yoctoproject.org
> wrote:
> > From: Tim Orling <tim.orling@konsulko.com>
> >
> > Add the push-containers infrastructure that drives the
> > post-build steps for the 'containers-library' job. After
> > each build step the runtime container store is harvested
> > and pushed to one or more registries with derived per-step
> > tags.
> >
> > * config.json: add CONTAINER_REGISTRIES, CONTAINER_AUTH_CONFIG,
> > CONTAINER_RUNTIME, CONTAINER_TAG_CMDS, CONTAINER_VERSION_RECIPE
> > and CONTAINER_IMAGE_MAP configuration knobs.
> > * scripts/run-config: drive push-containers as a post-step
> > action. Tags are generated from recipe and distro metadata
> > (yocto- tag uses major.minor on snapshots and full PV on
> > releases) with CONTAINER_VERSION_RECIPE allowing a step to
> > source PV from a different recipe than the image itself.
> > * Registry auth is staged via .../config.json or podman
> > .../auth.json using CONTAINER_AUTH_CONFIG, replacing an
> > interactive login that could hang. CONTAINER_RUNTIME picks
> > between vdkr (Docker-compatible) and vpdmn (Podman) runtimes.
> > * Robustness: skip gracefully when no registries are configured,
> > fix the OCI directory path, handle memres already running,
> > and avoid hanging when memres has not yet come up.
> >
> > AI-Generated: Claude Cowork Opus 4.7
> > Signed-off-by: Tim Orling <tim.orling@konsulko.com>
> > ---
> > config.json | 5 ++
> > scripts/run-config | 129 +++++++++++++++++++++++++++++++++++++++++++++
> > 2 files changed, 134 insertions(+)
>
> This is adding way too much code for a specific use case to run-config.
> This needs to be in a separate script (in the same way docs builds or
> other things are).
>
>
Agreed, and I already had the sense that this "wall of text" was too much.
Broken out into a dedicated 'run-pull-containers' script in v3.
Cheers,
>
> Richard
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#4152):
> https://lists.yoctoproject.org/g/yocto-patches/message/4152
> Mute This Topic: https://lists.yoctoproject.org/mt/119603248/924729
> Group Owner: yocto-patches+owner@lists.yoctoproject.org
> Unsubscribe:
> https://lists.yoctoproject.org/g/yocto-patches/leave/13169857/924729/1023951714/xyzzy
> [ticotimo@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>
>
[-- Attachment #2: Type: text/html, Size: 4096 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* [yocto-autobuilder-helper][PATCH v2 09/10] scripts/run-config: push_containers workaround
2026-06-01 23:17 [yocto-autobuilder-helper][PATCH v2 00/10] Implement 'containers' jobs tim.orling
` (7 preceding siblings ...)
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 08/10] scripts: add container registry push, auth, tagging, runtime selection tim.orling
@ 2026-06-01 23:18 ` tim.orling
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 10/10] config.json: add 'containers-library' build job tim.orling
9 siblings, 0 replies; 23+ messages in thread
From: tim.orling @ 2026-06-01 23:18 UTC (permalink / raw)
To: yocto-patches
From: Tim Orling <tim.orling@konsulko.com>
Workaround for:
Error: reading blob sha256:<hash>: file integrity checksum failed for "<file>"
The first time the blob (e.g. bin/bash.bash) is copied, it succeeds. All subsequent
tries fail. Might possibly be an issue with sstate or a recent change to vcontainer-common
to allow for multiarch containers.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
scripts/run-config | 2 ++
1 file changed, 2 insertions(+)
diff --git a/scripts/run-config b/scripts/run-config
index 0fe0385..03d921d 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -381,6 +381,8 @@ def handle_stepnum(stepnum):
# password from stdin (see login case in vcontainer-common.sh).
"trap '%s-$(arch) memres stop 2>/dev/null || true' EXIT" % runtime,
"%s-$(arch) --config %s memres restart </dev/null" % (runtime, auth_config),
+ # workaround for 'Error: reading blob sha256:<hash>: file integrity checksum failed for "<file>"'
+ "%s-$(arch) image rm --all 2>/dev/null" % (runtime),
]
tag_cmds = utils.getconfiglist("CONTAINER_TAG_CMDS", ourconfig, args.target, stepnum)
version_recipe = utils.getconfigvar("CONTAINER_VERSION_RECIPE", ourconfig, args.target, stepnum)
--
2.43.0
^ permalink raw reply related [flat|nested] 23+ messages in thread* [yocto-autobuilder-helper][PATCH v2 10/10] config.json: add 'containers-library' build job
2026-06-01 23:17 [yocto-autobuilder-helper][PATCH v2 00/10] Implement 'containers' jobs tim.orling
` (8 preceding siblings ...)
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 09/10] scripts/run-config: push_containers workaround tim.orling
@ 2026-06-01 23:18 ` tim.orling
2026-06-02 11:02 ` [yocto-patches] " Paul Barker
9 siblings, 1 reply; 23+ messages in thread
From: tim.orling @ 2026-06-01 23:18 UTC (permalink / raw)
To: yocto-patches
From: Tim Orling <tim.orling@konsulko.com>
Add 'containers-library' build job that build container images on top
of the vcontainer-tarball SDK:
* original container images from the intial "container-cross-install"
branch of meta-virtualization
- container-base
- app-container-curl
* additional images modelled after docker.io/library/*
- app-container-python
- app-container-mosquitto
- app-container-valkey
- app-container-nginx
* Tag containers with versions based on the recipe to which they are
mapped, e.g. python:3, python:3.14, python:3.14.5.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
config.json | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 55 insertions(+)
diff --git a/config.json b/config.json
index 9f85a7d..dae55d6 100644
--- a/config.json
+++ b/config.json
@@ -1929,6 +1929,61 @@
"VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vpdmn ${BUILDDIR} ${BUILDDIR}/../meta-virtualization"
]
}
+ },
+ "containers-library": {
+ "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"],
+ "ADDLAYER" : [
+ "${BUILDDIR}/../meta-openembedded/meta-oe",
+ "${BUILDDIR}/../meta-openembedded/meta-python",
+ "${BUILDDIR}/../meta-openembedded/meta-networking",
+ "${BUILDDIR}/../meta-openembedded/meta-filesystems",
+ "${BUILDDIR}/../meta-openembedded/meta-webserver",
+ "${BUILDDIR}/../meta-virtualization"
+ ],
+ "extravars" : [
+ "DISTRO_FEATURES:append = ' virtualization vcontainer'"
+ ],
+ "CONTAINER_TAG_CMDS" : [
+ "_PV_MAJOR=$(echo $_PV | cut -d. -f1)",
+ "_PV_MAJOR_MINOR=$(echo $_PV | cut -d. -f1,2)",
+ "_EXTRA_TAGS=\"$_PV_MAJOR $_PV_MAJOR_MINOR\""
+ ],
+ "step1" : {
+ "shortname" : "Build 'base' container",
+ "BBTARGETS" : "container-base",
+ "CONTAINER_IMAGE_MAP" : {"container-base": "base"},
+ "CONTAINER_VERSION_RECIPE" : "base-files"
+ },
+ "step2" : {
+ "shortname" : "Build 'curl' container",
+ "BBTARGETS" : "app-container-curl",
+ "CONTAINER_IMAGE_MAP" : {"app-container-curl": "curl"},
+ "CONTAINER_VERSION_RECIPE" : "curl"
+ },
+ "step3" : {
+ "shortname" : "Build 'python' container",
+ "BBTARGETS" : "app-container-python",
+ "CONTAINER_IMAGE_MAP" : {"app-container-python": "python"},
+ "CONTAINER_VERSION_RECIPE" : "python3"
+ },
+ "step4" : {
+ "shortname" : "Build 'mosquitto' container",
+ "BBTARGETS" : "app-container-mosquitto",
+ "CONTAINER_IMAGE_MAP" : {"app-container-mosquitto": "mosquitto"},
+ "CONTAINER_VERSION_RECIPE" : "mosquitto"
+ },
+ "step5" : {
+ "shortname" : "Build 'valkey' container",
+ "BBTARGETS" : "app-container-valkey",
+ "CONTAINER_IMAGE_MAP" : {"app-container-valkey": "valkey"},
+ "CONTAINER_VERSION_RECIPE" : "valkey"
+ },
+ "step6" : {
+ "shortname" : "Build 'nginx' container",
+ "BBTARGETS" : "app-container-nginx",
+ "CONTAINER_IMAGE_MAP" : {"app-container-nginx": "nginx"},
+ "CONTAINER_VERSION_RECIPE" : "nginx"
+ }
}
},
"repo-defaults" : {
--
2.43.0
^ permalink raw reply related [flat|nested] 23+ messages in thread* Re: [yocto-patches] [yocto-autobuilder-helper][PATCH v2 10/10] config.json: add 'containers-library' build job
2026-06-01 23:18 ` [yocto-autobuilder-helper][PATCH v2 10/10] config.json: add 'containers-library' build job tim.orling
@ 2026-06-02 11:02 ` Paul Barker
2026-06-02 15:11 ` Tim Orling
0 siblings, 1 reply; 23+ messages in thread
From: Paul Barker @ 2026-06-02 11:02 UTC (permalink / raw)
To: yocto-patches
[-- Attachment #1: Type: text/plain, Size: 976 bytes --]
On Mon, 2026-06-01 at 16:18 -0700, Tim Orling via lists.yoctoproject.org
wrote:
> From: Tim Orling <tim.orling@konsulko.com>
>
> Add 'containers-library' build job that build container images on top
> of the vcontainer-tarball SDK:
>
> * original container images from the intial "container-cross-install"
> branch of meta-virtualization
> - container-base
> - app-container-curl
> * additional images modelled after docker.io/library/*
> - app-container-python
> - app-container-mosquitto
> - app-container-valkey
> - app-container-nginx
> * Tag containers with versions based on the recipe to which they are
> mapped, e.g. python:3, python:3.14, python:3.14.5.
These version tags may exist in docker library/hardened images, but are
they actually appropriate for the images we create?
I would expect python:master, python:blacksail, etc tags which track our
branches.
Best regards,
--
Paul Barker
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 252 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [yocto-patches] [yocto-autobuilder-helper][PATCH v2 10/10] config.json: add 'containers-library' build job
2026-06-02 11:02 ` [yocto-patches] " Paul Barker
@ 2026-06-02 15:11 ` Tim Orling
0 siblings, 0 replies; 23+ messages in thread
From: Tim Orling @ 2026-06-02 15:11 UTC (permalink / raw)
To: yocto-patches
[-- Attachment #1: Type: text/plain, Size: 2026 bytes --]
On Tue, Jun 2, 2026 at 4:02 AM Paul Barker via lists.yoctoproject.org <paul=
pbarker.dev@lists.yoctoproject.org> wrote:
> On Mon, 2026-06-01 at 16:18 -0700, Tim Orling via lists.yoctoproject.org
> wrote:
> > From: Tim Orling <tim.orling@konsulko.com>
> >
> > Add 'containers-library' build job that build container images on top
> > of the vcontainer-tarball SDK:
> >
> > * original container images from the intial "container-cross-install"
> > branch of meta-virtualization
> > - container-base
> > - app-container-curl
> > * additional images modelled after docker.io/library/*
> > - app-container-python
> > - app-container-mosquitto
> > - app-container-valkey
> > - app-container-nginx
> > * Tag containers with versions based on the recipe to which they are
> > mapped, e.g. python:3, python:3.14, python:3.14.5.
>
> These version tags may exist in docker library/hardened images, but are
> they actually appropriate for the images we create?
>
They are tagged with 'wrynose' and 'yocto-6.0' (and release builds would be
tagged with full 'yocto-6.0.n').
We need to think about container consumers, not just Yocto users. Container
consumers want to know
what version of the tool is in the container they are consuming, so we
should follow the same tagging
pattern. Tags are cheap (they are just an alias for the same hash).
>
> I would expect python:master, python:blacksail, etc tags which track our
> branches.
>
> Best regards,
>
> --
> Paul Barker
>
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#4112):
> https://lists.yoctoproject.org/g/yocto-patches/message/4112
> Mute This Topic: https://lists.yoctoproject.org/mt/119603250/924729
> Group Owner: yocto-patches+owner@lists.yoctoproject.org
> Unsubscribe:
> https://lists.yoctoproject.org/g/yocto-patches/leave/13169857/924729/1023951714/xyzzy
> [ticotimo@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>
>
[-- Attachment #2: Type: text/html, Size: 3505 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread