[PATCH 1/2] archiver.bbclass: Improve the indentation of the documentation

[PATCH 1/2] archiver.bbclass: Improve the indentation of the documentation

[Peter Kjellerstedt]

This makes the class documentation's indentation consistent.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 meta/classes/archiver.bbclass | 26 +++++++++++++-------------
 1 file changed, 13 insertions(+), 13 deletions(-)

diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
index 3124b7040d..f38409774d 100644
--- a/meta/classes/archiver.bbclass
+++ b/meta/classes/archiver.bbclass
@@ -19,12 +19,12 @@
 #  - The recipe (.bb and .inc): ARCHIVER_MODE[recipe] = "1"
 #  - Filter the license, the recipe whose license in
 #    COPYLEFT_LICENSE_INCLUDE will be included, and in
-#    COPYLEFT_LICENSE_EXCLUDE will be excluded.
-#    COPYLEFT_LICENSE_INCLUDE = 'GPL* LGPL*'
-#    COPYLEFT_LICENSE_EXCLUDE = 'CLOSED Proprietary'
-# - The recipe type that will be archived:
-#    COPYLEFT_RECIPE_TYPES = 'target'
-# - The source mirror mode:
+#    COPYLEFT_LICENSE_EXCLUDE will be excluded:
+#     COPYLEFT_LICENSE_INCLUDE = 'GPL* LGPL*'
+#     COPYLEFT_LICENSE_EXCLUDE = 'CLOSED Proprietary'
+#  - The recipe type that will be archived:
+#     COPYLEFT_RECIPE_TYPES = 'target'
+#  - The source mirror mode:
 #    ARCHIVER_MODE[mirror] = "split" (default): Sources are split into
 #    per-recipe directories in a similar way to other archiver modes.
 #    Post-processing may be required to produce a single mirror directory.
@@ -33,13 +33,13 @@
 #    ARCHIVER_MODE[mirror] = "combined": All sources are placed into a single
 #    directory suitable for direct use as a mirror. Duplicate sources are
 #    ignored.
-# - Source mirror exclusions:
-#     ARCHIVER_MIRROR_EXCLUDE is a list of prefixes to exclude from the mirror.
-#     This may be used for sources which you are already publishing yourself
-#     (e.g. if the URI starts with 'https://mysite.com/' and your mirror is
-#     going to be published to the same site). It may also be used to exclude
-#     local files (with the prefix 'file://') if these will be provided as part
-#     of an archive of the layers themselves.
+#  - Source mirror exclusions:
+#    ARCHIVER_MIRROR_EXCLUDE is a list of prefixes to exclude from the mirror.
+#    This may be used for sources which you are already publishing yourself
+#    (e.g. if the URI starts with 'https://mysite.com/' and your mirror is
+#    going to be published to the same site). It may also be used to exclude
+#    local files (with the prefix 'file://') if these will be provided as part
+#    of an archive of the layers themselves.
 #
 
 # Create archive for all the recipe types

[PATCH 2/2] archiver.bbclass: Add ARCHIVER_MIRROR_INCLUDE

[Peter Kjellerstedt]

When using the archiver's mirror mode, ARCHIVER_MIRROR_INCLUDE contains
a list of URI prefixes to always include in the mirror. This may be
useful if recipes that are included in the mirror (due to their
licenses) depend on some recipe that otherwise would not be included in
the mirror, and that recipe uses files that are normally only available
on a private server.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 meta/classes/archiver.bbclass            | 35 ++++++++++++++++++++++--
 meta/lib/oeqa/selftest/cases/archiver.py | 24 ++++++++++++++++
 2 files changed, 57 insertions(+), 2 deletions(-)

diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
index f38409774d..db8b60916a 100644
--- a/meta/classes/archiver.bbclass
+++ b/meta/classes/archiver.bbclass
@@ -33,6 +33,12 @@
 #    ARCHIVER_MODE[mirror] = "combined": All sources are placed into a single
 #    directory suitable for direct use as a mirror. Duplicate sources are
 #    ignored.
+#  - Source mirror inclusions:
+#    ARCHIVER_MIRROR_INCLUDE is a list of URI prefixes to always include in
+#    the mirror. This may be useful if recipes that are included in the mirror
+#    (due to their licenses) depend on some recipe that otherwise would not be
+#    included in the mirror, and that recipe uses files that are normally only
+#    available on a private server.
 #  - Source mirror exclusions:
 #    ARCHIVER_MIRROR_EXCLUDE is a list of prefixes to exclude from the mirror.
 #    This may be used for sources which you are already publishing yourself
@@ -72,8 +78,9 @@ do_ar_original[dirs] = "${ARCHIVER_OUTDIR} ${ARCHIVER_WORKDIR}"
 # This is a convenience for the shell script to use it
 
 def include_package(d, pn):
-
-    included, reason = copyleft_should_include(d)
+    included, reason = archiver_should_include(d)
+    if not included:
+        included, reason = copyleft_should_include(d)
     if not included:
         bb.debug(1, 'archiver: %s is excluded: %s' % (pn, reason))
         return False
@@ -94,6 +101,29 @@ def include_package(d, pn):
 
     return True
 
+def archiver_should_include(d):
+    ar_src = d.getVarFlag('ARCHIVER_MODE', 'src')
+    if ar_src == "mirror":
+        src_uri = (d.getVar('SRC_URI') or '').split()
+        if len(src_uri) == 0:
+            return False, None
+
+        mirror_inclusions = (d.getVar('ARCHIVER_MIRROR_INCLUDE') or '').split()
+
+        def is_included(url):
+            for prefix in mirror_inclusions:
+                if url.startswith(prefix):
+                    return True
+            return False
+
+        fetcher = bb.fetch2.Fetch(src_uri, d)
+
+        for ud in fetcher.expanded_urldata():
+            if is_included(ud.url):
+                return True, "URL matches ARCHIVER_MIRROR_INCLUDE"
+
+    return False, None
+
 python () {
     pn = d.getVar('PN')
     assume_provided = (d.getVar("ASSUME_PROVIDED") or "").split()
@@ -154,6 +184,7 @@ python () {
 do_ar_prepare[vardeps] += " \
     ARCHIVER_MODE \
     ARCHIVER_MIRROR_EXCLUDE \
+    ARCHIVER_MIRROR_INCLUDE \
     COPYLEFT_LICENSE_EXCLUDE \
     COPYLEFT_LICENSE_INCLUDE \
     COPYLEFT_PN_EXCLUDE \
diff --git a/meta/lib/oeqa/selftest/cases/archiver.py b/meta/lib/oeqa/selftest/cases/archiver.py
index 8c5aa75e9b..78118db448 100644
--- a/meta/lib/oeqa/selftest/cases/archiver.py
+++ b/meta/lib/oeqa/selftest/cases/archiver.py
@@ -249,6 +249,30 @@ class Archiver(OESelftestTestCase):
         archive_path = os.path.join(glob_result[0], target_file_name)
         self.assertFalse(os.path.exists(archive_path), 'Failed to exclude archive file %s' % (target_file_name))
 
+    def test_archiver_mode_mirror_include(self):
+        """
+        Test that `ARCHIVER_MIRROR_INCLUDE` causes a source URL to be included
+        in the mirror even when the recipe would otherwise be excluded by the
+        copyleft license filter.
+        """
+
+        target = 'selftest-ed'
+        target_file_name = 'ed-1.21.1.tar.lz'
+
+        features = 'INHERIT += "archiver"\n'
+        features += 'ARCHIVER_MODE[src] = "mirror"\n'
+        features += 'ARCHIVER_MODE[mirror] = "combined"\n'
+        features += 'BB_GENERATE_MIRROR_TARBALLS = "1"\n'
+        features += 'COPYLEFT_LICENSE_INCLUDE = "CLOSED"\n'
+        features += 'ARCHIVER_MIRROR_INCLUDE = "${GNU_MIRROR}"\n'
+        self.write_config(features)
+
+        bitbake('-c deploy_archives %s' % (target))
+
+        bb_vars = get_bb_vars(['DEPLOY_DIR_SRC'])
+        target_path = os.path.join(bb_vars['DEPLOY_DIR_SRC'], 'mirror', target_file_name)
+        self.assertTrue(os.path.exists(target_path), 'Missing archive file %s' % (target_file_name))
+
     def test_archiver_mode_mirror_combined(self):
         """
         Test that the archiver works with `ARCHIVER_MODE[src] = "mirror"`

Re: [OE-core] [PATCH 2/2] archiver.bbclass: Add ARCHIVER_MIRROR_INCLUDE

[Richard Purdie]

On Tue, 2026-08-25 at 21:07 +0200, Peter Kjellerstedt via lists.openembedded.org wrote:
> When using the archiver's mirror mode, ARCHIVER_MIRROR_INCLUDE contains
> a list of URI prefixes to always include in the mirror. This may be
> useful if recipes that are included in the mirror (due to their
> licenses) depend on some recipe that otherwise would not be included in
> the mirror, and that recipe uses files that are normally only available
> on a private server.
> 
> Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> ---
>  meta/classes/archiver.bbclass            | 35 ++++++++++++++++++++++--
>  meta/lib/oeqa/selftest/cases/archiver.py | 24 ++++++++++++++++
>  2 files changed, 57 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
> index f38409774d..db8b60916a 100644
> --- a/meta/classes/archiver.bbclass
> +++ b/meta/classes/archiver.bbclass
> @@ -33,6 +33,12 @@
>  #    ARCHIVER_MODE[mirror] = "combined": All sources are placed into a single
>  #    directory suitable for direct use as a mirror. Duplicate sources are
>  #    ignored.
> +#  - Source mirror inclusions:
> +#    ARCHIVER_MIRROR_INCLUDE is a list of URI prefixes to always include in
> +#    the mirror. This may be useful if recipes that are included in the mirror
> +#    (due to their licenses) depend on some recipe that otherwise would not be
> +#    included in the mirror, and that recipe uses files that are normally only
> +#    available on a private server.

This wording is a rather ambiguous to me :/.

It reads like things would normally be included but this is a list of
extra special things to include. That isn't how it works.

The "depend on some recipe" suggests it somehow also follows
dependencies into other recipes, which as far as I know, it doesn't. I
don't see where a private server fits into things.

>  #  - Source mirror exclusions:
>  #    ARCHIVER_MIRROR_EXCLUDE is a list of prefixes to exclude from the mirror.
>  #    This may be used for sources which you are already publishing yourself
> @@ -72,8 +78,9 @@ do_ar_original[dirs] = "${ARCHIVER_OUTDIR} ${ARCHIVER_WORKDIR}"
>  # This is a convenience for the shell script to use it
>  
>  def include_package(d, pn):
> -
> -    included, reason = copyleft_should_include(d)
> +    included, reason = archiver_should_include(d)
> +    if not included:
> +        included, reason = copyleft_should_include(d)

It overrides copyleft includes? That should probably be documented.
It's relationship with exclude is also not documented. Which wins? I
suspect exclude does, but should it?

>      if not included:
>          bb.debug(1, 'archiver: %s is excluded: %s' % (pn, reason))
>          return False
> @@ -94,6 +101,29 @@ def include_package(d, pn):
>  
>      return True
>  
> +def archiver_should_include(d):
> +    ar_src = d.getVarFlag('ARCHIVER_MODE', 'src')
> +    if ar_src == "mirror":
> +        src_uri = (d.getVar('SRC_URI') or '').split()
> +        if len(src_uri) == 0:
> +            return False, None
> +
> +        mirror_inclusions = (d.getVar('ARCHIVER_MIRROR_INCLUDE') or '').split()
> +
> +        def is_included(url):
> +            for prefix in mirror_inclusions:
> +                if url.startswith(prefix):
> +                    return True
> +            return False
> +
> +        fetcher = bb.fetch2.Fetch(src_uri, d)
> +
> +        for ud in fetcher.expanded_urldata():
> +            if is_included(ud.url):
> +                return True, "URL matches ARCHIVER_MIRROR_INCLUDE"
> +
> +    return False, None
> +
>  python () {
>      pn = d.getVar('PN')
>      assume_provided = (d.getVar("ASSUME_PROVIDED") or "").split()
> @@ -154,6 +184,7 @@ python () {
>  do_ar_prepare[vardeps] += " \
>      ARCHIVER_MODE \
>      ARCHIVER_MIRROR_EXCLUDE \
> +    ARCHIVER_MIRROR_INCLUDE \
>      COPYLEFT_LICENSE_EXCLUDE \
>      COPYLEFT_LICENSE_INCLUDE \
>      COPYLEFT_PN_EXCLUDE \

I get the feeling this was discussed before and one of the concerns was
that the include and exclude code are in two different places, working
in two quite different ways? That isn't going to help maintainability
of this code, which already has a poor reputation amongst developers.

Cheers,

Richard