[PATCH 1/2] archiver.bbclass: Improve the indentation of the documentation

[PATCH 1/2] archiver.bbclass: Improve the indentation of the documentation

[Peter Kjellerstedt]

This makes the class documentation's indentation consistent.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 meta/classes/archiver.bbclass | 26 +++++++++++++-------------
 1 file changed, 13 insertions(+), 13 deletions(-)

diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
index 3124b7040d..f38409774d 100644
--- a/meta/classes/archiver.bbclass
+++ b/meta/classes/archiver.bbclass
@@ -19,12 +19,12 @@
 #  - The recipe (.bb and .inc): ARCHIVER_MODE[recipe] = "1"
 #  - Filter the license, the recipe whose license in
 #    COPYLEFT_LICENSE_INCLUDE will be included, and in
-#    COPYLEFT_LICENSE_EXCLUDE will be excluded.
-#    COPYLEFT_LICENSE_INCLUDE = 'GPL* LGPL*'
-#    COPYLEFT_LICENSE_EXCLUDE = 'CLOSED Proprietary'
-# - The recipe type that will be archived:
-#    COPYLEFT_RECIPE_TYPES = 'target'
-# - The source mirror mode:
+#    COPYLEFT_LICENSE_EXCLUDE will be excluded:
+#     COPYLEFT_LICENSE_INCLUDE = 'GPL* LGPL*'
+#     COPYLEFT_LICENSE_EXCLUDE = 'CLOSED Proprietary'
+#  - The recipe type that will be archived:
+#     COPYLEFT_RECIPE_TYPES = 'target'
+#  - The source mirror mode:
 #    ARCHIVER_MODE[mirror] = "split" (default): Sources are split into
 #    per-recipe directories in a similar way to other archiver modes.
 #    Post-processing may be required to produce a single mirror directory.
@@ -33,13 +33,13 @@
 #    ARCHIVER_MODE[mirror] = "combined": All sources are placed into a single
 #    directory suitable for direct use as a mirror. Duplicate sources are
 #    ignored.
-# - Source mirror exclusions:
-#     ARCHIVER_MIRROR_EXCLUDE is a list of prefixes to exclude from the mirror.
-#     This may be used for sources which you are already publishing yourself
-#     (e.g. if the URI starts with 'https://mysite.com/' and your mirror is
-#     going to be published to the same site). It may also be used to exclude
-#     local files (with the prefix 'file://') if these will be provided as part
-#     of an archive of the layers themselves.
+#  - Source mirror exclusions:
+#    ARCHIVER_MIRROR_EXCLUDE is a list of prefixes to exclude from the mirror.
+#    This may be used for sources which you are already publishing yourself
+#    (e.g. if the URI starts with 'https://mysite.com/' and your mirror is
+#    going to be published to the same site). It may also be used to exclude
+#    local files (with the prefix 'file://') if these will be provided as part
+#    of an archive of the layers themselves.
 #
 
 # Create archive for all the recipe types

[PATCH 2/2] archiver.bbclass: Add ARCHIVER_MIRROR_INCLUDE

[Peter Kjellerstedt]

When using the archiver's mirror mode, ARCHIVER_MIRROR_INCLUDE contains
a list of URI prefixes to always include in the mirror. This may be
useful if recipes that are included in the mirror (due to their
licenses) depend on some recipe that otherwise would not be included in
the mirror, and that recipe uses files that are normally only available
on a private server.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 meta/classes/archiver.bbclass            | 35 ++++++++++++++++++++++--
 meta/lib/oeqa/selftest/cases/archiver.py | 24 ++++++++++++++++
 2 files changed, 57 insertions(+), 2 deletions(-)

diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
index f38409774d..db8b60916a 100644
--- a/meta/classes/archiver.bbclass
+++ b/meta/classes/archiver.bbclass
@@ -33,6 +33,12 @@
 #    ARCHIVER_MODE[mirror] = "combined": All sources are placed into a single
 #    directory suitable for direct use as a mirror. Duplicate sources are
 #    ignored.
+#  - Source mirror inclusions:
+#    ARCHIVER_MIRROR_INCLUDE is a list of URI prefixes to always include in
+#    the mirror. This may be useful if recipes that are included in the mirror
+#    (due to their licenses) depend on some recipe that otherwise would not be
+#    included in the mirror, and that recipe uses files that are normally only
+#    available on a private server.
 #  - Source mirror exclusions:
 #    ARCHIVER_MIRROR_EXCLUDE is a list of prefixes to exclude from the mirror.
 #    This may be used for sources which you are already publishing yourself
@@ -72,8 +78,9 @@ do_ar_original[dirs] = "${ARCHIVER_OUTDIR} ${ARCHIVER_WORKDIR}"
 # This is a convenience for the shell script to use it
 
 def include_package(d, pn):
-
-    included, reason = copyleft_should_include(d)
+    included, reason = archiver_should_include(d)
+    if not included:
+        included, reason = copyleft_should_include(d)
     if not included:
         bb.debug(1, 'archiver: %s is excluded: %s' % (pn, reason))
         return False
@@ -94,6 +101,29 @@ def include_package(d, pn):
 
     return True
 
+def archiver_should_include(d):
+    ar_src = d.getVarFlag('ARCHIVER_MODE', 'src')
+    if ar_src == "mirror":
+        src_uri = (d.getVar('SRC_URI') or '').split()
+        if len(src_uri) == 0:
+            return False, None
+
+        mirror_inclusions = (d.getVar('ARCHIVER_MIRROR_INCLUDE') or '').split()
+
+        def is_included(url):
+            for prefix in mirror_inclusions:
+                if url.startswith(prefix):
+                    return True
+            return False
+
+        fetcher = bb.fetch2.Fetch(src_uri, d)
+
+        for ud in fetcher.expanded_urldata():
+            if is_included(ud.url):
+                return True, "URL matches ARCHIVER_MIRROR_INCLUDE"
+
+    return False, None
+
 python () {
     pn = d.getVar('PN')
     assume_provided = (d.getVar("ASSUME_PROVIDED") or "").split()
@@ -154,6 +184,7 @@ python () {
 do_ar_prepare[vardeps] += " \
     ARCHIVER_MODE \
     ARCHIVER_MIRROR_EXCLUDE \
+    ARCHIVER_MIRROR_INCLUDE \
     COPYLEFT_LICENSE_EXCLUDE \
     COPYLEFT_LICENSE_INCLUDE \
     COPYLEFT_PN_EXCLUDE \
diff --git a/meta/lib/oeqa/selftest/cases/archiver.py b/meta/lib/oeqa/selftest/cases/archiver.py
index 8c5aa75e9b..78118db448 100644
--- a/meta/lib/oeqa/selftest/cases/archiver.py
+++ b/meta/lib/oeqa/selftest/cases/archiver.py
@@ -249,6 +249,30 @@ class Archiver(OESelftestTestCase):
         archive_path = os.path.join(glob_result[0], target_file_name)
         self.assertFalse(os.path.exists(archive_path), 'Failed to exclude archive file %s' % (target_file_name))
 
+    def test_archiver_mode_mirror_include(self):
+        """
+        Test that `ARCHIVER_MIRROR_INCLUDE` causes a source URL to be included
+        in the mirror even when the recipe would otherwise be excluded by the
+        copyleft license filter.
+        """
+
+        target = 'selftest-ed'
+        target_file_name = 'ed-1.21.1.tar.lz'
+
+        features = 'INHERIT += "archiver"\n'
+        features += 'ARCHIVER_MODE[src] = "mirror"\n'
+        features += 'ARCHIVER_MODE[mirror] = "combined"\n'
+        features += 'BB_GENERATE_MIRROR_TARBALLS = "1"\n'
+        features += 'COPYLEFT_LICENSE_INCLUDE = "CLOSED"\n'
+        features += 'ARCHIVER_MIRROR_INCLUDE = "${GNU_MIRROR}"\n'
+        self.write_config(features)
+
+        bitbake('-c deploy_archives %s' % (target))
+
+        bb_vars = get_bb_vars(['DEPLOY_DIR_SRC'])
+        target_path = os.path.join(bb_vars['DEPLOY_DIR_SRC'], 'mirror', target_file_name)
+        self.assertTrue(os.path.exists(target_path), 'Missing archive file %s' % (target_file_name))
+
     def test_archiver_mode_mirror_combined(self):
         """
         Test that the archiver works with `ARCHIVER_MODE[src] = "mirror"`

Re: [OE-core] [PATCH 2/2] archiver.bbclass: Add ARCHIVER_MIRROR_INCLUDE

[Richard Purdie]

On Tue, 2026-08-25 at 21:07 +0200, Peter Kjellerstedt via lists.openembedded.org wrote:
> When using the archiver's mirror mode, ARCHIVER_MIRROR_INCLUDE contains
> a list of URI prefixes to always include in the mirror. This may be
> useful if recipes that are included in the mirror (due to their
> licenses) depend on some recipe that otherwise would not be included in
> the mirror, and that recipe uses files that are normally only available
> on a private server.
> 
> Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> ---
>  meta/classes/archiver.bbclass            | 35 ++++++++++++++++++++++--
>  meta/lib/oeqa/selftest/cases/archiver.py | 24 ++++++++++++++++
>  2 files changed, 57 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
> index f38409774d..db8b60916a 100644
> --- a/meta/classes/archiver.bbclass
> +++ b/meta/classes/archiver.bbclass
> @@ -33,6 +33,12 @@
>  #    ARCHIVER_MODE[mirror] = "combined": All sources are placed into a single
>  #    directory suitable for direct use as a mirror. Duplicate sources are
>  #    ignored.
> +#  - Source mirror inclusions:
> +#    ARCHIVER_MIRROR_INCLUDE is a list of URI prefixes to always include in
> +#    the mirror. This may be useful if recipes that are included in the mirror
> +#    (due to their licenses) depend on some recipe that otherwise would not be
> +#    included in the mirror, and that recipe uses files that are normally only
> +#    available on a private server.

This wording is a rather ambiguous to me :/.

It reads like things would normally be included but this is a list of
extra special things to include. That isn't how it works.

The "depend on some recipe" suggests it somehow also follows
dependencies into other recipes, which as far as I know, it doesn't. I
don't see where a private server fits into things.

>  #  - Source mirror exclusions:
>  #    ARCHIVER_MIRROR_EXCLUDE is a list of prefixes to exclude from the mirror.
>  #    This may be used for sources which you are already publishing yourself
> @@ -72,8 +78,9 @@ do_ar_original[dirs] = "${ARCHIVER_OUTDIR} ${ARCHIVER_WORKDIR}"
>  # This is a convenience for the shell script to use it
>  
>  def include_package(d, pn):
> -
> -    included, reason = copyleft_should_include(d)
> +    included, reason = archiver_should_include(d)
> +    if not included:
> +        included, reason = copyleft_should_include(d)

It overrides copyleft includes? That should probably be documented.
It's relationship with exclude is also not documented. Which wins? I
suspect exclude does, but should it?

>      if not included:
>          bb.debug(1, 'archiver: %s is excluded: %s' % (pn, reason))
>          return False
> @@ -94,6 +101,29 @@ def include_package(d, pn):
>  
>      return True
>  
> +def archiver_should_include(d):
> +    ar_src = d.getVarFlag('ARCHIVER_MODE', 'src')
> +    if ar_src == "mirror":
> +        src_uri = (d.getVar('SRC_URI') or '').split()
> +        if len(src_uri) == 0:
> +            return False, None
> +
> +        mirror_inclusions = (d.getVar('ARCHIVER_MIRROR_INCLUDE') or '').split()
> +
> +        def is_included(url):
> +            for prefix in mirror_inclusions:
> +                if url.startswith(prefix):
> +                    return True
> +            return False
> +
> +        fetcher = bb.fetch2.Fetch(src_uri, d)
> +
> +        for ud in fetcher.expanded_urldata():
> +            if is_included(ud.url):
> +                return True, "URL matches ARCHIVER_MIRROR_INCLUDE"
> +
> +    return False, None
> +
>  python () {
>      pn = d.getVar('PN')
>      assume_provided = (d.getVar("ASSUME_PROVIDED") or "").split()
> @@ -154,6 +184,7 @@ python () {
>  do_ar_prepare[vardeps] += " \
>      ARCHIVER_MODE \
>      ARCHIVER_MIRROR_EXCLUDE \
> +    ARCHIVER_MIRROR_INCLUDE \
>      COPYLEFT_LICENSE_EXCLUDE \
>      COPYLEFT_LICENSE_INCLUDE \
>      COPYLEFT_PN_EXCLUDE \

I get the feeling this was discussed before and one of the concerns was
that the include and exclude code are in two different places, working
in two quite different ways? That isn't going to help maintainability
of this code, which already has a poor reputation amongst developers.

Cheers,

Richard

RE: [OE-core] [PATCH 2/2] archiver.bbclass: Add ARCHIVER_MIRROR_INCLUDE

[Peter Kjellerstedt]

> -----Original Message-----
> From: Richard Purdie <richard.purdie@linuxfoundation.org>
> Sent: den 26 augusti 2026 09:13
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH 2/2] archiver.bbclass: Add ARCHIVER_MIRROR_INCLUDE
> 
> On Tue, 2026-08-25 at 21:07 +0200, Peter Kjellerstedt via
> lists.openembedded.org wrote:
> > When using the archiver's mirror mode, ARCHIVER_MIRROR_INCLUDE contains
> > a list of URI prefixes to always include in the mirror. This may be
> > useful if recipes that are included in the mirror (due to their
> > licenses) depend on some recipe that otherwise would not be included in
> > the mirror, and that recipe uses files that are normally only available
> > on a private server.
> >
> > Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> > ---
> >  meta/classes/archiver.bbclass            | 35 ++++++++++++++++++++++--
> >  meta/lib/oeqa/selftest/cases/archiver.py | 24 ++++++++++++++++
> >  2 files changed, 57 insertions(+), 2 deletions(-)
> >
> > diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
> > index f38409774d..db8b60916a 100644
> > --- a/meta/classes/archiver.bbclass
> > +++ b/meta/classes/archiver.bbclass
> > @@ -33,6 +33,12 @@
> >  #    ARCHIVER_MODE[mirror] = "combined": All sources are placed into a single
> >  #    directory suitable for direct use as a mirror. Duplicate sources are
> >  #    ignored.
> > +#  - Source mirror inclusions:
> > +#    ARCHIVER_MIRROR_INCLUDE is a list of URI prefixes to always include in
> > +#    the mirror. This may be useful if recipes that are included in the mirror
> > +#    (due to their licenses) depend on some recipe that otherwise would not be
> > +#    included in the mirror, and that recipe uses files that are normally only
> > +#    available on a private server.
> 
> This wording is a rather ambiguous to me :/.
> 
> It reads like things would normally be included but this is a list of
> extra special things to include. That isn't how it works.
> 
> The "depend on some recipe" suggests it somehow also follows
> dependencies into other recipes, which as far as I know, it doesn't. I
> don't see where a private server fits into things.

What I'm trying to convey is the situation where you have a recipe that 
is copyleft and thus it will be included in the archives that are 
produced. If that recipe in turn depends on some recipe that does not 
use a copyleft license, that second recipe will by default not be 
included in the saved archives. For such recipes that fetch their code 
from the Internet, this is not a problem as Bitbake will just fetch the 
sources as normal. However, if the second recipe is a private one and it 
has its files on an internal server, those files must be included in the 
produced artifacts or the copyleft recipe will not be able to be built.

> 
> >  #  - Source mirror exclusions:
> >  #    ARCHIVER_MIRROR_EXCLUDE is a list of prefixes to exclude from the mirror.
> >  #    This may be used for sources which you are already publishing yourself
> > @@ -72,8 +78,9 @@ do_ar_original[dirs] = "${ARCHIVER_OUTDIR} ${ARCHIVER_WORKDIR}"
> >  # This is a convenience for the shell script to use it
> >
> >  def include_package(d, pn):
> > -
> > -    included, reason = copyleft_should_include(d)
> > +    included, reason = archiver_should_include(d)
> > +    if not included:
> > +        included, reason = copyleft_should_include(d)
> 
> It overrides copyleft includes? That should probably be documented.

You are correct, this should be documented better. The assumption 
from my side is that if you specify ARCHIVER_MIRROR_INCLUDE, then 
you know what you are doing and the sources that match the prefix(es) 
should be included regardless of their licenses.

> It's relationship with exclude is also not documented. Which wins? I
> suspect exclude does, but should it?

Yes, ARCHIVER_MIRROR_EXCLUDE will win over ARCHIVER_MIRROR_INCLUDE. 
I think it is a bit of a corner case though. It does not make much 
sense to specify the same prefixes for both. But it should be 
documented.

> >      if not included:
> >          bb.debug(1, 'archiver: %s is excluded: %s' % (pn, reason))
> >          return False
> > @@ -94,6 +101,29 @@ def include_package(d, pn):
> >
> >      return True
> >
> > +def archiver_should_include(d):
> > +    ar_src = d.getVarFlag('ARCHIVER_MODE', 'src')
> > +    if ar_src == "mirror":
> > +        src_uri = (d.getVar('SRC_URI') or '').split()
> > +        if len(src_uri) == 0:
> > +            return False, None
> > +
> > +        mirror_inclusions = (d.getVar('ARCHIVER_MIRROR_INCLUDE') or '').split()
> > +
> > +        def is_included(url):
> > +            for prefix in mirror_inclusions:
> > +                if url.startswith(prefix):
> > +                    return True
> > +            return False
> > +
> > +        fetcher = bb.fetch2.Fetch(src_uri, d)
> > +
> > +        for ud in fetcher.expanded_urldata():
> > +            if is_included(ud.url):
> > +                return True, "URL matches ARCHIVER_MIRROR_INCLUDE"
> > +
> > +    return False, None
> > +
> >  python () {
> >      pn = d.getVar('PN')
> >      assume_provided = (d.getVar("ASSUME_PROVIDED") or "").split()
> > @@ -154,6 +184,7 @@ python () {
> >  do_ar_prepare[vardeps] += " \
> >      ARCHIVER_MODE \
> >      ARCHIVER_MIRROR_EXCLUDE \
> > +    ARCHIVER_MIRROR_INCLUDE \
> >      COPYLEFT_LICENSE_EXCLUDE \
> >      COPYLEFT_LICENSE_INCLUDE \
> >      COPYLEFT_PN_EXCLUDE \
> 
> I get the feeling this was discussed before and one of the concerns was
> that the include and exclude code are in two different places, working
> in two quite different ways? That isn't going to help maintainability
> of this code, which already has a poor reputation amongst developers.

If it was discussed before, then that is news to me. 

I mimicked ARCHIVER_MIRROR_EXCLUDE when I implemented 
ARCHIVER_MIRROR_INCLUDE (since we use mirror mode this was enough). 
However, ARCHIVER_MIRROR_INCLUDE could probably be useful regardless 
of which mode the archiver is in. So maybe it should be 
ARCHIVER_INCLUDE instead? And then the fact that the implementations 
are in totally different parts of the code aren't that much of a 
concern anymore?

> 
> Cheers,
> 
> Richard

//Peter