Documentation around adding custom fetchers to layer

Documentation around adding custom fetchers to layer

[seth.raymond]

[-- Attachment #1: Type: text/plain, Size: 2032 bytes --]

Hello,

I am attempting to write a couple of custom fetcher that pull secrets from our password manager (using its CLI). To start, I wrote Python classes that inherit from bb.fetch2.FetchMethod and registered them in poky/bitbake/lib/bb/fetch2/__init__.py by importing them and appending them to methods. I confirmed that the fetchers themselves are working as expected and the secrets are getting pulled during the recipe build process.

Of course, I don't (necessarily) want to contribute these fetchers upstream to poky. I would like to add them to one of our company's layers. Our directory structure is laid out like:

top-dir/
|--- build/
|--- layers/
|          |--- poky/
|          |--- meta-matrixspace/
|          |                |--- meta-mtrx-core/  (this is the layer I want to add my fetchers to)
|          |                |--- meta-mtrx-other-layer/
|          |                |--- meta-mtrx-yet-another-layer/

We're based off of scarthgap. I've put my fetchers into top-dir/layers/meta-matrixspace/meta-mtrx-core/lib/bb/fetch2/op.py and updated top-dir/layers/meta-matrixspace/meta-mtrx-core/conf/layer.conf to include

PYTHONPATH .= ":${LAYERDIR}/lib"

When building recipes that use the fetchers, I get parsing errors saying that no fetchers could be found to support the URL:

bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception NoMethodError: Could not find a fetcher which supports the URL: 'optemplate://aes_key.tpl;output=aes.key'
The variable dependency chain for the failure is: fetcher_hashes_dummyfunc[vardepvalue]

I'm not seeing any documentation on how to add a custom fetcher to a layer, and the fetchers in poky don't provide enough information on how to register these fetchers. Is this functionality supported, or will we have to fork poky and keep it up-to-date to add a new fetcher?

Thanks,
Seth

[-- Attachment #2: Type: text/html, Size: 2638 bytes --]

Re: [yocto] Documentation around adding custom fetchers to layer

[Ross Burton]

On 16 Sep 2025, at 21:04, seth.raymond via lists.yoctoproject.org <seth.raymond=matrixspace.com@lists.yoctoproject.org> wrote:
> 
> Hello,
>   I am attempting to write a couple of custom fetcher that pull secrets from our password manager (using its CLI). To start, I wrote Python classes that inherit from bb.fetch2.FetchMethod and registered them in poky/bitbake/lib/bb/fetch2/__init__.py by importing them and appending them to methods. I confirmed that the fetchers themselves are working as expected and the secrets are getting pulled during the recipe build process.
>   Of course, I don't (necessarily) want to contribute these fetchers upstream to poky. I would like to add them to one of our company's layers. Our directory structure is laid out like:
...
> PYTHONPATH .= ":${LAYERDIR}/lib"
>   When building recipes that use the fetchers, I get parsing errors saying that no fetchers could be found to support the URL:
> 
> bb.data_smart.ExpansionError: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception NoMethodError: Could not find a fetcher which supports the URL: 'optemplate://aes_key.tpl;output=aes.key'
> The variable dependency chain for the failure is: fetcher_hashes_dummyfunc[vardepvalue]
> 
> I'm not seeing any documentation on how to add a custom fetcher to a layer, and the fetchers in poky don't provide enough information on how to register these fetchers. Is this functionality supported, or will we have to fork poky and keep it up-to-date to add a new fetcher?

This is a bitbake issue, so I’ve copied in the bitbake list.

There’s no standard way to add out-of-bitbake fetchers, as you say you need to add the fetcher instance to bb.fetch2.methods.

I guess you might be able to write a do_fetch prefunc that creates an instance of your fetcher and adds it to the list, but I would not be surprised if there were more places where that would be needed.

python use_my_fetcher() {
  bb.fetch2.methods.append(MyFetcher())
}
do_fetch[prefuncs] += “use_my_fetcher”

Ross

Re: [yocto] Documentation around adding custom fetchers to layer

[Alexander Kanavin]

On Wed, 17 Sept 2025 at 11:15, Ross Burton via lists.yoctoproject.org
<ross.burton=arm.com@lists.yoctoproject.org> wrote:
> > I'm not seeing any documentation on how to add a custom fetcher to a layer, and the fetchers in poky don't provide enough information on how to register these fetchers. Is this functionality supported, or will we have to fork poky and keep it up-to-date to add a new fetcher?
>
> This is a bitbake issue, so I’ve copied in the bitbake list.
>
> There’s no standard way to add out-of-bitbake fetchers, as you say you need to add the fetcher instance to bb.fetch2.methods.
>
> I guess you might be able to write a do_fetch prefunc that creates an instance of your fetcher and adds it to the list, but I would not be surprised if there were more places where that would be needed.
>
> python use_my_fetcher() {
>   bb.fetch2.methods.append(MyFetcher())
> }
> do_fetch[prefuncs] += “use_my_fetcher”

To be honest, I was going to say that custom fetchers are not
supported. Doing that would fragment the ecosystem and make people
invent all kinds of private magic in their builds, and subvert the
promises that standard fetchers make.

Perhaps it would help to step back and look at what this customer
fecher does. Can we see its source? Or an extended description of how
the password manager is used, what data it takes and produces, and how
that data is used in actual fetching? Then we'd think of how to do
this with regular fetchers.

Alex

Re: [yocto] Documentation around adding custom fetchers to layer

[seth.raymond]

[-- Attachment #1.1: Type: text/plain, Size: 1765 bytes --]

Hi Alex,

Sure, I can share the source. It grabs secrets from our password manager (1Password) and injects them into the build system. We currently have two use cases for this:
1. We use AES keys for encrypting/decrypting our SWU payloads. Those key/iv pairs are held in our password manager, so we need to take one of them out at build-time and hand it to the swupdate bbclass to encrypt the payload. We do the same thing for a signing .pem key
2. We want to install an SSH key into our OS so that we can SSH into our devices. 1Password also acts as an SSH agent, so it makes life seamless when we store the keys in there.

My source is attached. The use case in a recipe would be:

SRC_URI = " \
op://MyVault/SSHKey/public%20key ;output=ssh_key.pub \
optemplate://aes_key.tpl;output=aes_key \
"

The output would be:
1. The content of the MyVault > SSHKey > public key dumped into a file called ssh_key.pub in the $WORKDIR
2. Depending on the template ( https://developer.1password.com/docs/cli/secrets-template-syntax/ ), ${WORKDIR}/aes_key would be filled in with referenced secrets

Someone at my last company wrote a custom fetcher to fetch conan packages, but I unfortunately wasn't super plugged in to what he was doing. I vaguely remember he did something with a bbclass and might have used something similar to the proposed do_fetch[prefuncs] method.

We currently have a hacky workaround right now where we have a *-init-build-env script that pulls the secrets from the 1Password vault(s) and put them in the correct place. But persisting secrets on-disk, unencrypted is non-ideal. My preference would be for us to only grab the secrets as we need them and then clean them up from build/ after they've been consumed.

Cheers,
Seth

[-- Attachment #1.2: Type: text/html, Size: 2260 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: op.py --]
[-- Type: text/x-python; name="op.py", Size: 4612 bytes --]

import os
from pathlib import Path

import bb
from bb.fetch2 import FetchError, FetchMethod, runfetchcmd


class OpItem(FetchMethod):
    """Fetch method that runs `op read` on a 1Password CLI item path. URI should be in the format of:
    op://<Vault>/<Item>/[<Section>/]<Field>

    Note that all spaces in the string should be replaced with %20 so that bitbake can parse it properly.
    """

    def supports(self, ud, d):
        # Recognize URLs like: op://Vault/Item/[Section/]Field
        return ud.type in ["op"]

    def recommends_checksum(self, urldata):
        return False

    def supports_checksum(self, urldata):
        return False

    def urldata_init(self, ud, d):
        # The 1Password path is given as the 'path' part of the URL.
        # For example: op://Vault/Item/[Section/]Field
        ud.path = ud.url.split("://", 1)[1].split(";")[0]
        if not ud.path:
            raise FetchError("Invalid op:// URI")
        ud.path = f'op://{ud.path.replace("%20", " ")}'

        # Optional query parameter: output=<filename>
        ud.output = ud.parm.get("output") or os.path.basename(ud.path) + ".rendered"

        # Store local destination path
        ud.localfile = ud.output

    def download(self, ud, d):
        """
        Run `op read` to dump the contents of a field into a file.
        """
        dl_dir = d.getVar("DL_DIR")
        dest = os.path.join(dl_dir, ud.output)
        os.environ["OP_SERVICE_ACCOUNT_TOKEN"] = d.getVar("OP_SERVICE_ACCOUNT_TOKEN")

        cmd = f"/usr/local/bin/op read --out-file {dest} '{ud.path}'"
        runfetchcmd(cmd, d)

        if not os.path.exists(dest):
            raise FetchError("op read did not produce an output file")

        # The localfile is the file BitBake will checksum and stage.
        ud.localpath = dest

    def unpack(self, ud, destdir, d):
        """
        Standard fetcher API: copy the injected file into the unpacked source dir.
        """
        bb.utils.copyfile(ud.localpath, os.path.join(destdir, ud.output))

    def checkstatus(self, fetch, urldata, d):
        return True


class OpTemplate(FetchMethod):
    """Fetch method that runs `op inject` on a 1Password CLI injectable template."""

    def supports(self, ud, d):
        # Recognize URLs like: optemplate://path/to/template
        return ud.type in ["optemplate"]

    def recommends_checksum(self, urldata):
        return False

    def supports_checksum(self, urldata):
        return False

    def urldata_init(self, ud, d):
        # The template file is given as the 'path' part of the URL.
        # For example: optemplate:///absolute/path/to/template
        ud.template = ud.url.split("://", 1)[1].split(";")[0]
        if not ud.template:
            raise FetchError("Missing template path in optemplate:// URL")

        # Optional query parameter: output=<filename>
        ud.output = ud.parm.get("output") or os.path.basename(ud.template) + ".rendered"

        # Store local destination path
        ud.localfile = ud.output

    def download(self, ud, d):
        """
        Run `op inject` to resolve secrets in the template and
        produce the formatted output file.
        """
        template_path = ""
        if not os.path.exists(ud.template):
            filespath = d.getVar("FILESPATH")
            for location in filter(None, [Path(p) for p in filespath.split(":")]):
                for glob_hit in location.glob(ud.template):
                    if glob_hit.exists():
                        template_path = location / ud.template
                        bb.note(f"Found template file at {template_path}")
                        break
            if not template_path:
                raise FetchError(f"Template file not found: {ud.template}")

        dl_dir = d.getVar("DL_DIR")
        dest = os.path.join(dl_dir, ud.output)
        os.environ["OP_SERVICE_ACCOUNT_TOKEN"] = d.getVar("OP_SERVICE_ACCOUNT_TOKEN")

        bb.note(f"Injecting 1Password template: {ud.template} -> {dest}")
        cmd = f"/usr/local/bin/op inject -f -i {template_path} -o {dest}"
        runfetchcmd(cmd, d)

        if not os.path.exists(dest):
            raise FetchError("op inject did not produce output file")

        # The localfile is the file BitBake will checksum and stage.
        ud.localpath = dest

    def unpack(self, ud, destdir, d):
        """
        Standard fetcher API: copy the injected file into the unpacked source dir.
        """
        bb.utils.copyfile(ud.localpath, os.path.join(destdir, ud.output))

    def checkstatus(self, fetch, urldata, d):
        return True

Re: [yocto] Documentation around adding custom fetchers to layer

[Alexander Kanavin]

On Wed, 17 Sept 2025 at 16:45, seth.raymond via Lists.Yoctoproject.Org
<seth.raymond=matrixspace.com@lists.yoctoproject.org> wrote:
> Sure, I can share the source. It grabs secrets from our password manager (1Password) and injects them into the build system. We currently have two use cases for this:
> 1. We use AES keys for encrypting/decrypting our SWU payloads. Those key/iv pairs are held in our password manager, so we need to take one of them out at build-time and hand it to the swupdate bbclass to encrypt the payload. We do the same thing for a signing .pem key
> 2. We want to install an SSH key into our OS so that we can SSH into our devices. 1Password also acts as an SSH agent, so it makes life seamless when we store the keys in there.
>
> My source is attached. The use case in a recipe would be:
>
> SRC_URI = " \
>     op://MyVault/SSHKey/public%20key;output=ssh_key.pub \
>     optemplate://aes_key.tpl;output=aes_key \
> "
>
> The output would be:
> 1. The content of the MyVault > SSHKey > public key dumped into a file called ssh_key.pub in the $WORKDIR
> 2. Depending on the template (https://developer.1password.com/docs/cli/secrets-template-syntax/), ${WORKDIR}/aes_key would be filled in with referenced secrets

But this doesn't require a fetcher implementation. You can do it in a
task, put the task into a class, and write recipes that inherit the
class. The meat of the fetcher is this:

        cmd = f"/usr/local/bin/op read --out-file {dest} '{ud.path}'"

and this can simply write directly into ${S} from a task.

Alex

Re: [yocto] Documentation around adding custom fetchers to layer

[seth.raymond]

[-- Attachment #1: Type: text/plain, Size: 130 bytes --]

Yep, I'm pivoting to just make this a bbclass instead. It feels less elegant, but it does the job. Thanks for the tips!

-Seth

[-- Attachment #2: Type: text/html, Size: 169 bytes --]

Re: [yocto] Documentation around adding custom fetchers to layer

[Chuck Wolber]

On Wed Sep 17, 2025 at 2:53 PM UTC, seth.raymond via lists.yoctoproject.org wrote:
>
> Sure, I can share the source. It grabs secrets from our password manager
> (1Password) and injects them into the build system. We currently have two use
> cases for this:

It is probably worth noting that the 1Password v8 CLI and SDK both require a
round trip to the cloud even if you run the desktop 1P client locally (which
maintains a local copy of your vaults). The 1P CLI and SDK do _some_ caching,
but they still require a round trip for every call. My testing has shown that
this round-trip adds about a second when taking full advantage of what they
cache locally.

For a tool meant to help with automation, this falls into the "egregious"
category of oversights as far as I am concerned. After 12 years of 1P use, I am
actively exploring other options.

..Ch:W..

Re: [bitbake-devel] [yocto] Documentation around adding custom fetchers to layer

[Richard Purdie]

On Wed, 2025-09-17 at 11:19 +0200, Alexander Kanavin via lists.openembedded.org wrote:
> On Wed, 17 Sept 2025 at 11:15, Ross Burton via lists.yoctoproject.org
> <ross.burton=arm.com@lists.yoctoproject.org> wrote:
> > > I'm not seeing any documentation on how to add a custom fetcher to a layer, and the fetchers in poky don't provide enough information on how to register these fetchers. Is this functionality supported, or will we have to fork poky and keep it up-to-date to add a new fetcher?
> > 
> > This is a bitbake issue, so I’ve copied in the bitbake list.
> > 
> > There’s no standard way to add out-of-bitbake fetchers, as you say you need to add the fetcher instance to bb.fetch2.methods.
> > 
> > I guess you might be able to write a do_fetch prefunc that creates an instance of your fetcher and adds it to the list, but I would not be surprised if there were more places where that would be needed.
> > 
> > python use_my_fetcher() {
> >   bb.fetch2.methods.append(MyFetcher())
> > }
> > do_fetch[prefuncs] += “use_my_fetcher”
> 
> To be honest, I was going to say that custom fetchers are not
> supported. Doing that would fragment the ecosystem and make people
> invent all kinds of private magic in their builds, and subvert the
> promises that standard fetchers make.
> 
> Perhaps it would help to step back and look at what this customer
> fecher does. Can we see its source? Or an extended description of how
> the password manager is used, what data it takes and produces, and how
> that data is used in actual fetching? Then we'd think of how to do
> this with regular fetchers.

I'm struggling with time/bandwidth so I'll just mention what I can
remember without digging too much.

There have been cases of successfully injecting extra fetchers from the
metadata, particularly if you do it from the global class code,
probably in an event handler. I think some of the early npm/node/crate
code did this (I don't remember which) before we merged things into
bitbake itself.

There was an open bug for making it easier to define new fetchers from
the metadata and we did get most things working. Sadly I can't spot the
bug. The one area that was problematic was AUTOREV since that happened
before you could inject the fetcher in the class code in some cases. We
never did work out an API which would allow AUTOREV to work and allow a
general API.

Cheers,

Richard