proposal: build pipelines in bitbake-setup

proposal: build pipelines in bitbake-setup

[Alexander Kanavin]

Hello,

bitbake-setup currently doesn't support describing build pipelines: a
sequence of commands that actually produce something useful. For a
true end-to-end solution this gap should be filled.

Here's my proposal, which probably has obvious shortcomings, that I
would like to hear about :)

0. Design goals:

- something suitable for most situations, including newcomers running
a local build, and CI doing a nightly job
- avoid custom scripts in most situations
- self-documented; everything must be described
- extensible; if there's something missing that no one thought about,
it can be added later in a way that supplements existing features
without replacing them.

1. Specification in a build configuration would look like this:

"build-targets": {
    "steps" : [
        { "name": "build-apple", "description": "Build an apple for
the purpose of making apple pie", "command": "bitbake apple-image"},
        { "name": "build-orange", "description": "Build an orange for
the purpose of making orange juice", "command": "bitbake
orange-image"},
        { "name": "test-apple", "description": "Test an apple using
testimage class", "command": "bitbake -c testimage apple-image"},
        { "name": "test-orange", "description": "Test an orange using
testimage class", "command": "bitbake -c testimage orange-image"},
        { "name": "qemu-console", "description": "Start the last built
image in qemu with console UI", "command": "runqemu kvm snapshot
nographic"},
        { "name": "publish", "description":"Copy build artefacts to a
public download server", "command":
"path/to/publish-image-executable"}
    ],
    "pipelines": [
        {"name": "publish-apple", "description": "Build, test, and
publish apple (requires publishing rights on the download server)",
"steps": ["build-apple", "test-apple", "publish"]},
        {"name": "publish-orange", "description": "Build, test, and
publish orange  (requires publishing rights on the download server)",
"steps": ["build-orange", "test-orange", "publish"]},
        {"name": "qemu-apple", "description": "Build and start apple
in qemu with console UI", "steps": ["build-apple", "qemu-console"]},
        {"name": "qemu-orange", "description": "Build and start orange
in qemu with console UI", "steps": ["build-orange", "qemu-console"]}
    ]
}

Configurations would include a list of possible pipelines:
        "configurations": [
        {
            "name": "fruitgarden",
            "description": "Alex's fruit garden",
            "bb-layers": ["meta-fruit"],
            "oe-fragments": [ "distro/fruit-garden", "machine/earth" ],
            "pipelines": ["qemu-apple", "qemu-orange",
"publish-apple", "publish-orange"]
        }

2. Nomenclature:

- "step" is a single command that is also given a short name and a
longer description. Steps are building blocks for pipelines and can't
be individually executed.
- "pipeline" is a sequence of steps, that is also given a short name
and a longer description. Pipelines can mix and match defined steps as
they please.
- each configuration includes a list of possible targets, and once it
is set up, any of them can be executed (see below for the UI).

3. User Interface to pipelines

Once there is a setup, there are two ways to execute a pipeline:

a) For each pipeline there is a shell script that can be run directly:

/path/to/bitbake-builds/fruitgarden/qemu-apple
/path/to/bitbake-builds/fruitgarden/publish-apple
etc.

qemu-apple script would look something like:

#!/bin/sh
# Build and start apple in qemu with console UI
. init-build-env
# Build an apple for the purpose of making apple pie
bitbake apple-image
# Start the last built image in qemu with console UI
runqemu kvm snapshot nographic

e.g. assembled from pipeline steps and their descriptions.

b) Interactive UI :'bitbake-setup build'

This would present an interactive UI similar to 'bitbake-setup init', e.g:
$ bitbake-setup build
Available build pipelines:
1. qemu-apple  Build and start apple in qemu with console UI
2. qemu-orange Build and start orange in qemu with console UI
... etc
Please choose one of the above: 1

The following commands will be executed:
# Build an apple for the purpose of making apple pie
bitbake apple-image
# Start the last built image in qemu with console UI
runqemu kvm snapshot nographic

Proceed? [y/N]
etc.

4. Implementation

This should actually be fairly straightforward. There's no need to add
new features to bitbake (e.g. 'standard target definitions' in layers
was considered at some point, but after more consideration I think it
can be postponed or perhaps altogether avoided), or modify oe-core.

Alex

Re: [Openembedded-architecture] proposal: build pipelines in bitbake-setup

[Trevor Gamblin]

On 2026-05-05 06:17, Alexander Kanavin via lists.openembedded.org wrote:
> Hello,
>
> bitbake-setup currently doesn't support describing build pipelines: a
> sequence of commands that actually produce something useful. For a
> true end-to-end solution this gap should be filled.
>
> Here's my proposal, which probably has obvious shortcomings, that I
> would like to hear about :)

Hi,

Thanks for submitting this proposal. I will try to summarize some of my 
thoughts from the public call and otherwise in-line.

>
> 0. Design goals:
>
> - something suitable for most situations, including newcomers running
> a local build, and CI doing a nightly job
> - avoid custom scripts in most situations
> - self-documented; everything must be described
> - extensible; if there's something missing that no one thought about,
> it can be added later in a way that supplements existing features
> without replacing them.
>
> 1. Specification in a build configuration would look like this:
>
> "build-targets": {
>      "steps" : [
>          { "name": "build-apple", "description": "Build an apple for
> the purpose of making apple pie", "command": "bitbake apple-image"},
>          { "name": "build-orange", "description": "Build an orange for
> the purpose of making orange juice", "command": "bitbake
> orange-image"},
>          { "name": "test-apple", "description": "Test an apple using
> testimage class", "command": "bitbake -c testimage apple-image"},
>          { "name": "test-orange", "description": "Test an orange using
> testimage class", "command": "bitbake -c testimage orange-image"},
>          { "name": "qemu-console", "description": "Start the last built
> image in qemu with console UI", "command": "runqemu kvm snapshot
> nographic"},
>          { "name": "publish", "description":"Copy build artefacts to a
> public download server", "command":
> "path/to/publish-image-executable"}
>      ],
>      "pipelines": [
>          {"name": "publish-apple", "description": "Build, test, and
> publish apple (requires publishing rights on the download server)",
> "steps": ["build-apple", "test-apple", "publish"]},
>          {"name": "publish-orange", "description": "Build, test, and
> publish orange  (requires publishing rights on the download server)",
> "steps": ["build-orange", "test-orange", "publish"]},
>          {"name": "qemu-apple", "description": "Build and start apple
> in qemu with console UI", "steps": ["build-apple", "qemu-console"]},
>          {"name": "qemu-orange", "description": "Build and start orange
> in qemu with console UI", "steps": ["build-orange", "qemu-console"]}
>      ]
> }
>
> Configurations would include a list of possible pipelines:
>          "configurations": [
>          {
>              "name": "fruitgarden",
>              "description": "Alex's fruit garden",
>              "bb-layers": ["meta-fruit"],
>              "oe-fragments": [ "distro/fruit-garden", "machine/earth" ],
>              "pipelines": ["qemu-apple", "qemu-orange",
> "publish-apple", "publish-orange"]
>          }
I think it's worth considering for the nightly-job-in-CI case whether 
people will want to use bitbake-setup to cover that entire scope (i.e., 
build-test-publish). There are a lot of existing CI systems out there 
(Buildbot, Tekton, GitHub/Gitea/Forgejo Actions, GitLab CI, etc.) where 
you already define pipelines as a series of stages/steps, and each one 
is generally intended to cover either a single command or a handful of 
closely-associated ones. I am not sure that users would want to invoke a 
bitbake-setup pipeline as a single step in these systems, as that would 
potentially obfuscate failures and configuration issues just by having 
too many things happening per step. However, I can see it being useful 
to people who want a fire-and-forget reproducer for typical build/test 
configurations.

As a side note, it feels like stepping outside of the "build setup" 
space and including things like runtime testing, artifact publishing, 
etc. feels like a clash with bitbake-setup's name.
>
> 2. Nomenclature:
>
> - "step" is a single command that is also given a short name and a
> longer description. Steps are building blocks for pipelines and can't
> be individually executed.
Why not? Why not have the ability to run something like 'bitbake-setup 
run-step <step_name'? That would actually be an advantage this feature 
could provide over many other CI systems, as it's usually difficult or 
impossible to rerun individual steps (you have to re-run a whole job or 
pipeline). I can see this being very useful for example when you need to 
run 'bitbake -c testimage' in a loop many times to try and catch 
intermittent failures.
> - "pipeline" is a sequence of steps, that is also given a short name
> and a longer description. Pipelines can mix and match defined steps as
> they please.
> - each configuration includes a list of possible targets, and once it
> is set up, any of them can be executed (see below for the UI).
>
> 3. User Interface to pipelines
>
> Once there is a setup, there are two ways to execute a pipeline:
>
> a) For each pipeline there is a shell script that can be run directly:
>
> /path/to/bitbake-builds/fruitgarden/qemu-apple
> /path/to/bitbake-builds/fruitgarden/publish-apple
> etc.
>
> qemu-apple script would look something like:
>
> #!/bin/sh
> # Build and start apple in qemu with console UI
> . init-build-env
> # Build an apple for the purpose of making apple pie
> bitbake apple-image
> # Start the last built image in qemu with console UI
> runqemu kvm snapshot nographic
>
> e.g. assembled from pipeline steps and their descriptions.
Similarly here - maybe a 'bitbake-setup run-pipeline <pipeline_name>' 
would be more effective?
>
> b) Interactive UI :'bitbake-setup build'
>
> This would present an interactive UI similar to 'bitbake-setup init', e.g:
> $ bitbake-setup build
> Available build pipelines:
> 1. qemu-apple  Build and start apple in qemu with console UI
> 2. qemu-orange Build and start orange in qemu with console UI
> ... etc
> Please choose one of the above: 1
>
> The following commands will be executed:
> # Build an apple for the purpose of making apple pie
> bitbake apple-image
> # Start the last built image in qemu with console UI
> runqemu kvm snapshot nographic
>
> Proceed? [y/N]
> etc.
>
> 4. Implementation
>
> This should actually be fairly straightforward. There's no need to add
> new features to bitbake (e.g. 'standard target definitions' in layers
> was considered at some point, but after more consideration I think it
> can be postponed or perhaps altogether avoided), or modify oe-core.
My other main concern about this is around maintenance - depending on 
how it is ultimately implemented, it could end up looking like a 
miniature CI/CD system for Yocto builds - that is certainly how I find 
myself seeing it, and I've written my responses in that mindset. I can 
see the utility in having the feature available more generally to reduce 
time spent reproducing a common build/test case, but I wonder if it'd 
get used often enough under its currently proposed form.

Trevor
>
> Alex
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#2324): https://lists.openembedded.org/g/openembedded-architecture/message/2324
> Mute This Topic: https://lists.openembedded.org/mt/119158252/7611679
> Group Owner: openembedded-architecture+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub [tgamblin@baylibre.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [Openembedded-architecture] proposal: build pipelines in bitbake-setup

[Alexander Kanavin]

On Tue, 5 May 2026 at 18:17, Trevor Gamblin <tgamblin@baylibre.com> wrote:
> > 2. Nomenclature:
> >
> > - "step" is a single command that is also given a short name and a
> > longer description. Steps are building blocks for pipelines and can't
> > be individually executed.
> Why not? Why not have the ability to run something like 'bitbake-setup
> run-step <step_name'? That would actually be an advantage this feature
> could provide over many other CI systems, as it's usually difficult or
> impossible to rerun individual steps (you have to re-run a whole job or
> pipeline). I can see this being very useful for example when you need to
> run 'bitbake -c testimage' in a loop many times to try and catch
> intermittent failures.

The reason I suggested that steps can't be run individually is that I
wanted to keep the UI simple, and couldn't think of use cases for
individual steps. One has to start somewhere, my starting point is
baffled yocto newcomers, and poorly written custom build scripts :)

Anyway! This was very useful to read, and I agree with what you say.
Here's how the UI could look like:

1. Interactively choose a pipeline and run it:
$ bitbake-setup build

2. Run a pipeline given on command line:
$ bitbake-setup build <pipeline-name>

3. List pipeline steps
$ bitbake-setup build --list-steps <pipeline-name>
(this can be used to avoid hardcoding step names and their sequence
into CI-specific definitions, e.g. CI would first list the steps, then
iterate over them, avoiding the 'giant single step' problem - this of
course requires support for creating steps 'on the fly', which I think
at least buildbot can do)

4. Run a single step:
$ bitbake-setup build <pipeline-name> <step-name>

5. Run a pipeline, starting at a particular step:
$ bitbake-setup build <pipeline-name> --start-at <step-name>

6. Run a pipeline, stopping after a particular step:
$ bitbake-setup build <pipeline-name> --stop-after <step-name>

Etc. How's this looking?

Alex

Re: [Openembedded-architecture] proposal: build pipelines in bitbake-setup

[Daiane Angolini]

On Tue, May 5, 2026 at 3:42 PM Alexander Kanavin via
lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org>
wrote:
>

> 6. Run a pipeline, stopping after a particular step:
> $ bitbake-setup build <pipeline-name> --stop-after <step-name>
>
> Etc. How's this looking?
>

I like the term `steps`. I would like to add that this should be
optional, not required. One can still decide to avoid creating a
pipeline.

Regarding a setup tool for builds, why not use `bitbake
<pipeline-name>` instead of `bitbake-setup build <pipeline-name>`?

Daiane
> Alex
>

Re: [Openembedded-architecture] proposal: build pipelines in bitbake-setup

[Alexander Kanavin]

On Wed, 6 May 2026 at 12:23, Daiane Angolini
<daiane.angolini@foundries.io> wrote:
> > 6. Run a pipeline, stopping after a particular step:
> > $ bitbake-setup build <pipeline-name> --stop-after <step-name>
> >
> > Etc. How's this looking?
> >
>
> I like the term `steps`. I would like to add that this should be
> optional, not required. One can still decide to avoid creating a
> pipeline.

Yes of course.

> Regarding a setup tool for builds, why not use `bitbake
> <pipeline-name>` instead of `bitbake-setup build <pipeline-name>`?

bitbake operates on recipes using a well-known command line interface.
We can't bolt on something entirely different.

That said, pipeline functions don't have to be in bitbake-setup, they
can be in a separate tool (Trevor mentioned that 'bitbake-setup' as a
name doesn't fit that functionality very well). I just don't have any
ideas off the top of my head. 'bitbake-pipeline' ?

Alex

Re: [Openembedded-architecture] proposal: build pipelines in bitbake-setup

[Trevor Gamblin]

On 2026-05-06 08:43, Alexander Kanavin via lists.openembedded.org wrote:
> On Wed, 6 May 2026 at 12:23, Daiane Angolini
> <daiane.angolini@foundries.io> wrote:
>>> 6. Run a pipeline, stopping after a particular step:
>>> $ bitbake-setup build <pipeline-name> --stop-after <step-name>
>>>
>>> Etc. How's this looking?
>>>
>> I like the term `steps`. I would like to add that this should be
>> optional, not required. One can still decide to avoid creating a
>> pipeline.
> Yes of course.
>
>> Regarding a setup tool for builds, why not use `bitbake
>> <pipeline-name>` instead of `bitbake-setup build <pipeline-name>`?
> bitbake operates on recipes using a well-known command line interface.
> We can't bolt on something entirely different.
>
> That said, pipeline functions don't have to be in bitbake-setup, they
> can be in a separate tool (Trevor mentioned that 'bitbake-setup' as a
> name doesn't fit that functionality very well). I just don't have any
> ideas off the top of my head. 'bitbake-pipeline' ?
I do like this version of the proposal better. Having read the responses 
from the two of you, I think 'bitbake-setup run' might be a reasonable 
compromise for running steps and pipelines. What do you think?
>
> Alex
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#2328): https://lists.openembedded.org/g/openembedded-architecture/message/2328
> Mute This Topic: https://lists.openembedded.org/mt/119158252/7611679
> Group Owner: openembedded-architecture+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub [tgamblin@baylibre.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [Openembedded-architecture] proposal: build pipelines in bitbake-setup

[Daiane Angolini]

On Wed, May 6, 2026 at 10:29 AM Trevor Gamblin <tgamblin@baylibre.com> wrote:
>
> I do like this version of the proposal better. Having read the responses
> from the two of you, I think 'bitbake-setup run' might be a reasonable
> compromise for running steps and pipelines. What do you think?

To be honest, I think that we download bitbake to install bitbake
(using bitbake-setup) so we can use another tool (bitbake-pipeline) to
build bitbake. Although I understand why, it looks confusing.

Why not:
$ bitbake-setup --non-interactive <all your choices>
$ ./<pipeline-name> all/step1/from-step15

Can I pip install bitbake-pipeline? For example.

I understand `bitbake <pipeline-name>` is not what we want because
some of the steps will not be bitbake dependable (such as a deploy
which is only cp a image to a storage), that's why this pipeline isn't
strictly a Yocto recipe (even though a Yocto recipe is just a list of
steps). So the pipeline is a list of steps that may or may not depend
on bitbake, which is kinda bash script for example?

I like that the bitbake configuration template may provide a set of
usable steps, but I don't understand what this other tool does (other
than wrapping the build script)

Daiane
> >
> > Alex
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#2328): https://lists.openembedded.org/g/openembedded-architecture/message/2328
> > Mute This Topic: https://lists.openembedded.org/mt/119158252/7611679
> > Group Owner: openembedded-architecture+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub [tgamblin@baylibre.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >

Re: [Openembedded-architecture] proposal: build pipelines in bitbake-setup

[Alexander Kanavin]

On Wed, 6 May 2026 at 16:03, Daiane Angolini
<daiane.angolini@foundries.io> wrote:
> > I do like this version of the proposal better. Having read the responses
> > from the two of you, I think 'bitbake-setup run' might be a reasonable
> > compromise for running steps and pipelines. What do you think?
>
> I like that the bitbake configuration template may provide a set of
> usable steps, but I don't understand what this other tool does (other
> than wrapping the build script)

I was confused by your response which quotes Trevor's suggestion and
then starts talking about a separate tool. But if you are actually
responding to me, then the arguments make sense.

I think we can drop the 'separate tool' idea, and use 'bitbake-setup
run'. If someone wants straightforward shell scripts that just run all
pipeline steps one after another that can be done too, but rich
functionality should be inside the 'run' command.

Alex

Re: [bitbake-devel] proposal: build pipelines in bitbake-setup

[Ross Burton]

On 5 May 2026, at 11:17, Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote:
> 
> bitbake-setup currently doesn't support describing build pipelines: a
> sequence of commands that actually produce something useful. For a
> true end-to-end solution this gap should be filled.

To be honest I think I disagree with you here.  There’s already a load of “build pipeline” tools from sh scripts to Makefiles to the multitude of  task runners written in every language under the run (invoke, rake, etc).

Does the world need another one?

Ross

Re: [bitbake-devel] proposal: build pipelines in bitbake-setup

[Alexander Kanavin]

On Thu, 7 May 2026 at 14:37, Ross Burton <Ross.Burton@arm.com> wrote:

> To be honest I think I disagree with you here.  There’s already a load of “build pipeline” tools from sh scripts to Makefiles to the multitude of  task runners written in every language under the run (invoke, rake, etc).
>
> Does the world need another one?

The idea is that bitbake-setup should help users do something useful
with a build they set up, by providing a standard UI. You'd type
$ bitbake-setup run
and it will give you a selection of what's possible, and then proceed
to do it. It's the equivalent of 'kas build', just more flexible :)

If someone has written their pipeline with make or a shell script,
then bitbake-setup can simply run that (e.g. a pipeline containing a
single step), but I think there's also value in having 'native'
support for multi-step pipelines. They're simple (e.g. executable
commands are static strings that can't be parametrized), and if that's
not flexible enough, there's indeed plenty of industrial-grade tools.

Alex

Re: [Openembedded-architecture] [bitbake-devel] proposal: build pipelines in bitbake-setup

[Richard Purdie]

On Thu, 2026-05-07 at 15:13 +0200, Alexander Kanavin via lists.openembedded.org wrote:
> On Thu, 7 May 2026 at 14:37, Ross Burton <Ross.Burton@arm.com> wrote:
> 
> > To be honest I think I disagree with you here.  There’s already a
> > load of “build pipeline” tools from sh scripts to Makefiles to the
> > multitude of  task runners written in every language under the run
> > (invoke, rake, etc).
> > 
> > Does the world need another one?
> 
> The idea is that bitbake-setup should help users do something useful
> with a build they set up, by providing a standard UI. You'd type
> $ bitbake-setup run
> and it will give you a selection of what's possible, and then proceed
> to do it. It's the equivalent of 'kas build', just more flexible :)

The trouble is there are multiple things a user may want to do and
which one is a hard choice to make. If you have some narrow layer with
a specific image and application, that is easy. If you have a general
layer like meta-virtualization, it is unclear which target(s) a given
user may want. nodistro/poky face the same challenge, the thing the
user might want "depends".

> If someone has written their pipeline with make or a shell script,
> then bitbake-setup can simply run that (e.g. a pipeline containing a
> single step), but I think there's also value in having 'native'
> support for multi-step pipelines. They're simple (e.g. executable
> commands are static strings that can't be parametrized), and if that's
> not flexible enough, there's indeed plenty of industrial-grade tools.

I believe if we implement and do something, we need to do it well and
it does need to have a specific target/focus and use. If this
functionality is too hard or inappropriate to use for too many of the
application areas, you have to question whether we should be doing it
at all...

Cheers,

Richard

Re: [Openembedded-architecture] [bitbake-devel] proposal: build pipelines in bitbake-setup

[Alexander Kanavin]

On Thu, 7 May 2026 at 15:20, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> The trouble is there are multiple things a user may want to do and
> which one is a hard choice to make. If you have some narrow layer with
> a specific image and application, that is easy. If you have a general
> layer like meta-virtualization, it is unclear which target(s) a given
> user may want. nodistro/poky face the same challenge, the thing the
> user might want "depends".

For such general layers two things come to mind:

1. An example/demo/starting point for one or a handful of most common
use cases. E.g. poky/nodistro would showcase two or three most common
core- images.

2. Running tests for locally made changes, before submitting a patch.
For oe-core/poky that's perhaps unrealistic, but I can imagine
something domain-specific like meta-virtualization could define steps
for this: $ bitbake-setup run test

Alex