Yocto Project Status 1 July. 2025 (WW26)

Yocto Project Status 1 July. 2025 (WW26)

[Stephen K Jolley]

[-- Attachment #1: Type: text/plain, Size: 6903 bytes --]

Current Dev Position: YP 5.3 M2

Next Deadline: YP 5.3 M2 Build date 2025-07-21

Next Team Meetings:

   -

   Bug Triage meeting - Thursday July 3rd 7:30 am PDT (
   https://zoom.us/j/454367603?pwd=ZGxoa2ZXL3FkM3Y0bFd5aVpHVVZ6dz09)
   -

   Weekly Project Engineering Sync  - Tuesday July 1st 8 am PDT (
   https://zoom.us/j/990892712?pwd=cHU1MjhoM2x6ck81bkcrYjRrcmJsUT09)
   <https://zoom.us/j/990892712>
   -

   Yocto Project Patch Review - Thursday July 3rd 10am GMT and Monday July
   7th 9am PDT (
   https://zoom.us/j/97762397148?pwd=1xk0iC9hp9SjEonaTaONwTb6iWry4Eb.1
   <https://zoom.us/j/97762397148?pwd=1xkiC9hp9SjEonaTaONwTb6iWry4Eb.1>)


Project data dashboard: https://dashboard.yoctoproject.org/

Key Status/Updates:

   -

   YP 5.3 M1 is due to be released. One issue was found which needs to be
   fixed during M2: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15915

Help in debugging and fixing this would be very welcome.

   -

   YP 4.0.28 is in QA.
   -

   Patches continue to merge into master for upgrades and fixes.
   -

   There were git tag improvements meaning we can now start adding these to
   SRC_URIs in all cases as shallow clone tarballs now account for them.
   -

   The clang build time and size increases continue to cause significant
   concern.
   -

   Patches to improve gomod support in devtool merged.
   -

   The main mirroring issues left are around “downloadfilename” support in
   urls which likely needs to be rewritten.
   -

   OE_SHARED_UMASK was added to control the umask used for files created in
   the shared DL_DIR and SSTATE_DIR directories.
   -

   For M2, the following changes have been mentioned as possibilities:
   -

      bitbake-setup
      -

      wic split from OE-Core
      -

      RISC-V tune changes (part 2)
      -

      switch classextend to use [filter] support from bitbake


Ways to contribute:

   -

   As people are likely aware, the project has a number of components which
   are either unmaintained, or have people with little to no time trying to
   keep them alive. These components include: devtool, toaster, wic, oeqa,
   autobuilder, CROPs containers, pseudo and more. Many have open bugs. Help
   is welcome in trying to better look after these components!
   -

   There is an issue open upstream with the openssl project related to
   making path relocation of openssl easier (as used in our buildtools tarball
   and SDK). We’d love assistance in moving this forward and getting some kind
   of upstream feature merged to make this easier:
   https://github.com/openssl/openssl/pull/19260
   -

   There are bugs identified as possible for newcomers to the project:
   https://dashboard.yoctoproject.org/bugtriage/#newcomer-container
   -

   There are bugs that are currently unassigned for YP 5.3. See:
   https://wiki.yoctoproject.org/wiki/Bug_Triage#Medium+_5.3_Unassigned_Enhancements/Bugs
   -

   We’d welcome new maintainers for recipes in OE-Core. Please see the list
   at:
   http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/conf/distro/include/maintainers.inc
   and discuss with the existing maintainer, or ask on the OE-Core mailing
   list. We will likely move a chunk of these to “Unassigned” soon to help
   facilitate this.
   -

   Help is very much welcome in trying to resolve our autobuilder
   intermittent issues. You can see the list of failures we’re continuing to
   see by searching for the “AB-INT” tag in bugzilla:
   https://bugzilla.yoctoproject.org/buglist.cgi?quicksearch=AB-INT.
   -

   Help us resolve CVE issues: CVE metrics
   <https://autobuilder.yocto.io/pub/non-release/patchmetrics/>
   -

   We have a growing number of bugs in bugzilla, any help with them is
   appreciated.
   -

   Regarding bugs, even if you can’t fix a bug, submitting a failing test
   case that can reproduce the issue significantly improves the chances it
   might get fixed.
   -

   Further help on optimizing build disk usage would be most welcome.
   -

   Ongoing project development plans are being developed in this document:
   https://docs.google.com/document/d/1xsnN_HcaMhqg6Dn1P_19AnumDaUMQSdFZ_I4rjD830A/edit?usp=sharing

We need to continue to develop this as it will allow us to potentially find
ways to fund specific work items.

Tracking Metrics:

   -

   WDD 2770 (last week 2778) (
   https://wiki.yoctoproject.org/charts/combo.html)
   -

   OE-Core/Poky Patch Metrics
   -

      Total patches found: 1091 (last week 1093)
      -

      Patches in the Pending State: 157 (14%) [last week 157 (14%)]
      -

      https://autobuilder.yocto.io/pub/non-release/patchmetrics/


YP 5.3 Milestone Dates:

   -

   YP 5.3 M1 is ready for release.
   -

   YP 5.3 M2 Build Date 2025-07-21
   -

   YP 5.3 M2 Release Date 2025-08-01
   -

   YP 5.3 M3 Build Date 2025-09-01
   -

   YP 5.3 M3 Release Date 2025-09-12
   -

   YP 5.3 M4 Build Date 2025-10-06
   -

   YP 5.3 M4 Release Date 2025-10-31


Upcoming dot releases:.

   -

   YP 4.0.28 is in QA.
   -

   YP 5.2.2 Build Date 2025-07-07
   -

   YP 5.2.2 Release Date 2025-07-18
   -

   YP 5.0.11 Build Date 2025-07-14
   -

   YP 5.0.11 Release Date 2025-07-25
   -

   YP 4.0.29 Build Date 2025-08-11
   -

   YP 4.0.29 Release Date 2025-08-25
   -

   YP 5.2.3 Build Date 2025-08-18
   -

   YP 5.2.3 Release Date 2025-08-29
   -

   YP 5.0.12 Build Date 2025-08-25
   -

   YP 5.0.12 Release Date 2025-09-05
   -

   YP 4.0.30 Build Date 2025-09-22
   -

   YP 4.0.30 Release Date 2025-10-03
   -

   YP 5.2.4 Build Date 2025-09-29
   -

   YP 5.2.4 Release Date 2025-10-10
   -

   YP 5.0.13 Build Date 2025-10-13
   -

   YP 5.0.13 Release Date 2025-10-24
   -

   YP 4.0.31 Build Date 2025-11-03
   -

   YP 4.031 Release Date 2025-11-14
   -

   YP 5.0.14 Build Date 2025-11-17
   -

   YP 5.0.14 Release Date 2025-11-28
   -

   YP 4.0.32 Build Date 2025-12-15
   -

   YP 4.0.32 Release Date 2025-12-24
   -

   YP 5.0.15 Build Date 2026-01-05
   -

   YP 5.0.15 Release Date 2026-01-16


The Yocto Project’s technical governance is through its Technical Steering
Committee, more information is available at:

https://wiki.yoctoproject.org/wiki/TSC

The Status reports are now stored on the wiki at:
https://wiki.yoctoproject.org/wiki/Weekly_Status

[If anyone has suggestions for other information you’d like to see on this
weekly status update, let us know!]

Thanks,



*Stephen K. Jolley*

*Yocto Project Program Manager*

(    *Cell*:                (208) 244-4460

* *Email*:                 *s
<stephen.k.jolley@intel.com>jolley.yp.pm@gmail.com <jolley.yp.pm@gmail.com>*

[-- Attachment #2: Type: text/html, Size: 57799 bytes --]

Re: [OE-core] Yocto Project Status 1 July. 2025 (WW26)

[Alexander Kanavin]

On Tue, 1 Jul 2025 at 16:44, Stephen Jolley via lists.openembedded.org
<sjolley.yp.pm=gmail.com@lists.openembedded.org> wrote:
> For M2, the following changes have been mentioned as possibilities:
>
> bitbake-setup

This was discussed on the call; the next step would be to start using
it on the autobuilder. That would significantly boost the credibility
of the project, and validate it in a real, mission-critical setting :)

Before that can happen, the tool still needs additional functionality:

- nested bitbake configurations, that would be an equivalent of
'templates' in autobuilder's config.json. E.g. a top level base config
that sets autobuilder-specific fragments for all builds, then an
intemediate config that sets what images and other targets get build,
then final configs that set target machines.

- ability to override git repo locations/revisions from the command
line. E.g. the config file sets master branches for all repos, but one
wants to build master-next.

- ability to select a subset of bitbake configurations for
initialization/update. An autobuilder config would specify dozens of
configurations, but any particular worker machine would only work on
one of them. There is no need to initialize every bitbake config, and
it would only add delays and clutter.

Once all of this in place, config.json could be gradually transitioned
to defer to bitbake-setup:

       "qemux86-64" : {
            "BBSETUP" : "qemux86-64",
        },

and the code would run that with appropriate build-specific
parameters. Config.json also has custom local.conf tweaks all over the
place which would need to be reviewed and converted to fragments.
There's also nomenclature for targets
(BBTARGETS/SANITYTARGETS/EXTRACMDS), multi-step builds, and possibly
other things I haven't yet stumbled into, and that all needs to be
carried over to bitbake-setup without loss of functionality.

I'll get to it after the holiday, e.g. at the end of July or so.

Thanks,
Alex

Re: [OE-core] Yocto Project Status 1 July. 2025 (WW26)

[Richard Purdie]

On Mon, 2025-07-07 at 09:02 +0200, Alexander Kanavin wrote:
> On Tue, 1 Jul 2025 at 16:44, Stephen Jolley via lists.openembedded.org
> <sjolley.yp.pm=gmail.com@lists.openembedded.org> wrote:
> > For M2, the following changes have been mentioned as possibilities:
> > 
> > bitbake-setup
> 
> This was discussed on the call; the next step would be to start using
> it on the autobuilder. That would significantly boost the credibility
> of the project, and validate it in a real, mission-critical setting :)
> 
> Before that can happen, the tool still needs additional functionality:
> 
> - nested bitbake configurations, that would be an equivalent of
> 'templates' in autobuilder's config.json. E.g. a top level base config
> that sets autobuilder-specific fragments for all builds, then an
> intemediate config that sets what images and other targets get build,
> then final configs that set target machines.

I've been looking into this a bit more. I don't think we need to
replace the template functionality in the autobuilder, at least not at
this time. What we need to do is just switch over to using bitbake-
setup to setup poky instead of needing the combo-layer derived repo.

> - ability to override git repo locations/revisions from the command
> line. E.g. the config file sets master branches for all repos, but one
> wants to build master-next.

There are two pieces to this. One is being able to select/override the
repo/branch/revision as you mention. Another key piece is being able to
point at a local cache of repositories. The tool may handle the latter,
I'm just struggling to page in the context.

> - ability to select a subset of bitbake configurations for
> initialization/update. An autobuilder config would specify dozens of
> configurations, but any particular worker machine would only work on
> one of them. There is no need to initialize every bitbake config, and
> it would only add delays and clutter.

When I did try the tool, I was a bit surprised to see the multiple
build directories being created. This also won't scale well.

I suspect the more user friendly workflow is going to be to create a
single build directory and then point the user at the configurations
available, i.e. the features (which include the machine/distro
selection).

This is going to be the part of the workflow which is going to make or
break the tool and it isn't quite right yet. We have all the underlying
pieces now, we just need to expose it to the user in the right way. As
it stands, it isn't feeling quite right yet.

> Once all of this in place, config.json could be gradually transitioned
> to defer to bitbake-setup:
> 
>        "qemux86-64" : {
>             "BBSETUP" : "qemux86-64",
>         },
> 
> and the code would run that with appropriate build-specific
> parameters.

Or perhaps this just becomes a list of features...

>  Config.json also has custom local.conf tweaks all over the
> place which would need to be reviewed and converted to fragments.
> There's also nomenclature for targets
> (BBTARGETS/SANITYTARGETS/EXTRACMDS), multi-step builds, and possibly
> other things I haven't yet stumbled into, and that all needs to be
> carried over to bitbake-setup without loss of functionality.

I'm less worried about these pieces, they key piece is whether we still
need combo-layer.

Cheers,

Richard

Re: [OE-core] Yocto Project Status 1 July. 2025 (WW26)

[Alexander Kanavin]

On Mon, 21 Jul 2025 at 12:27, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> > Before that can happen, the tool still needs additional functionality:
> >
> > - nested bitbake configurations, that would be an equivalent of
> > 'templates' in autobuilder's config.json. E.g. a top level base config
> > that sets autobuilder-specific fragments for all builds, then an
> > intemediate config that sets what images and other targets get build,
> > then final configs that set target machines.
>
> I've been looking into this a bit more. I don't think we need to
> replace the template functionality in the autobuilder, at least not at
> this time. What we need to do is just switch over to using bitbake-
> setup to setup poky instead of needing the combo-layer derived repo.

That's fine and reduces the time pressure; I somehow thought you
expect a full end-to-end autobuilder demo of bitbake-setup handling
both the layer checkout and the build configuration.

I'll still implement nested bitbake configurations, they will help
reduce repetition and clutter e.g. here:
https://github.com/kanavin/bitbake-setup-configurations/blob/main/yocto-master.conf.json#L49

This will look much neater and readable as a tree where nested
configurations add details to the parent (e.g. adding a BSP layer and
picking a particular machine from it). I think that's a rather common
use case.

> > - ability to override git repo locations/revisions from the command
> > line. E.g. the config file sets master branches for all repos, but one
> > wants to build master-next.
>
> There are two pieces to this. One is being able to select/override the
> repo/branch/revision as you mention. Another key piece is being able to
> point at a local cache of repositories. The tool may handle the latter,
> I'm just struggling to page in the context.

I guess you're looking for 'bitbake-setup change-setting default
dl-dir /path/to/dl-dir' ?

(I know the tool currently offers no help in discovering its own settings).

> > - ability to select a subset of bitbake configurations for
> > initialization/update. An autobuilder config would specify dozens of
> > configurations, but any particular worker machine would only work on
> > one of them. There is no need to initialize every bitbake config, and
> > it would only add delays and clutter.
>
> When I did try the tool, I was a bit surprised to see the multiple
> build directories being created. This also won't scale well.
>
> I suspect the more user friendly workflow is going to be to create a
> single build directory and then point the user at the configurations
> available, i.e. the features (which include the machine/distro
> selection).
>
> This is going to be the part of the workflow which is going to make or
> break the tool and it isn't quite right yet. We have all the underlying
> pieces now, we just need to expose it to the user in the right way. As
> it stands, it isn't feeling quite right yet.

I agree; initializing everything was simply the easiest way to produce
the working proof of concept. I didn't want to have to 'decide'
upfront how this should work.

Here's the proposal:

- 'bitbake-setup init' will not initialize every bitbake config that
the json specifies; rather it will print a list of them (if len > 1)
and ask the user to choose (trivial in python and does not require
curses or other ui libraries; oe-setup-build already does this with
templates);

- it can also take an optional second parameter to setup a particular
bitbake configuration non-interactively;

- it can also take --all to set up everything like it does now.

> > Once all of this in place, config.json could be gradually transitioned
> > to defer to bitbake-setup:
> >
> >        "qemux86-64" : {
> >             "BBSETUP" : "qemux86-64",
> >         },
> >
> > and the code would run that with appropriate build-specific
> > parameters.
>
> Or perhaps this just becomes a list of features...

The goal for me is allowing easy replication of autobuilder failures
by anyone who submits patches, so they never have to find the right
bits to copy-paste in config.json or other obscure places. One idea is
to have a step that is clearly marked on the buildbot UI: 'look here
if you want to replicate this build'. When expanded, there are clear
instructions that require typing as few and as short commands as
possible - ideally just a single bitbake-setup invocation followed by
applying the patches-under-test, and then it's also using CDN sstate
to hopefully get to the fail as quickly as possible.

Thanks for looking,
Alex

Re: [OE-core] Yocto Project Status 1 July. 2025 (WW26)

[Richard Purdie]

On Mon, 2025-07-21 at 15:09 +0200, Alexander Kanavin wrote:
> On Mon, 21 Jul 2025 at 12:27, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > > Before that can happen, the tool still needs additional
> > > functionality:
> > > 
> > > - nested bitbake configurations, that would be an equivalent of
> > > 'templates' in autobuilder's config.json. E.g. a top level base
> > > config
> > > that sets autobuilder-specific fragments for all builds, then an
> > > intemediate config that sets what images and other targets get
> > > build,
> > > then final configs that set target machines.
> > 
> > I've been looking into this a bit more. I don't think we need to
> > replace the template functionality in the autobuilder, at least not
> > at
> > this time. What we need to do is just switch over to using bitbake-
> > setup to setup poky instead of needing the combo-layer derived
> > repo.
> 
> That's fine and reduces the time pressure; I somehow thought you
> expect a full end-to-end autobuilder demo of bitbake-setup handling
> both the layer checkout and the build configuration.
> 
> I'll still implement nested bitbake configurations, they will help
> reduce repetition and clutter e.g. here:
> https://github.com/kanavin/bitbake-setup-configurations/blob/main/yocto-master.conf.json#L49
> 
> This will look much neater and readable as a tree where nested
> configurations add details to the parent (e.g. adding a BSP layer and
> picking a particular machine from it). I think that's a rather common
> use case.

It can be summarised a bit more succinctly i.e.:

distro: ['distro/poky', 'distro/poky-altcfg', 'distro/poky-tiny']
machine: ['machine/qemux86-64', 'machine/qemuarm64', 'machine/qemuriscv64']

which is why I'm thinking we should perhaps work with this in terms of
fragments/config selection.

> > > - ability to override git repo locations/revisions from the
> > > command
> > > line. E.g. the config file sets master branches for all repos,
> > > but one
> > > wants to build master-next.
> > 
> > There are two pieces to this. One is being able to select/override
> > the
> > repo/branch/revision as you mention. Another key piece is being
> > able to
> > point at a local cache of repositories. The tool may handle the
> > latter,
> > I'm just struggling to page in the context.
> 
> I guess you're looking for 'bitbake-setup change-setting default
> dl-dir /path/to/dl-dir' ?
> 
> (I know the tool currently offers no help in discovering its own
> settings).

No, I mean something different.

In order to speed up builds and reduce network usage, the autobuilder
has a cache of commonly used git repos such as poky, bitbake and some
other metadata layers. When cloning, it will first use those cached
repos, then just do a final pull from upstream, just to ensure
everything is up to date.

Those cache directories are updated by a cron job. I'd like a way of
telling bitbake-setup to use such a cache to speed up checkouts and
reduce network usage.

This feature doesn't need to be that user visible, just available for
the CI setups to configure. The code for this is in yocto-autobuilder-
helper.


> > > - ability to select a subset of bitbake configurations for
> > > initialization/update. An autobuilder config would specify dozens
> > > of
> > > configurations, but any particular worker machine would only work
> > > on
> > > one of them. There is no need to initialize every bitbake config,
> > > and
> > > it would only add delays and clutter.
> > 
> > When I did try the tool, I was a bit surprised to see the multiple
> > build directories being created. This also won't scale well.
> > 
> > I suspect the more user friendly workflow is going to be to create
> > a
> > single build directory and then point the user at the
> > configurations
> > available, i.e. the features (which include the machine/distro
> > selection).
> > 
> > This is going to be the part of the workflow which is going to make
> > or
> > break the tool and it isn't quite right yet. We have all the
> > underlying
> > pieces now, we just need to expose it to the user in the right way.
> > As
> > it stands, it isn't feeling quite right yet.
> 
> I agree; initializing everything was simply the easiest way to
> produce the working proof of concept. I didn't want to have to
> 'decide' upfront how this should work.
> 
> Here's the proposal:
> 
> - 'bitbake-setup init' will not initialize every bitbake config that
> the json specifies; rather it will print a list of them (if len > 1)
> and ask the user to choose (trivial in python and does not require
> curses or other ui libraries; oe-setup-build already does this with
> templates);
> 
> - it can also take an optional second parameter to setup a particular
> bitbake configuration non-interactively;
> 
> - it can also take --all to set up everything like it does now.

I'm not convinced anyone will ever really want to configure "all", it
isn't what users generally want to do.

I suspect what we need is a single base default configuration and a
prompt saying "run XXX" to configure fragments X or Y.

We then allow init to take a list of fragments to configure? That would
mean:

bitbake-setup init poky "machine/qemux86-64 distro/poky-tiny"

would work and we have a way to add new things to the commandline
relatively easily.



> 
> > > Once all of this in place, config.json could be gradually
> > > transitioned
> > > to defer to bitbake-setup:
> > > 
> > >        "qemux86-64" : {
> > >             "BBSETUP" : "qemux86-64",
> > >         },
> > > 
> > > and the code would run that with appropriate build-specific
> > > parameters.
> > 
> > Or perhaps this just becomes a list of features...
> 
> The goal for me is allowing easy replication of autobuilder failures
> by anyone who submits patches, so they never have to find the right
> bits to copy-paste in config.json or other obscure places. One idea
> is to have a step that is clearly marked on the buildbot UI: 'look
> here if you want to replicate this build'. When expanded, there are
> clear instructions that require typing as few and as short commands
> as possible - ideally just a single bitbake-setup invocation followed
> by applying the patches-under-test, and then it's also using CDN
> sstate to hopefully get to the fail as quickly as possible.

Currently the autobuilder dumps the local.conf settings in one of the
steps and it is quite clear about which commands it runs, if you know
where to look. I agree it would be nice to simplify that and go a step
or two further and make it clearer.

Keep in mind that many of the builder targets are series of commands,
not just a single command.

Cheers,

Richard

Re: [OE-core] Yocto Project Status 1 July. 2025 (WW26)

[Alexander Kanavin]

On Mon, 21 Jul 2025 at 15:41, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> > This will look much neater and readable as a tree where nested
> > configurations add details to the parent (e.g. adding a BSP layer and
> > picking a particular machine from it). I think that's a rather common
> > use case.
>
> It can be summarised a bit more succinctly i.e.:
>
> distro: ['distro/poky', 'distro/poky-altcfg', 'distro/poky-tiny']
> machine: ['machine/qemux86-64', 'machine/qemuarm64', 'machine/qemuriscv64']
>
> which is why I'm thinking we should perhaps work with this in terms of
> fragments/config selection.

So the above would create a kind of 'config generator' that would
allow any combination of the entries from the two lists?

I'm ok with that, but I would generalize the keyword for such lists to
something like 'fragments-choose-one-of', as the values are just
fragment names and do not require special treatment. It would then
also easily extend to other choose-one-of things like toolchain/clang,
init/systemd, or libc/musl.

> > (I know the tool currently offers no help in discovering its own
> > settings).
>
> No, I mean something different.
>
> In order to speed up builds and reduce network usage, the autobuilder
> has a cache of commonly used git repos such as poky, bitbake and some
> other metadata layers. When cloning, it will first use those cached
> repos, then just do a final pull from upstream, just to ensure
> everything is up to date.
>
> Those cache directories are updated by a cron job. I'd like a way of
> telling bitbake-setup to use such a cache to speed up checkouts and
> reduce network usage.
>
> This feature doesn't need to be that user visible, just available for
> the CI setups to configure. The code for this is in yocto-autobuilder-
> helper.

But bitbake-setup is using bitbake fetcher for network operations,
e.g. fetching and unpacking layers. And as far as I see the bitbake
fetcher provides exactly same download cache function as this custom
mechanism. So you could configure bitbake-setup (with the command I
mentioned in the previous email) to use DL_DIR that is shared via nfs
between all workers and bitbake-setup instances (or even is the same
as autobuilder's global DL_DIR for recipe fetching), and then it will
not touch the network at all if the needed revision is already in
cache, or do a quick incremental git fetch if it isn't (or revision is
floating like master-next or other branch name).

> I'm not convinced anyone will ever really want to configure "all", it
> isn't what users generally want to do.

I'd want to use it if I need to locally run builds for all products of
some company, and there aren't many specified in the config. It's an
option, no one is forced into it.

> I suspect what we need is a single base default configuration and a
> prompt saying "run XXX" to configure fragments X or Y.
>
> We then allow init to take a list of fragments to configure? That would
> mean:
>
> bitbake-setup init poky "machine/qemux86-64 distro/poky-tiny"
>
> would work and we have a way to add new things to the commandline
> relatively easily.

Yes, this should work, as long as the fragments on command line are
all from the choose-one-of lists in the config file.

bitbake-setup should then make a record of what fragments came from
the base config, and what were hand-picked (this probably matters in
status/update operations, and in general should be preserved).

Cheers,
Alex