Re: [OE-core] [RFC PATCH 0/4] Clean AUTOREV usage, add revision check for every SCM URI

["Mathieu Dubois-Briand" <mathieu.dubois-briand@bootlin.com>]

On Wed Mar 4, 2026 at 11:43 AM CET, Corentin Guillevic via lists.openembedded.org wrote:
> This patch series introduces an additional "insane" test to check whether every
> URI within the SRC_URI variable has a revision. This is the purpose of the last
> two patches.
>
> However, the new test raises a "deep" issue when AUTOREV is set:
> "AUTOREV/SRCPV set too late for the fetcher to work properly". This occurs
> because the test accesses to the SRCREV variable, at the line:
>
> if d.getVar('SRCREV') == "INVALID":
>
> The following self-tests can raise this issue (if my test is enabled):
>
> $ oe-selftest -Kv -r bblayers.BitbakeLayers \
>     -r pokybleeding.PokyBleeding.test_poky_bleeding_autorev \
>     -r devtool.DevtoolUpgradeTests
>
> Or a shorter one:
>
> $ recipetool --color=auto create --devtool -o /tmp/devtool4819z5wk \
>     'http://downloads.yoctoproject.org/mirror/sources/i2c-tools-3.1.2.tar.bz2' \
>     -x ${BUILDDIR}/workspace/sources/devtoolsrcdcviuvn9
>
> The common point is AUTOREV, which is used as default value by devtool and
> recipetool. When AUTOREV is parsed, this triggers the call to the get_autorev()
> function (see [1]) which sets the internal __BBAUTOREV_SEEN variable.
>
> If the associated URI is a SCM one, there is no problem. Otherwise, the above
> error will occur (see [2]) because the fetcher code doesn't take AUTOREV into
> account (see [3]).
>
> Why has this error never occurred before? In my opinion, it is related to how
> devtool/recipetool parse and run their crafted recipes. Firstly, these tools
> unconditionally set AUTOREV as default value for srcrev. Despite this value is
> not written into the temporary recipe if SRC_URI doesn't have a SCM URI, the
> tools seem to keep it in memory.
>
> Next, the crafted recipe is run. Without my patch series, no error is raised
> because I think there is no use of SRCREV in the execution flow. With my insane
> test only, which uses SRCREV, the test code is parsed and expanded. As SRCREV
> is being used, its value AUTOREV is expanded and leads to function
> get_autorev(). As the URI is no a SCM one, no code handles AUTOREV and the
> above error occurs.
>
> This is why the first two patches: their purpose is to set srcrev to AUTOREV
> only if the submitted URI is a SCM one. These patches resolve, the
> "AUTOREV/SRCPV set too late for the fetcher to work properly" error disappears,
> as AUTOREV is only set when required.
>
> I sent this patch series as RFC because I need feedback on this issue. What
> are your thoughts on the error and my fix? Is my theory correct about
> devtool/recipetool keeping some variables in memory before running the crafted
> recipe?
>
> Thank you
>
> Corentin
>

Hi Corentin,

Thanks for the update. I know this is still an RFC, but I still picked
it for a run on the autobuilder.

I noted just a single test fail, that seems linked with this series:

2026-03-10 16:29:38,455 - oe-selftest - INFO - devtool.DevtoolAddTests.test_devtool_add_git_local (subunit.RemotedTestCase)
2026-03-10 16:29:38,457 - oe-selftest - INFO -  ... FAIL
...
ERROR: Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/bitbake/lib/bb/data_smart.py", line 465, in expandWithRefs
    s = __expand_python_regexp__.sub(varparse.python_sub, s)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/bitbake/lib/bb/data_smart.py", line 154, in python_sub
    value = utils.better_eval(codeobj, DataContext(self.d), {'d' : self.d})
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/bitbake/lib/bb/utils.py", line 489, in better_eval
    return eval(source, ctx, locals)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "Var <fetcher_hashes_dummyfunc[vardepvalue]>", line 1, in <module>
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/bitbake/lib/bb/fetch2/__init__.py", line 853, in get_hashvalue
    pkgv, revs = _get_srcrev(d, method_name=method_name)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/bitbake/lib/bb/fetch2/__init__.py", line 798, in _get_srcrev
    fetcher = Fetch(d.getVar('SRC_URI').split(), d)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/bitbake/lib/bb/fetch2/__init__.py", line 1821, in __init__
    self.ud[url] = FetchData(url, d, localonly)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/bitbake/lib/bb/fetch2/__init__.py", line 1357, in __init__
    self.method.urldata_init(self, d)
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/bitbake/lib/bb/fetch2/git.py", line 257, in urldata_init
    ud.setup_revisions(d)
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/bitbake/lib/bb/fetch2/__init__.py", line 1390, in setup_revisions
    self.revision = srcrev_internal_helper(self, d, self.name)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/bitbake/lib/bb/fetch2/__init__.py", line 1251, in srcrev_internal_helper
    raise FetchError("Please set a valid SRCREV for url %s (possible key names are %s, or use a ;rev=X URL parameter)" % (str(attempts), ud.url), ud.url)
bb.fetch2.FetchError: Fetcher failure for URL: 'git://git.yoctoproject.org/git/dbus-wait;protocol=https;branch=master'. Please set a valid SRCREV for url ['SRCREV_default:pn-dbus-wait', 'SRCREV_default', 'SRCREV:pn-dbus-wait', 'SRCREV'] (possible key names are git://git.yoctoproject.org/git/dbus-wait;protocol=https;branch=master, or use a ;rev=X URL parameter)


https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3490
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3371
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3261

Thanks,
Mathieu

-- 
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com