Re: [docs] [PATCH 1/4] tools/obsolete-variables: add script

[Quentin Schulz <quentin.schulz@cherry.de>]

Hi Antonin,

On 8/27/26 9:46 AM, Antonin Godard via lists.yoctoproject.org wrote:
> Add a script that outputs missing variables from
> OE-Core/meta-yocto/BitBake, i.e. for which grepping returned nothing.
> 
> There are a few exceptions, which are listed in VAR_EXCEPTIONS.
> 
> The script currently returns:
> 
>    ERROR: Variable CVSDIR not found anywhere
>    ERROR: Variable FIT_KERNEL_COMP_ALG_EXTENSION not found anywhere
>    ERROR: Variable USERMOD_PARAMS not found anywhere
> 
> Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
> ---
>   documentation/tools/obsolete-variables | 142 +++++++++++++++++++++++++++++++++
>   1 file changed, 142 insertions(+)
> 
> diff --git a/documentation/tools/obsolete-variables b/documentation/tools/obsolete-variables
> new file mode 100755
> index 000000000..bc97cbb71
> --- /dev/null
> +++ b/documentation/tools/obsolete-variables
> @@ -0,0 +1,142 @@
> +#!/usr/bin/env python3
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +# Author: Antonin Godard <antonin.godard@bootlin.com>
> +#
> +# Copyright (C) 2026 Bootlin
> +#
> +
> +import argparse
> +import logging
> +import subprocess
> +import sys
> +
> +from pathlib import Path
> +from sphinx.util.inventory import InventoryFile

I would recommend splitting modules that aren't from core Python into a 
separate section. pathlib and typing are python core modules, but 
sphinx.util... isn't.

> +from typing import List
> +
> +
> +DOCS_DIR = Path(__file__).parent.parent
> +# False positives:
> +# - variables we know exist but have a specific syntax
> +# - we keep documentation for it here, already saying it is obsolete
> +VAR_EXCEPTIONS = (
> +    "CONFLICT_IMAGE_FEATURES",
> +    "CONFLICT_TUNE_FEATURES",
> +    "FEATURE_PACKAGES",
> +    "LAYERRECOMMENDS",
> +    "REQUIRED_IMAGE_FEATURES",
> +    "VIRTUAL-RUNTIME",
> +    "module_autoload",
> +    "module_conf",
> +)
> +ERR_MSG = "Variable %s not found in OE-Core, meta-yocto, or BitBake"
> +
> +
> +def parse_arguments() -> argparse.Namespace:
> +    parser = argparse.ArgumentParser(description="Use the Sphinx's inventory to "
> +                                     "check for variables not present in "
> +                                     "OE-Core/meta-yocto/BitBake")
> +
> +    parser.add_argument("--debug",
> +                        action="store_true",
> +                        help="Print debug messages")
> +
> +    parser.add_argument("--yocto-docs-inv",
> +                        type=Path,
> +                        default=DOCS_DIR / "_build/html/objects.inv",
> +                        help="Input yocto-docs inventory file")
> +
> +    parser.add_argument("--bitbake-inv",
> +                        type=Path,
> +                        default=DOCS_DIR / "_build/doctrees/__intersphinx_cache__/bitbake_objects.inv",
> +                        help="Input bitbake inventory file")

Consider printing the default values with %(default)s, c.f. 
https://docs.python.org/3/library/argparse.html#help.

> +
> +    parser.add_argument("oecore_dir",
> +                        type=Path,
> +                        help="Path to openembedded-core")
> +
> +    parser.add_argument("meta_yocto_dir",
> +                        type=Path,
> +                        help="Path to meta-yocto")
> +
> +    parser.add_argument("bitbake_dir",
> +                        type=Path,
> +                        help="Path to bitbake")
> +

Please specify those need to be *git* repositories (otherwise you cannot 
run git-grep on them).

> +    return parser.parse_args()
> +
> +
> +def var_exists_in(var: str, gitdir: Path) -> bool:
> +    """
> +    In gitdir, check if a grepping for "<var>" return something (then return
> +    True, False otherwise).
> +
> +    Special case where in OE-Core we can have:
> +        BB_RENAMED_VARIABLES[<var>] = "..."
> +    Then exclude that.
> +    """
> +    cmd = [
> +        "git", "-C", gitdir, "grep", "--extended-regexp", fr"\<{var}\>",
> +    ]
> +    _out = ""
> +    try:
> +        _out = subprocess.check_output(cmd, encoding="utf-8")
> +    except subprocess.CalledProcessError:
> +        pass
> +
> +    out = ""
> +    for line in _out.splitlines():
> +        if not line.startswith(f"BB_RENAMED_VARIABLES[{var}]"):
> +            out += f"{line}\n"
> +
> +    if out:
> +        logging.debug(f"{var} found in {gitdir.name}:\n{out}")
> +        return True
> +

Can't you simplify this to

git grep --extended-regexp -e '\<{var}\>' --and --not -e 
'BB_RENAMED_VARIABLES\[{var}\]'

?

If this works for you, please consider adding --max-count=1 so that it 
exits as soon as possible if there's at least one match. You probably 
also want to pass --quiet as we won't care about the output anymore.

> +    return False
> +
> +
> +def var_exists(var: str, repos: List[Path]):
> +    return any(var_exists_in(var, d) for d in repos)
> +
> +
> +def check_inventory(inv_p: Path, uri: str, repos: List[Path]) -> int:
> +    exit_code = 0
> +    inv = InventoryFile.loads(inv_p.read_bytes(), uri="")
> +    for entry, inv_item in sorted(inv.data["std:term"].items()):
> +        if inv_item.uri.startswith(uri) \
> +                and entry not in VAR_EXCEPTIONS \
> +                and not var_exists(entry, repos):

Typo here, it should be var_exists_in().

> +            exit_code = 1
> +            logging.error(ERR_MSG % entry)

Does it make sense to have a variable (far from where it's used) for 
something that's used only once?

> +    return exit_code
> +
> +
> +def main():
> +    exit_code = 0
> +    args = parse_arguments()
> +
> +    if args.debug:
> +        logging.basicConfig(format="%(levelname)s: %(message)s", level=logging.DEBUG)
> +    else:
> +        logging.basicConfig(format="%(levelname)s: %(message)s")
> +
> +    if not (args.yocto_docs_inv.exists() and args.bitbake_inv.exists()):
> +        logging.error(f"yocto-docs and bitbake inventories not found at {args.yocto_docs_inv}/"
> +                      f"{args.bitbake_inv}. Build the documentation or use the "
> +                      "--yocto-docs-inv/--bitbake-inv options")
> +        sys.exit(1)
> +
> +    exit_code = check_inventory(args.yocto_docs_inv, "ref-manual/variables.html#term-",
> +                                (args.oecore_dir, args.meta_yocto_dir, args.bitbake_dir))
> +

This exit_code is never tested. I'm assuming you may want to do += for 
the next one (or store the result in another variable).

> +    exit_code = check_inventory(args.bitbake_inv, "bitbake-user-manual/bitbake-user-manual-ref-variables.html#term-",
> +                                (args.oecore_dir, args.meta_yocto_dir, args.bitbake_dir))
> +
> +    sys.exit(exit_code)
> +
> +
> +if __name__ == "__main__":
> +    main()
> 
Cheers,
Quentin