[PATCH v2 00/13] bitbake-setup PyPI Packaging

[Rob Woolley <rob.woolley@windriver.com>]

This is a new series to add PyPI packaging for bitbake-setup.  It is based on
my development branch here: https://github.com/robwoolley/bitbake/tree/add-pypi-v6

UPDATE: This series is based on the HEAD of bitbake master.  It repeats any of
the previous linting fixes that have not yet been applied.  As such this v2
stands alone independent of any previous emails sent by me recently.

The main challenge has been to package bitbake-setup independently from the
rest of bitbake.  This necessitates creating a separate staging directory
with pyproject.toml at the root.  Details are in the commit messages.

I have tested using these changes to package all of bitbake to make sure that
these changes didn't inhibit that from working. That is beyond the scope of
work for what I was trying to do.

There are 2 challenges I had to overcome:

1. Ensuring that all primary workflows worked:
   (a) Running bitbake and bitbake-setup directly from git
   (b) Allowing developers to use pip install -e . to develop in-tree
   (c) Enabling pip install in virtual environments and at the system-level

2. Historically,  he sys.path modificiations made it easy to refer to the
   vendorized modules without worrying about system-wide package conflicts.
   Adding packaging support required that the modules be properly
   vendorized so they didn't conflict with the upstream packages.
   Vendorizing the bb module resulted in too much churn as we import bb
   and its sub-modules across all the files in lib/bb/*
   Given that the bb package on PyPI is obscure and hasn't been updated in
   a very long time, I chose to bundle bb instead of vendorize it.


Here are the detailed testing steps that I used to verify this series before
submitting it to bitbake-devel:

$ python3 -m venv venv

$ source venv/bin/activate

$ pip install build

$ BB_SKIP_PYPI_TESTS=no bin/bitbake-selftest -v bb.tests.setup.PyPIPackagingTest

bb.tests.setup.PyPIPackagingTest
test_console_script_help (bb.tests.setup.PyPIPackagingTest)
Verify bitbake-setup --help runs successfully. ... ok
test_console_script_list (bb.tests.setup.PyPIPackagingTest)
Verify bitbake-setup list runs successfully. ... ok
test_entry_points (bb.tests.setup.PyPIPackagingTest)
Verify console script entry points are correctly defined. ... ok
test_imports (bb.tests.setup.PyPIPackagingTest)
Verify all expected modules can be imported from installed package. ... ok
test_package_metadata (bb.tests.setup.PyPIPackagingTest)
Verify package metadata is correctly set. ... ok
test_vendored_dependencies (bb.tests.setup.PyPIPackagingTest)
Verify vendored dependencies (bs4, ply, progressbar, simplediff) are not bundled in package. ... ok
test_version_from_wheel (bb.tests.setup.PyPIPackagingTest)
Verify version is set correctly (not fallback 0.0.0 unless expected). ... ok
test_wheel_metadata_file (bb.tests.setup.PyPIPackagingTest)
Verify wheel METADATA file contains required fields. ... ok

----------------------------------------------------------------------
Ran 8 tests in 18.014s

OK

$ contrib/pypi/package-bitbake-setup.py

$ cd packaging_workspace/

$ pip install -e .

$ which bitbake-setup
/ala-lpggp31/rwoolley/bitbake-testing/bitbake/venv/bin/bitbake-setup

$ bitbake-setup --version
bitbake-setup 2.16.0

$ bitbake-setup list
NOTE: Fetching configuration registry
    git://git.openembedded.org/bitbake;protocol=https;branch=master;rev=master
into
    /tmp/bitbake-setup-list-v1m6p7s3/.bitbake-setup-cache/configurations
Available configurations:
oe-nodistro-master      OpenEmbedded - 'nodistro' basic configuration
oe-nodistro-whinlatter  OpenEmbedded - 'nodistro' basic configuration, release 5.3 'whinlatter' (supported until 2026-05-31)
poky-master     Poky - The Yocto Project testing distribution configurations and hardware test platforms
poky-whinlatter Poky - The Yocto Project testing distribution configurations and hardware test platforms, release 5.3 'whinlatter' (supported until 2026-05-31)

Run 'init' with one of the above configuration identifiers to set up a build.

$ pip uninstall bitbake-setup

$ python3 -m build

$ pip install dist/bitbake_setup-2.16.0-py3-none-any.whl
Processing ./dist/bitbake_setup-2.16.0-py3-none-any.whl
Installing collected packages: bitbake-setup
Successfully installed bitbake-setup-2.16.0

$ cd $(mktemp -d)

$ bitbake-setup init

... 

The bitbake configuration files (local.conf, bblayers.conf and more) can be found in
    /tmp/tmp.s1MXZQ5XiF/bitbake-builds/poky-whinlatter/build/conf