[PATCH 00/15] Developer workflow tools

Paul Eggleton
2014-12-19T11:42:05Z

I've been looking at how to make it easier for application and system
component developers to get their work done using the tools we provide,
and I believe this patchset is a piece of the solution. There's still a
number of other pieces to come, but this should be usable on its own.

The first part of this patchset makes some changes to OE to accommodate
the tools. One of these is to extend the PATCHTOOL = "git" code to
ensure that we're able to apply all patches even if "git am" and
"git apply" can't handle them, that we do a commit to the repository per
patch, and that there is a way to track these commits back to the patch
from which they were added. There are also some additions to shared
library code that may well be useful for other tools.

I've then added a new recipe auto-creation script, recipetool, which can
take a source tree or URL and create a skeleton recipe to build it.
(In case anyone is wondering about the existing scripts/create-recipe,
frankly I consider it a dead end - it's written in Perl, which makes it
a bit difficult to integrate with the rest of our code; it's also
GPLv3-only which makes any such integration pretty much impossible from
another angle.)

Then we add devtool. This allows you to:

* Add a new piece of software (auto-create the skeleton of a recipe
using the aforementioned recipetool and point the build to an
external source tree)
* Modify the source for an existing recipe (point the build to an
external source tree, possibly creating that tree in the same step
and managing it with git)
* Deploy the installed files from a recipe from ${D} to a target using
scp.
* Apply any modifications to the commits in the external source tree
back to the recipe (by updating/adding/removing patches and updating
SRC_URI within the recipe).

There will obviously be extensions to these tools, but I hope they are
already functional enough to be useful at the state they are in at the
moment.

(Note that this series depends on another series of patches to BitBake
sent at the same time - these can be found on the paule/devtool-bb2
branch on poky-contrib.)

Changes since the earlier RFC:

For those that looked at the earlier RFC series, there have been quite a
few improvements since that, almost too many to list - but other than
what's visible in the actual commits, here goes:

* recipetool now uses autoscan to pick up dependencies from Makefile-only
software
* recipetool is much smarter about library dependencies (making use of
collected shared lib provider and pkgdata information rather than a
limited hardcoded map) and detects more licenses.
* devtool now creates workspace layer as needed if it doesn't exist
* devtool can now write back commits to patches next to the original
recipe
* Compilation of recipes in the workspace now happens every time rather
than just once, and causes dependent tasks (e.g. images) to be rebuilt
as well.
* recipetool: if we can't find any license files at all, set LICENSE to
"CLOSED" (with a comment warning it needs to be set more appropriately)
so that the user can at least start building the recipe.
* devtool can extract the source for linux-yocto but externalsrc usage
may still be problematic - something that still needs to be addressed.
* Branch and tag the external source git repository
* Improved error reporting, help and general messages

(Thanks to Trevor Woerner and others for earlier testing & feedback.)

The following changes since commit 8d0e56a850579f9a6d501266deeef9b257ce4780:

serf: readded md5sum (2014-12-11 11:34:22 +0000)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib paule/devtool
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/devtool

Junchun Guan (1):
scripts/devtool: Support deploy/undeploy function

Paul Eggleton (14):
meta-environment: don't mark tasks as nostamp
classes/package: move read_shlib_providers() to a common unit
lib/oe/patch: fall back to patch if git apply fails
lib/oe/patch: auto-commit when falling back from git am
lib/oe/patch: use --keep-cr with git am
lib/oe/patch.py: abort "git am" if it fails
lib/oe/patch: add support for extracting patches from git tree
lib/oe: add recipeutils module
oeqa/utils: make get_bb_var() more reliable
classes/externalsrc: set do_compile as nostamp
scripts/recipetool: Add a recipe auto-creation script
scripts: add scriptutils module
scripts/devtool: add development helper tool
devtool: add QA tests

meta/classes/externalsrc.bbclass | 3 +
meta/classes/package.bbclass | 24 +-
meta/lib/oe/package.py | 26 ++
meta/lib/oe/patch.py | 158 ++++++++-
meta/lib/oe/recipeutils.py | 279 +++++++++++++++
meta/lib/oeqa/selftest/devtool.py | 239 +++++++++++++
meta/lib/oeqa/utils/commands.py | 4 +-
meta/recipes-core/meta/meta-environment.bb | 2 -
scripts/devtool | 255 ++++++++++++++
scripts/lib/devtool/__init__.py | 78 +++++
scripts/lib/devtool/deploy.py | 100 ++++++
scripts/lib/devtool/standard.py | 545 +++++++++++++++++++++++++++++
scripts/lib/recipetool/__init__.py | 0
scripts/lib/recipetool/create.py | 413 ++++++++++++++++++++++
scripts/lib/recipetool/create_buildsys.py | 319 +++++++++++++++++
scripts/lib/scriptutils.py | 60 ++++
scripts/recipetool | 99 ++++++
17 files changed, 2572 insertions(+), 32 deletions(-)
create mode 100644 meta/lib/oe/recipeutils.py
create mode 100644 meta/lib/oeqa/selftest/devtool.py
create mode 100755 scripts/devtool
create mode 100644 scripts/lib/devtool/__init__.py
create mode 100644 scripts/lib/devtool/deploy.py
create mode 100644 scripts/lib/devtool/standard.py
create mode 100644 scripts/lib/recipetool/__init__.py
create mode 100644 scripts/lib/recipetool/create.py
create mode 100644 scripts/lib/recipetool/create_buildsys.py
create mode 100644 scripts/lib/scriptutils.py
create mode 100755 scripts/recipetool

--
1.9.3