[PATCH 0/5] Jobserver support

["Martin Hundebøll" <martin@geanix.com>]

The parallelism of bitbake easily uses every available core on the build
host. But since every task is run with the same number of parallel
threads/processes, multiple tasks might load the CPU excessively, which
in turn slows down the build due to scheduling overhead.

This patch series adds a class that creates a fifo filled with
PARALLEL_MAKE tokens. The path to the created fifo is then configured in
the MAKEFLAGS environment variable, which is read by make and a patched
ninja (and gcc if doing lto).

The benefits from using the jobserver depends on the set of executed
tasks: running multiple large do_compile tasks simultaneously benefits
more than multiple do_fetch tasks. A simple test building the following
tasks (and all their dependencies) yields a ~5% improvement in build
time (20:20 -> 19:20):

  nodejs-native
  rust-llvm-native
  rust-native
  linux-yocto
  qemu-native

On build machines shared by multiple users, a single jobserver can be
shared between multiple builds (using the JOBSERVER_FIFO variable).
Running the above build in two different build directories at the same
time gives a ~12% improvement (43:17 -> 37:55).

Finally, the memory pressure from e.g. compiling multiple c++ based
projects is also reduced. In our case, a cloud based build machine (with
32 cores and 32GB RAM) fails to compile llvm-rust-native (in parallel to
nodejs) without the jobserver due to a lack of memory.

This patch set is roughly based on previous work by Richard[1]. That
patch lists three TODO items, which are all addressed by these patches:
 * The fifo path defaults to TMPDIR/jobserver_fifo, but can be
   configured using JOBSERVER_FIFO.
 * The number of make threads defaults to the value from PARALLEL_MAKE
   (which is then redundant).
 * If PARALLEL_MAKE is unset, the jobserver functionality is skipped.

Further work in addition to this patch set could be to make bitbake
tasks jobserver aware.

Changes since the RFC[2]:
 * The ninja src uri change in patch 3 is converted to a set of patches
 * The qemu fix in patch 4 is converted to a submitted patch

[1] https://lore.kernel.org/openembedded-core/1423223184.20217.15.camel@linuxfoundation.org/
[2] https://lore.kernel.org/openembedded-core/20230828124834.376779-1-martin@geanix.com/

Martin Hundebøll (5):
  classes: jobserver: support gnu make fifo jobserver
  scripts: build-env: allow passing JOBSERVER_FIFO from environment
  ninja: build modified version with GNU Make jobserver support
  qemu: enable parallel builds when using the jobserver class
  contrib: add python service and systemd unit to run shared jobserver

 contrib/jobserver/jobserver.py                |   78 +
 contrib/jobserver/jobserver.service           |   10 +
 meta/classes-global/jobserver.bbclass         |   80 +
 meta/conf/bitbake.conf                        |    2 +-
 ...dd-GNU-make-jobserver-client-support.patch |  494 +++++++
 ...l-monitoring-to-SubprocessSet-DoWork.patch |  560 +++++++
 ...er-when-jN-is-forced-on-command-line.patch |  198 +++
 .../files/0004-Honor-lN-from-MAKEFLAGS.patch  |  134 ++
 ...e-LinePrinter-for-TokenPool-messages.patch |  128 ++
 .../files/0006-Prepare-PR-for-merging.patch   |  156 ++
 .../files/0007-Add-tests-for-TokenPool.patch  |  237 +++
 ...0008-Add-tests-for-subprocess-module.patch |  121 ++
 .../0009-Add-tests-for-build-module.patch     |  397 +++++
 ...-implementation-for-GNUmakeTokenPool.patch | 1283 +++++++++++++++++
 .../0011-Prepare-PR-for-merging-part-II.patch |  744 ++++++++++
 ...ename-TokenPool-Setup-to-SetupClient.patch |  109 ++
 ...013-Add-TokenPool-SetupMaster-method.patch |   78 +
 ...mmand-line-option-m-tokenpool-master.patch |  101 ++
 ...plement-GNUmakeTokenPool-SetupMaster.patch |  152 ++
 ...mplement-GNUmakeTokenPool-CreatePool.patch |   87 ++
 ...-Add-tests-for-TokenPool-SetupMaster.patch |  145 ++
 ...-jobserver-fifo-style-client-support.patch |  265 ++++
 ...ional-argument-to-m-tokenpool-master.patch |  368 +++++
 ...-jobserver-fifo-style-master-support.patch |  287 ++++
 meta/recipes-devtools/ninja/ninja_1.11.1.bb   |   33 -
 meta/recipes-devtools/ninja/ninja_1.12.0.bb   |   55 +
 meta/recipes-devtools/qemu/qemu.inc           |    1 +
 ...e-jobserver-auth-argument-when-calli.patch |   37 +
 scripts/oe-buildenv-internal                  |    2 +-
 29 files changed, 6307 insertions(+), 35 deletions(-)
 create mode 100644 contrib/jobserver/jobserver.py
 create mode 100644 contrib/jobserver/jobserver.service
 create mode 100644 meta/classes-global/jobserver.bbclass
 create mode 100644 meta/recipes-devtools/ninja/files/0001-Add-GNU-make-jobserver-client-support.patch
 create mode 100644 meta/recipes-devtools/ninja/files/0002-Add-TokenPool-monitoring-to-SubprocessSet-DoWork.patch
 create mode 100644 meta/recipes-devtools/ninja/files/0003-Ignore-jobserver-when-jN-is-forced-on-command-line.patch
 create mode 100644 meta/recipes-devtools/ninja/files/0004-Honor-lN-from-MAKEFLAGS.patch
 create mode 100644 meta/recipes-devtools/ninja/files/0005-Use-LinePrinter-for-TokenPool-messages.patch
 create mode 100644 meta/recipes-devtools/ninja/files/0006-Prepare-PR-for-merging.patch
 create mode 100644 meta/recipes-devtools/ninja/files/0007-Add-tests-for-TokenPool.patch
 create mode 100644 meta/recipes-devtools/ninja/files/0008-Add-tests-for-subprocess-module.patch
 create mode 100644 meta/recipes-devtools/ninja/files/0009-Add-tests-for-build-module.patch
 create mode 100644 meta/recipes-devtools/ninja/files/0010-Add-Win32-implementation-for-GNUmakeTokenPool.patch
 create mode 100644 meta/recipes-devtools/ninja/files/0011-Prepare-PR-for-merging-part-II.patch
 create mode 100644 meta/recipes-devtools/ninja/files/0012-Rename-TokenPool-Setup-to-SetupClient.patch
 create mode 100644 meta/recipes-devtools/ninja/files/0013-Add-TokenPool-SetupMaster-method.patch
 create mode 100644 meta/recipes-devtools/ninja/files/0014-Add-command-line-option-m-tokenpool-master.patch
 create mode 100644 meta/recipes-devtools/ninja/files/0015-Implement-GNUmakeTokenPool-SetupMaster.patch
 create mode 100644 meta/recipes-devtools/ninja/files/0016-Implement-GNUmakeTokenPool-CreatePool.patch
 create mode 100644 meta/recipes-devtools/ninja/files/0017-Add-tests-for-TokenPool-SetupMaster.patch
 create mode 100644 meta/recipes-devtools/ninja/files/0018-Add-GNU-make-jobserver-fifo-style-client-support.patch
 create mode 100644 meta/recipes-devtools/ninja/files/0019-Add-optional-argument-to-m-tokenpool-master.patch
 create mode 100644 meta/recipes-devtools/ninja/files/0020-Add-GNU-make-jobserver-fifo-style-master-support.patch
 delete mode 100644 meta/recipes-devtools/ninja/ninja_1.11.1.bb
 create mode 100644 meta/recipes-devtools/ninja/ninja_1.12.0.bb
 create mode 100644 meta/recipes-devtools/qemu/qemu/0013-Makefile-preserve-jobserver-auth-argument-when-calli.patch

-- 
2.44.0